Camera monitor system comprising an automated HMI adjustment
By determining the trailer reference point in the CMS of commercial vehicles and adjusting the HMI element position in real time, the position inaccuracy caused by camera tilt or deflection is solved, and the accuracy of image information and the field of view of the vehicle operator are improved.
Patent Information
- Application Number
- CN202380033583.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-13
- Filing Date
- 2023-04-13
- Publication Date
- 2024-11-12
AI Technical Summary
In commercial vehicles, images provided by camera monitor systems (CMSs) may cause inaccurate position and orientation of human-machine interface (HMI) elements on the display due to tilting or deflection of the camera, affecting the transfer of information and field of view coverage.
By displaying an image from a fixed camera on an electronic display of a commercial vehicle, determining the reference point position of the trailer and adjusting the position of the HMI element according to that position to minimize or avoid overlap of the HMI element with the trailer in the image, using the camera arm The rotation speed and extension time detect the change in the reference position and adjust the position of the HMI element in real time.
Ensure the accurate position and orientation of HMI elements in the image, provide more accurate vehicle surroundings, and enhance vehicle operators' field of view coverage and operational safety.
Smart Images

Figure CN118946482A_ABST
Abstract
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 330,460, filed on April 13, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field
[0002] The present disclosure relates to a camera monitor system (CMS) for use in a commercial vehicle including a trailer, and more particularly to a method for adjusting the position of a human machine interface (HMI) element within a commercial vehicle display. Background Art
[0003] Rearview mirror replacement systems and camera systems for supplementing the rearview mirror view are utilized in commercial vehicles to enhance the vehicle operator's ability to see the surrounding environment. A camera monitor system (CMS) utilizes one or more cameras to provide an enhanced field of view to the vehicle operator. In some examples, the rearview mirror replacement system covers a larger field of view than a conventional rearview mirror, or includes views that cannot be fully obtained through a conventional rearview mirror.
[0004] The CMS uses multiple displays located in the vehicle cab to display rearview mirror alternatives and / or supplementary views to the vehicle operator. In some cases, the display includes human-machine interface (HMI) elements overlaid on the image. The HMI elements can provide relevant position information, vehicle feature identification, and / or other information to the vehicle operator. This information supplements the view provided by the camera and, in some cases, allows the CMS display to provide better functionality than a traditional rearview mirror system. The CMS transmits at least a portion of the supplementary information by positioning the HMI elements in a defined position and / or orientation relative to objects within the image. However, when the camera image is tilted or skewed relative to the expected position, the HMI element may be overlaid at a position different from the expected position and / or in an orientation different from the expected orientation. Summary of the invention
[0005] A method for adjusting a position of a human machine interface (HMI) element in a commercial vehicle display according to an example embodiment of the present disclosure includes: displaying an image from a camera fixed to the commercial vehicle on an electronic display of the commercial vehicle, the image depicting a trailer of the commercial vehicle and its surroundings; determining a reference position of a reference point of the trailer in the image; superimposing an HMI element into the image on the electronic display at the HMI position determined relative to the reference position; and detecting whether the reference position has changed in the image, and adjusting the HMI position in response to the change.
[0006] In another embodiment of the foregoing embodiment, the HMI element is a distance marker that is superimposed to extend laterally away from the side of the trailer in the image.
[0007] In another embodiment of any of the foregoing embodiments, adjusting the HMI position includes adjusting the HMI position to minimize or avoid overlap of HMI elements and the trailer in the image.
[0008] In a further embodiment of any of the foregoing embodiments, the reference point of the trailer is a rear corner of the trailer.
[0009] In another embodiment of any of the foregoing embodiments, determining a reference position of a reference point of the trailer is based on a first trailer angle and a first arc, wherein the first trailer angle is the angle between a longitudinal axis of the trailer and a longitudinal axis of a tractor connected to the trailer, and the first arc represents a predetermined range of possible positions of the reference point in the image.
[0010] In another embodiment of any of the foregoing embodiments, the reference point of the trailer is a first reference point, and determining the reference position of the first reference point includes: determining a trailer image angle based on a trailer angle, the trailer image angle being an angle defined in an image relative to a trailer reference line in the image, the trailer reference line extending from a second reference point of the commercial vehicle toward a third reference point of the trailer. The second reference point is closer to the front of the trailer than the third reference point, and the third reference point is closer to the first reference point than the second reference point. Determining the reference position of the first reference point also includes: determining the reference position of the first reference point based on an intersection of the trailer image angle, the trailer reference line, and a second arc representing a predetermined range of possible positions of the third reference point in the image, and a distance between the first arc and the second arc in the image.
[0011] In another embodiment of any of the foregoing embodiments, the third reference point is located on the rear wheel of the trailer, the first reference point is located near the front lower corner of the trailer, and the distance between the first arc and the second arc in the image is the vertical distance between the first arc and the second arc.
[0012] In another embodiment of any of the foregoing embodiments, the camera is mounted on a camera arm, the camera arm is configured to rotate from a retracted position to an extended position at a predetermined rotational speed during normal operation, and the detection includes: measuring the rotational speed of the camera arm as it extends from a first position designated as the retracted position to a second position designated as the extended position; and determining that the reference position has changed in the image based on the measured rotational speed being different from the predetermined rotational speed.
[0013] In another embodiment of any of the foregoing embodiments, the camera is mounted on a camera arm, and the camera arm is configured to rotate from a retracted position to an extended position at a predetermined angular velocity during normal operation, so that the camera arm takes a predetermined duration to extend from the retracted position to the extended position during normal operation, and the detection includes: measuring the duration taken for the camera arm to extend from a first position designated as the retracted position to a second position designated as the extended position; and determining that the reference position has changed in the image based on the measured duration being different from the predetermined duration.
[0014] A camera monitor system (CMS) for a vehicle according to an example embodiment of the present disclosure includes a camera configured to record an image of a trailer of a commercial vehicle and its surroundings, an electronic display, and a CMS controller. The CMS controller is configured to display the image on the electronic display, determine a reference position of a reference point of the trailer in the image, superimpose a human machine interface (HMI) element into the image on the electronic display at an HMI position determined relative to the reference position, and detect whether the reference position has changed in the image, and adjust the HMI position of the HMI element in response to the change.
[0015] In another embodiment of the foregoing embodiment, the HMI element is a distance marker that is superimposed to extend laterally away from the side of the trailer in the image.
[0016] In another embodiment of any of the foregoing embodiments, to adjust the HMI position of the HMI element on the electronic display, the CMS controller is configured to adjust the HMI position to minimize or avoid overlap of the HMI element and the trailer in the image.
[0017] In a further embodiment of any of the foregoing embodiments, the reference point of the trailer is a rear corner of the trailer.
[0018] In another embodiment of any of the foregoing embodiments, the CMS controller is configured to determine a reference position of the reference point based on a first trailer angle and a first arc, wherein the first trailer angle is the angle between the longitudinal axis of the trailer and the longitudinal axis of the tractor connected to the trailer, and the first arc represents a predetermined range of possible positions of the reference point in the image.
[0019] In another embodiment of any of the foregoing embodiments, the reference point of the trailer is a first reference point, and to determine the reference position of the first reference point, the CMS controller is configured to determine a trailer image angle based on a trailer angle, the trailer image angle being an angle defined in an image relative to a trailer reference line in the image, the trailer reference line extending from a second reference point of the commercial vehicle toward a third reference point of the trailer. The second reference point is closer to the front of the trailer than the third reference point, and the third reference point is closer to the first reference point than the second reference point. The CMS controller is further configured to determine the reference position of the first reference point based on the trailer image angle, the intersection of the trailer reference line and a second arc representing a predetermined range of possible positions of the third reference point in the image, and a distance between the first arc and the second arc in the image.
[0020] In another embodiment of any of the foregoing embodiments, the third reference point is located on the rear wheel of the trailer, the first reference point is located near the front lower corner of the trailer, and the distance between the first arc and the second arc in the image is the vertical distance between the first arc and the second arc.
[0021] In another embodiment of any of the foregoing embodiments, the camera is mounted on a camera arm, and the camera arm is configured to rotate from a retracted position to an extended position at a predetermined rotational speed during normal operation. To detect whether the reference position has changed in the image, the CMS controller is configured to measure the rotational speed of the camera arm when it extends from a first position designated as the retracted position to a second position designated as the extended position; and determine that the reference position has changed in the image based on the measured rotational speed being different from the predetermined rotational speed.
[0022] In another embodiment of any of the foregoing embodiments, the camera is mounted on a camera arm, and the camera arm is configured to rotate from a retracted position to an extended position at a predetermined angular velocity during normal operation, so that the camera arm takes a predetermined duration to extend from the retracted position to the extended position during normal operation. To detect whether the reference position has changed in the image, the CMS controller is configured to measure a duration it takes for the camera arm to extend from a first position designated as the retracted position to a second position designated as the extended position; and determine that the reference position has changed in the image based on the measured duration being different from the predetermined duration.
[0023] A method for a camera rearview mirror system according to an example embodiment of the present disclosure includes: initiating an extension process for a camera arm, the extension process being configured to rotate the camera arm from a retracted position to an extended position at a predetermined rotational speed during normal operation, so that the camera arm takes a predetermined duration to extend from the retracted position to the extended position during normal operation; measuring the rotational speed of the camera arm during the extension process or the duration taken by the camera arm to complete the extension process; and determining that the camera arm has not yet reached the extended position based on the rotational speed or the duration differing from an expected value by more than a predetermined threshold, and adjusting a position of a human-machine interface (HMI) element within a vehicle display.
[0024] In another embodiment of the aforementioned embodiment, the measurement includes measuring the rotation speed of the camera arm during the camera arm extension process and the duration taken by the camera arm to complete the extension process; and determining that the camera arm has not yet reached the extended position is based on the measured rotation speed differing from an expected predetermined rotation speed threshold by more than a predetermined rotation speed threshold or the measured duration differing from an expected predetermined duration by more than a predetermined time threshold.
[0025] The embodiments, examples and alternatives of the preceding paragraphs, claims or the following description and drawings (including any of its various aspects or corresponding individual features) may be taken independently or in any combination. Features described in conjunction with one embodiment are applicable to all embodiments, unless these features are incompatible. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The present disclosure may be further understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:
[0027] 1A is a schematic front view of a commercial truck having a camera mirror system (CMS) for providing at least Class II and Class IV views.
[0028] Figure 1B is a schematic elevation view of the commercial truck of FIG. 1 when the trailer angle is zero.
[0029] Figure 1C is a schematic elevation view of the commercial truck of FIG. 1 when the trailer angle is non-zero.
[0030] 1D is a schematic top elevation view of the commercial truck of FIG. 1 , wherein the CMS provides Class II, Class IV, Class V, and Class VI views.
[0031] 2 is a schematic top perspective view of a vehicle cab including a display and an interior camera.
[0032] FIG. 3A schematically illustrates a simplified camera arm in a retracted position.
[0033] FIG. 3B schematically illustrates the simplified camera arm of FIG. 3A in an extended position.
[0034] FIG. 4 schematically illustrates an example CMS display including a plurality of human-machine interface (HMI) elements.
[0035] FIG. 5 schematically illustrates an example method for adjusting an HMI element.
[0036] FIG. 6 schematically illustrates an example implementation of the steps from the method of FIG. 5 .
[0037] FIG. 7 schematically illustrates an example technique for determining HMI adjustments.
[0038] FIG8 schematically illustrates an example method of determining that an HMI adjustment is required. DETAILED DESCRIPTION
[0039] In Figure 1A, Figure 1B , Figure 1C 1D , a schematic diagram of a commercial vehicle 10 is shown. The vehicle 10 includes a vehicle cab or “tractor 12” for towing a trailer 14. Although the vehicle 10 is depicted in this disclosure as a commercial truck, it should be understood that other types of vehicles may be used, and it should be understood that other configurations may be used for the vehicle cab 12 and / or trailer 14.
[0040] like Figure 1B and Figure 1C As shown, the hitch 11 mounts the trailer 14 to the tractor 12 and allows the trailer 14 to pivot relative to the tractor 12 during steering. The tractor 12 has a central longitudinal axis L1 and the trailer 14 has a central longitudinal axis L2. Figure 1B As shown, when the tractor 12 is not turning, the axes L1 and L2 are parallel or coaxial, and there is no angle between the axes L1 and L2. Figure 1C As shown, when the tractor 12 turns, an angle θ is formed between the axes L1 and L2. T The angle between the axes L1 and L2 (in Figure 1C The angle between the trailer and the trailer will be referred to herein as the "trailer angle".
[0041] 2 and continuing with FIGS. 1A and 1D , a camera mirror system (CMS) 15 includes a driver-side camera arm 16 a and a passenger-side camera arm 16 b ( FIG. 1A ) mounted on the exterior of a vehicle cab 12 . If desired, the camera arms 16 a, 16 b may also include conventional rearview mirrors integrated therewith, although the CMS 15 may be used to completely replace the rearview mirrors. In another example, multiple camera arms may be included on each side, each arm housing one or more cameras and / or rearview mirrors. As shown in FIG. 2 , the CMS 15 includes a CMS controller 13 , which includes processing circuitry that supports the operation of the CMS 15 and is operably connected to a memory. The processing circuitry may include one or more microprocessors, microcontrollers, application specific integrated circuits (ASICs), and the like.
[0042] 2 , a first electronic display and a second electronic display, which may be video displays 18 a, 18 b, are arranged on or near A-pillars 19 a, 19 b on each of the driver and passenger sides within the vehicle cab 12 to display Class II and Class IV views on respective sides of the vehicle 10 that provide rearward-facing side views along the vehicle 10 captured by external cameras 20 a, 20 b.
[0043] If video of Class V and Class VI views is also desired, the camera housing 16 c and camera 20 c may be disposed at or near the front of the vehicle 10 to provide these views ( Figure 1B ). A third display 18 c disposed within the cab 12 near the top center of the windshield may be used to display Class V and Class VI views toward the front of the vehicle 10 to the driver.
[0044] If video of a Class VIII view is desired, camera housings may be disposed on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII area of the vehicle 10. In such an example, the third display 18c may include one or more frames that display a Class VIII view. Alternatively, an additional display may be added adjacent to the first display 18a, the second display 18b, and the third display 18c that provides a display dedicated to providing a Class VIII view. The displays 18a, 18b, 18c face a driver area 34 within the cab 32 where an operator sits in a driver's seat 36.
[0045] 1A and 1D and 2, FIG3A schematically illustrates an example of a camera arm 16a in a retracted position, and FIG3B schematically illustrates a camera arm 16A in an extended position. Each camera arm 16A, 16B includes at least one rearward-facing camera 20 and is mounted to a respective base 24A, 24B mounted on the tractor 12. Each camera 20 is connected to its respective base 24 by a respective connection 28A, 28B (e.g., a ball joint), and the camera arms 16A, 16B are configured to rotate about their respective connections 28A, 28B between their respective retracted positions (shown in FIG3A for the camera arm 16A) and extended positions (shown in FIG3B for the camera arm 16A). In particular, each camera arm 16 is configured to extend from the retracted position to the extended position through an extension process, and is configured to retract from the extended position to the retracted position through a retraction process.
[0046] During normal operation, each camera arm 16 is configured to rotate at a predetermined rotational speed during its extension process, so that the camera arm 16 takes a predetermined duration to complete the extension process and to extend from the retracted position to the extended position. Similarly, during normal operation, the camera arm 16 is configured to rotate at the same or another predetermined rotational speed during the retraction process, so that the camera arm takes the same or a different predetermined duration to complete the retraction process and to retract from the extended position to the retracted position.
[0047] Each camera arm includes a corresponding angular motion sensor 26A, 26B, which is configured to measure the rotational angular velocity of the camera arm 16 rotating between the retracted position and the extended position. In one example, the angular motion sensor 22 is an inertial measurement unit (IMU). In another example, the angular motion sensor 22 is an accelerometer. In other examples, the angular motion sensor 222 can be any sensor or combination of sensors capable of detecting the relevant movement (including angular movement) of the sensing unit. The CMS controller 13 is configured to utilize the angular motion sensors 26A, 26B to measure the rotational velocity of the camera arm 16 during the extension and / or retraction process. It should be understood that for illustrative purposes, the camera arms 16A, 16B and connectors 26A, 26B are simplified, and the positioning of the cameras 20a, 20b described herein can be achieved using more complex support structures including articulated arms, electric articulated joints, and similar devices.
[0048] Continuing with the CMS 15 described in FIGS. 1-3 , FIG. 4 shows an example Class II image 204 from the camera 20 a displayed to the vehicle operator (e.g., on the electronic display 18 a). The image 204 includes a view of the trailer 210, the trailer wheels 220, and the road 230 and surroundings. Overlaid on the image 204 are a number of human machine interface (HMI) elements, including the distance line 142 (a projection of the trailer end onto the ground), the distance line 144 (30 m from the distance line 142), and the distance line 146 (100 m from the distance line 142). The wheel centers 150A, 150B, 150C, the wheel center 150, the rear lower trailer corner 160A, the rear upper trailer corner 160B, and the rear edge 162 of the wheel 220 are also shown. In alternative embodiments, alternative HMI elements, such as a trailer top end indicator, a reverse trajectory, etc., may be included in addition to or in place of some or all of the HMI elements shown in FIG. 4 , and the HMI elements and configurations shown are purely exemplary in nature.
[0049] Each HMI element is displayed at a specific location in the image 204 that is determined relative to a reference location in the image 204 (eg, the rear lower trailer corner 160A or the rear upper trailer corner 106B).
[0050] The reference position in image 204 may change due to various reasons, such as vehicle turning (forward or reverse), vehicle loading (which may cause the trailer to be more compressed on its tires and therefore closer to the ground than when unloaded), changes in road grade (which may cause camera 20 to tilt up or down relative to a horizontal ground plane), collision of external objects (e.g., tree branches) with camera arms 16A, 16B, and similar events.
[0051] For example, if the camera 20 is tilted downward relative to gravity without a change in orientation, objects in the image appear to be shifted upward and will be located higher in the field of view. The opposite is true for a camera 20 that is tilted upward. Similarly, if the camera 20 is rotated toward the vehicle 10, skew may occur, where objects near the side of the vehicle 10 appear larger than objects farther from the side of the vehicle 10.
[0052] When the reference position (e.g., of corners 160A and / or 160B) shifts (e.g., due to one of the events discussed above), this results in a change in the image 204 presented to the operator on the corresponding display (e.g., display 18A and / or 18B), which may result in inaccurate placement of the HMI elements 142, 144, 146. For example, if static positions are maintained for the HMI elements 142, 144, 146, these elements may overlap on the trailer 14 during a turn. When the cameras 20a, 20b are used to generate Class II / Class IV images, the above changes can apply to Class II and Class IV fields of view.
[0053] Many HMI elements, including those shown in FIG4 , convey information to the operator about the vehicle 10 and / or surroundings based at least in part on the relative position of the HMI element and the portion of the vehicle 10 that is in view. For example, the distance lines 142, 144, 146 represent the distance to the rear edge 162 of the trailer 110, and the position of the lines 142, 144, 146 relative to the end of the trailer 14. In addition, the orientation of the lines as they extend away from the end of the trailer 14 provides an indication of which aspects of the image are at the distance from the lines 142, 144, 146.
[0054] Movement of the reference position in the image (e.g., trailer corner 160A) relative to the camera 20, whether caused by movement of the trailer 14 or unintended movement of the camera 20 (e.g., due to contact with a tree branch causing the camera arm 16 to not reach the intended extended position but to reach some other extended position), will result in the HMI element being placed at an incorrect position in the CMS image without adjusting the position of the HMI element. As discussed in more detail below, the CMS controller 13 is configured to detect, using software within the CMS 15, when the reference position has changed in the image and to adjust the HMI position in response to the change so that the HMI elements within the image are moved to account for the shifted position and allow for continued accuracy of the information being conveyed.
[0055] 5 is a flow chart of an example method 400 for adjusting the position of an HMI element in an electronic display 18 of a commercial vehicle. An image depicting a trailer 14 of a commercial vehicle 10 and its surroundings is obtained from a camera 20 (step 402). The image is displayed on an electronic display 18 of the commercial vehicle 10 (step 404). A reference position of a first reference point of the trailer 14 in the image is determined (e.g., by a CMS controller 13) (step 406). HMI elements 142, 144, 146 are superimposed into the image on the electronic display 18 at the HMI position determined relative to the reference position (step 408).
[0056] A determination is made (e.g., by the CMS controller 13) as to whether the reference position has changed in the image (step 410). If the reference position has changed ("yes" at step 410), the HMI position is adjusted in response to the change (step 412). Otherwise, if the reference position has not changed in the image ("no" at step 410), the HMI position is maintained (step 414). In one example, the change must exceed a predetermined threshold (e.g., a minimum pixel change) in order to trigger the adjustment in step 412. As discussed in more detail below, the determination in 410 as to whether the reference position has changed may be based on image analysis and / or based on movement of the camera arm 16. Furthermore, the method 400 may be performed for multiple HMI elements, each HMI element having its own corresponding HMI position determined relative to the reference position.
[0057] As described above, the HMI elements may be distance markers 142, 144, and / or 146 that are superimposed to extend laterally in the image away from the sides of the trailer 14 (as represented by the rear edge 162), and the reference point of the trailer 14 may be, for example, the rear corner 160A or 160B. In one example, adjusting the HMI position in step 412 includes adjusting the HMI position horizontally and / or vertically to minimize or avoid overlap of the HMI element(s) and the trailer 14 in the image, and instead overlapping the HMI element with a road or similar portion of the image.
[0058] FIG. 6 illustrates an example implementation 450 of step 406 from FIG. 4 for determining a reference position of a reference point of the trailer 14 in the CMS image.
[0059] FIG. 7 shows two annotated CMS images 206A, 206B that will be used to discuss an implementation 450 of step 406. For purposes of illustration, images 206A, 206B are presented adjacent to each other in FIG. 7 (even though in reality they may be provided on separate CMS displays 18A, 18B). Image 206A represents a CMS image from camera 20A and image 206B represents a CMS image from camera 20B. Point 612 represents a calibration point of a commercial vehicle (e.g., the front lower corner of trailer 14) having coordinates (x t ,y t ) represents the rear corner of the trailer (the first reference point from step 406) (eg, the lower rear corner 160A of the trailer) and has coordinates (x i ), y i ) represents a third reference point of the commercial vehicle 10 (eg, wheel center 150A, 150B, or 150C of FIG. 4 ).
[0060] Arc 650 represents the trajectory / range of possible values for a third reference point in the image (e.g., the range of possible values for one wheel center 150), and arc 640 represents the trajectory / range of possible values for a first reference point in the CMS image (e.g., the range of possible values for the lower rear trailer corner 160A). The labeled distance ΔY represents the vertical distance between arcs 640, 650. Trailer reference line 614 extends through first reference point 616, second reference point 612, and third reference point 618, and represents the orientation of trailer 14 relative to tractor 12. Trailer image angle θ T1 y represents an angle defined relative to trailer line 614 (which in the example of FIG. 7 is defined relative to image edge 620 ).
[0061] Referring again to FIG. 6 , the location of a trailer calibration point (e.g., second reference point 612) is determined in step 452. For example, this step can be performed using conventional image analysis. The trailer calibration point is the trailer location relative to which the trailer reference line 614 is drawn. In one example, the calibration point 612 remains substantially constant throughout vehicle operation and can be detected using trailer markings and / or marking independent image analysis.
[0062] Determine the trailer angle θ between the tractor longitudinal axis L1 and the trailer longitudinal axis L2 T (See Figure 1C ) (Step 454). Trailer angle θ T The trailer angle θ may be detected from a trailer angle sensor (not shown), a derived trailer angle determined from image analysis, composite data from multiple vehicle sensors, or any combination thereof. In the case of implementation 450, the trailer angle θ T represents the three-dimensional angle, and the trailer image angle θ T1 Represents a two-dimensional angle.
[0063] Based on the trailer angle θ T The trailer image angle θ is determined (e.g., using a lookup table) T1 (Step 456). A third reference point 618 is determined corresponding to the intersection of trailer reference line 614 and arc 650 (Step 458). A first reference position of first reference point 616 is determined based on the intersection of trailer reference line 614 and arc 650 and the distance ΔY (pixel deviation) between arcs 640, 650 (Step 460).
[0064] Therefore, step 406 may be based at least on the trailer angle θ T and arc 640, which represents a predetermined range of possible positions of the first reference point in the image.
[0065] In addition, the execution of step 406 may include determining the trailer angle θ based on the trailer angle θ. TDetermine the trailer image angle θ T1 , and determining a reference position of the first reference point based on the intersection of the trailer reference line 614 and the arc 650 and the distance between the arcs 640 and 650.
[0066] The process described and illustrated in Figures 6 and 7 may be repeated to dynamically maintain the correct positioning of the trailer end line. Furthermore, the process of Figures 6 and 7 may be applied to other HMI elements and is not limited to the trailer end line.
[0067] FIG8 schematically illustrates an example method 800 for determining that an HMI position adjustment is required. As described above, the camera arm 16 extension process extends the camera arm from a retracted position (see, e.g., FIG3A ) to an extended position (see, e.g., FIG3B ). During normal operation, the extension is performed at a predetermined rotational speed and takes a predetermined duration to complete. If the extension process is performed at an irregular speed or for an irregular duration, this may be interpreted as an indication that the camera arm 16 is obstructed by an external object (e.g., a tree branch) or that the camera arm 16 is damaged (e.g., due to contact with an external object) and thus the camera 20 of the camera arm 16 is not properly positioned (e.g., properly positioned and oriented).
[0068] 8 , a camera arm extension process is initiated (step 702). The rotational speed of the camera arm 16 during the extension process is measured using the angular motion sensor 26 of the camera arm 16, and the duration taken for the camera arm 16 to complete the extension process is also measured (step 704). The measured rotational speed is compared to an expected predetermined rotational speed, and it is determined whether the difference between the measured rotational speed and the expected rotational speed exceeds a speed threshold (step 706).
[0069] If the difference exceeds the threshold (“yes” at step 706), it is determined that the camera arm 16 has not reached the extended position, and the position of the HMI element within the vehicle display 18 is adjusted (step 708). Step 708 also represents determining that the first reference point of step 406 of method 400 has changed. If the difference does not exceed the threshold (“no” at step 706), the method proceeds to step 710.
[0070] In step 710, the measured duration is compared with the expected predetermined duration, and it is determined whether the difference between the measured duration and the expected duration exceeds the duration threshold (step 7010). If the difference exceeds the duration threshold (step 710 is "yes"), the method proceeds to step 708. Otherwise, if the difference does not exceed the duration threshold (step 710 is "no"), it is determined that the camera arm 16 has correctly reached the extended position, and accordingly, it is determined that the camera 20 is correctly positioned and oriented.
[0071] Although the measurement and comparison of both rotation speed and duration are discussed in steps 704, 706 and 710, it should be understood that the measurement and comparison of rotation speed can be omitted so that the determination of step 708 is based on duration rather than rotation speed, or the measurement and comparison of duration can be omitted so that the determination of step 708 is based on rotation speed rather than duration.
[0072] The above features describe how to automatically position and reposition HMI elements based on various factors that cause relative motion of the trailer reference point in the CMS image (e.g., vehicle turning, vehicle positioning, road grade changes, and / or irregular camera arm extensions).
[0073] Although example embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. Therefore, the following claims should be studied to determine their true scope and content.
Claims
1. A method for adjusting the position of a human machine interface (HMI) element in a commercial vehicle display, comprising: displaying an image from a camera fixed to the commercial vehicle on an electronic display of the commercial vehicle, the image depicting the trailer of the commercial vehicle and its surroundings; determining a reference position of a reference point of the trailer in the image; superimposing an HMI element into the image on the electronic display at an HMI position determined relative to the reference position; as well as It is detected whether the reference position has changed in the image, and the HMI position is adjusted in response to the change. 2 . The method of claim 1 , wherein the HMI element is a distance marker that is superimposed to extend laterally away from a side of the trailer in the image. 3 . The method of claim 2 , wherein adjusting the HMI position comprises adjusting the HMI position to minimize or avoid overlap of the HMI element and the trailer in the image.
4. The method of claim 1, wherein the reference point of the trailer is a rear corner of the trailer.
5. The method of claim 1 , wherein determining the reference position of the reference point of the trailer is based on: a first trailer angle, the first trailer angle being the angle between a longitudinal axis of the trailer and a longitudinal axis of a tractor connected to the trailer; and A first arc represents a predetermined range of possible positions of the reference point in the image.
6. The method of claim 5, wherein the reference point of the trailer is a first reference point, and determining the reference position of the first reference point comprises: determining a trailer image angle based on the trailer angle, the trailer image angle being an angle defined in one of the images relative to a trailer reference line in the image, the trailer reference line extending from a second reference point of the commercial vehicle toward a third reference point of the trailer, wherein the second reference point is closer to a front of the trailer than the third reference point, and the third reference point is closer to the first reference point than the second reference point; and The reference position of the first reference point is determined based on the trailer image angle, an intersection of the trailer reference line and a second arc representing a predetermined range of possible positions of the third reference point in the image, and a distance between the first arc and the second arc in the image.
7. The method according to claim 6, wherein: The third reference point is located on the rear wheels of the trailer; The first reference point is located near the front lower corner of the trailer; and The distance between the first arc and the second arc in the image is a perpendicular distance between the first arc and the second arc.
8. The method of claim 1 , wherein the camera is mounted on a camera arm, the camera arm being configured to rotate from a retracted position to an extended position at a predetermined rotational speed during normal operation, and the detecting comprises: measuring a rotational speed of the camera arm as it extends from a first position designated as the retracted position to a second position designated as the extended position; as well as The reference position is determined to have changed in the image based on the measured rotational speed being different from the predetermined rotational speed.
9. The method of claim 1 , wherein the camera is mounted on a camera arm, the camera arm being configured to rotate from a retracted position to an extended position at a predetermined angular velocity during normal operation, such that the camera arm takes a predetermined duration to extend from the retracted position to the extended position during normal operation, and the detecting comprises: measuring a duration it takes the camera arm to extend from a first position designated as the retracted position to a second position designated as the extended position; as well as The reference position is determined to have changed in the image based on the measured duration being different from the predetermined duration.
10. A camera monitoring system (CMS) for a vehicle, comprising: a camera configured to record images of the trailer of the commercial vehicle and its surroundings; Electronic displays; as well as The CMS controller is configured to: displaying the image on the electronic display; determining a reference position of a reference point of the trailer in the image; superimposing a human machine interface (HMI) element into the image on the electronic display at an HMI position determined relative to the reference position; and It is detected whether the reference position has changed in the image, and the HMI position of the HMI element is adjusted in response to the change.
11. The CMS of claim 10, wherein the HMI element is a distance marker superimposed to extend laterally away from a side of the trailer in the image.
12. The CMS of claim 11, wherein to adjust the HMI position of the HMI element on the electronic display, the CMS controller is configured to adjust the HMI position to minimize or avoid overlap of the HMI element and the trailer in the image.
13. The CMS of claim 10, wherein the reference point of the trailer is a rear corner of the trailer.
14. The CMS of claim 10, wherein the CMS controller is configured to determine the reference position of the reference point based on: a first trailer angle, the first trailer angle being the angle between a longitudinal axis of the trailer and a longitudinal axis of a tractor connected to the trailer; and A first arc represents a predetermined range of possible positions of the reference point in the image.
15. The CMS of claim 14, wherein the reference point of the trailer is a first reference point, and to determine the reference position of the first reference point, the CMS controller is configured to: determining a trailer image angle based on the trailer angle, the trailer image angle being an angle defined in one of the images relative to a trailer reference line in the image, the trailer reference line extending from a second reference point of the commercial vehicle toward a third reference point of the trailer, wherein the second reference point is closer to a front of the trailer than the third reference point, and the third reference point is closer to the first reference point than the second reference point; and The reference position of the first reference point is determined based on the trailer image angle, an intersection of the trailer reference line and a second arc representing a predetermined range of possible positions of the third reference point in the image, and a distance between the first arc and the second arc in the image.
16. The CMS of claim 15, wherein: in: The third reference point is located on the rear wheels of the trailer; The first reference point is located near the front lower corner of the trailer; and The distance between the first arc and the second arc in the image is a perpendicular distance between the first arc and the second arc.
17. The CMS of claim 10, wherein: The camera is mounted on a camera arm, the camera arm being configured to rotate from a retracted position to an extended position at a predetermined rotational speed during normal operation; and In order to detect whether the reference position has changed in the image, the CMS controller is configured to: measuring a rotational speed of the camera arm as it extends from a first position designated as the retracted position to a second position designated as the extended position; and The reference position is determined to have changed in the image based on the measured rotational speed being different from the predetermined rotational speed.
18. The CMS of claim 10, wherein: The camera is mounted on a camera arm, the camera arm being configured to rotate from a retracted position to an extended position at a predetermined angular velocity during normal operation, such that the camera arm takes a predetermined duration to extend from the retracted position to the extended position during normal operation; and In order to detect whether the reference position has changed in the image, the CMS controller is configured to: measuring a duration it takes the camera arm to extend from a first position designated as the retracted position to a second position designated as the extended position; and The reference position is determined to have changed in the image based on the measured duration being different from the predetermined duration.
19. A method for a camera rearview mirror system: initiating an extension process for the camera arm, the extension process being configured to rotate the camera arm from a retracted position to an extended position at a predetermined rotational speed during normal operation such that the camera arm takes a predetermined duration to extend from the retracted position to the extended position during normal operation; measuring a rotation speed of the camera arm during the extension process or a duration taken by the camera arm to complete the extension process; as well as Based on the rotation speed or duration differing from an expected value by more than a predetermined threshold, determining that the camera arm has not reached the extended position and adjusting a position of a human machine interface (HMI) element within a vehicle display.
20. The method of claim 19, wherein: The measuring includes measuring the rotation speed of the camera arm during the camera arm extension process and the duration taken by the camera arm to complete the extension process; and Determining that the camera arm has not reached the extended position is based on a measured rotational speed differing from an expected predetermined rotational speed threshold by more than a predetermined rotational speed threshold or a measured duration differing from an expected predetermined duration by more than a predetermined time threshold.
