Visual field follow-up calculation method based on CMS camera automatic learning

Abstract

The invention relates to a visual field follow-up calculation method based on CMS camera automatic learning, and the method comprises the following steps: camera calibration: obtaining internal parameters and external parameters; estimating a vehicle motion trail, including acquiring a tractor motion trail and a trailer motion trail; visual feature detection: outputting an observation pixel coordinate sequence of the trailer in an image coordinate system; calculating the tail position of the trailer, wherein a theoretical pixel coordinate sequence is obtained based on the trailer motion trail; performing alignment matching on the observation pixel coordinate sequence and the theoretical pixel coordinate sequence in time; and outputting the optimized trailer tail position. According to the invention, the tail position of the trailer is synchronized to the CMS system, manual input is not needed, and the labor cost and errors are solved fundamentally. The position of the tail of the trailer is calculated in the CMS system and then displayed in the CMS system, and time delay and data transmission errors are avoided.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a computational method for field-of-view tracking based on automatic learning from a CMS camera. Background Technology

[0002] Commercial trucks typically consist of a tractor unit and a trailer articulated to it. To improve automated control capabilities and safety, it is now common practice to calculate the trailer's field of vision. For example, the trailer's field of vision is a key parameter affecting the vehicle's turning radius, lateral stability, and trajectory prediction accuracy, significantly impacting both driver assistance and autonomous driving functions. Furthermore, the CMS (Content Management System) needs to accurately display the trailer's blind spots during steering, requiring both the trailer's field of vision and steering angle.

[0003] Currently, there are two main methods to accurately obtain the field of vision of a vehicle (especially a trailer). The first is based on the external dimensions (length, width, and height) of the trailers that are permitted to be attached, which are registered on the vehicle registration certificate at the time of manufacture. This is the most authoritative and common method. However, for large commercial trailers, there are situations such as mid-journey trailer swapping. After the swap, we can only obtain the trailer's field of vision based on the trailer's information. The second method is to measure the trailer's field of vision at the vehicle inspection point using visual or infrared light curtain methods. However, it is difficult for these two methods to synchronize the trailer's field of vision information to the tractor's infotainment system. As a result, after the trailer's field of vision changes, the Type II field of vision of the CMS cannot accurately display the trailer's blind spot in the turning field of vision. Therefore, this application directly integrates visual and motion information into the CMS system to accurately calculate the field of vision.

[0004] To overcome the existing limitations, it is necessary to develop a calculation method for field-of-view tracking based on automatic learning from a CMS camera. Summary of the Invention

[0005] This application provides one or more embodiments of a field-of-view calculation method based on CMS camera automatic learning, which can automatically, accurately, and in real time estimate the field of view of a trailer during normal driving, avoiding manual intervention and improving the vehicle's intelligence level.

[0006] To achieve the above objectives, the main technical solutions adopted in this application include: A method for calculating field-of-view tracking based on CMS camera automatic learning includes the following steps: S1. Camera calibration, wherein the camera calibration includes acquiring internal and external parameters installed on the tractor unit of the vehicle, wherein the external parameters characterize the spatial transformation relationship between the camera coordinate system and the ground coordinate system; S2. Estimate the vehicle trajectory, wherein estimating the vehicle trajectory includes obtaining the tractor trajectory and trailer trajectory based on the acquired vehicle equipment information and vehicle motion information, wherein the vehicle equipment information includes the currently stored trailer tail position. S3. Visual feature detection, wherein the visual feature detection includes analyzing the images acquired by the camera in real time, identifying trailer information in the images, and thus outputting the sequence of observed pixel coordinates of the trailer in the image coordinate system; S4. Calculate the rear position of the trailer, wherein calculating the rear position of the trailer includes: S41: Using the internal parameters and external parameters obtained in step S1, the trailer motion trajectory in the ground coordinate system obtained in S2 is transformed to the image coordinate system to obtain the theoretical pixel coordinate sequence; S42: Align and match the observed pixel coordinate sequence output in step S3 with the theoretical pixel coordinate sequence obtained in step S41 in time; S43: Using the currently stored trailer rear position as the optimization variable, construct and solve an optimization model that minimizes the difference between the theoretical pixel coordinate sequence and the observed pixel coordinate sequence, and output the optimized trailer rear position.

[0007] Preferably, in step S5, a closed-loop update is performed, where the optimized trailer field of view estimate output by step S43 is fed back to step S2 as the new currently stored trailer rear position.

[0008] Preferably, in step S2, a bicycle model is used to calculate the trajectory of the tractor, and a single-axle trailer model is used to calculate the trajectory of the trailer.

[0009] Preferably, in step S2, the vehicle equipment information includes the length of the tractor vehicle, the width of the tractor vehicle, the wheelbase of the front and rear wheels of the tractor vehicle, the track width of the front and rear wheels of the tractor vehicle, and the position of the articulation point; the vehicle motion information includes vehicle speed and steering wheel angle data.

[0010] Preferably, in step S3, a target detection neural network is used to detect trailer components in the image, and the observed pixel coordinate sequence is calculated based on the detected wheel or vehicle body bounding boxes.

[0011] Preferably, the YOLO model is used to detect the trailer body and trailer wheels.

[0012] Preferably, in step S1, the internal parameters are obtained using the OpenCV checkerboard calibration algorithm.

[0013] Preferably, in step S1, the external parameters are obtained by using a perspective point transformation solution algorithm.

[0014] This application discloses a computer device, including: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the method described above.

[0015] This application discloses a computer-readable storage medium storing computer instructions for causing a computer to perform the above-described method.

[0016] One or more embodiments provided in this application have the following advantages: 1. Fully automatic and self-learning, no need for manual input of the field of view range, the system automatically learns and updates the field of view range while the vehicle is in motion; 2. Low cost and high practicality: It only uses the CMS camera and body equipment information installed in the vehicle, without the need to install any additional sensors, which is low cost and easy to deploy on a large scale. 3. By using motion model prediction and deep learning, and applying optimization theory for parameter identification, the computational accuracy and system stability are improved. 4. Online measurement does not affect transportation efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart of an embodiment of this application is shown.

[0019] Figure 2 The calibration plate pattern is shown.

[0020] Figure 3 The location of the calibration plate is shown.

[0021] Figure 4 The required vehicle equipment information for the truck is displayed.

[0022] Figure 5A and 5B The results of the trailer body wheel inspection are shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Please see Figure 1 As shown in the figure, this application discloses a method for calculating field of view tracking based on automatic learning of a CMS camera, including the following steps: S1. Calibration, wherein the calibration includes acquiring internal and external parameters installed on the tractor unit of the vehicle, wherein the external parameters characterize the spatial transformation relationship between the camera coordinate system and the ground coordinate system of the camera; S2. Estimate the vehicle trajectory, wherein estimating the vehicle trajectory includes obtaining the tractor trajectory and trailer trajectory based on the acquired vehicle equipment information and vehicle motion information, wherein the vehicle equipment information includes the currently stored trailer tail position. S3. Visual feature detection, wherein the visual feature detection includes analyzing the images acquired by the camera in real time, identifying trailer information in the images, and thus outputting the sequence of observed pixel coordinates of the trailer in the image coordinate system; S4. Calculate the rear position of the trailer, wherein calculating the rear position of the trailer includes: S41: Using the internal parameters and external parameters obtained in step S1, the trailer motion trajectory in the ground coordinate system obtained in S2 is transformed to the image coordinate system to obtain the theoretical pixel coordinate sequence; S42: Align and match the observed pixel coordinate sequence output in step S3 with the theoretical pixel coordinate sequence obtained in step S41 in time; S43: Using the currently stored trailer rear position as the optimization variable, construct and solve an optimization model that minimizes the difference between the theoretical pixel coordinate sequence and the observed pixel coordinate sequence, and output the optimized trailer rear position.

[0025] Specifically, the calculation method for trailer visibility tracking mainly includes camera calibration, estimation of vehicle trajectory, visual feature detection, and calculation of the trailer's rear position.

[0026] The purpose of camera calibration is to obtain the camera's internal parameters and the external parameters between the ground and the camera. A calibration plate may look like this. Figure 1 As shown, the calibration plate is positioned as follows: Figure 2 As shown. The calibration method can use OpenCV's checkerboard calibration algorithm to calibrate the internal parameter matrix K. Among them, f x f y C is the focal length (in pixels). x C y The coordinates of the main point.

[0027] The calibration method can employ the Perspective-n-Points algorithm to solve for the external parameters between the CMS camera and the ground. These external parameters can include, but are not limited to, the rotation matrix R and the translation vector t.

[0028] The vehicle trajectory includes the trajectory of the tractor and the trailer. Based on vehicle equipment information (including but not limited to: tractor length, tractor width, tractor wheelbase, tractor track width, and articulation point location), specific vehicle body data to be considered can be as follows: Figure 3 As shown, by combining vehicle motion information (including but not limited to vehicle speed and steering wheel angle data), the trajectory of the tractor and the traction direction at the articulation point are calculated in real time. For the trajectory calculation of the trailer, the vehicle length can be assumed to be a fixed 10 meters. Then, based on the trajectory of the tractor and the traction direction at the articulation point, the angle between the trailer and the tractor is calculated. It should be noted that the angle varies depending on the field of vision of different trailers, because the angle is closely related to the field of vision of the trailer, and the field of vision of the trailer is the main factor affecting the angle between the trailer and the tractor.

[0029] For example, the trajectory of the tractor can be obtained using a bicycle model, calculated by taking the pose of a point on the tractor (which could be the center of the rear axle) as the basis for the calculation (x). tc y tc , Φ tc ), Where v represents vehicle speed, k represents time, and L represents the wheelbase of the tractor. Indicates the sampling interval; Indicates the front wheel steering angle; Indicates the transmission ratio.

[0030] Update the position of the hinge point, that is, update the position of the hinge point from (x... h (k), y h (k) is updated to (x) h (k+1), y h (k+1)): Where, d h It is a fixed value representing the longitudinal distance between the articulation point and a certain point on the tractor (which can be the center of the rear axle).

[0031] Calculate the hinge angle at time k: ; in, This represents the heading angle of the trailer at time k. This represents the heading angle of the tractor at time k.

[0032] Then, update the trailer's heading angle at time k+1 using the articulation angle: .

[0033] Calculate the trailer position at time K+1 based on the articulation point position and the trailer's heading angle: Among them, L t This is the current stored position of the trailer's rear.

[0034] Therefore, a trailer movement trajectory with a timestamp can be output: , where t k Represents a timestamp.

[0035] Meanwhile, steps S3 and S2 run in parallel to directly obtain the trailer's observation information from the image. A pre-trained YOLO object detection network is used, which can identify the trailer body and wheels. Since the position of the trailer's rear is estimated, the detection results are only included in the next step of automatic learning, i.e., selected into the observation pixel coordinate sequence, if both the trailer body and wheels are detected. , where t m Represents a timestamp.

[0036] Where u and v represent coordinates; m represents the sequence number of the observed sample.

[0037] S41, using the internal and external parameters calibrated in step S1, calculate the coordinates of the theoretical point on the image. Based on the trailer's movement trajectory Including intrinsic and extrinsic parameters, the coordinates of each theoretical point on the image are obtained. .

[0038] S42, Read all valid observation points output from step S3 within the same time period. Based on t m and t k Alignment is performed for each valid observation point. Find the coordinates of the theoretical point with the smallest time difference .

[0039] S43. In order to calculate the rear position of the trailer so that the theoretical pixel coordinates calculated based on the rear position are closest to the observed pixel coordinates, the gradient descent method is used to optimize the position based on the observation results in step S3 to obtain the optimal rear position of the trailer.

[0040] S5. After obtaining the optimal trailer tail position, the obtained optimal trailer tail position can be updated to the currently stored trailer tail position in step S2, and then the process can be repeated.

[0041] This invention has the following three advantages: 1. By synchronizing trailer information to the CMS system, manual input is no longer required, thus fundamentally solving the problems of labor costs and errors.

[0042] 2. The position of the trailer's rear is calculated within the CMS system and then displayed in the CMS system, avoiding delays and data transmission errors.

[0043] 3. Based on the default 10-meter trailer parameters, it automatically learns by combining visual detection results. Therefore, compared with the existing method of calculating the running trajectory of the rear position of the vehicle, it is more adaptable and has better results.

[0044] The application also provides a computer-readable storage medium. The methods described in the embodiments of this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the embodiments described above.

[0045] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of any embodiment of this application.

[0046] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

[0047] The apparatus or unit described in the above embodiments can be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0048] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0049] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0051] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0056] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A calculation method for field-of-view tracking based on CMS camera automatic learning, characterized in that, Includes the following steps: S1. Camera calibration, wherein the camera calibration includes acquiring internal and external parameters installed on the tractor unit of the vehicle, wherein the external parameters characterize the spatial transformation relationship between the camera coordinate system and the ground coordinate system; S2. Estimate the vehicle trajectory, wherein estimating the vehicle trajectory includes obtaining the tractor trajectory and trailer trajectory based on the acquired vehicle equipment information and vehicle motion information, wherein the vehicle equipment information includes the currently stored trailer tail position. S3. Visual feature detection, wherein the visual feature detection includes analyzing the images acquired by the camera in real time, identifying trailer information in the images, and thus outputting the sequence of observed pixel coordinates of the trailer in the image coordinate system; S4. Calculate the rear position of the trailer, wherein calculating the rear position of the trailer includes: S41: Using the internal parameters and external parameters obtained in step S1, the trailer motion trajectory in the ground coordinate system obtained in S2 is transformed to the image coordinate system to obtain the theoretical pixel coordinate sequence; S42: Align and match the observed pixel coordinate sequence output in step S3 with the theoretical pixel coordinate sequence obtained in step S41 in time; S43: Using the currently stored trailer rear position as the optimization variable, construct and solve an optimization model that minimizes the difference between the theoretical pixel coordinate sequence and the observed pixel coordinate sequence, and output the optimized trailer rear position.

2. The method according to claim 1, characterized in that, Also includes: S5, Closed-loop update: Feed back the optimized trailer tail position estimate output by S43 to S2 as the new currently stored trailer tail position.

3. The method according to claim 1, characterized in that, In step S2, a bicycle model is used to calculate the trajectory of the tractor, and a single-axle trailer model is used to calculate the trajectory of the trailer.

4. The method according to claim 3, characterized in that, In step S2, the vehicle equipment information includes the length of the tractor vehicle, the width of the tractor vehicle, the wheelbase of the front and rear wheels of the tractor vehicle, the track width of the front and rear wheels of the tractor vehicle, and the position of the articulation point; the vehicle motion information includes vehicle speed and steering wheel angle data.

5. The method according to claim 1, characterized in that, In step S3, a target detection neural network is used to detect trailer components in the image, and the sequence of observed pixel coordinates is calculated based on the detected wheel or vehicle body bounding boxes.

6. The method according to claim 5, characterized in that, The YOLO model is used to detect the trailer body and trailer wheels.

7. The method according to claim 1, characterized in that, In step S1, the internal parameters are obtained using the OpenCV checkerboard calibration algorithm.

8. The method according to claim 1, characterized in that, In step S1, the external parameters are obtained using the perspective point transformation solution algorithm.

9. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 8.

Images