Flexible trailer detection method and system based on monocular vision

Abstract

The application provides a flexible towing detection method and system based on monocular vision, wherein the method comprises: acquiring monocular vision information and 3D point cloud information of a towing vehicle respectively; fusing the monocular vision information and the 3D point cloud information to obtain image information, wherein the image information at least includes state information of a pose, a position and a motion of a flexible towing part; performing feature extraction and target detection on the image information to obtain target information of the flexible towing part; performing monocular direction estimation of a front pose on the target information to obtain pose information of the flexible towing part; and generating a control instruction to control vehicle tracking based on the pose information and in combination with dynamic changes of a front vehicle. The application realizes efficient detection of a flexible towing object by using monocular vision technology, and improves the safety and operation efficiency of a towing vehicle.

Description

Technical Field

[0001] The present invention relates to the field of monocular vision technology, and in particular to a flexible trailer detection method and system based on monocular vision. Background Art

[0002] With the development of the logistics and transportation industries, flexible trailer vehicles play a vital role in cargo transportation. However, traditional flexible trailer detection systems have limitations, such as reliance on a large number of sensors, complex installation locations, high costs, and stringent equipment requirements. Current flexible trailer detection systems primarily rely on sensors such as lidar and camera arrays to monitor and detect trailer vehicles. While these sensors offer a certain degree of accuracy, they are expensive to install and maintain, and they can introduce blind spots and errors during flexible trailer detection.

[0003] Therefore, existing flexible trailer detection systems usually have the following technical defects:

[0004] (1) High cost: Traditional flexible towed vehicle detection systems rely on multiple sensors, which have complex installation locations and increase installation and maintenance costs.

[0005] (2) Large blind spots and errors: The sensor layout in traditional systems may have blind spots or dead angles, resulting in certain errors in the monitoring and detection of flexible towed objects, reducing the accuracy and reliability of the system.

[0006] (3) Installation location restrictions: The sensor installation location of traditional systems is usually restricted and needs to be installed in a fixed or specific location, which may limit the applicability and flexibility of the system in different scenarios and environments.

[0007] The search found:

[0008] The Chinese invention patent application, "A Collision Detection Method, Device, Electronic Device, and Storage Medium," with application number 202211667015.4 and filing date December 22, 2022, describes a method for predicting a trailer collision by obtaining the obstacle's trajectory, establishing a trailer motion model, performing feature matching, detecting whether obstacles intersect, performing convex optimization on intersecting obstacles, and marking the target obstacle's collision zone. Its technical features include extracting point cloud information, determining the obstacle's contour and trajectory, establishing a trailer motion model using an on-board radar sensor, performing feature matching based on factors such as the tractor's speed and angle, targeting intersecting obstacles as target obstacles, determining their hinge points, and performing convex optimization on the target obstacles to obtain the collision zone. This method can be used to formulate driving routes and issue warnings, thereby improving vehicle safety. However, this method fails to account for the complex and changing real-world road conditions and unexpected accidents, and the trailer motion model may contain certain simplifications and assumptions, failing to fully and accurately reflect the actual situation.

[0009] The Chinese invention patent application "Gap Measurement for Vehicle Escort" with application number 202211662662.6 and application date 2017-10-26 describes a variety of methods, controllers, and algorithms for identifying specific vehicles (such as the rear of a platoon companion) and tracking their tails. This technology can be used in combination with various distance measurement technologies and is applicable to vehicle platooning and convoy systems, and describes a technology for fusing different vehicle sensor data for at least partially automatic control. The described method is suitable for use in combination with various vehicle control applications, including platooning, convoy, and connected driving applications. The technology also involves the use of a distance measurement module to measure and maintain the desired gap between the lead vehicle and the following vehicle. However, this technology still fails to solve the problems of tracking errors, environmental interference, and algorithm optimization for complex vehicle platooning or convoy situations. In practical applications, integration and compatibility with other vehicle systems also need to be considered.

[0010] Therefore, how to accurately predict trailer collisions in complex and ever-changing real-world road environments, taking into account unexpected situations while optimizing the accuracy and adaptability of trailer motion models, has become an urgent challenge in the field. Currently, no collision detection methods or reports similar to the present invention have been found, nor have any relevant domestic or international data been collected that can fully address the aforementioned issues. Summary of the Invention

[0011] In view of the above-mentioned deficiencies in the prior art of flexible trailer vehicle detection and control technology, the object of the present invention is to provide a flexible trailer detection method and system based on monocular vision.

[0012] To achieve the above object, the present invention is implemented through the following technical solutions:

[0013] According to one aspect of the present invention, a flexible trailer detection method based on monocular vision is provided, comprising:

[0014] Obtain monocular vision information and 3D point cloud information of the trailer vehicle respectively;

[0015] fusing the monocular vision information and the 3D point cloud information to obtain image information, wherein the image information includes at least information on the posture, position, and motion of the flexible towing part;

[0016] Performing feature extraction and target detection on the image information to obtain target information of the flexible trailer part;

[0017] Performing monocular direction estimation of the forward posture of the target information to obtain posture information of the flexible towing part;

[0018] Based on the posture information and combined with the dynamic changes of the vehicle ahead, path planning and decision adjustments are performed to generate control instructions to control vehicle tracking.

[0019] Preferably, the separately acquiring monocular visual information and 3D point cloud information of the towing vehicle includes:

[0020] A monocular vision camera is used to capture real-time image data of the trailer vehicle. The image data is used to display the shape, contour, and texture information of the flexible trailer part, thereby obtaining preliminary morphology and position information of the flexible trailer part, i.e., obtaining monocular vision information.

[0021] A lidar sensor is used to acquire point cloud data of the trailer vehicle. The point cloud data is used to describe the three-dimensional structure, spatial position, and speed information of the flexible trailer part, that is, to obtain 3D point cloud information, providing auxiliary depth information and spatial positioning reference for the image data.

[0022] Preferably, fusing the monocular vision information and the 3D point cloud information includes:

[0023] The monocular vision information and the 3D point cloud information are deeply fused and processed to obtain current state and behavior information for accurately identifying the flexible trailer part, that is, image information.

[0024] Preferably, the performing feature extraction and target detection on the image information includes:

[0025] Providing a pre-trained deep learning model, utilizing the model to perform feature extraction, trailer part recognition, and leading vehicle direction estimation on the image information;

[0026] The trailer part is subjected to contour extraction and target detection, and combined with the estimation result of the leading vehicle direction, the flexible trailer part is preliminarily positioned and morphologically analyzed to obtain target information.

[0027] Preferably, performing monocular direction estimation of the forward posture of the target information to obtain the posture information of the flexible towing part includes:

[0028] A monocular direction estimation model is provided, and the target information is used as input to the model to extract key feature points of the target information. The monocular direction estimation model estimates the direction of each individual body in the target information, and then optimizes the directions by combining the directions of the respective rigid bodies with the intersection constraints between the rigid bodies to obtain the direction along the arc of the tow, i.e., the key feature points of the target information.

[0029] Matching the key feature points with the prior pose information of the vehicle ahead, and evaluating the motion state of the target information;

[0030] Calculating estimated position information of the flexible trailer part according to the motion state of the target information, including position coordinates (x, y, z) and attitude (roll, pitch, yaw);

[0031] The estimated position and posture information of the flexible towing part is smoothed, and final position and posture information of the flexible towing part is output.

[0032] Preferably, the path planning and decision adjustment based on the posture information and in combination with the dynamic changes of the vehicle ahead include:

[0033] Obtaining parameter information of the length, angle, and speed of the flexible trailer portion based on the posture information, and adjusting the vehicle's driving trajectory and speed in real time based on the parameter information so that the flexible trailer portion remains stable during driving, and / or issuing an early warning based on the posture state;

[0034] Based on the current road conditions and the dynamic changes of the preceding vehicle, path planning and evaluation are performed, the optimal path is selected and control decisions are adjusted, and instruction information is generated to control vehicle tracking.

[0035] Preferably, it also includes:

[0036] The monocular vision information, 3D point cloud information, image information, target information, posture information, path information and / or decision information are sent to other terminals in real time for monitoring, management and / or information sharing and feedback.

[0037] According to another aspect of the present invention, a flexible trailer detection system based on monocular vision is provided, comprising: a monocular vision camera, an image processing and analysis module, a decision and control module, and an interactive feedback module; wherein:

[0038] Monocular vision detection module, which is used to obtain monocular vision information and 3D point cloud information of the trailer vehicle respectively;

[0039] An image processing and analysis module is configured to fuse the monocular visual information with the 3D point cloud information to obtain image information, wherein the image information includes at least information about the posture, position, and motion of the flexible trailer; perform feature extraction and target detection on the image information to obtain target information of the flexible trailer; and perform monocular direction estimation of the forward position on the target information to obtain position information of the flexible trailer;

[0040] The decision-making and control module performs path planning and decision adjustment based on the posture information and in combination with the dynamic changes of the vehicle in front, and generates control instructions to control vehicle tracking.

[0041] Preferably, the monocular vision detection module includes: a monocular vision camera and a laser radar sensor; wherein:

[0042] The monocular vision camera is used to capture real-time image data of the trailer vehicle, and the image data is used to display the shape, contour and texture information of the flexible trailer part, and obtain preliminary form and position information of the flexible trailer part, that is, to obtain monocular vision information;

[0043] The lidar sensor is used to obtain point cloud data of the towing vehicle. The point cloud data is used to describe the three-dimensional structure, spatial position and speed information of the flexible towing part, that is, to obtain 3D point cloud information, providing auxiliary depth information and spatial positioning reference for the image data.

[0044] Preferably, any one or more of the following are also included:

[0045] -Interactive feedback module, which is used to share information and / or provide information feedback with other terminals;

[0046] - A status monitoring module, which is used to monitor key vehicle parameters in real time, including the position information of the flexible trailer and the driving trajectory of the trailer;

[0047] -Data processing and analysis module, used to analyze and process key vehicle parameters and provide early warning of potential safety hazards;

[0048] - An adaptive adjustment module, which is used to dynamically adjust the parameters of the image processing and analysis module and / or the paths and decisions of the decision and control module according to the actual operating conditions of the vehicle;

[0049] - A user operation module, which is in communication with the image processing and analysis module and the decision and control module, and is used for real-time data interaction, real-time status display, operation instruction input, and early warning information prompts;

[0050] - Trailer tracking optimization module, which optimizes the tracking accuracy of the flexible trailer part based on the analysis and processing results obtained in the data processing and analysis module, so that the flexible trailer part remains stable during driving;

[0051] -Remote monitoring module, used to remotely monitor and manage the vehicle's operating status, and to display the tracking status of the flexible trailer, early warning information, and / or statistical data of various information in real time;

[0052] -Communication module, used for information transmission and command issuance between the monocular vision detection module, flexible towing vehicle and remote monitoring module.

[0053] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0054] The present invention achieves accurate detection and tracking of the flexible trailer part by combining a monocular vision camera and high-precision lidar data assistance, thereby improving the safety and stability of the trailer vehicle in complex road environments.

[0055] The present invention realizes stable control of the flexible towing vehicle through the coordinated work of the image processing and analysis module and the decision and control module, thereby improving the integration and compatibility, reliability and efficiency of the towing system.

[0056] The present invention utilizes interactive feedback technology to enhance the interactive experience between the user and the towing system, optimizes the adaptability of the trailer motion model, and enables the user to more comprehensively understand and control the flexible towing vehicle.

[0057] The present invention has broad application prospects in practical applications and can be applied to various scenarios requiring towing transportation, such as logistics, agriculture, military and other fields, providing efficient and safe towing solutions for related industries.

[0058] Of course, not all products implementing this invention need to achieve all of the advantages described above. However, overall, this invention utilizes advanced vision and radar technologies, combined with efficient image processing and control algorithms, to achieve precise detection and stable control of flexible trailers, providing strong support for the development of related industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0060] Figure 1 The figure is a workflow diagram of a flexible trailer detection method based on monocular vision in a preferred embodiment of the present invention.

[0061] Figure 2 This is a structural block diagram of a flexible trailer detection system based on monocular vision in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention.

[0063] One embodiment of the present invention provides a monocular vision-based flexible trailer detection method. This method utilizes high-precision lidar technology as an auxiliary to provide an accurate reference for monocular vision technology, thereby improving the safety and stability of flexible trailer vehicles in complex road environments, achieving efficient and accurate detection and intelligent control of flexible trailer vehicles, and ensuring the safety and operating efficiency of trailer vehicles through information exchange with the driver or other relevant systems through interactive feedback technology. This method is particularly suitable for the detection and management of flexible trailer vehicles in complex road environments and variable working conditions. Through data processing and analysis, trailer tracking optimization, and remote monitoring, it can effectively improve the tracking accuracy of flexible trailer vehicles, reduce safety hazards caused by inaccurate tracking, and enhance the driving safety and operating efficiency of flexible trailer vehicles.

[0064] Specifically, if Figure 1 As shown, the flexible trailer detection method based on monocular vision provided in this embodiment may include the following operations:

[0065] S1, obtains the monocular visual information and 3D point cloud information of the towing vehicle respectively;

[0066] S2, fusing the monocular vision information and the 3D point cloud information to obtain image information, wherein the image information includes at least: posture, position, and motion status information of the flexible towing part;

[0067] S3, performing feature extraction and target detection on the image information to obtain target information of the flexible trailer part;

[0068] S4, performing monocular direction estimation of the forward position of the target information to obtain the position information of the flexible trailer;

[0069] S5, based on the posture information and combined with the dynamic changes of the vehicle in front, performs path planning and decision adjustment, and generates control instructions to control vehicle tracking.

[0070] In some preferred embodiments, the above S1 may further include:

[0071] S11, using a monocular vision camera to capture real-time image data of the trailer vehicle, the image data is used to display the shape, contour and texture information of the flexible trailer part, and obtain preliminary form and position information of the flexible trailer part, that is, obtain monocular vision information;

[0072] S12 uses a lidar sensor to obtain point cloud data of the trailer vehicle. The point cloud data is used to describe the three-dimensional structure, spatial position, and speed information of the flexible trailer part, that is, to obtain 3D point cloud information, providing auxiliary depth information and spatial positioning reference for the image data.

[0073] In some preferred embodiments, the above S2 may further include:

[0074] Multimodal fusion technology is used to deeply fuse and process (integrate) monocular vision information and 3D point cloud information to improve recognition accuracy and obtain the current state and behavior information for accurately identifying the flexible trailer part, that is, image information.

[0075] In some preferred embodiments, the above S3 may further include:

[0076] S31, providing a pre-trained deep learning model, and utilizing the model to extract features from image information, identify the trailer part, and estimate the direction of the preceding vehicle;

[0077] S32, performing contour extraction and target detection on the trailer part, and combining the estimation result of the direction of the preceding vehicle, performing preliminary positioning and morphological analysis on the flexible trailer part to obtain target information.

[0078] In this preferred embodiment, the morphological analysis specifically includes:

[0079] S321, extracting edge and contour features of the flexible trailer part to evaluate its shape and size;

[0080] S322, analyzing the geometric shape, proportions, and deformation characteristics of the flexible trailer to identify possible abnormalities or damage;

[0081] S323: Based on the target detection results, the motion characteristics of the trailer are evaluated, such as acceleration, speed change, and movement posture direction.

[0082] In some preferred embodiments, the above S4 may further include:

[0083] S41, providing a monocular direction estimation model, taking the target information as input to the model, and extracting key feature points of the target information. Conventional monocular estimation only estimates the direction of a single rigid body, while the monocular direction estimation model used in this step estimates the direction of each individual body of the target information (i.e., the flexible towed portion), and then optimizes the directions of each rigid body in conjunction with the intersection constraints between the rigid bodies to obtain the direction of the towed arc. Simply put, the arc direction formed by the articulation of multiple rigid bodies with their own directions is obtained, i.e., the key feature points of the target information are obtained. Further preferably:

[0084] Estimating the direction of each monomer of the target information (i.e., the flexible towing part) may further include the following operations:

[0085] Estimate the direction of each unit in the flexible trailer. Here, "unit" refers to each rigid body module in the trailer system. The monocular direction estimation model can estimate the direction of the rigid body using a visual sensor.

[0086] By combining the directions of the respective rigid bodies with the intersection constraints between the rigid bodies, a constraint optimization between the directions is formed, thereby obtaining the direction of the towed arc. The following operations can also be further included:

[0087] Combined with intersection constraints: Consider the connection relationship between each rigid body. The individual units of the flexible towing system are connected by hinged structures, which means that there is a certain constraint relationship between them. Mathematically, this relationship can be described by establishing a direction constraint equation. That is, the direction of each rigid body is not only determined by its own characteristics, but also by its connection relationship with other rigid bodies.

[0088] Directional constraint optimization: After combining the independent directions of each rigid body and the intersection constraints between them, the system will perform directional constraint optimization. The purpose of this step is to solve the overall direction of the flexible trailer system while satisfying the direction and intersection constraints of each individual body. In this way, the directional characteristics of the trailer part in the arc can be obtained.

[0089] To obtain the key feature points of the target information, the following operations can be further performed

[0090] Extraction of key feature points: Through direction estimation and constraint optimization, the overall arc direction of the flexible trailer can be determined. This direction information can be regarded as the key feature points of the target information. The acquisition of key feature points is very important for subsequent target tracking and positioning because they describe the motion characteristics and posture of the flexible trailer in space.

[0091] S42, matching the key feature points with the prior pose information of the vehicle ahead, using an optical flow method or a feature point tracking algorithm to evaluate the motion state of the target information; wherein the prior pose information of the vehicle ahead is directly detected in 3D by the monocular camera of the vehicle;

[0092] S43, calculating estimated position information of the flexible trailer part through triangulation and geometric transformation based on the motion state of the target information, including position coordinates (x, y, z) and attitude (roll, pitch, yaw);

[0093] S44, using optimization algorithms such as Kalman filtering or particle filtering to smooth the estimated posture information of the flexible trailer part to reduce the impact of noise and improve estimation accuracy, and output the final posture information of the flexible trailer part to provide a basis for subsequent path planning and decision adjustments.

[0094] In this preferred embodiment, optimization of posture information is achieved: algorithms such as Kalman filtering or particle filtering are applied to optimize the obtained posture information to reduce noise and uncertainty and improve the accuracy of estimation; dynamic environment adaptation is also achieved: the posture estimation is adjusted in real time based on the speed, acceleration and road condition information of the vehicle in front to ensure stable tracking in complex and dynamic environments.

[0095] In some preferred embodiments, the above S5 may further include:

[0096] S51, obtaining parameter information of the length, angle, and speed of the flexible trailer according to the posture information, and adjusting the vehicle's driving trajectory and speed in real time based on the parameter information to ensure that the flexible trailer remains stable during driving; and further providing an early warning based on the posture status;

[0097] S52, combining the current road conditions and the dynamic changes of the preceding vehicle, performs path planning and evaluation, selects the optimal path and adjusts the control decision, and generates instruction information to control vehicle tracking.

[0098] In some preferred embodiments, the above method may further include:

[0099] S6, sending monocular vision information, 3D point cloud information, image information, target information, posture information, path information and / or decision information to other terminals in real time for monitoring, management and / or information sharing and feedback.

[0100] The above-mentioned embodiment of the present invention provides a flexible trailer detection method based on monocular vision. When tracking and making decisions on flexible trailers, the method first uses a monocular vision camera to capture real-time image data to reveal details such as the shape, contour, and texture of the flexible trailer. At the same time, with the help of point cloud information generated by sensors such as high-precision lidar, the three-dimensional structure and spatial position of the flexible trailer are accurately depicted. Through deep fusion and processing of information, the system can accurately identify the current state and behavior of the flexible trailer.

[0101] When driving, flexible trailer vehicles must adapt to dynamic changes such as acceleration, deceleration, and lane changes caused by the vehicle ahead, as well as fluctuating road conditions. Relying on monocular vision data, flexible trailer tracking is adjusted in real time to ensure stable and safe driving. Combined with the orientation estimation from monocular direction estimation, the tracking path is precisely adjusted based on the posture changes of the leading vehicle, flexibly adapting to the dynamics of the leading vehicle and ensuring safe and stable driving. Simultaneously, multiple candidate tracking paths are evaluated, taking into account factors such as tracking accuracy, safety, stability of the flexible trailer, and driving efficiency. This evaluation process also considers external factors such as road conditions and the traffic environment to develop a more comprehensive and reasonable decision-making strategy.

[0102] Based on the evaluation results, the system selects the optimal tracking path and formulates corresponding decision-making strategies, including adjusting the vehicle's acceleration and deceleration and coordinating vehicle steering. These decisions are designed to ensure that the flexible towed vehicle maintains high accuracy, safety, and convenience while tracking the vehicle ahead.

[0103] One embodiment of the present invention provides a monocular vision-based flexible trailer detection system. This system utilizes a high-precision laser radar as an auxiliary to provide an accurate reference for a monocular vision camera, thereby improving the safety and stability of flexible trailer vehicles in complex road environments. This system enables efficient and accurate detection and intelligent control of flexible trailer vehicles. Through an interactive feedback module, information exchange with the driver or other relevant systems ensures the safety and operating efficiency of trailer vehicles. This system is particularly suitable for detecting and managing flexible trailer vehicles in complex road environments and under variable operating conditions. Through a data processing and analysis module, a trailer tracking optimization module, and a remote monitoring module, the system can effectively improve the tracking accuracy of flexible trailer vehicles, reduce safety hazards caused by inaccurate tracking, and enhance the driving safety and operating efficiency of flexible trailer vehicles.

[0104] Specifically, if Figure 2 As shown, the flexible trailer detection system based on monocular vision provided in this embodiment may include:

[0105] Monocular vision camera, image processing and analysis module, decision and control module, and interactive feedback module; among which:

[0106] Monocular vision detection module, which is used to obtain monocular vision information and 3D point cloud information of the trailer vehicle respectively;

[0107] An image processing and analysis module is used to fuse monocular visual information with 3D point cloud information to obtain image information, which includes at least: the posture, position, and motion status information of the flexible trailer; perform feature extraction and target detection on the image information to obtain target information of the flexible trailer; perform monocular direction estimation of the forward posture of the target information to obtain the posture information of the flexible trailer;

[0108] The decision-making and control module performs path planning and decision adjustments based on posture information and the dynamic changes of the vehicle in front, and generates control instructions to control vehicle tracking.

[0109] In some preferred embodiments, the monocular vision detection module may further include: a monocular vision camera and a laser radar sensor; wherein:

[0110] A monocular vision camera is used to capture real-time image data of the trailer vehicle. The image data is used to display the shape, contour, and texture information of the flexible trailer part, and obtain preliminary form and position information of the flexible trailer part, that is, to obtain monocular vision information;

[0111] The lidar sensor is used to obtain point cloud data of the trailer vehicle. The point cloud data is used to describe the three-dimensional structure, spatial position and speed information of the flexible trailer part, that is, to obtain 3D point cloud information, providing auxiliary depth information and spatial positioning reference for the image data.

[0112] In some preferred embodiments, the above system may further include any one or more of the following modules:

[0113] -Interactive feedback module, which is used to share information and / or provide information feedback with other terminals;

[0114] - A status monitoring module, which is used to monitor key vehicle parameters in real time, including the position information of the flexible trailer and the driving trajectory of the trailer;

[0115] - The data processing and analysis module is used to analyze and process key vehicle parameters and provide early warnings for potential safety hazards. This data analysis and processing mainly includes: anomaly detection: applying anomaly detection algorithms to identify abnormal vehicle operating conditions, such as sudden braking, sudden acceleration, and deviation from the track; early warning mechanism: based on the analysis results, a threshold trigger mechanism is set to provide real-time early warnings for potential safety hazards and send alarm information to the driver or monitoring system;

[0116] - An adaptive adjustment module, which is used to dynamically adjust the parameters of the image processing and analysis module and / or the paths and decisions of the decision and control module according to the actual operating conditions of the vehicle;

[0117] - User operation module, which is connected to the image processing and analysis module and the decision and control module respectively, and is used for real-time data interaction, real-time status display, operation instruction input and early warning information prompt;

[0118] The trailer tracking optimization module optimizes the tracking accuracy of the flexible trailer using advanced algorithms and strategies based on the analysis and processing results obtained in the data processing and analysis module (including the flexible trailer's real-time position information, driving trajectory, condition monitoring data, and safety assessment results, which are used to guide subsequent tracking optimization). This ensures that the flexible trailer remains stable during driving. These advanced algorithms and strategies include: Kalman filtering, which updates the flexible trailer's position estimate in real time and provides more accurate position information by modeling noise; and model predictive control, which adjusts control inputs based on predicted future states to achieve smooth tracking of the flexible trailer. This entire process optimizes tracking within the trailer.

[0119] -Remote monitoring module, used to remotely monitor and manage the vehicle's operating status, and to display the tracking status of the flexible trailer, early warning information, and / or statistical data of various information in real time;

[0120] -Communication module, used for information transmission and command issuance between the monocular vision detection module, flexible towing vehicle and remote monitoring module.

[0121] It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, and will not be elaborated here.

[0122] The following further describes the working contents of each module of the flexible trailer detection system based on monocular vision provided by the above embodiment of the present invention.

[0123] The monocular vision camera is used to capture real-time image data of the trailer vehicle, paying special attention to the deformation, position and motion state of the flexible trailer part, and providing preliminary shape and position information of the trailer part.

[0124] Data from a high-precision LiDAR sensor is used as a supplement. By emitting laser beams and receiving reflected signals, it obtains precise distance, speed, and angle information of the trailer, providing depth information and spatial positioning reference for the monocular camera. The LiDAR sensor enhances the accuracy of preceding vehicle detection. Tightly integrated with other modules, it precisely measures key information such as distance, speed, and direction of the preceding vehicle. This module complements the monocular camera to form an efficient and accurate preceding vehicle detection system.

[0125] The image processing and analysis module receives data from a monocular vision camera and a high-precision lidar, and performs image processing and feature extraction through deep learning algorithms or monocular direction estimation. By performing feature extraction, target detection, and 3D pose estimation on the image, it achieves precise identification and positioning of the flexible trailer part, thereby achieving efficient and accurate detection of the flexible trailer part.

[0126] The decision-making and control module performs path planning and decision adjustments based on the output results of the image processing and analysis module and the dynamic changes of the vehicle in front, and generates corresponding control instructions to perform corresponding control operations on the towing vehicle, including but not limited to issuing warning signals, adjusting the driving speed, adjusting the posture of the towing vehicle to stabilize the flexible towing part, or triggering emergency braking and other safety measures.

[0127] The introduction of high-precision LiDAR data provides crucial auxiliary information for the monocular vision camera. Using the depth information and spatial positioning references acquired by LiDAR, the image processing and analysis module can more accurately identify and locate the flexible trailer, thereby improving the accuracy and reliability of the entire detection system. Furthermore, the decision-making and control module can adjust the vehicle's driving state in a timely manner based on real-time data and path planning, ensuring stable operation of the flexible trailer in complex road conditions.

[0128] The interactive feedback module is responsible for interacting with the user, displaying real-time status information, receiving user operation instructions, and providing warning information prompts to ensure the user's comprehensive control and safe use of the flexible trailer vehicle. The existence of the interactive feedback module allows users to understand the vehicle status in real time, receive warning information, and make necessary operational adjustments, further enhancing the system's security and user experience. It specifically includes the following functions:

[0129] Feedback the detection results, status information and control instructions of the flexible trailer vehicle to the user in real time through interface display, sound prompts, etc., so that the user can promptly understand the operating status of the trailer vehicle and take corresponding measures;

[0130] Receive input instructions from users or other operators, such as adjusting detection parameters and control strategies, to meet detection requirements in different scenarios;

[0131] Exchange data and transmit instructions with other related systems or equipment to achieve information sharing and collaborative work. For example, collaborate with the navigation system to optimize the driving path and avoid unnecessary vibration or impact on the flexible trailer part.

[0132] The status monitoring module of the flexible towing vehicle is used to monitor the key parameters such as the trajectory of the flexible towing part and the position of the towing vehicle in real time, and through data analysis and processing, it warns of possible safety hazards.

[0133] The adaptive adjustment module can dynamically adjust the algorithm parameters of the image processing and analysis module or the decision-making strategy of the decision-making and control module according to the actual operating conditions of the flexible towing vehicle to improve detection accuracy and response speed.

[0134] Through the design of the above structure and functions, the flexible trailer detection system based on monocular vision can achieve efficient and accurate detection and intelligent control of flexible trailer vehicles. Through the interactive feedback module, it interacts with the driver or other relevant systems to ensure the safety and operating efficiency of the trailer vehicle. It is particularly suitable for the detection and management of flexible trailer vehicles in complex road environments and variable working conditions.

[0135] The user operation module is connected to the flexible trailer vehicle and interacts with the monocular vision detection module, image processing and analysis module, and decision-making and control module in real time through wireless communication. At the same time, the user operation module provides functions such as real-time status display, operation command input, and early warning information prompts to ensure the user's comprehensive control and safe use of the flexible trailer vehicle.

[0136] The data processing and analysis module is responsible for receiving and processing data from the monocular vision detection module, and accurately identifying and analyzing the real-time status of the flexible trailer.

[0137] The trailer tracking optimization module optimizes the tracking accuracy of the flexible trailer part based on the analysis and processing results obtained in the data processing and analysis module, ensuring efficient and stable tracking of the flexible trailer part during driving;

[0138] The communication module is responsible for information transmission and command issuance between the monocular vision detection module, the flexible towing vehicle, and the remote monitoring module, ensuring the real-time nature of the data and the accuracy of the commands;

[0139] The remote monitoring module is used to remotely monitor and manage the operating status of flexible trailer vehicles, display the tracking status, warning information and statistical data of the flexible trailer part in real time, and provide comprehensive monitoring and decision support for managers.

[0140] Through the above design and implementation, the tracking accuracy of flexible trailer vehicles can be effectively improved, the safety hazards caused by inaccurate tracking can be reduced, and the driving safety and operating efficiency of flexible trailer vehicles can be improved.

[0141] The image processing and analysis module and the decision-making and control module implement the control decision-making process and undertake the key task of issuing instructions. Using a monocular vision camera, they accurately capture the position of the preceding vehicle and make a detailed assessment of the current state of the flexible trailer and its relative position to the preceding vehicle. Based on this information, they develop a safe and efficient driving path and action strategy to maintain a stable connection between the flexible trailer and ensure smooth and safe driving.

[0142] The above-described embodiments of the present invention not only monitor changes in the vehicle's surrounding environment but also address multi-target vehicle management. Using monocular vision technology, the system captures and processes visual information from multiple targets, including the flexible trailer vehicle, surrounding vehicles, and obstacles, in real time. In complex traffic environments, the system can accurately distinguish and detect individual targets, analyzing their dynamic characteristics and interactions, providing comprehensive and accurate data support for route planning and decision-making.

[0143] The working process of the above-mentioned flexible trailer detection system based on monocular vision includes the initialization stage, the flexible trailer detection stage, and the decision-making and control stage.

[0144] During the initialization phase:

[0145] Perform initial configuration and calibration of the monocular vision camera and high-precision lidar to ensure that the camera can accurately capture real-time image data of the flexible trailer, the lidar can accurately obtain depth information and spatial positioning data of the trailer, and that all modules work normally and coordinate with each other.

[0146] The image processing and analysis module parameters are set to adapt to the flexible trailer detection requirements in different scenarios. The decision-making and control module also requires initial configuration to ensure that it can perform path planning and decision adjustments based on the output of the image processing and analysis module and the dynamic changes of the vehicle ahead.

[0147] Finally, the above is tested to test the functionality of the interactive feedback module to ensure that it can normally display real-time status information, receive user operation instructions, and provide early warning information prompts.

[0148] During the flexible towing detection phase:

[0149] First, a monocular camera is activated to capture real-time image data of the flexible trailer, providing preliminary information on its shape and position. Simultaneously, a high-precision LiDAR (LiDAR) begins operating, emitting a laser beam and receiving reflected signals to obtain precise depth, speed, and angle information of the trailer. This information provides the image processing and analysis module with crucial precision depth information and spatial positioning references, helping to further improve the recognition and positioning accuracy of the flexible trailer. The image processing and analysis module receives data from the monocular camera and LiDAR and utilizes a pre-trained deep learning model for feature extraction, trailer recognition, and leading vehicle direction estimation. By performing contour extraction and morphological analysis on the trailer in the image, combined with specific direction estimation and prediction, preliminary positioning and morphological analysis of the flexible trailer are performed. Efficient image processing and feature extraction are then performed to achieve precise recognition and positioning of the flexible trailer.

[0150] During the trailer control phase:

[0151] Based on the output of the image processing and analysis module, the decision-making and control module performs path planning and decision adjustments in conjunction with current road conditions and the dynamic changes of the preceding vehicle. By generating appropriate control commands, it achieves stable control of the flexible trailer vehicle. Furthermore, the decision-making and control module continuously monitors the status of the trailer and makes real-time adjustments as needed to ensure the safety and stability of the trailer vehicle in complex road environments. Specifically, the decision-making and control module determines parameters such as the trailer's length, angle, and speed based on the flexible trailer's position information, and adjusts the vehicle's trajectory and speed to ensure the trailer remains stable during driving. Furthermore, the decision-making and control module monitors the trailer's status in real time and, if any anomalies or potential risks are detected, immediately takes appropriate measures to make adjustments or issues warnings.

[0152] Throughout the entire control process, the interactive feedback module maintains real-time interaction with the user. It displays real-time status information of the flexible trailer, such as length, angle, and speed, as well as the vehicle's trajectory and path planning. Users can access this information through the interactive feedback module and make adjustments as needed. Furthermore, the interactive feedback module provides early warning notifications, ensuring that users can take timely action when encountering potential risks. The module receives user instructions and provides necessary early warning notifications, ensuring comprehensive control and safe use of the flexible trailer vehicle.

[0153] It's worth noting that during the flexible trailer detection phase, the image processing and analysis module may employ advanced algorithms and technologies to improve the accuracy of flexible trailer identification and positioning. Furthermore, the decision-making and control module may consider various factors, such as road conditions, traffic flow, and wind resistance, when controlling the flexible trailer vehicle to ensure safe and stable operation in various scenarios.

[0154] Through the above implementation methods, the method and system provided by the above embodiments of the present invention can achieve accurate detection and stable control of the flexible trailer part, thereby improving the safety and stability of the trailer vehicle in complex road environments.

[0155] An embodiment of the present invention also provides a computer detection terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the terminal can be used to execute any one of the methods described above in the embodiments of the present invention, or to execute any one of the systems described above in the embodiments of the present invention.

[0156] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc. The above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0157] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0158] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0159] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0160] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0161] The flexible trailer detection method and system based on monocular vision provided by the above-mentioned embodiment of the present invention is intended to break through the technical bottleneck of traditional trailer detection systems in flexible object detection. The system cleverly integrates monocular vision detection, image processing and analysis, decision-making and control, and interactive feedback functions to build an efficient and accurate flexible trailer detection system. Flexible trailer objects are detected and tracked through deep learning algorithms or monocular direction estimation technology, and single-trailer direction detection is achieved for multiple trailers. Through monocular direction estimation technology, flexible trailer objects are effectively detected and tracked, and single-trailer direction detection is achieved in the case of multiple trailers, thereby realizing flexible direction judgment for fully flexible trailers. Corresponding control operations are performed based on the detection results, and seamless connection is achieved with other terminals (systems or vehicle-mounted equipment) to achieve information interconnection and intelligent collaboration. The above-mentioned embodiment of the present invention uses monocular vision technology to achieve efficient detection of flexible trailer objects, thereby improving the safety and operating efficiency of trailer vehicles.

[0162] Matters not mentioned in the above embodiments of the present invention are well known in the art.

[0163] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A flexible trailer detection method based on monocular vision, characterized in that: include: Obtain monocular vision information and 3D point cloud information of the trailer vehicle respectively; fusing the monocular vision information and the 3D point cloud information to obtain image information, wherein the image information includes at least information on the posture, position, and motion of the flexible towing part; Performing feature extraction and target detection on the image information to obtain target information of the flexible trailer part, the target information including deformation, position and motion state; Performing monocular direction estimation of the forward posture of the target information to obtain posture information of the flexible towing part; Based on the posture information and in combination with the dynamic changes of the vehicle ahead, path planning and decision adjustment are performed to generate control instructions to control vehicle tracking; The performing feature extraction and target detection on the image information includes: Providing a pre-trained deep learning model, utilizing the model to perform feature extraction, trailer part recognition, and leading vehicle direction estimation on the image information; Performing contour extraction and target detection on the trailer part, and combining the estimation result of the leading vehicle direction, performing preliminary positioning and morphological analysis on the flexible trailer part to obtain target information; The monocular direction estimation of the forward posture of the target information to obtain the posture information of the flexible towing part includes: A monocular direction estimation model is provided, and the target information is used as input to the model to extract key feature points of the target information. The monocular direction estimation model estimates the direction of each individual body in the target information, and then optimizes the directions by combining the directions of the respective rigid bodies with the intersection constraints between the rigid bodies to obtain the direction along the arc of the tow, i.e., the key feature points of the target information. Matching the key feature points with the prior pose information of the vehicle ahead, and evaluating the motion state of the target information; Calculating estimated position information of the flexible towing part according to the motion state of the target information, including: position coordinates and posture; The estimated position and posture information of the flexible towing part is smoothed, and final position and posture information of the flexible towing part is output.

2. The flexible trailer detection method based on monocular vision according to claim 1 is characterized in that: The step of respectively acquiring monocular visual information and 3D point cloud information of the trailer vehicle includes: A monocular vision camera is used to capture real-time image data of the trailer vehicle. The image data is used to display the shape, contour, and texture information of the flexible trailer part, thereby obtaining preliminary morphology and position information of the flexible trailer part, i.e., obtaining monocular vision information. A lidar sensor is used to acquire point cloud data of the trailer vehicle. The point cloud data is used to describe the three-dimensional structure, spatial position, and speed information of the flexible trailer part, that is, to obtain 3D point cloud information, providing auxiliary depth information and spatial positioning reference for the image data.

3. The flexible trailer detection method based on monocular vision according to claim 1 is characterized in that: The fusing the monocular vision information and the 3D point cloud information includes: The monocular vision information and the 3D point cloud information are deeply fused and processed to obtain current state and behavior information for accurately identifying the flexible trailer part, that is, image information.

4. The flexible trailer detection method based on monocular vision according to claim 1 is characterized in that: The path planning and decision adjustment based on the posture information and combined with the dynamic changes of the vehicle ahead include: Obtaining parameter information of the length, angle, and speed of the flexible trailer portion based on the posture information, and adjusting the vehicle's driving trajectory and speed in real time based on the parameter information so that the flexible trailer portion remains stable during driving, and / or issuing an early warning based on the posture state; Based on the current road conditions and the dynamic changes of the preceding vehicle, path planning and evaluation are performed, the optimal path is selected and control decisions are adjusted, and instruction information is generated to control vehicle tracking.

5. The flexible trailer detection method based on monocular vision according to any one of claims 1 to 4, characterized in that: Also includes: The monocular vision information, 3D point cloud information, image information, target information, posture information, path information and / or decision information are sent to other terminals in real time for monitoring, management and / or information sharing and feedback.

6. A flexible trailer detection system based on monocular vision, characterized in that: include: Monocular vision camera, image processing and analysis module, decision and control module, and interactive feedback module; among which: Monocular vision detection module, which is used to obtain monocular vision information and 3D point cloud information of the trailer vehicle respectively; An image processing and analysis module is configured to fuse the monocular visual information with the 3D point cloud information to obtain image information, wherein the image information includes at least information about the posture, position, and motion state of the flexible towed portion; perform feature extraction and target detection on the image information to obtain target information of the flexible towed portion, wherein the target information includes deformation, position, and motion state; and perform monocular direction estimation of the forward posture on the target information to obtain posture information of the flexible towed portion. A decision-making and control module, which performs path planning and decision adjustments based on the posture information and in combination with the dynamic changes of the vehicle ahead, and generates control instructions to control vehicle tracking; The image processing and analysis module performs feature extraction and target detection on the image information, including: Providing a pre-trained deep learning model, utilizing the model to perform feature extraction, trailer part recognition, and leading vehicle direction estimation on the image information; Performing contour extraction and target detection on the trailer part, and combining the estimation result of the leading vehicle direction, performing preliminary positioning and morphological analysis on the flexible trailer part to obtain target information; The image processing and analysis module performs monocular direction estimation of the forward posture of the target information to obtain the posture information of the flexible trailer part, including: A monocular direction estimation model is provided, and the target information is used as input to the model to extract key feature points of the target information. The monocular direction estimation model estimates the direction of each individual body in the target information, and then optimizes the directions by combining the directions of the respective rigid bodies with the intersection constraints between the rigid bodies to obtain the direction along the arc of the tow, i.e., the key feature points of the target information. Matching the key feature points with the prior pose information of the vehicle ahead, and evaluating the motion state of the target information; Calculating estimated position information of the flexible towing part according to the motion state of the target information, including: position coordinates and posture; The estimated position and posture information of the flexible towing part is smoothed, and final position and posture information of the flexible towing part is output.

7. The flexible trailer detection system based on monocular vision according to claim 6 is characterized in that: The monocular vision detection module includes: a monocular vision camera and a laser radar sensor; wherein: The monocular vision camera is used to capture real-time image data of the trailer vehicle, and the image data is used to display the shape, contour and texture information of the flexible trailer part, and obtain preliminary form and position information of the flexible trailer part, that is, to obtain monocular vision information; The lidar sensor is used to obtain point cloud data of the towing vehicle. The point cloud data is used to describe the three-dimensional structure, spatial position and speed information of the flexible towing part, that is, to obtain 3D point cloud information, providing auxiliary depth information and spatial positioning reference for the image data.

8. The flexible trailer detection system based on monocular vision according to claim 6 or 7, characterized in that: Also includes any one or more of the following: -Interactive feedback module, which is used to share information and / or provide information feedback with other terminals; - A status monitoring module, which is used to monitor key vehicle parameters in real time, including the position information of the flexible trailer and the driving trajectory of the trailer; -Data processing and analysis module, used to analyze and process key vehicle parameters and provide early warning of potential safety hazards; - An adaptive adjustment module, which is used to dynamically adjust the parameters of the image processing and analysis module and / or the paths and decisions of the decision and control module according to the actual operating conditions of the vehicle; - A user operation module, which is in communication with the image processing and analysis module and the decision and control module, and is used for real-time data interaction, real-time status display, operation instruction input, and early warning information prompts; - Trailer tracking optimization module, which optimizes the tracking accuracy of the flexible trailer part based on the analysis and processing results obtained in the data processing and analysis module, so that the flexible trailer part remains stable during driving; -Remote monitoring module, used to remotely monitor and manage the vehicle's operating status, and to display the tracking status of the flexible trailer, early warning information, and / or statistical data of various information in real time; -Communication module, used for information transmission and command issuance between the monocular vision detection module, flexible towing vehicle and remote monitoring module.

Images