Trailer view imaging method, apparatus, device, and storage medium

Abstract

This invention relates to the field of intelligent vehicle perception technology, and discloses a method, apparatus, device, and storage medium for trailer view imaging. The method includes: semantically segmenting multi-camera images of a trailer to obtain a trailer outline and a target image; jointly calibrating and fusing the trailer outline and the target image to obtain a panoramic bird's-eye view; matching the measured distance parameters and the trailer outline in a model dataset to obtain a trailer model; and fusing the trailer model and the panoramic bird's-eye view to obtain a panoramic trailer view. This invention processes acquired multi-camera images to obtain a panoramic bird's-eye view, then highlights the identified trailer to replace it with a trailer model, and fuses the distance parameters with the panoramic bird's-eye view to obtain a panoramic trailer view. This makes the view closer to the actual shape of the trailer, allowing the driver to better observe the surrounding environment from a bird's-eye view perspective. This not only reduces the difficulty of towing a trailer for the driver but also improves driving safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle perception technology, and in particular to a trailer view imaging method, apparatus, device, and storage medium. Background Technology

[0002] With the increasing prevalence of vehicles, they are used for a wide variety of purposes. Vehicles are frequently used as towing trailers to transport various types of goods, hence the term "tractor unit." Examples of such trailers include, but are not limited to, travel trailers, snowmobile trailers, boat trailers, and car trailers. In many cases, the height of the trailer towed by the tractor unit means that the driver of the tractor unit can only see the trailer when looking in the tractor unit's rearview mirror. This can make it difficult to observe the vehicle or the surrounding environment of the trailer during operation, negatively impacting the driver's ability to tow the trailer safely and conveniently.

[0003] Traditional methods for sensing trailer size typically involve stitching together images from the vehicle's rear-view and side views or performing radar scans. However, traditional trailer view imaging confirmation methods are overly complex, carrying the risk of overlapping and misalignment in stitching, making it difficult to perceive the entire trailer.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a trailer view imaging method, apparatus, device, and storage medium, which aims to solve the technical problems of traditional trailer view imaging confirmation conditions being too complex, having the risk of splicing overlap and misalignment, and being difficult to perceive the entire trailer.

[0006] To achieve the above objectives, the present invention provides a trailer view imaging method, the method comprising the following steps:

[0007] Semantic segmentation is performed on the multi-camera images of the trailer to obtain the trailer outline and target image;

[0008] The trailer outline and the target image are jointly calibrated and fused to obtain a panoramic bird's-eye view of the trailer.

[0009] The trailer model is obtained by matching the measured distance parameters and the trailer outline in the model dataset.

[0010] The trailer model and the panoramic bird's-eye view are merged to obtain a panoramic trailer view.

[0011] Optionally, the step of matching the measured distance parameters and the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer includes:

[0012] Acquire the target image corresponding to the target on the trailer, wherein the target is set in front of the trailer and the target is used for distance measurement;

[0013] The distance between the trailer and the tractor is obtained by measuring the distance between the trailer and the tractor based on the target image and the actual size of the target.

[0014] The trailer model corresponding to the trailer is obtained by matching the distance parameter and the trailer outline in the model dataset.

[0015] Optionally, the step of measuring the distance between the trailer and the tractor based on the target image and the actual size of the target to obtain the distance parameter between the trailer and the tractor includes:

[0016] The target image is measured to obtain the pixel size of the target in the target image;

[0017] The camera corresponding to the target image is determined, and the focal length parameter of the camera and the width parameter of the sensor in the camera are obtained;

[0018] The distance between the trailer and the tractor is obtained by calculating the pixel size, the actual target size, the focal length parameter, and the width parameter using the principle of triangle similarity.

[0019] Optionally, the step of semantic segmentation of the multi-camera images of the trailer to obtain the trailer outline and target image includes:

[0020] The trailer's surrounding environment is captured by multiple onboard cameras.

[0021] Semantic segmentation is performed on the multi-camera images to obtain the trailer outline and target image, wherein the target image includes lane images, vehicle images and road landscape images from the multi-camera images;

[0022] The step of jointly calibrating and fusing the trailer outline and the target image to obtain a panoramic bird's-eye view corresponding to the trailer includes:

[0023] The trailer outline, the lane image, the vehicle image, and the road landscape image are jointly calibrated and fused to obtain a panoramic bird's-eye view corresponding to the trailer.

[0024] Optionally, the step of jointly calibrating and fusing the trailer outline, the lane image, the vehicle image, and the road landscape image to obtain a panoramic bird's-eye view corresponding to the trailer includes:

[0025] The trailer outline, the lane image, the vehicle image, and the road landscape image are input into a deep learning network for fusion to obtain an initial fused image;

[0026] Obtain the field of view parameters of each image in the multi-camera images;

[0027] Based on the field of view parameters, a perspective transformation is performed on the initial fused image to obtain a panoramic bird's-eye view of the trailer.

[0028] Optionally, the step of performing perspective transformation on the initial fused image based on the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer includes:

[0029] Projection mapping is performed on the trailer outline, the lane image, the vehicle image, and the road landscape image to obtain the homography matrix corresponding to each image;

[0030] Based on the homography matrix and the field of view parameters, a perspective transformation is performed on the initial fused image to obtain a panoramic bird's-eye view of the trailer.

[0031] Optionally, after acquiring multi-camera images of the surrounding environment of the trailer from the multiple onboard cameras of the trailer, the method further includes:

[0032] The trailer image in the multi-camera images is labeled to obtain the labeled image corresponding to the trailer;

[0033] After performing semantic segmentation on the multi-camera images to obtain the trailer outline and target image, the target image includes lane images, vehicle images, and road landscape images from the multi-camera images, and further includes:

[0034] The labeled image is compared with the trailer outline, and the trailer outline is corrected based on the comparison results to obtain the corrected trailer outline.

[0035] Furthermore, to achieve the above objectives, the present invention also proposes a trailer view imaging device, the device comprising:

[0036] The semantic segmentation module is used to perform semantic segmentation on the multi-camera images of the trailer to obtain the trailer outline and target image.

[0037] An image fusion module is used to perform joint calibration and fusion of the trailer outline and the target image to obtain a panoramic bird's-eye view of the trailer.

[0038] The model matching module is used to match the measured distance parameters and the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer.

[0039] The trailer view module is used to merge the trailer model and the panoramic bird's-eye view to obtain a panoramic trailer view.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a trailer view imaging device, the device comprising: a memory, a processor, and a trailer view imaging program stored in the memory and executable on the processor, the trailer view imaging program being configured to implement the steps of the trailer view imaging method as described above.

[0041] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a trailer view imaging program, which, when executed by a processor, implements the steps of the trailer view imaging method as described above.

[0042] This invention obtains the trailer outline and target image by semantically segmenting multi-camera images of a trailer; then, it performs joint calibration and fusion of the trailer outline and the target image to obtain a panoramic bird's-eye view of the trailer; next, it matches the measured distance parameters and the trailer outline in a model dataset to obtain a trailer model; finally, it fuses the trailer model and the panoramic bird's-eye view to obtain a panoramic view of the trailer. This invention obtains a panoramic bird's-eye view by processing the acquired multi-camera images, highlighting the identified trailer, replacing it with a trailer model, and fusing it with the measured distance parameters into the panoramic bird's-eye view to obtain an overall panoramic view of the trailer. This makes the view closer to the actual shape of the trailer, allowing the driver to better observe the vehicle or trailer's surroundings from a bird's-eye view perspective. This not only reduces the difficulty of towing the trailer but also improves driving safety. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of the trailer view imaging device in the hardware operating environment involved in the embodiments of the present invention;

[0044] Figure 2 This is a flowchart illustrating the first embodiment of the trailer view imaging method of the present invention;

[0045] Figure 3 This is a schematic diagram of a panoramic trailer view in the first embodiment of the trailer view imaging method of the present invention;

[0046] Figure 4 This is a flowchart illustrating the second embodiment of the trailer view imaging method of the present invention;

[0047] Figure 5 This is a schematic diagram of the target scene in the second embodiment of the trailer view imaging method of the present invention;

[0048] Figure 6 This is a flowchart illustrating the third embodiment of the trailer view imaging method of the present invention;

[0049] Figure 7This is a schematic diagram of a scenario in the third embodiment of the trailer view imaging method of the present invention, showing multiple vehicle-mounted cameras acquiring multi-camera images;

[0050] Figure 8 This is a schematic diagram of a scene where the image fusion module based on deep learning fuses images from four directions in the third embodiment of the trailer view imaging method of the present invention.

[0051] Figure 9 This is a structural block diagram of the first embodiment of the trailer view imaging device of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0054] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a trailer view imaging device in the hardware operating environment involved in the embodiments of the present invention.

[0055] like Figure 1 As shown, the trailer-mounted view imaging device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the trailer view imaging device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] like Figure 1As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a trailer view imaging program.

[0058] exist Figure 1 In the trailer view imaging device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the trailer view imaging device of the present invention can be set in the trailer view imaging device, and the trailer view imaging device calls the trailer view imaging program stored in the memory 1005 through the processor 1001 and executes the trailer view imaging method provided in the embodiment of the present invention.

[0059] This invention provides a trailer view imaging method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the trailer view imaging method of the present invention.

[0060] In this embodiment, the trailer view imaging method includes the following steps:

[0061] Step S10: Perform semantic segmentation on the multi-camera images of the trailer to obtain the trailer outline and target image.

[0062] It should be noted that the executing entity of the method in this embodiment can be a computing service device with image fusion and data processing capabilities, such as an in-vehicle computer or a driver assistance system, or other electronic devices capable of performing the same or similar functions, such as the aforementioned trailer view imaging device. This embodiment does not limit the scope of the invention. This embodiment and the following embodiments will be specifically described using the aforementioned trailer view imaging device (hereinafter referred to as the imaging device).

[0063] Understandably, with the increasing prevalence of vehicles, they are used for a wider range of purposes. Vehicles used to tow and transport various types of goods are called tractor units, and vehicles towed by tractor units are called trailers. Examples of trailers include, but are not limited to, travel trailers, snowmobile trailers, and car trailers. However, in many cases, the height of the trailer towed by the tractor unit can make it difficult for the tractor unit driver to see the trailer in their rearview mirror. This can hinder the driver's ability to observe the surrounding environment of the vehicle or trailer during operation, posing a safety hazard to the driver and the operation of towing the trailer. Therefore, this paper proposes a trailer view imaging method that allows the driver to drive from a bird's-eye view, enabling better observation of the vehicle or trailer's surroundings and reducing the difficulty of towing the trailer.

[0064] It should be noted that the tractor unit needs to connect to the trailer when towing, for example, via a trailer hitch. Trailer hitchs typically consist of several pins, and the standard may vary slightly depending on the vehicle model and region. The trailer hitch is a crucial component during towing, ensuring proper electrical signal transmission between the tractor and trailer, allowing for effective control of trailer lights and other equipment, thus improving driving safety. In this embodiment, the trailer hitch functions like a switch; if the hitch is working properly, it indicates that the tractor unit is towing the trailer correctly. At this point, a trailer view image can be generated, allowing the driver to better observe the vehicle or trailer's surroundings from a bird's-eye view.

[0065] It should be understood that multi-camera images are images captured by cameras mounted on the vehicle. These images reflect the environment surrounding the trailer, including images of surrounding vehicles, lanes, and the surrounding road landscape, from directions such as front, rear, left and right sides, and rear side. Multi-camera images can be obtained from the trailer's cameras, the towing vehicle's cameras, or a combination of both; this embodiment does not impose any limitations. Acquiring images using multiple cameras reduces errors between different cameras, improving the accuracy and stability of the image visuals.

[0066] In practical implementation, the multi-camera images acquired by the cameras mounted on the vehicle include images of various objects. Therefore, the imaging equipment needs to annotate and segment these images to extract the necessary images for the trailer view, including the trailer outline and target images. For example, segmenting the trailer outline is used to accurately display the trailer size; the segmented target image can include images of other vehicles, lanes, or road landscapes from the multi-camera images, which can better remind the driver of the distance between the trailer and other vehicles, and between the tractor and the trailer.

[0067] Step S20: Perform joint calibration and fusion on the trailer outline and the target image to obtain a panoramic bird's-eye view corresponding to the trailer.

[0068] It's important to note that a panoramic bird's-eye view (also known as a BVE view) is a natural and direct candidate view. It's an image obtained by fusing the trailer outline with the target image. Compared to front or perspective views, panoramic bird's-eye views do not suffer from the occlusion and scale issues common in two-dimensional images. During trailer towing, it allows for better identification of vehicles with obstructions or intersecting traffic, and provides the driver with a better view of the vehicle or trailer's surroundings.

[0069] In practice, the imaging device performs spatial joint calibration and fusion on the above-mentioned trailer outline and target image, and then processes the image data to obtain a panoramic bird's-eye view that reflects the three-dimensional top view of the vehicle's surrounding environment, so that the driver can better observe the environment around the vehicle or trailer while driving from a bird's-eye view.

[0070] Step S30: Match the measured distance parameters and the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer.

[0071] It should be noted that the distance parameter reflects the distance between the trailer and the tractor. This distance parameter can better reflect the distance between the trailer and the tractor in the panoramic bird's-eye view, so as to remind the driver to control the distance and improve driving safety.

[0072] It should be understood that the imaging equipment measures the distance between the trailer and the tractor. A target for distance measurement can be pre-set on the trailer, and the distance parameters between the trailer and the tractor can be calculated based on the actual size and view size of the target. Alternatively, the distance can be determined by subtracting the overlap length between the trailer boom and the vehicle from the actual boom length. For example, trailer boom lengths generally come in three common specifications: 48 inches, 60 inches, and 72 inches (1 inch equals 2.54 centimeters). A 48-inch boom is approximately 122 centimeters long and is suitable for light trailers and some small heavy-duty vehicles; a 60-inch boom is approximately 152 centimeters long and is suitable for medium-duty trailers and some large heavy-duty vehicles; a 72-inch boom is approximately 183 centimeters long and is suitable for heavy-duty trailers and some extra-large heavy-duty vehicles. This embodiment does not impose any limitations on these specifications.

[0073] Understandably, the model dataset is an ensemble of models corresponding to various trailers. Depending on the actual situation, the trailer models in the model dataset include, but are not limited to, different types of RVs, trailer pallets of different sizes (trailer pallets that can carry different capacities of cargo at the same time), etc.

[0074] In the specific implementation, the trailer outline is matched in the model dataset to obtain the trailer model corresponding to the trailer. Since different trailers use different trailer boom lengths, the measured distance parameters can also be used as a reference to further determine the trailer model of the trailer, thereby improving the accuracy of obtaining the trailer model.

[0075] Step S40: Merge the trailer model and the panoramic bird's-eye view to obtain a panoramic trailer view.

[0076] In practice, the identified trailers are highlighted and replaced with trailer models, and the measured distance parameters are fused into the panoramic bird's-eye view to obtain an overall panoramic view of the trailers. This makes the view closer to the actual shape of the trailers, and the driver can better observe the vehicle or the environment around the trailer from the bird's-eye view.

[0077] Furthermore, when the towing vehicle is parking, it can also obtain a panoramic view of the trailer. Cameras on the vehicle can capture images of surrounding vehicles, the parking environment, and distance parameters between vehicles, thus better alerting the driver to the distances of the trailer and towing vehicle to other vehicles. By highlighting the identified trailer and replacing it with a trailer model, and then integrating it into the panoramic bird's-eye view to obtain a complete panoramic view of the trailer, the view becomes closer to the actual shape of the trailer. From this bird's-eye view, the driver can better observe the surrounding environment while parking; making the view closer to the actual shape of the trailer reduces the difficulty for the driver when parking the towing vehicle.

[0078] For ease of understanding, a panoramic trailer view obtained from a simulated scenario is used for illustration, but this does not limit the scope of this solution. (Reference) Figure 3 , Figure 3 This is a schematic diagram of a panoramic trailer view in the first embodiment of the trailer view imaging method of the present invention. As shown in the figure, in the acquired panoramic trailer view, the towing vehicle is pulling the trailer, and there are other vehicles around, such as buses. The road and road landscape are also color-coded. After the trailer model is fused, the trailer is displayed in a highlighted state. From a bird's-eye view, the driver can better observe the vehicle or the environment around the trailer, and better remind the driver to pay attention to the distance between the trailer and other vehicles, and between the towing vehicle and the trailer.

[0079] This embodiment acquires multi-camera images, including images of various objects, using cameras mounted on the vehicle. These images need to be labeled and segmented to extract the necessary images for the trailer view, including the trailer outline and target images. For example, the trailer outline is segmented to accurately display the trailer size; the segmented target image can include images of other vehicles, lanes, or road landscapes from the multi-camera images, better alerting the driver to the distance between the trailer and other vehicles, and between the tractor and trailer. Then, the trailer outline and target images are spatially jointly calibrated and fused, and image data is processed to obtain a panoramic bird's-eye view reflecting a three-dimensional, top-down perspective of the vehicle's surroundings. This allows the driver to better observe the environment around the vehicle or trailer from a bird's-eye view. Next, the trailer outline is matched against a model dataset to obtain the corresponding trailer model. Since different trailers use different tow bar lengths, measured distance parameters can be used as a reference to further determine the trailer model, improving the accuracy of obtaining the trailer model. Finally, the identified trailer is highlighted and replaced with a trailer model, which is then fused with the measured distance parameters into the panoramic bird's-eye view to obtain a complete panoramic view of the trailer. This makes the view closer to the actual shape of the trailer, allowing the driver to better observe the vehicle or trailer's surroundings from a bird's-eye view. Compared to traditional methods that rely on hardware output signals, this invention eliminates the need for additional cameras, thus reducing costs. Furthermore, compared to traditional lidar ranging methods that measure the dimensions of a single trailer facade, this invention can also perceive the overall size of the trailer, making the view closer to its actual shape. The bird's-eye view allows the driver to better observe the vehicle or trailer's surroundings, not only reducing the difficulty of towing the trailer for the driver but also improving driving safety.

[0080] refer to Figure 4 , Figure 4 This is a schematic flowchart of the second embodiment of the trailer view imaging method of the present invention.

[0081] Based on the first embodiment described above, in this embodiment, considering the accuracy of measuring the distance between the trailer and the tractor, step S30 includes:

[0082] Step S31: Obtain the target image corresponding to the target on the trailer, wherein the target is set in front of the trailer and the target is used for distance measurement.

[0083] It should be noted that the target is a marker that is pre-set in front of the trailer for distance measurement. The target used for distance measurement can be a variety of different objects or markers, depending on the purpose of distance measurement and the distance measurement equipment used, such as a calibration plate, a reflector, etc.

[0084] Understandably, the target image is based on an image taken by the rear-view camera on the tractor unit. Different cameras capture images of different sizes, and the distance between the trailer and the tractor unit can be determined based on the difference between the image size and the actual size.

[0085] Step S32: Measure the distance between the trailer and the tractor based on the target image and the actual size of the target to obtain the distance parameters between the trailer and the tractor.

[0086] Step S33: Match the distance parameter and the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer.

[0087] It should be noted that the actual target size refers to the actual size of the target. Due to different distances and camera specifications, the size of the captured target image will vary. The distance between the trailer and the tractor can be determined based on the difference between the image size and the actual size.

[0088] In practice, the imaging device acquires the target image corresponding to the target on the trailer using a rear-view camera. Then, it measures the distance between the trailer and the towing vehicle based on the image size and the actual size of the target. The distance parameter between the trailer and the towing vehicle is determined based on the difference between the image size and the actual size. Since different trailers use different trailer boom lengths, after obtaining the trailer model corresponding to the trailer, the measured distance parameter can be used as a reference to further determine the trailer model, thereby improving the accuracy of obtaining the trailer model.

[0089] Furthermore, considering the accuracy of distance parameter calculation, step S32 in this embodiment includes: measuring the target image to obtain the pixel size of the target in the target image; determining the camera corresponding to the target image and obtaining the focal length parameter of the camera and the width parameter of the sensor in the camera; and calculating the pixel size, the actual size of the target, the focal length parameter, and the width parameter using the principle of triangle similarity to obtain the distance parameter between the trailer and the tractor.

[0090] It should be noted that pixel size refers to the image size of the target within the target image. Due to different distances and camera specifications, the size of the captured target image will vary. Therefore, it is also necessary to determine the camera corresponding to the target image and obtain the camera's focal length parameters and the width parameters of the sensor within the camera.

[0091] Understandably, the principle of triangle similarity states that two or more triangles have equal relative angles and equal proportions of corresponding sides. Based on the principle of triangle similarity and the data obtained above, the distance parameters between the trailer and the tractor can be calculated.

[0092] For ease of understanding, a three-circle target calibration plate is used for illustration, but this does not limit the scope of this method. (Reference) Figure 5 , Figure 5 This is a schematic diagram of the target scene in the second embodiment of the trailer view imaging method of the present invention. First, the target (e.g., Figure 5 The actual physical size of the target (as shown) is determined, i.e., its actual dimensions such as width or diameter in the real world; then, the pixel size of the target is measured in the target image, i.e., the pixel dimensions of the target's width or diameter in the image; next, information such as the focal length and resolution of the camera that captured the target image is obtained, and the horizontal and vertical viewing angles of the camera are calculated; finally, the principle of triangle similarity is applied, and the following formula can be obtained using the principle of triangle similarity:

[0093]

[0094] Where Distance represents the distance between the target and the camera; Actual width represents the actual physical size of the target; Pixel width represents the pixel size of the target in the image; Focal length represents the camera's focal length; and Sensorwidth represents the width of the camera sensor. Then, the trailers are classified to obtain trailer models. The trailer outlines and distance parameters are matched with the model dataset to select a suitable trailer model.

[0095] In this embodiment, the imaging device acquires the target image corresponding to the target on the trailer using a rear-view camera. Then, it measures the distance between the trailer and the tractor based on the image size and actual target size. The pixel size, actual target size, focal length parameter, and width parameter are calculated to obtain the distance parameter between the trailer and the tractor. At the same time, since different trailers use different trailer boom lengths, after obtaining the trailer model corresponding to the trailer, the measured distance parameter can be used as a reference to further determine the trailer model, thereby improving the accuracy of obtaining the trailer model.

[0096] refer to Figure 6 , Figure 6 This is a flowchart illustrating the third embodiment of the trailer view imaging method of the present invention.

[0097] Based on the above embodiments, in this embodiment, considering the accuracy of the panoramic bird's-eye view, step S10 includes:

[0098] Step S11: Obtain multi-camera images of the environment surrounding the trailer based on the multiple on-board cameras of the trailer.

[0099] Step S12: Perform semantic segmentation on the multi-camera images to obtain the trailer outline and target image. The target image includes lane images, vehicle images, and road landscape images from the multi-camera images.

[0100] Step S20 includes: Step S21: Jointly calibrate and fuse the trailer outline, the lane image, the vehicle image and the road landscape image to obtain a panoramic bird's-eye view corresponding to the trailer.

[0101] It should be noted that multi-camera images are images taken by cameras mounted on the vehicle. Multi-camera images reflect the environment around the trailer vehicle, including images of surrounding vehicles, lanes, and surrounding road landscapes, in directions including front, rear, left and right sides, and rear side.

[0102] In the specific implementation, a common 8-bit camera position and BEV perspective example are used for illustration, but this does not limit the solution. References Figure 7 , Figure 7 This is a schematic diagram illustrating a scenario where multiple vehicle-mounted cameras acquire multi-camera images in the third embodiment of the trailer view imaging method of the present invention. As shown in the figure, multiple vehicle-mounted cameras on the vehicle acquire images of the surrounding environment from multiple directions, including the front, rear, left, and right sides. These images include images of surrounding vehicles, lanes, and surrounding road landscapes. Semantic segmentation processing has been performed on these images, with a focus on segmenting vehicles, lanes, and the surrounding road landscapes. The trailer outline, lane images, vehicle images, and road landscape images obtained from semantic segmentation are jointly calibrated and fused to obtain a panoramic bird's-eye view of the trailer. This makes the view closer to the real-world form, allowing the driver to better observe the environment around the vehicle or trailer and improving driving safety.

[0103] Furthermore, considering the accuracy of the fusion process, in this embodiment, step S21 includes: inputting the trailer outline, the lane image, the vehicle image, and the road landscape image into a deep learning network for fusion to obtain an initial fused image; obtaining the field of view parameters of each image in the multi-camera image; and performing perspective transformation on the initial fused image according to the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer.

[0104] It's important to note that deep learning networks are a type of machine learning network composed of multiple layers of neural networks, such as the UNetXST network and the DeepLab Xception network. They contain multiple hidden layers to handle complex inputs and outputs. The core component of deep learning networks is the neural network, which consists of many connected artificial neurons (also called nodes) that mimic the behavior of biological neurons. Each neuron receives input from neurons in the previous layer and passes it to neurons in the next layer. By adjusting the connection weights and biases between neurons, the neural network can learn complex mappings between inputs and outputs.

[0105] Understandably, the field of view parameter is the spatial field of view parameter represented by each image in the multi-camera imagery. To ensure spatial consistency, it is necessary to capture the same field of view as the actual ground-based BEV view from each image in the multi-camera imagery.

[0106] In the specific implementation, the trailer outline, lane image, vehicle image and road landscape image are input into a deep learning network for fusion to obtain an initial fused image; then, the same field of view parameters as the real BEV view on the ground are captured from each image of the multi-camera image; the initial fused image is subjected to perspective transformation according to the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer, so as to ensure the consistency of the panoramic bird's-eye view space and improve the realism of the panoramic bird's-eye view.

[0107] Furthermore, in this embodiment, the step of performing perspective transformation on the initial fused image based on the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer includes: performing projection mapping on the trailer outline, the lane image, the vehicle image, and the road landscape image to obtain the homography matrix corresponding to each image; and performing perspective transformation on the initial fused image based on the homography matrix and the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer.

[0108] It's important to note that Inverse Perspective Mapping (IPM) is an image processing technique used to convert an image from a perspective projection viewpoint to a top-down projection viewpoint. Perspective projection refers to the phenomenon in real-world scenes where objects farther from the observer appear smaller, while objects closer to the observer appear larger, due to the observer's perspective and the position of objects within the scene. Perspective transformation corrects this distortion by performing the inverse operation, remapping the perspective projection image to a top-down projection viewpoint.

[0109] It should be understood that the homography matrix is ​​a commonly used transformation matrix in computer vision, used to describe the perspective transformation relationship between two planes, that is, mapping a point on one plane to a point on another plane.

[0110] In the specific implementation, the imaging device projects and maps the trailer outline, lane image, vehicle image and road landscape image to obtain the homography matrix corresponding to each image, which is used to describe the perspective transformation relationship between image planes; then, according to the homography matrix and field of view parameters, the initial fused image is subjected to perspective transformation to convert the image to the perspective of top-down projection to obtain the panoramic bird's-eye view corresponding to the trailer.

[0111] During perspective transformation, the input feature maps from the preceding convolutional layer undergo projection transformation. Since the transformations differ between the input streams of different cameras, different homography matrices are obtained through IPM perspective transformation. Simultaneously, to ensure spatial consistency, all transformed feature maps capture the same field of view as the actual ground-based BEV (bird's-eye view). Finally, these transformed feature maps are concatenated into a single feature map. The image is then transformed to a top-down projection perspective to obtain a panoramic bird's-eye view of the trailer.

[0112] Furthermore, for ease of understanding, such as Figure 8 As shown, Figure 8 This is a schematic diagram illustrating a scene where a deep learning network fuses images from four directions in the third embodiment of the trailer view imaging method of the present invention. In this embodiment, the vehicle-mounted camera acquires images from four directions simultaneously, such as... Figure 8 As shown, the front camera image 1 is obtained from the front camera, the rear camera image 2 is obtained from the rear camera, the left camera image 3 is obtained from the left camera, and the right camera image 4 is obtained from the right camera (the number of images is not limited in this embodiment; images from four directions are used for illustration, but this embodiment does not limit this). Then, the images from the four cameras in the four directions are input into a deep learning-based image fusion module (such as DeepLab Xception, uNetXST, etc.) for fusion to obtain the panoramic trailer view shown in the figure. The more images acquired by the cameras, the more image information is obtained, and the closer the fused result is to reality. As shown in the figure, in the panoramic trailer view, the towing vehicle is pulling the trailer, and there are other vehicles around, such as buses. After fusing the trailer model, the trailer is displayed with a highlighted mark. The panoramic trailer view is displayed from a bird's-eye view, allowing the driver to better observe the environment around the vehicle or trailer and better remind the driver of the distance between the trailer and other vehicles, and between the towing vehicle and the trailer.

[0113] Furthermore, considering the accuracy of the segmented trailer outline, this embodiment, after step S11, further includes: annotating the trailer image in the multi-camera images to obtain an annotated image corresponding to the trailer. After step S12, it further includes: comparing the annotated image with the trailer outline, and correcting the trailer outline based on the comparison result to obtain a corrected trailer outline.

[0114] In the specific implementation, the trailer can be labeled before semantic segmentation results are processed; and after semantic segmentation, the segmented trailer outline can be compared with the actual labeled image to correct the segmentation results and obtain the corrected trailer outline, thereby improving the accuracy of the trailer outline.

[0115] This embodiment acquires images of the surrounding environment from multiple directions, including the front, rear, left, and right sides, using multiple onboard cameras mounted on the vehicle. These images include images of surrounding vehicles, lanes, and the surrounding road landscape. Semantic segmentation has been performed on these images, with a focus on segmenting vehicles, lanes, and the surrounding road landscape. The trailer outline, lane images, vehicle images, and road landscape images obtained from semantic segmentation are jointly calibrated and fused to obtain a panoramic bird's-eye view of the trailer. This makes the view closer to the real-world form, allowing the driver to better observe the environment around the vehicle or trailer and improving driving safety. Furthermore, the trailer outline, lane images, vehicle images, and road landscape images can be input into a deep learning network for fusion to obtain an initial fused image. Then, the same field-of-view parameters as the real BEV view on the ground are captured from the images from the multiple cameras. Perspective transformation is performed on the initial fused image based on the field-of-view parameters to obtain the panoramic bird's-eye view of the trailer, ensuring spatial consistency and improving the realism of the panoramic bird's-eye view. Furthermore, before processing the semantic segmentation results, the trailer can be labeled; and after semantic segmentation, the segmented trailer outline can be compared with the actual labeled image to correct the segmentation results, obtain the corrected trailer outline, and improve the accuracy of the trailer outline.

[0116] Furthermore, this embodiment of the invention also proposes a storage medium storing a trailer view imaging program, which, when executed by a processor, implements the steps of the trailer view imaging method described above.

[0117] Reference Figure 9 , Figure 9 This is a structural block diagram of the first embodiment of the trailer view imaging device of the present invention.

[0118] like Figure 9 As shown, the trailer view imaging device proposed in this embodiment of the invention includes:

[0119] Semantic segmentation module 901 is used to perform semantic segmentation on the multi-camera images of the trailer to obtain the trailer outline and target image;

[0120] Image fusion module 902 is used to perform joint calibration and fusion of the trailer outline and the target image to obtain a panoramic bird's-eye view of the trailer.

[0121] The model matching module 903 is used to match the measured distance parameters and the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer.

[0122] The trailer view module 904 is used to fuse the trailer model and the panoramic bird's-eye view to obtain a panoramic trailer view.

[0123] This embodiment acquires multi-camera images, including images of various objects, using cameras mounted on the vehicle. These images need to be labeled and segmented to extract the necessary images for the trailer view, including the trailer outline and target images. For example, the trailer outline is segmented to accurately display the trailer size; the segmented target image can include images of other vehicles, lanes, or road landscapes from the multi-camera images, better alerting the driver to the distance between the trailer and other vehicles, and between the tractor and trailer. Then, the trailer outline and target images are spatially jointly calibrated and fused, and image data is processed to obtain a panoramic bird's-eye view reflecting a three-dimensional, top-down perspective of the vehicle's surroundings. This allows the driver to better observe the environment around the vehicle or trailer from a bird's-eye view. Next, the trailer outline is matched against a model dataset to obtain the corresponding trailer model. Since different trailers use different tow bar lengths, measured distance parameters can be used as a reference to further determine the trailer model, improving the accuracy of obtaining the trailer model. Finally, the identified trailer is highlighted and replaced with a trailer model, which is then fused with the measured distance parameters into the panoramic bird's-eye view to obtain a complete panoramic view of the trailer. This makes the view closer to the actual shape of the trailer, allowing the driver to better observe the vehicle or trailer's surroundings from a bird's-eye view. Compared to traditional methods that rely on hardware output signals, this invention eliminates the need for additional cameras, thus reducing costs. Furthermore, compared to traditional lidar ranging methods that measure the dimensions of a single trailer facade, this invention can also perceive the overall size of the trailer, making the view closer to its actual shape. The bird's-eye view allows the driver to better observe the vehicle or trailer's surroundings, not only reducing the difficulty of towing the trailer for the driver but also improving driving safety.

[0124] Based on the first embodiment of the trailer view imaging device of the present invention described above, a second embodiment of the trailer view imaging device of the present invention is proposed.

[0125] In this embodiment, the model matching module 903 is further configured to acquire a target image corresponding to the target on the trailer, wherein the target is set in front of the trailer and the target is used for distance measurement; to measure the distance between the trailer and the tractor by measuring the distance between the trailer and the tractor by measuring the distance between the trailer and the tractor by measuring the distance between the trailer and the tractor by measuring the distance between the trailer and the tractor by measuring the distance between the trailer and the trailer outline by matching the distance parameters with the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer.

[0126] Furthermore, the model matching module 903 is also used to measure the target image to obtain the pixel size of the target in the target image; determine the camera corresponding to the target image, and obtain the focal length parameter of the camera and the width parameter of the sensor in the camera; calculate the distance parameter between the trailer and the tractor by using the principle of triangle similarity to obtain the pixel size, the actual size of the target, the focal length parameter and the width parameter.

[0127] Furthermore, the semantic segmentation module 901 is also used to acquire multi-camera images corresponding to the environment around the trailer based on the multiple on-board cameras of the trailer; perform semantic segmentation on the multi-camera images to obtain the trailer outline and target image, wherein the target image includes lane image, vehicle image and road landscape image in the multi-camera images.

[0128] The image fusion module 902 is also used to perform joint calibration and fusion of the trailer outline, the lane image, the vehicle image and the road landscape image to obtain a panoramic bird's-eye view corresponding to the trailer.

[0129] Furthermore, the image fusion module 902 is also used to input the trailer outline, the lane image, the vehicle image and the road landscape image into a deep learning network for fusion to obtain an initial fused image; obtain the field of view parameters of each image in the multi-camera image; and perform perspective transformation on the initial fused image according to the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer.

[0130] Furthermore, the image fusion module 902 is also used to perform projection mapping on the trailer outline, the lane image, the vehicle image and the road landscape image to obtain the homography matrix corresponding to each image; and to perform perspective transformation on the initial fused image according to the homography matrix and the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer.

[0131] Furthermore, the trailer view imaging device also includes a contour correction module 905, which is used to annotate the trailer image in the multi-camera images to obtain an annotated image corresponding to the trailer.

[0132] The contour correction module 905 is further configured to compare the labeled image with the trailer contour, and correct the trailer contour according to the comparison result to obtain the corrected trailer contour.

[0133] Other embodiments or specific implementations of the trailer view imaging device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0134] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0135] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0137] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method of trailer view imaging, the method comprising: The trailer view imaging method includes: Semantic segmentation is performed on the multi-camera images of the trailer to obtain the trailer outline and target image; The trailer outline and the target image are jointly calibrated and fused to obtain a panoramic bird's-eye view of the trailer. The trailer model is obtained by matching the measured distance parameters and the trailer outline in the model dataset. The trailer model and the panoramic bird's-eye view are merged to obtain a panoramic trailer view; The step of matching the measured distance parameters and the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer includes: Acquire the target image corresponding to the target on the trailer, wherein the target is set in front of the trailer and the target is used for distance measurement; The target image is measured to obtain the pixel size of the target in the target image; The camera corresponding to the target image is determined, and the focal length parameter of the camera and the width parameter of the sensor in the camera are obtained; The distance parameter between the trailer and the tractor is obtained by calculating the pixel size, the actual target size, the focal length parameter, and the width parameter using the principle of triangle similarity. The distance parameter is equal to the quotient of the actual target size and the pixel size multiplied by the quotient of the focal length parameter and the width parameter. The trailer model corresponding to the trailer is obtained by matching the distance parameter and the trailer outline in the model dataset.

2. The method of claim 1, wherein, The step of semantic segmentation of the multi-camera images of the trailer to obtain the trailer outline and target image includes: The trailer's surrounding environment is captured by multiple onboard cameras. Semantic segmentation is performed on the multi-camera images to obtain the trailer outline and target image, wherein the target image includes lane images, vehicle images and road landscape images from the multi-camera images; The step of jointly calibrating and fusing the trailer outline and the target image to obtain a panoramic bird's-eye view corresponding to the trailer includes: The trailer outline, the lane image, the vehicle image, and the road landscape image are jointly calibrated and fused to obtain a panoramic bird's-eye view corresponding to the trailer.

3. The method of claim 2, wherein, The step of jointly calibrating and fusing the trailer outline, the lane image, the vehicle image, and the road landscape image to obtain a panoramic bird's-eye view corresponding to the trailer includes: The trailer outline, the lane image, the vehicle image, and the road landscape image are input into a deep learning network for fusion to obtain an initial fused image; Obtain the field of view parameters of each image in the multi-camera images; Based on the field of view parameters, a perspective transformation is performed on the initial fused image to obtain a panoramic bird's-eye view of the trailer.

4. The trailer view imaging method as described in claim 3, characterized in that, The step of performing perspective transformation on the initial fused image based on the field of view parameters to obtain a panoramic bird's-eye view corresponding to the trailer includes: Projection mapping is performed on the trailer outline, the lane image, the vehicle image, and the road landscape image to obtain the homography matrix corresponding to each image; Based on the homography matrix and the field of view parameters, a perspective transformation is performed on the initial fused image to obtain a panoramic bird's-eye view of the trailer.

5. The trailer view imaging method as described in claim 2, characterized in that, After acquiring multi-camera images of the surrounding environment of the trailer based on multiple onboard cameras of the trailer, the method further includes: The trailer image in the multi-camera images is labeled to obtain the labeled image corresponding to the trailer; After performing semantic segmentation on the multi-camera images to obtain the trailer outline and target image, the target image includes lane images, vehicle images, and road landscape images from the multi-camera images, and further includes: The labeled image is compared with the trailer outline, and the trailer outline is corrected based on the comparison results to obtain the corrected trailer outline.

6. A trailer view imaging device, characterized in that, The device includes: The semantic segmentation module is used to perform semantic segmentation on the multi-camera images of the trailer to obtain the trailer outline and target image. An image fusion module is used to perform joint calibration and fusion of the trailer outline and the target image to obtain a panoramic bird's-eye view of the trailer. The model matching module is used to match the measured distance parameters and the trailer outline in the model dataset to obtain the trailer model corresponding to the trailer. The trailer view module is used to merge the trailer model and the panoramic bird's-eye view to obtain a panoramic trailer view; The model matching module is also used for: Acquire the target image corresponding to the target on the trailer, wherein the target is set in front of the trailer and the target is used for distance measurement; The target image is measured to obtain the pixel size of the target in the target image; The camera corresponding to the target image is determined, and the focal length parameter of the camera and the width parameter of the sensor in the camera are obtained; The distance parameter between the trailer and the tractor is obtained by calculating the pixel size, the actual target size, the focal length parameter, and the width parameter using the principle of triangle similarity. The distance parameter is equal to the quotient of the actual target size and the pixel size multiplied by the quotient of the focal length parameter and the width parameter. The trailer model corresponding to the trailer is obtained by matching the distance parameter and the trailer outline in the model dataset.

7. A trailer view imaging device, characterized in that, The device includes: a memory, a processor, and a trailer view imaging program stored in the memory and executable on the processor, the trailer view imaging program being configured to implement the steps of the trailer view imaging method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a trailer view imaging program, which, when executed by a processor, implements the steps of the trailer view imaging method as described in any one of claims 1 to 5.

Images