Cms-based vehicle turning blind area compensation method and device and vehicle
By installing multiple cameras on the front of large vehicles and on both sides of the trailer, and then stitching together the images, the blind spot problem caused by the vehicle body obstructing the view when large vehicles turn is solved, thus improving driving safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING KANKAN INTELLIGENT TECH CO LTD
- Filing Date
- 2023-09-28
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, camera monitoring systems cannot effectively solve the blind spot problem caused by the vehicle body obstructing the view of large vehicles when they turn.
Multiple cameras are installed on both sides of the vehicle's front and both sides of the trailer. By acquiring and stitching together the images from each camera, blind spot information is supplemented to form a complete display image.
It improves the driver's field of vision, reduces blind spots when turning, and enhances safety during driving.
Smart Images
Figure CN117400860B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle detection sensor technology, and in particular to a vehicle turning blind spot compensation method, device, and vehicle based on CMS. Background Technology
[0002] With the development of technology, vehicles are ubiquitous in daily life. However, with the increase in the number of vehicles, blind spots are easily created when turning, leading to more and more accidents. Current technology uses Camera Monitor Systems (CMS) to detect and compensate for blind spots when turning. Cameras are placed on either side of the driver's seat or the left and right sides of the rearview mirror to capture images of the left and right sides of the vehicle. For small vehicles, both cameras can capture images of the left and right sides during turns, which can eliminate blind spots to some extent. However, for large vehicles, the vehicle body obstructs the view of the cameras during turns, creating blind spots. Camera monitor systems cannot solve this type of blind spot problem caused by vehicle body obstruction. Summary of the Invention
[0003] This application provides a vehicle turning blind spot compensation method, device and vehicle based on CMS, which can avoid blind spots when the vehicle is turning.
[0004] In a first aspect, embodiments of this application provide a vehicle turning blind spot compensation method based on CMS, characterized in that the vehicle includes a front end, a trailer attached to the front end, a first camera symmetrically arranged on the right side of the front end and a second camera symmetrically arranged on the left side, and a third camera and a fourth camera symmetrically arranged on the left and right sides of the trailer. The vehicle turning blind spot compensation method based on CMS includes the following steps: when the vehicle turns right, acquiring a first image captured by the first camera and a third image captured by the third camera; when the vehicle turns left, acquiring a second image captured by the second camera and a fourth image captured by the fourth camera; performing blind spot detection on the first image or the second image to obtain a first blind spot or a second blind spot; determining the area of the first blind spot in the third image based on the first blind spot to obtain a first region of interest, or determining the area of the second blind spot in the fourth image based on the second blind spot to obtain a second region of interest; stitching the first image and the first region of interest together to obtain a first display screen, or stitching the third image and the second region of interest together to obtain a second display screen.
[0005] Secondly, embodiments of this application provide a vehicle turning blind spot compensation device, comprising: an acquisition module, configured to acquire a first image captured by a first camera and a third image captured by a third camera when the vehicle turns right, and to acquire a second image captured by a second camera and a fourth image captured by a fourth camera when the vehicle turns left; a detection module, configured to perform blind spot detection on the first image or the third image to obtain a first blind spot or a second blind spot; a confirmation module, configured to determine the area of the first blind spot in the second image based on the first blind spot to obtain a first region of interest, or to determine the area of the second blind spot in the fourth image based on the second blind spot to obtain a second region of interest; a stitching module, configured to stitch the first image and the first region of interest together to obtain a first display screen, or to stitch the third image and the second region of interest together to obtain a second display screen; and an output module, configured to output the first display screen or the second display screen.
[0006] Thirdly, embodiments of this application provide a vehicle, including: a camera device, including a first camera and a second camera disposed on the left and right sides of the front of the vehicle, and a third camera and a fourth camera respectively symmetrically disposed on the left and right sides of the trailer and on the right side of the trailer; a vehicle turning blind spot compensation device, including: a memory for storing program instructions of the CMS-based vehicle turning blind spot compensation method; and a processor for executing the program instructions of the CMS-based vehicle turning blind spot compensation method to implement the CMS-based vehicle turning blind spot compensation method as described in the first aspect.
[0007] The aforementioned vehicle turning blind spot compensation method, device, and vehicle based on CMS, by setting up multiple cameras, can compensate for blind spots caused by vehicles obstructing the view of the vehicle's cameras during turning, thereby improving vehicle safety to a certain extent. Attached Figure Description
[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0009] Figure 1 A flowchart of a vehicle turning blind spot compensation method based on CMS provided in this application embodiment.
[0010] Figure 2 This is a schematic diagram of a vehicle turning blind spot compensation device provided in an embodiment of this application.
[0011] Figure 3 This is a schematic diagram of a vehicle structure provided in an embodiment of this application.
[0012] Figure 4 This is a schematic diagram of a vehicle scenario provided in an embodiment of this application.
[0013] Figure 5 This is a diagram showing the effect of stitching together vehicle images provided in an embodiment of this application.
[0014] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0016] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar planned objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data are interchangeable where appropriate; in other words, the described embodiments are implemented according to a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, may also include other content; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0017] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0018] Please refer to Figure 1This document presents a flowchart of a vehicle turning blind spot compensation method based on a CMS (Camera Monitor System) according to an embodiment of this application. This method is applied to vehicle turning blind spot compensation devices and large vehicles, used to compensate for blind spots caused by vehicle body obstruction during turning. The vehicle CMS typically refers to an electronic exterior rearview mirror system, which uses an external camera to capture the field of view, sends signals to the electronic control unit (ECU) for further processing, and finally provides the driver with the view via a monitor. It can also integrate blind spot warning and obstacle indication functions. The vehicle CMS system compensates for blind spots by installing cameras on both sides of the front rearview mirrors, namely the first camera 10a and the second camera 10b. A third camera 10c and a fourth camera 10d are installed on both sides of a trailer attached to the front of the vehicle to compensate for the blind spots captured by the first camera 10a and the second camera 10b. How the third camera 10c and the fourth camera 10d compensate for blind spots using the images captured by the CMS system will be described in detail below.
[0019] When the vehicle turns right, the first camera 10a captures an image of the right side of the vehicle. Due to the obstruction of the vehicle body, the first camera 10a cannot capture an image of the left side of the vehicle, and the second camera 10b also cannot capture an image of the left side of the vehicle. At this time, a blind spot is created on the left side of the vehicle. By setting a third camera 10c on the left side of the trailer attached to the front of the vehicle, an image of the left side of the vehicle is captured. The image captured by the third camera 10c is used to fill in the blind spot by filling in the image captured by the first camera 10a, and the blind spot image is displayed on the vehicle's display screen.
[0020] When the vehicle turns left, the second camera 10b captures an image of the left side of the vehicle. Due to the vehicle's obstruction, the second camera 10b cannot capture an image of the right side of the vehicle, nor can the first camera 10a. A fourth camera 10d is installed on the right side of the trailer attached to the front of the vehicle to capture an image of the right side of the vehicle. The image captured by the fourth camera 10d is used to fill in the blind spot by supplementing the image captured by the second camera 10b, resulting in a blind spot compensation image, which is then displayed on the vehicle's display screen. By installing cameras on both sides of the vehicle and on both sides of the trailer attached to the front of the vehicle, blind spots are avoided when the vehicle turns. Specifically, this CMS-based vehicle turning blind spot compensation method includes the following steps.
[0021] Step S101: When the vehicle turns right, acquire the first image captured by the first camera and the third image captured by the third camera; when the vehicle turns left, acquire the second image captured by the second camera and the fourth image captured by the fourth camera.
[0022] In step S101, the first camera 10a is located on the right side of the front of the vehicle, the second camera 10b is located on the left side of the front of the vehicle, the third camera 10c is located on the left side of the trailer attached to the front of the vehicle, and the fourth camera 10d is located on the right side of the trailer attached to the front of the vehicle.
[0023] When the vehicle turns right, the vehicle body partially obstructs the field of view of the first camera 10a. The first image captured by the first camera 10a cannot capture an image of the left side of the vehicle, and the second image captured by the second camera 10b also cannot capture an image of the left side of the vehicle, thus creating a blind spot on the left side of the vehicle. A third camera 10c is installed on the left side of the vehicle's trailer to capture a third image of the left side of the vehicle when it turns right, thus compensating for the blind spot created by the first image captured by the first camera 10a. When the vehicle turns left, the vehicle body obstructs the field of view of the second camera 10b. The second image captured by the second camera 10b cannot capture an image of the right side of the vehicle, and the first image captured by the first camera 10a also cannot capture an image of the right side of the vehicle, thus creating a blind spot on the right side of the vehicle. A fourth camera 10d is installed on the right side of the vehicle's trailer to capture a fourth image of the right side of the vehicle, thus compensating for the blind spot created by the second image captured by the second camera 10b. The cameras include, but are not limited to, monocular, binocular, and depth cameras. All images (first, second, third, and fourth) are 2D images.
[0024] By installing cameras on both sides of the vehicle's front and on both sides of the trailer attached to the vehicle, the cameras can capture complete images of the left and right sides of the vehicle, regardless of whether the vehicle turns left or right. This not only improves the driver's field of vision during driving but also, to some extent, avoids the occurrence of blind spots when turning.
[0025] Step S102: Perform blind zone detection on the first image or the second image to obtain a first blind zone or a second blind zone.
[0026] In step S102, when the vehicle turns right, edge detection is performed on the first image captured by the first camera 10a using an edge detection algorithm. The edge of the vehicle body on the right side in the first image is identified to obtain the edge of the vehicle body on the right side in the first image. The edge of the vehicle body on the right side is marked to obtain key points of the edge of the vehicle body on the right side. The key points of the edge of the vehicle body on the right side are used for subsequent image stitching. Based on the edge of the vehicle body on the right side in the first image, the area that the first camera 10a cannot capture due to the vehicle body is identified and the area that the first camera 10a cannot capture is recorded as the first blind zone.
[0027] When the vehicle turns left, edge detection algorithms are used to perform edge detection on the second image captured by the second camera 10b. The edges of the vehicle body on the left side of the second image are identified, and the left edge is marked to obtain key points. These key points are used for subsequent image stitching. Based on the left edge of the vehicle in the second image, areas that the second camera 10b cannot capture due to vehicle occlusion are identified and recorded as the second blind spot. The edge detection algorithms include, but are not limited to, the Canny edge detection algorithm, the Sobel operator, and the Laplacian operator.
[0028] For example, when the vehicle turns right, the first camera 10a captures an image of the right side of the vehicle, but due to the obstruction of the vehicle body, the first camera 10a cannot capture an image of the left side of the vehicle. At this time, the left side of the vehicle is recorded as the first blind spot. When the vehicle turns left, the second camera 10b captures an image of the left side of the vehicle, but due to the obstruction of the vehicle body, the second camera 10b cannot capture an image of the right side of the vehicle. At this time, the right side of the vehicle is recorded as the second blind spot.
[0029] Step S103: Determine the region of the first blind zone in the third image based on the first blind zone to obtain a first region of interest, or determine the region of the second blind zone in the fourth image based on the second blind zone to obtain a second region of interest.
[0030] In step S103, a first region of interest (ROI) is obtained by identifying the area of the first blind zone in the third image based on the first blind zone; that is, an image in the third image that shares the same area as the first blind zone is identified as the first ROI. Similarly, a second ROI is obtained by identifying the area of the second blind zone in the fourth image based on the second blind zone; that is, an image in the fourth image that shares the same area as the second blind zone is identified as the second ROI.
[0031] The first region of interest (ROI) is identified in the third image as the region that shares the same area as the first blind zone, and the second region of interest is identified in the fourth image as the region that shares the same area as the second blind zone. This is mainly achieved through mapping relationships, whereby the first blind zone is mapped onto the third image to obtain the first ROI, or the second blind zone is mapped onto the fourth image to obtain the second ROI. How to map the first or second blind zone is described in detail below.
[0032] When the vehicle turns right, a mapping relationship from 2D to 3D image is first established based on the intrinsic and extrinsic parameters of the first camera 10a. The first blind spot is then transformed into a 3D image. Similarly, a mapping relationship from 3D to 2D image is established based on the intrinsic and extrinsic parameters of the third camera 10c. The 3D image of the first blind spot is then mapped onto the third image to obtain the first region of interest. The purpose of mapping the 3D image of the first blind spot to the third image is to obtain the image within the first blind spot, i.e., the image within the first region of interest, for subsequent stitching and fusion.
[0033] When the vehicle turns left, a mapping relationship from 2D to 3D image is first established based on the intrinsic and extrinsic parameters of the second camera 10b. The second blind spot is then transformed into a 3D image. Similarly, a mapping relationship from 3D to 2D image is established based on the intrinsic and extrinsic parameters of the fourth camera 10d. The 3D image of the second blind spot is then mapped onto the fourth image to obtain the second region of interest. The purpose of mapping the 3D image of the second blind spot to the fourth image is to obtain the image within the second blind spot, i.e., the image within the second region of interest, for subsequent stitching and fusion.
[0034] The first camera 10a, the second camera 10b, the third camera 10c, and the fourth camera 10d each contain intrinsic and extrinsic parameters. The intrinsic parameters include the focal length *f* and the principal point coordinates (Cx, Cy), while the extrinsic parameters include the rotation matrix *R* and the translation vector *t*. Based on the intrinsic and extrinsic parameters of the cameras, 2D to 3D or 3D to 2D image conversions are performed. Taking the first camera 10a and the third camera 10c as examples, a 2D to 3D mapping relationship is established based on the intrinsic and extrinsic parameters of the first camera 10a, and the first blind zone is 3D converted to obtain a 3D image of the first blind zone. Similarly, based on the intrinsic and extrinsic parameters of the third camera 10c, a 3D to 2D mapping relationship is established, and the 3D image of the first blind zone is mapped to the third image to obtain the first region of interest.
[0035] Based on the intrinsic and extrinsic parameters of the first camera 10a, a 2D-to-3D mapping relationship is established, and the first blind zone is transformed into a 3D image to obtain the first blind zone. The specific steps are as follows.
[0036] (1) Given the extrinsic and intrinsic parameters of the first camera 10a, and the coordinates of point A in the first blind zone as (u, v), calculate the coordinates (X, Y) of the normalized plane point A1 using the coordinates of point A:
[0037] X = (u - Cx) / f
[0038] Y = (v - Cy) / f
[0039] Where “f” is the focal length of the first camera 10a, and Cx and Cy are the x and y coordinates of the principal point of the first camera 10a.
[0040] (2) Convert the coordinates (X,Y) of point A1 on the normalized plane to the coordinates (x1,y1,z1) of point A2 in the camera coordinate system:
[0041] x1=X*z1
[0042] y1=Y*z1
[0043] z1 = 1
[0044] (3) By applying the inverse matrix of the extrinsic parameters of the first camera 10a, we can reverse the calculation of the coordinates of point A2 in the camera coordinate system as the coordinates of point A3 in the 3D image of the first blind zone:
[0045] A3 = R^(-1)*(A2-t)
[0046] Where “R^(-1)” is the inverse of the rotation matrix R of the first camera 10a, and t is the translation vector of the first camera 10a.
[0047] Through the above steps (1), (2), and (3), based on the intrinsic and extrinsic parameters of the first camera 10a, the first blind zone can be transformed into a 3D image of the first blind zone.
[0048] Based on the intrinsic and extrinsic parameters of the third camera 10c, a 3D-to-2D mapping relationship is established, and the 3D image of the first blind zone is mapped to the third image to obtain the first region of interest. The specific steps are as follows.
[0049] a. Assume that point B in the 3D image of the first blind zone is B1 in the camera coordinate system. Through the extrinsic parameters of the third camera, we can obtain B1 = R1*B + t1, where "R1" is the rotation matrix of the third camera 10c, t1 is the translation vector of the third camera 10c, and the coordinates of B1 are (x2, y2, z2).
[0050] b. Project point B1(x2,y2,z2) onto the camera image plane to obtain the coordinates (x3,y3) of point B2 on the normalized plane:
[0051] x3=x2 / z2
[0052] y3=y2 / z2
[0053] c. Based on the intrinsic parameters of the third camera 10c, transform point B2 on the normalized plane into point B3(u1,v1) in the first region of interest:
[0054] u1=f1*x3+Cx1
[0055] v1 = f1 * y3 + Cy1
[0056] Wherein, "f1" is the focal length of the third camera 10c, and Cx1 and Cy1 are the horizontal and vertical coordinates of the principal point coordinates of the third camera 10c.
[0057] Through the above steps a, b, and c, based on the intrinsic and extrinsic parameters of the third camera 10c, the 3D image of the first blind zone is mapped onto the third image to obtain the first region of interest.
[0058] Step S104: The first image and the first region of interest are stitched together to obtain a first display screen, or the third image and the second region of interest are stitched together to obtain a second display screen.
[0059] In step S104, the right edge of the vehicle body in the first image captured by the first camera 10a is detected using an edge detection algorithm, resulting in key points on the right edge of the vehicle body in the first image, denoted as the first key points. The left edge of the vehicle body in the image of the first region of interest is detected using the same edge detection algorithm, resulting in key points on the left edge of the vehicle body in the first region of interest, denoted as the second key points. The first and second key points are collectively referred to as first-class stitching points. The homography matrix H in the RANSAC algorithm is used to transform the image of the first region of interest to the same viewpoint as the first image, and the first and second key points are connected and stitched together to obtain the first stitched image. The image of the vehicle body region in the first stitched image is extracted, and the image of the vehicle body region is semi-transparent using alpha image fusion technology to obtain the first display image.
[0060] The left edge of the vehicle body in the second image captured by the second camera 10b is detected using an edge detection algorithm, resulting in key points on the left edge of the vehicle body in the first image, denoted as the third key point. The right edge of the vehicle body in the image of the second region of interest is detected using the same edge detection algorithm, resulting in key points on the right edge of the vehicle body in the second region of interest, denoted as the fourth key point. The third and fourth key points are collectively referred to as second-class stitching points. The homography matrix H in the RANSAC algorithm is used to transform the image of the second region of interest to the same viewpoint as the second image, and the third and fourth key points are connected and stitched together to obtain the second stitched image. The vehicle body region image is extracted from the second stitched image, and alpha image fusion technology is used to semi-transparently process the vehicle body region image to obtain the second display image.
[0061] Specifically, when the vehicle turns right, the first display screen shows the left and right sides of the vehicle. When the vehicle turns left, the second display screen shows the left and right sides of the vehicle. This way, regardless of whether the vehicle is turning left or right, the side view of the vehicle can be displayed, which improves the driver's safety to a certain extent and reduces the occurrence of accidents caused by blind spots.
[0062] Step S105: Display the first display screen or the second display screen on the display screen.
[0063] In step S105, the display screen is set on the vehicle's center console for the driver to view. When the vehicle turns right, the display screen displays the first display screen, and when the vehicle turns left, the display screen displays the second display screen.
[0064] Please refer to Figure 2 This is a structural schematic diagram of a vehicle turning blind spot compensation device provided in an embodiment of this application.
[0065] The vehicle turning blind spot compensation device 200 includes an acquisition module 210, a detection module 220, a confirmation module 230, a stitching module 240, and an output module 250. The acquisition module 210 acquires a first image captured by the first camera 10a and a second image captured by the third camera 10c when the vehicle turns right, and a third image captured by the second camera 10b and a fourth image captured by the fourth camera 10d when the vehicle turns left. The first camera 10a is located on the right side of the vehicle's front, the second camera 10b is located on the left side of the vehicle's front, the third camera 10c is located on the left side of the trailer attached to the front of the vehicle, and the fourth camera 10d is located on the right side of the trailer attached to the front of the vehicle. When the vehicle turns right, the third image captured by the third camera 10c is used to compensate for the blind spot in the first image captured by the first camera 10a; when the vehicle turns left, the fourth image captured by the fourth camera 10d is used to compensate for the blind spot in the second image.
[0066] The detection module 220 is used to perform blind spot detection on the first image or the third image to obtain a first blind spot or a second blind spot. When the vehicle turns right, edge detection is performed on the first image captured by the first camera 10a using an edge detection algorithm to obtain the right side edge of the vehicle body in the first image. Based on the right side edge of the vehicle body in the first image, the first blind spot is obtained. When the vehicle turns left, edge detection is performed on the second image captured by the second camera 10b using an edge detection algorithm to obtain the left side edge of the vehicle body in the second image. Based on the left side edge of the vehicle body in the second image, the second blind spot is obtained.
[0067] The confirmation module 230 is used to determine the region of the first blind zone in the second image based on the first blind zone to obtain a first region of interest, or to determine the region of the second blind zone in the fourth image based on the second blind zone to obtain a second region of interest.
[0068] When the vehicle turns right, a 2D-to-3D image mapping relationship is established based on the intrinsic and extrinsic parameters of the first camera 10a. The first blind spot is then transformed into a 3D image. Based on the intrinsic and extrinsic parameters of the third camera 10c, a 3D-to-2D image mapping relationship is established. The 3D image of the first blind spot is mapped onto the third image to obtain the first region of interest, which is used for subsequent stitching and fusion. When the vehicle turns left, a 2D-to-3D image mapping relationship is established based on the intrinsic and extrinsic parameters of the second camera 10b. The second blind spot is then transformed into a 3D image. Based on the intrinsic and extrinsic parameters of the fourth camera 10d, a 3D-to-2D image mapping relationship is established. The 3D image of the second blind spot is mapped onto the fourth image to obtain the second region of interest, which is used for subsequent stitching and fusion.
[0069] The stitching module 240 is used to stitch the first image and the first region of interest (ROI) to obtain a first display image, or to stitch the third image and the second ROI to obtain a second display image. An edge detection algorithm is used to detect vehicle edges in the first image captured by the first camera 10a and the image of the first ROI, obtaining key points on the right edge of the vehicle in the first image (denoted as the first key point) and key points on the left edge of the vehicle in the image of the first ROI (denoted as the second key point). The homography matrix H in the RANSAC algorithm is used to switch the image of the first ROI to the same viewpoint as the first image, and the first and second key points are connected and stitched together to obtain the first stitched image. An alpha image fusion algorithm is used to semi-transparent the vehicle edge region in the first stitched image to obtain the first display image.
[0070] Edge detection algorithms are used to detect vehicle edges in the second image and the second region of interest (ROI) image captured by the second camera 10b. Key points on the left edge of the vehicle in the second image are identified and designated as the third key point, and key points on the right edge of the vehicle in the second ROI image are identified and designated as the fourth key point. The homography matrix H in the RANSAC algorithm is used to switch the image of the second ROI to the same viewpoint as the second image, and the third and fourth key points are connected and stitched together to obtain the second stitched image. An alpha image fusion algorithm is then used to semi-transparently process the vehicle edge regions in the second stitched image to obtain the second display image.
[0071] The output module 250 is used to output either the first display screen or the second display screen. The display screen is installed on the vehicle's center console for the driver to view. When the vehicle turns right, the display screen displays the first display screen, and when the vehicle turns left, the display screen displays the second display screen.
[0072] Please refer to Figure 3 This is a schematic diagram of a vehicle structure provided in an embodiment of this application.
[0073] The vehicle 30 includes a camera device 10 and a vehicle turning blind spot compensation device 20. The camera device 10 includes a first camera 10a and a second camera 10b disposed on the left and right sides of the front of the vehicle, and a third camera 10c symmetrically disposed on the left and right sides of the trailer and a fourth camera 10d on the right side of the trailer. The vehicle turning blind spot compensation device 20 includes a memory 201 and a processor 202. The memory 201 is used to store the program instructions of the CMS-based vehicle turning blind spot compensation method; the processor 202 is used to execute the program instructions of the CMS-based vehicle turning blind spot compensation method to implement the CMS-based vehicle turning blind spot compensation method.
[0074] Since vehicle 30 adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.
[0075] Please refer to Figure 4 This is a schematic diagram of a vehicle scenario provided in an embodiment of this application.
[0076] Taking a vehicle turning right as an example, where "R1" represents "first camera 10a", "L1" represents "second camera 10b", "L2" represents "third camera", and "R2" represents "fourth camera 10d". The following example illustrates how, when a vehicle turns right, camera R1 captures an image of the right side of the vehicle. Region "B" represents the area that camera R1 can capture. Due to vehicle body obstruction, neither camera R1 nor L1 can capture an image of the left side of the vehicle. Therefore, camera L2 is placed on the left side of the vehicle's trailer to capture an image of the left side. This image from camera L2 is used to fill in the blind spot in the image captured by camera R1. Edge detection technology is used to detect the vehicle body edges in the image captured by camera R1, obtaining the first blind spot. Based on the intrinsic and extrinsic parameters of cameras R1 and L2, the first blind spot is mapped onto the image captured by camera L2, yielding the first region of interest (ROI). Figure 4 The image within the diagonal region "A" is stitched together with the image captured by camera R1 to obtain the first display image.
[0077] Please refer to Figure 5This is a vehicle image stitching effect diagram provided in the embodiments of this application.
[0078] Taking a vehicle turning right as an example, region "B" is the area captured by camera R1, and region "A" is the first region of interest. Vehicle edge detection is performed on the image captured by camera R1 and the image in the first region of interest "A" to obtain stitching key points. Based on the stitching key points, the image captured by camera R1, i.e., the image in region "B", is stitched and merged with the image in the first region of interest "A" to obtain the first display image.
[0079] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.
[0080] The above-listed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.
Claims
1. A vehicle turning blind spot compensation method based on CMS, characterized in that, The vehicle includes a front end, a trailer attached to the front end, a first camera symmetrically positioned on the right side of the front end and a second camera symmetrically positioned on the left side, and a third camera symmetrically positioned on the left and right sides of the trailer and a fourth camera on the trailer. The CMS-based vehicle turning blind spot compensation method includes: When the vehicle turns right, acquire the first image captured by the first camera and the third image captured by the third camera; When the vehicle turns left, acquire the second image captured by the second camera and the fourth image captured by the fourth camera; Blind spot detection is performed on the first image or the second image to obtain a first blind spot or a second blind spot; Determining a first region of interest (ROI) based on the first blind zone in the third image, or determining a second region of interest based on the second blind zone in the fourth image, includes: performing a three-dimensional spatial transformation on the first blind zone or the second blind zone to obtain a 3D image of the first blind zone or the second blind zone; mapping the 3D image of the first blind zone to the third image to obtain the first ROI, or mapping the 3D image of the second blind zone to the fourth image to obtain the second ROI. The method of stitching the first image and the first region of interest (ROI) together to obtain a first display image or stitching the third image and the second ROI together to obtain a second display image includes: performing edge detection on the first ROI and the vehicle body edges in the first image to obtain a first type of stitching points; performing edge detection on the second ROI and the vehicle body edges in the second image to obtain a second type of stitching points; stitching the first ROI with the first image based on the first type of stitching points or the second type of stitching points to obtain a first stitched image, or stitching the second ROI with the second image to obtain a second stitched image; and performing image fusion on the first stitched image to obtain a first display image or performing image fusion on the second stitched image to obtain a second display image.
2. The vehicle turning blind spot compensation method based on CMS as described in claim 1, wherein the vehicle further includes a display screen disposed on the center console for the driver to view, and the vehicle turning blind spot compensation method further includes: The first or second display screen is displayed on the screen.
3. The vehicle turning blind spot compensation method based on CMS as described in claim 1, characterized in that, The step of performing blind zone detection on the first image or the third image to obtain the first blind zone or the second blind zone includes: The vehicle body edge is obtained by performing edge-mapping processing on the first image or the second image; Identify and mark the vehicle body edges in the first image or the second image; Based on the vehicle's body edge, the first blind spot or the second blind spot of the first image or the second image is obtained.
4. The vehicle turning blind spot compensation method based on CMS as described in claim 3, characterized in that, The step of performing edge-mapping processing on the first image or the second image to obtain the vehicle body edge includes: The edge processing of the first image or the second image to obtain the vehicle body edge is achieved using an edge algorithm.
5. The vehicle turning blind spot compensation method based on CMS as described in claim 1, characterized in that, The first image, the second image, the third image, and the fourth image are 2D images.
6. The vehicle turning blind spot compensation method based on CMS as described in claim 5, characterized in that, The step of performing three-dimensional spatial transformation on the first blind zone or the second blind zone to obtain a 3D image of the first blind zone or the second blind zone includes: Based on the intrinsic and extrinsic parameters of the first camera, a mapping relationship from 2D image to 3D image is established; Based on the mapping relationship, a three-dimensional spatial transformation is performed on the first blind zone to obtain a 3D image of the first blind zone; Based on the intrinsic and extrinsic parameters of the second camera, a mapping relationship from 2D image to 3D image is established; Based on the mapping relationship, a three-dimensional spatial transformation is performed on the second blind zone to obtain a 3D image of the second blind zone.
7. The vehicle turning blind spot compensation method based on CMS as described in claim 1, characterized in that, The step of fusing the first stitched image to obtain the first display image or fusing the second stitched image to obtain the second display image includes: The first display image is obtained by making the vehicle body area in the first spliced image semi-transparent, or the second display image is obtained by making the vehicle body area in the second spliced image semi-transparent.
8. A vehicle turning blind spot compensation device, the vehicle comprising a front end, a trailer attached to the front end, a first camera symmetrically arranged on the right side and a second camera symmetrically arranged on the left side of the front end, and a third camera and a fourth camera symmetrically arranged on the left and right sides of the trailer, comprising: The acquisition module is used to acquire a first image captured by the first camera and a third image captured by the third camera when the vehicle turns right, and to acquire a second image captured by the second camera and a fourth image captured by the fourth camera when the vehicle turns left. The detection module is used to perform blind zone detection on the first image or the third image to obtain a first blind zone or a second blind zone; The confirmation module determines a first region of interest (ROI) based on the first blind zone in the second image, or a second region of interest based on the second blind zone in the fourth image. This includes: performing a three-dimensional spatial transformation on the first or second blind zone to obtain a 3D image of the first or second blind zone; mapping the 3D image of the first blind zone to the third image to obtain the first ROI, or mapping the 3D image of the second blind zone to the fourth image to obtain the second ROI. A stitching module is used to stitch the first image and the first region of interest (ROI) to obtain a first display image or to stitch the third image and the second ROI to obtain a second display image. The module includes: performing edge detection on the first ROI and the vehicle body edges in the first image to obtain first-type stitching points; performing edge detection on the second ROI and the vehicle body edges in the second image to obtain second-type stitching points; stitching the first ROI with the first image based on the first-type or second-type stitching points to obtain a first stitched image, or stitching the second ROI with the second image to obtain a second stitched image; and performing image fusion on the first stitched image to obtain a first display image or performing image fusion on the second stitched image to obtain a second display image. The output module is used to output either the first display screen or the second display screen.
9. A vehicle comprising: The camera device includes a first camera and a second camera disposed on the left and right sides of the front of the vehicle, and a third camera and a fourth camera disposed symmetrically on the left and right sides of the trailer and on the right side of the trailer, respectively. Vehicle blind spot compensation device for turning, including: A memory for storing the program instructions of the CMS-based vehicle turning blind spot compensation method; A processor is configured to execute program instructions for the CMS-based vehicle turning blind spot compensation method to implement the CMS-based vehicle turning blind spot compensation method as described in any one of claims 1 to 7.