Vehicle navigation panoramic video stitching method and system

Abstract

The invention provides a vehicle-mounted navigation panoramic video stitching method and system, and belongs to the technical field of auxiliary driving, and the method comprises the following steps: obtaining vehicle periphery original images of a vehicle body in multiple directions in real time; performing image distortion correction on the plurality of original images around the vehicle to obtain a plurality of overlook view angle images for splicing; and dividing the plurality of overlook view angle images into overlapped areas and non-overlapped areas by adopting a partition gradual-in and gradual-out weighted fusion method, retaining image information of the non-overlapped areas, performing weighted distribution on gray weight information of the overlapped areas, and fusing the overlapped areas according to a distributed gray value to obtain a spliced panoramic image around the vehicle body. According to the method, the visual blind area existing during turning, overtaking and side parking can be solved, potential safety hazards are greatly eliminated, meanwhile, ghosting or distortion occurring in the overlapping area can be eliminated, the splicing effect of the panoramic spliced image is improved, and a driver can make correct judgment conveniently.

Description

Technical Field

[0001] The present invention belongs to the technical field of assisted driving, and in particular relates to a method and system for splicing panoramic videos of vehicle-mounted navigation. Background Art

[0002] At present, ordinary vehicles have many blind spots when they are stationary or running. The field of view is very narrow, ranging from 30-60° in both horizontal and vertical directions, which is quite different from the perception range of the human eye and has poor scalability. This leads to inner wheel difference in turning and overtaking blind spots in dynamic situations, posing safety hazards.

[0003] In addition, when multiple cameras are used to reduce blind spots, the multiple images collected are displayed on split screens, and the field of vision is relatively scattered, which can easily cause driver visual fatigue and omission of vision; for this reason, panoramic stitching images came into being, which can provide a more comprehensive field of vision and help drivers better judge the surrounding environment. However, the overlapping areas of existing panoramic stitching images often have ghosting or distortion, and the stitching effect of the stitching boundary is not good, which affects the driver's visual judgment and driving concentration, and creates safety hazards. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a method for stitching panoramic videos of vehicle-mounted navigation, comprising the following steps:

[0005] Real-time acquisition of original images of the vehicle surrounding multiple positions of the vehicle body; the original images of the vehicle surrounding include images of the front visual blind spot, images of the visual blind spots on both sides of the front, images of the blind spots close to both sides of the vehicle body, and images of the rear blind spot;

[0006] Performing image distortion correction on multiple original images around the vehicle to obtain multiple top-view images for stitching;

[0007] A partitioned fade-in and fade-out weighted fusion method is used to stitch and fuse multiple bird's-eye view images. Specifically, the multiple bird's-eye view images are divided into overlapping areas and non-overlapping areas, the image information of the non-overlapping areas is retained, the grayscale weight information of the overlapping areas is weightedly allocated, and the overlapping areas are fused according to the allocated grayscale values ​​to obtain a stitched panoramic image around the vehicle body.

[0008] Preferably, the visual blind spot image of the front of the vehicle is obtained through a front camera installed on the top of the front license plate; the visual blind spot images on both sides of the front of the vehicle are obtained through cameras installed on the left and right rearview mirrors, and the blind spot images close to both sides of the vehicle body are obtained through cameras installed on both sides of the C-pillars; the blind spot image of the rear of the vehicle is obtained through a camera installed in the center above the rear license plate.

[0009] Preferably, before acquiring the original images of the vehicle body in multiple directions in real time, the method further includes calibrating parameters of a camera for acquiring the original images of the vehicle body, including the following steps:

[0010] Prepare multiple black and white grid reference templates of specified sizes and place them around the car;

[0011] Start all cameras to shoot a set of calibration test images;

[0012] Extract the coordinates of the feature points of the image on the reference template and establish the corresponding relationship between the world coordinate system and the pixel coordinate system;

[0013] The external displacement parameter matrix and the internal scale parameter matrix of the camera coordinate transformation are obtained; the matrix parameters of the external displacement parameter matrix and the internal scale parameter matrix are merged to obtain the optimized camera correction parameter model; and the optimized camera correction parameter model is used to calibrate the camera parameters.

[0014] Preferably, the image distortion correction of the multiple original images around the vehicle is performed by: converting the spatial points of the original images around the vehicle from the world coordinate system to the camera coordinate system, then projecting them to the plane coordinate system through proportional transformation, and finally converting them into the pixel coordinate system to obtain the corrected top-down perspective image.

[0015] The present invention also provides a vehicle-mounted navigation panoramic video stitching system, comprising:

[0016] An image acquisition module is used to obtain original images of the vehicle in multiple directions in real time; the original images of the vehicle include images of the front visual blind spot, images of the front visual blind spots on both sides of the vehicle, images of the blind spots close to both sides of the vehicle body, and images of the rear blind spot;

[0017] An image correction module is used to perform image distortion correction on multiple original images around the vehicle to obtain multiple top-view images for stitching;

[0018] The image stitching module is used to stitch and fuse multiple overhead view images using a partitioned fade-in and fade-out weighted fusion method. Specifically, the multiple overhead view images are divided into overlapping areas and non-overlapping areas, the image information of the non-overlapping areas is retained, the grayscale weight information of the overlapping areas is weightedly allocated, and the overlapping areas are fused according to the allocated grayscale values ​​to obtain a stitched panoramic image around the vehicle body.

[0019] Preferably, the image acquisition module is also used to obtain an image that expands the visible range in front of the cockpit and an image that records the driver's driving status in real time. The image that expands the visible range in front of the cockpit is obtained through a camera installed in the front center of the roof, and the image that records the driver's driving status in real time is obtained through a camera installed in the cockpit inside the vehicle.

[0020] Preferably, the image stitching module is an edge computing digital base mounted on the MineHong OS platform.

[0021] Preferably, it also includes a vehicle-mounted display flat panel, which is connected to the edge computing digital base signal, and the vehicle-mounted display flat panel is used to display a panoramic image around the vehicle body.

[0022] The vehicle-mounted navigation panoramic video stitching method and system provided by the present invention have the following beneficial effects:

[0023] The present invention adopts a partitioned fade-in and fade-out weighted fusion method to fuse the overlapping areas of the acquired images, thereby eliminating ghosting or distortion in the overlapping areas, improving the stitching effect of the panoramic stitching image, and providing the driver with a higher quality panoramic monitoring image around the vehicle body, which is conducive to the driver making correct judgments and reducing safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 The present invention is a process of the vehicle navigation panoramic video stitching method;

[0026] Figure 2 This is a schematic diagram of the installation position of the vehicle-mounted camera of the present invention;

[0027] Figure 3 This is a diagram of the image distortion correction process. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0029] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the technical solutions of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0030] In addition, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In the description of the present invention, unless otherwise specified, "plurality" means two or more, which will not be described in detail here.

[0031] Example

[0032] The present invention provides a method for stitching panoramic videos of vehicle-mounted navigation, specifically, Figure 1 As shown, the following steps are included:

[0033] Step 1: Calibrate the parameters of multiple cameras that obtain original images around the vehicle.

[0034] Through the calibration process in advance, the camera is calibrated according to the vehicle's camera properties and installation location, the calibration parameters are adjusted, the image correction model and the overlap range model are fitted, and the image data processing model adapted to the camera on the current vehicle is obtained to reduce the time loss in the subsequent processing process. The calibration process includes the following steps:

[0035] ① Prepare 8 black and white grid reference templates of specified size and place them around the car;

[0036] ② Start all cameras to capture a common set of calibration test images;

[0037] ③Internal processing and analysis to extract the coordinates of the feature points of the template image and establish the corresponding relationship between the world coordinate system and the pixel coordinate system;

[0038] ④ Fit the parameter model to obtain the external displacement parameter matrix and internal scale parameter matrix of the camera coordinate transformation;

[0039] ⑤Fuse matrix parameters to obtain the optimized camera correction parameter model.

[0040] The internal scale parameter matrix is ​​the internal parameters of the camera, including the camera focal length, image scale coefficient, distortion coefficient, etc., among which the distortion coefficient includes the lateral distortion coefficient and the longitudinal distortion coefficient; the external displacement parameter matrix includes the translation vector and rotation vector of the camera.

[0041] Step 2: Use the calibrated camera to obtain the original images of the vehicle in multiple directions in real time.

[0042] Use the calibrated camera to obtain real-time images of the front blind spot that cannot be seen due to the height of the cockpit and the hood, the blind spot on both sides of the front of the vehicle due to the obstruction of the A-pillar, the blind spot near the sides of the vehicle body, and the blind spot at the rear of the vehicle.

[0043] Step 3: Perform image distortion correction on multiple original images around the vehicle to obtain multiple top-view images for stitching.

[0044] In order to obtain a wider angle coverage, a wide-angle camera is used to capture real-time images, but the original video stream has wide-angle distortion; at the same time, the camera is generally installed at an angle downward, which causes the original image output by the camera to be not a bird's-eye view. To achieve a 360° stitched bird's-eye view effect, the image output by the camera needs to be subjected to distortion correction processing to correct the video images obtained from different directions into a stitchable bird's-eye view.

[0045] The image distortion correction process of the present invention is as follows: Figure 2 As shown in the figure, the spatial point is essentially converted from the world coordinate system to the camera coordinate system, then projected to the plane coordinate system through proportional transformation, and finally converted to the pixel coordinate system. This process uses the calibration parameter fitting process to place the pixel image collected by the camera into the world coordinate system to find the corresponding position relationship, thereby correcting the top-down perspective of the image and eliminating the barrel distortion problem of the original image caused by the properties of the wide-angle camera itself, providing high-quality images for building a 360° panoramic stitching effect.

[0046] In the process of converting from the world coordinate system to the camera coordinate system, suppose a point Q in the world coordinate system W (x W ,y W ,z W ), which corresponds to Q in the camera coordinate system C (x C ,y C ,z C ), the conversion model is defined as:

[0047]

[0048] Among them, R is a rotation matrix of size 3×3, t is a translation vector, and both R and t are related to the acquisition angle of the camera installation position and have nothing to do with the properties of the camera itself. The relevant parameters are obtained through initialization calibration.

[0049] In the process of converting from the camera coordinate system to the image coordinate system, similarly, point Q C (x C ,y C ,z C ) is mapped from the camera coordinate system to the image coordinate system according to the proportional relationship, corresponding to point Q P (x P ,y P ), the conversion model is defined as:

[0050]

[0051] The proportional coefficient m used in the proportional relationship is determined by the internal parameters of the camera. According to the spatial similarity of the projection, the proportional coefficient m is specified as the focal length of the camera. Here, the coefficient m is set during the initialization calibration according to the corresponding focal length properties of the selected camera and the actual adjustment effect.

[0052] In the process of converting from the image coordinate system to the pixel coordinate system, point Q P (x P ,y P ) is converted from the image coordinate system to the pixel coordinate system, corresponding to point Q S (u S ,v S ), the conversion process is:

[0053]

[0054] Among them, dx and dy represent the corresponding ratio between pixels and physical sizes, and u0 and v0 represent the adjustment offsets. In summary, the correction process from the world coordinate system to the pixel image is expressed as:

[0055]

[0056] in, is the internal parameter of the camera, are the external parameters of the camera, which are parameter matrices obtained through early initialization calibration.

[0057] In order to solve the problem of differences in brightness and chromaticity of images acquired at different camera positions due to factors such as installation position, illumination, and color temperature, the present invention converts the image into RGB space, constructs a grayscale world method to balance the chromaticity differences of images acquired by different cameras, and sets the median grayscale K value to 128. Then, the gain coefficients of each component are expressed as:

[0058]

[0059] According to the Von Kries diagonal model, the adjusted pixel value is:

[0060]

[0061] Here, the upper limit of the pixel value is set to 255. The adjusted chromaticity components are affected by the unified mean measurement, which significantly improves the chromaticity and brightness differences of the image groups collected by different cameras at the same time, optimizes the post-stitching effect, and greatly reduces the traces of post-image stitching.

[0062] Step 4: Use the partitioned fade-in and fade-out weighted fusion method to stitch and fuse multiple top-view images. Specifically, divide the multiple top-view images into overlapping areas and non-overlapping areas, retain the image information of the non-overlapping areas, weightedly allocate the grayscale weight information of the overlapping areas, and fuse the overlapping areas according to the allocated grayscale values ​​to obtain a stitched panoramic image around the vehicle body.

[0063] This process mainly processes the overlapping area to solve the ghosting problem of the overlapping area of ​​adjacent camera images in the image stitching process. Through this process, the overlapping area template of the image obtained by two adjacent cameras at the same time is calibrated and extracted, and the partitioned fade-in and fade-out weighted fusion method is used to retain the image information of the non-overlapping area; the overlapping area adopts the combined spatial grayscale value and performs weighted average distribution according to a certain weight to obtain the new grayscale value of the fused area. Among them, the partition representation of the partitioned gradual in and out weighted fusion method is divided into overlapping areas and non-overlapping areas. The purpose is to use different methods to process the two partitions. The non-overlapping area retains the original data information, and the overlapping area combines the spatial grayscale value. The weighted distribution is performed by adjusting the weight information of the overlapping area of ​​the two images; gradual in and out means that according to the weight distribution map, a transition rather than an obvious fault is presented at the junction of the two areas. The effect is determined by the grayscale value fusion formula of the overlapping area. The closer to the center, the higher the weight, and vice versa; weighted means that the grayscale value of the pixel point data in the overlapping area is weighted and corrected according to the position information, referring to the grayscale value fusion formula; fusion means that the grayscale value of a certain point finally obtained is the weighted sum of the grayscale values ​​of the same pixel point of the two weighted images.

[0064] Suppose the two images to be fused are I1 and I2 respectively, and the fused result image is I. Then the gray value I(u,v) of the pixel point in the overlapping area with coordinates (u,v) in image I can be expressed as:

[0065] I(u,v)=α·I1(u,v)+β·I2(u,v).

[0066] Among them, α and β are the grayscale weights corresponding to the pixels at the same position in the fused image of the two images to be fused, and α+β=1. The grayscale weights are allocated according to the fade-in and fade-out function method. The weights are allocated according to the distance between the pixels and the center in the overlapping area. The closer to the center, the higher the weight, and vice versa. The grayscale weight allocation coefficient of the fusion area is constructed. The effect image after weight processing is stitched to weaken or eliminate ghosting and improve the quality of the stitching boundary.

[0067] The specific installation scheme of the camera used to obtain the blind spot image in the image acquisition module is as follows (eg Figure 3As shown): a front camera 101 is installed on the upper part of the front license plate, a camera 102 and a camera 103 are installed on the left and right side rearview mirrors, a camera 104 and a camera 105 are installed on both C-pillars, a camera 106 is installed in the front center of the roof, a camera 107 is installed in the center above the rear license plate, and a camera 108 is installed in the cockpit inside the vehicle. Different cameras obtain fields of view at specific angles. A front camera 101 is installed on the upper part of the front license plate to obtain the visual blind spot of the front of the vehicle that cannot be seen due to the height of the cockpit and the front hood, and provide auxiliary video data of the front vehicle for ADAS, so that the driver can view the distance of the front vehicle in real time and ensure the safe driving distance of the front of the vehicle; a camera 102 and 103 are installed on the left and right side rearview mirrors respectively. By adjusting the installation angle of the camera, a field of view is provided to cover the headlight area on both sides of the vehicle, eliminating the visual blind spot in front of the cockpit caused by the A-pillar on both sides of the front of the vehicle; a camera 104 and 105 are installed on both sides of the C-pillar to obtain the visual of the rearview mirrors on both sides of the vehicle body. Blind spots, eliminating low-level visual blind spots near both sides of the vehicle body and blind spots on both sides of the taillights; installing a camera 106 in the front center of the roof to expand the visual range in front of the cockpit, providing a wider driving field of view, and facilitating the driver to make advance judgments with reference to the road conditions ahead; installing a camera 107 in the center above the rear license plate to obtain the field of view of the rear of the vehicle, assist in the reversing perspective, monitor the safe distance to the rear of the vehicle according to the corrected conversion distance, and provide a reversing safety warning; installing a camera 108 in the cockpit of the vehicle to record the driver's driving status video in real time, which is used to assist the DMS driver fatigue monitoring function, and issue an alarm in abnormal driving conditions to remind the driver to focus on safe driving. The image stitching module of the present invention is an edge computing digital base, and a multi-interface edge computing digital base, which is based on Rockchip RK3588 embedded development, and the multiple interfaces include a variety of wired and wireless interfaces, including but not limited to HDMI output, aviation plug interface AHD input, each channel supports a maximum resolution of 1920×1080@30fps, high-speed UART, USB3.0, RS232, RS485, type-C interface; wireless interfaces include but are not limited to BLE, WIFI interface, which is convenient for adapting to a variety of interface application scenarios. The edge computing digital base is developed through the underlying driver adaptation and equipped with the Kuanghong OS platform. Kuanghong OS can be freely tailored through soft bus technology and unified HCP protocol and other technologies, and supports the embedding of different types of devices, thereby realizing data sharing of multiple types of devices, building interconnection and real-time perception of the surrounding environment, and improving the safety of drivers.

[0068] The display part of the present invention adopts a vehicle-mounted display flat panel, which is connected to the edge computing digital base signal. The vehicle-mounted display flat panel is used to display a panoramic image around the vehicle body, and is connected to the multi-interface edge computing digital base through a type-C interface to display the panoramic vehicle screen in real time for the cockpit; it can be controlled by touch screen and is suitable for various types of brand cars.

[0069] The present invention inputs the images collected by multiple cameras into the edge computing digital base through the AHD interface. The edge computing digital base is equipped with the RK1808AI computing card to expand the NPU to improve computing power. By splicing and deploying 360-degree panoramic videos on the platform, fast algorithm reasoning is achieved; it supports multi-type data protocol parsing and provides a multi-functional expansion reserved interface. The edge computing digital base processes the scene videos inside and outside the car in real time through the 360-degree panoramic video fusion algorithm, and outputs the scene videos inside and outside the car to the on-board display tablet in real time, providing the driver with a panoramic monitoring of the surrounding area of ​​the car body and improving driving concentration. By optimizing and splicing the original images collected by the on-board camera, a 360-degree panoramic image of the vehicle and an image of the driver's status in the cockpit are obtained, which largely eliminates the visual blind spots around the vehicle, making it convenient for the driver to monitor the 360-degree all-round driving environment around the vehicle body in real time, record the driving status of the driver in the car, and provide auxiliary ADAS, DMS, and BSD functions, thereby improving driving safety.

[0070] The above embodiments are only preferred specific implementation methods of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belong to the protection scope of the present invention.

Claims

1. A method for stitching panoramic videos for vehicle navigation, characterized in that: The steps include: Real-time acquisition of original images of the vehicle surrounding multiple positions of the vehicle body; the original images of the vehicle surrounding include images of the front visual blind spot, images of the visual blind spots on both sides of the front, images of the blind spots close to both sides of the vehicle body, and images of the rear blind spot; Performing image distortion correction on multiple original images around the vehicle to obtain multiple top-view images for stitching; A partitioned fade-in and fade-out weighted fusion method is used to stitch and fuse multiple bird's-eye view images. Specifically, the multiple bird's-eye view images are divided into overlapping areas and non-overlapping areas, the image information of the non-overlapping areas is retained, the grayscale weight information of the overlapping areas is weightedly allocated, and the overlapping areas are fused according to the allocated grayscale values ​​to obtain a stitched panoramic image around the vehicle body.

2. The vehicle navigation panoramic video stitching method according to claim 1, characterized in that: The visual blind spot image of the front of the vehicle is obtained by the front camera installed on the top of the front license plate; the visual blind spot images on both sides of the front of the vehicle are obtained by cameras installed on the left and right rearview mirrors, and the blind spot images close to both sides of the vehicle body are obtained by cameras installed on both sides of the C-pillars; the blind spot image of the rear of the vehicle is obtained by the camera installed in the center above the rear license plate.

3. The vehicle navigation panoramic video stitching method according to claim 2, characterized in that: Before obtaining the original images of the vehicle body in multiple directions in real time, the method further includes calibrating parameters of a camera for obtaining the original images of the vehicle body, including the following steps: Prepare multiple black and white grid reference templates of specified sizes and place them around the car; Start all cameras to shoot a set of calibration test images; Extract the coordinates of the feature points of the image on the reference template and establish the corresponding relationship between the world coordinate system and the pixel coordinate system; The external displacement parameter matrix and the internal scale parameter matrix of the camera coordinate transformation are obtained; the matrix parameters of the external displacement parameter matrix and the internal scale parameter matrix are merged to obtain the optimized camera correction parameter model; and the optimized camera correction parameter model is used to calibrate the camera parameters.

4. The vehicle navigation panoramic video stitching method according to claim 1, characterized in that: The image distortion correction of the multiple original images around the vehicle is specifically performed as follows: the spatial points of the original images around the vehicle are converted from the world coordinate system to the camera coordinate system, then projected to the plane coordinate system through proportional transformation, and finally converted into the pixel coordinate system to obtain the corrected top-down perspective image.

5. A vehicle-mounted navigation panoramic video stitching system, characterized in that: include: An image acquisition module is used to obtain original images of the vehicle in multiple directions in real time; the original images of the vehicle include images of the front visual blind spot, images of the front visual blind spots on both sides of the vehicle, images of the blind spots close to both sides of the vehicle body, and images of the rear blind spot; An image correction module is used to perform image distortion correction on multiple original images around the vehicle to obtain multiple top-view images for stitching; The image stitching module is used to stitch and fuse multiple overhead view images using a partitioned fade-in and fade-out weighted fusion method. Specifically, the multiple overhead view images are divided into overlapping areas and non-overlapping areas, the image information of the non-overlapping areas is retained, the grayscale weight information of the overlapping areas is weightedly allocated, and the overlapping areas are fused according to the allocated grayscale values ​​to obtain a stitched panoramic image around the vehicle body.

6. The vehicle-mounted navigation panoramic video stitching system according to claim 5, characterized in that: The image acquisition module is also used to obtain an image that expands the visible range in front of the cockpit and an image that records the driver's driving status in real time. The image that expands the visible range in front of the cockpit is obtained through a camera installed in the front middle of the roof, and the image that records the driver's driving status in real time is obtained through a camera installed in the cockpit inside the vehicle.

7. The vehicle-mounted navigation panoramic video stitching system according to claim 5, characterized in that: The image stitching module is an edge computing digital base mounted on the Kuanghong OS platform.

8. The vehicle-mounted navigation panoramic video stitching system according to claim 5, characterized in that: It also includes a vehicle-mounted display flat panel, which is connected to the edge computing digital base signal, and the vehicle-mounted display flat panel is used to display a panoramic image around the vehicle body.

Images