A system for achieving panoramic surround vision

Connecting the camera group through an Ethernet switch, extracting and correcting distorted images, and combining virtual and real technology to generate non-distorted panoramic images, solving the distortion problem when the camera takes too close to objects in the prior art, improving the user's driving experience and image accuracy.

CN118018772BActive Publication Date: 2024-11-26JIANGSU SHENMOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410339918.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-25
Publication Date
2024-11-26
Estimated Expiration
2044-03-25

AI Technical Summary

Technical Problem

The existing panoramic surround view system has distortion problems when the camera takes objects too close to each other, which makes it impossible for users to determine the image, which increases the difficulty of driving and reduces the user experience.

Method used

The Ethernet switch is used to connect the camera group, and the distorted images are extracted through the data preprocessing module, transmitted to the data modeling module for correction and rendering, and combined with virtual and real technology to generate non-distorted panoramic images, and displayed through the touch display screen.

Benefits of technology

It effectively solves the problem of image distortion of users when driving too close to each other, improves the user's driving experience, and ensures image integrity and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118018772B_ABST
    Figure CN118018772B_ABST
Patent Text Reader

Abstract

The invention discloses a system and method for realizing panoramic surround view, belonging to the technical field of panoramic surround view systems. A vehicle-mounted panoramic surround view system comprises an Ethernet switch, and a camera group is connected via the Ethernet switch. The Ethernet switch is coupled with a data preprocessing module to stitch images to form a panoramic image. The data preprocessing module transmits the panoramic image to a coupled data recognition module to extract a part of the distorted image. The part of the distorted image is transmitted to a coupled data modeling module via the data recognition module. The part of the distorted image is corrected and rendered via the data modeling module. The distorted image after correction and rendering is transmitted to a virtual-reality combination module via the data modeling module to form a non-distorted panoramic image. The non-distorted panoramic image is transmitted to a touch display screen via the virtual-reality combination module for display.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a panoramic surround view system, and in particular to a system and method for realizing panoramic surround view, belonging to the technical field of panoramic surround view systems. Background Art

[0002] The panoramic view system in the prior art, such as a real-time video stitching method based on stitching lines disclosed in application number CN201711445383.3, includes reading the first frames of two surveillance videos as template frames, extracting feature points on the template frames and matching feature points, preliminarily screening matching point sets, and using the RANSAC (random sampling consensus) algorithm to screen out reliable matching point sets, thereby calculating the transformation matrix, using the transformation matrix to align the two frames of images, calculating the overlapping area, non-overlapping area and the final complete image size through the coordinates before and after the corner point transformation, and calculating the color intensity and geometric structure of the overlapping area. Stitching lines, stitching overlapping areas, stitching non-overlapping areas, calculating the complete image after stitching, and stitching subsequent frames of the video using the already obtained transformation matrix and stitching lines. This process is implemented using the CUDA platform for parallel acceleration. Although the panoramic function can be achieved, the problem of distortion caused by the camera shooting too close objects in the prior art has always been ignored and has not been solved. This causes the user to be unable to make a judgment when driving due to the distortion of the image that is too close, thereby increasing the difficulty of driving and reducing the user experience. For this reason, a system and method for achieving panoramic surround view are designed to solve the above problems. Summary of the invention

[0003] The main purpose of the present invention is to provide a system and method for achieving panoramic surround viewing.

[0004] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0005] A system for realizing panoramic surround view, applied to a vehicle-mounted panoramic surround view system, including a vehicle-mounted panoramic surround view system;

[0006] The vehicle-mounted panoramic surround view system includes an Ethernet switch, and the camera group is connected via the Ethernet switch;

[0007] The Ethernet switch is coupled to a data preprocessing module to stitch the images to form a panoramic image;

[0008] Features:

[0009] The data preprocessing module transmits the panoramic image to the coupled data recognition module to extract the partially distorted image;

[0010] transmitting the partially distorted image to a coupled data modeling module via a data recognition module;

[0011] Correct and render some distorted images through the data modeling module;

[0012] The distorted image after correction and rendering is transmitted to the virtual-real combination module through the data modeling module to form a non-distorted panoramic image;

[0013] The non-distorted panoramic image is transmitted to the touch screen through the virtual-reality combination module.

[0014] Preferably, the camera group includes cameras S1-S11;

[0015] Among them, cameras S1-S3 are set on both sides and ends of the front part of the vehicle;

[0016] Cameras S9-S11 are arranged on both sides and ends of the rear of the vehicle;

[0017] Cameras S4-S5 are arranged at the lower middle part and the upper middle part of one side of the vehicle;

[0018] Cameras S7-S8 are arranged at the lower middle part and the upper middle part of the other side of the vehicle;

[0019] A camera S6 is arranged in the middle of the top of the vehicle.

[0020] Preferably, measuring or obtaining the distortion coefficients of cameras S1-S11;

[0021] The distortion coefficient formula of camera group S1-S11 uses the radial distortion coefficient and tangential distortion coefficient formula as follows:

[0022] Radial distortion coefficient: k1×r 2 +k2×r 4 +k3×r 6 ;

[0023] Tangential distortion coefficient: p1×(2×x×y)+p2×(r 2 +2×x 2 );

[0024] Among them, r represents the distance from the pixel to the center of the image, x and y represent the horizontal and vertical coordinates of the pixel in the image coordinate system, k1, k2 and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0025] Preferably, the distortion coefficients are directly input into the data modeling module using the distortion coefficients provided in the instructions of cameras S1-S11;

[0026] Or use Zhang's calibration method to obtain the distortion coefficient.

[0027] Preferably, the data recognition module also includes cutting the videos obtained by shooting from different angles by other cameras and inputting them into the data modeling module to obtain a set of distorted image samples.

[0028] Preferably, when correcting the partially distorted image through the data modeling module, points with clear geometric features are detected based on image samples, and a rectangular coordinate system is established based on the points with clear geometric features;

[0029] A polynomial correction model is used for correction, including the following:

[0030] Using the radial distortion correction model:

[0031] x corr =x+k1×r 2 +k2×r 4 +k3×r 6 ;

[0032] y corr =y+k1×r 2 +k2×r 4 +k3×r 6 ;

[0033] Where x and y are the pixel coordinates in the original image, r is the distance from the pixel to the center of the image, and k1, k2, k3 are the radial distortion coefficients;

[0034] Tangential distortion correction model:

[0035] x corr =x+p1×r 2 +p2×r 4 +p3×r 6 ;

[0036] y corr =y+p1×r 2 +p2×r 4 +p3×r 6 ;

[0037] Among them, p1, p2, p3 are tangential distortion coefficients.

[0038] Preferably, when rendering the corrected virtual image, images of the part taken at different angles by other cameras are used for image comparison. After the comparison is made to be the image of the same object, the colors of corresponding positions in the compared images are identified. After identification, the colors of corresponding positions are extracted, and the extracted colors are rendered to the virtual image after the distorted part is corrected.

[0039] Preferably, when comparing images of the part taken at different angles by other cameras, the following method is specifically adopted:

[0040] First, obtain a virtual image of the virtual image after the distortion part is corrected at the same angle as other cameras;

[0041] The virtual image is assumed to be I1 and the contrast image is assumed to be I2, and the size is the same, both M×N;

[0042] For each pixel point (x, y), compare the pixel values ​​I1(x, y) and I2(x, y) of I1 and I2 at that position.

[0043] And use the absolute difference formula to calculate:

[0044] Absolute difference: D(x, y) = |I1(x, y) - I2(x, y)|;

[0045] Where D(x, y) is the absolute difference;

[0046] And further adopt the cumulative difference metric, for the entire image, the difference metric of each pixel is accumulated to obtain the total difference value:

[0047] Total difference:

[0048] According to the total difference value D total To judge the similarity between two images, set a threshold T. If D total If it is less than T, the two images are considered similar;

[0049] Otherwise, they are considered dissimilar.

[0050] Preferably, in Java, use the BufferedImage class to obtain the pixel color value of the image;

[0051] After obtaining the pixel color value, the collected pixel color value is rendered onto the virtual image after the distortion is corrected through the data modeling module.

[0052] Preferably, the virtual-reality combination module uses augmented reality AR technology to merge virtual elements with the real world, allowing users to see virtual images in a real environment, thereby constructing a complete and undistorted image, and transmitting the image to a touch screen for display.

[0053] Beneficial technical effects of the present invention:

[0054] The present invention provides a system and method for achieving panoramic surround vision. A camera group is connected via an Ethernet switch. The Ethernet switch is coupled to a data preprocessing module to stitch images to form a panoramic image. The data preprocessing module transmits the panoramic image to a coupled data recognition module to extract a part of the distorted image. The part of the distorted image is transmitted to a coupled data modeling module via the data recognition module. The part of the distorted image is corrected and rendered via the data modeling module. The distorted image after correction and rendering is transmitted to a virtual-reality combination module via the data modeling module to form a non-distorted panoramic image. The non-distorted panoramic image is transmitted to a touch screen via the virtual-reality combination module for display, thereby solving the problem in the prior art that when a user is driving, the image distortion is too close so that the user cannot make a judgment, thereby increasing the difficulty of driving and reducing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A system diagram of a preferred embodiment of a system and method for achieving panoramic surround viewing according to the present invention;

[0056] Figure 2 A schematic diagram of camera group distribution according to a preferred embodiment of a system and method for achieving panoramic surround viewing according to the present invention;

[0057] Figure 3 In accordance with a preferred embodiment of a system and method for implementing panoramic viewing of the present invention, in Java, the BufferedImage class is used to obtain a pixel color value map of an image.

[0058] Figure 4 The present invention is a flowchart of a preferred embodiment of a system and method for implementing panoramic surround viewing. DETAILED DESCRIPTION

[0059] In order to make the technical solution of the present invention more clear and specific to those skilled in the art, the present invention is further described in detail below in conjunction with embodiments and drawings, but the implementation manner of the present invention is not limited thereto.

[0060] like Figure 1 As shown, the vehicle-mounted panoramic surround view system uses an Ethernet switch to connect to multiple surround view cameras on the vehicle body, and the surround view cameras include Ethernet-based high-definition cameras, where the surround view cameras are installed at Figure 2 As shown in the camera group S1-S11, S1-S3 are set on both sides and ends of the front of the vehicle;

[0061] S9-S11 are set on both sides and ends of the rear part of the vehicle;

[0062] S4-S5 are set at the lower middle and upper middle parts of one side of the car;

[0063] S7-S8 are set at the lower middle and upper middle parts of the other side of the car;

[0064] Set S6 in the middle of the top of the car;

[0065] Through Figure 2 The video data used by the camera groups S1-S11 shown in FIG. 1 is transmitted to the Ethernet switch, and then transmitted to the data preprocessing module through the Ethernet switch;

[0066] Among them, the Ethernet switch and the data preprocessing module adopt the gigabit media independent interface RGMII. The data preprocessing module is connected to the Ethernet switch through RGMII, and receives the video data collected by each surround-view camera through the Ethernet switch. The video data collected by the camera group S1-S11 is received through the data preprocessing module. Multiple shooting angle ranges and a field of view range of several camera lenses are received. According to these shooting angle ranges and field of view ranges, a stitching number of several original images is calculated. According to the stitching number, adjacent parts of these original images are stitched respectively to obtain several local panoramic images.

[0067] The panoramic real-time monitoring video processed by the data preprocessing module is transmitted to the data recognition module, and the panoramic real-time monitoring video is recognized and processed. Specifically, the distorted video part in the image is recognized, and the distorted video part obtained by recognition is cut and collected, and transmitted to the data modeling module, and the distorted video part is restored and modeled. The modeling is based on the following method:

[0068] When installing the camera group S1-S11, first measure or obtain the distortion coefficient of the camera group S1-S11. The distortion coefficient formula of the camera group S1-S11 can be described using the radial distortion coefficient and the tangential distortion coefficient. The formula is as follows:

[0069] Radial distortion coefficient: k1×r 2 +k2×r 4 +k3×r 6 ;

[0070] Tangential distortion coefficient: p1×(2×x×y)+p2×(r 2 +2×x 2 );

[0071] Among them, r represents the distance from the pixel to the center of the image, x and y represent the horizontal and vertical coordinates of the pixel in the image coordinate system, k1, k2 and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0072] Input the calculated distortion coefficient into the data modeling module in advance, or the distortion coefficient provided in the camera manual can be directly input into the data modeling module, or the distortion coefficient can be obtained by using Zhang's calibration method. Therefore, the data modeling module knows the distortion coefficient for each camera and establishes the corresponding distortion model, which describes the distortion law of the pixels in the image.

[0073] Among them, establishing the corresponding distortion model also requires the collection of calibration samples, that is, the distorted video part, which is cut by the remaining cameras from different angles and input into the data modeling module to obtain a set of distorted image samples. Based on the image samples, points with clear geometric features are detected, and a rectangular coordinate system is established based on the points with clear geometric features.

[0074] A polynomial correction model is used for correction, including the following:

[0075] Using radial distortion correction model: Radial distortion is caused by the optical characteristics of the lens and can usually be described using a quadratic or higher order polynomial. The radial distortion correction model is expressed as:

[0076] x corr =x+k1×r 2 +k2×r 4 +k3×r 6 ;

[0077] y corr =y+k1×r 2 +k2×r 4 +k3×r 6 ;

[0078] Where x and y are the pixel coordinates in the original image, r is the distance from the pixel to the center of the image, and k1, k2, k3 are the radial distortion coefficients;

[0079] Tangential distortion correction model: Tangential distortion is caused by camera installation deviation or lens manufacturing error. The tangential distortion correction model is expressed as:

[0080] x corr =x+p1×r 2 +p2×r 4 +p3×r 6 ;

[0081] y corr =y+p1×r 2 +p2×r 4 +p3×r 6 ;

[0082] Among them, p1, p2, p3 are tangential distortion coefficients.

[0083] After the calculation is completed, for each pixel point that needs to be corrected, the corrected coordinates are calculated using the above correction model according to its coordinates and distortion coefficients in the original image. According to the corrected coordinates, the video of the distorted part is modeled to form a virtual image. Then, the virtual image of the distorted part is corrected and compared with the images of the part taken at different angles by other cameras. After the comparison is the same object image, the color of the corresponding position in the compared image is identified. After identification, the color of the corresponding position is extracted, and the extracted color is rendered into the virtual image of the distorted part to achieve a real image effect.

[0084] When comparing images of the part taken at different angles by other cameras, the following method is used:

[0085] First, obtain a virtual image of the virtual image after the distortion part is corrected at the same angle as other cameras;

[0086] The virtual image is assumed to be I1 and the comparison image is assumed to be I2, and they have the same size of M×N.

[0087] For each pixel point (x, y), compare the pixel values ​​I1(x, y) and I2(x, y) of I1 and I2 at that position.

[0088] And use the absolute difference formula to calculate:

[0089] Absolute difference: D(x, y) = |I1(x, y) - I2(x, y)|;

[0090] Where D(x, y) is the absolute difference;

[0091] And further adopt the cumulative difference metric, for the entire image, the difference metric of each pixel is accumulated to obtain the total difference value:

[0092] Total difference:

[0093] According to the total difference value D total To judge the similarity between two images, set a threshold T. If D total If it is less than T, the two images are considered similar;

[0094] Otherwise, they are considered dissimilar.

[0095] After determining that the images are similar, the colors of the corresponding positions of the comparison images after the determination is completed are identified;

[0096] Specific as Figure 3 As shown, in Java, use the BufferedImage class to get the pixel color value of the image;

[0097] After obtaining the pixel color value, the collected pixel color value is rendered to the virtual image after the distortion is corrected through the data modeling module, so that the virtual image after the distortion is corrected is closer to the real image taken by the camera;

[0098] Finally, the distorted part of the virtual image is corrected and rendered, and combined with the rest of the image to form a panoramic surround view system.

[0099] Specifically, augmented reality (AR) technology is used: with the help of AR technology, virtual elements are integrated with the real world, allowing users to see virtual images in the real environment, thereby constructing a complete and undistorted image, and transmitting the image to the touch screen for display.

[0100] In addition, the vehicle central control system host in this patent can obtain data on vehicle conditions in real time and feed the data back to the touch screen.

[0101] The above description is only a further embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the scope disclosed by the present invention, which belong to the protection scope of the present invention.

Claims

1. A system for realizing panoramic surround view, applied to a vehicle-mounted panoramic surround view system, including a vehicle-mounted panoramic surround view system; The vehicle-mounted panoramic surround view system includes an Ethernet switch, and the camera group is connected via the Ethernet switch; The Ethernet switch is coupled to a data preprocessing module to stitch the images to form a panoramic image; Features: The data preprocessing module transmits the panoramic image to the coupled data recognition module to extract the partially distorted image; transmitting the partially distorted image to a coupled data modeling module via a data recognition module; Correct and render some distorted images through the data modeling module; The distorted image after correction and rendering is transmitted to the virtual-real combination module through the data modeling module to form a non-distorted panoramic image; The non-distorted panoramic image is transmitted to the touch screen through the virtual-real combination module for display; The camera group includes cameras S1-S11; Among them, cameras S1-S3 are set on both sides and ends of the front part of the vehicle; Cameras S9-S11 are arranged on both sides and ends of the rear of the vehicle; Cameras S4-S5 are arranged at the lower middle part and the upper middle part of one side of the vehicle; Cameras S7-S8 are arranged at the lower middle part and the upper middle part of the other side of the vehicle; A camera S6 is arranged in the middle of the top of the vehicle; Measure or obtain the distortion coefficients of cameras S1-S11; The distortion coefficient formula of camera group S1-S11 uses the radial distortion coefficient and tangential distortion coefficient formula as follows: Radial distortion coefficient: k1×r 2 +k2×r 4 +k3×r 6 ; Tangential distortion coefficient: p1×(2×x×y)+p2×(r 2 +2×x 2 ); Where r represents the distance from the pixel to the center of the image, x and y represent the horizontal and vertical coordinates of the pixel in the image coordinate system, k1, k2 and k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients; The distortion coefficients are directly input into the data modeling module using the distortion coefficients provided in the instructions of cameras S1-S11; Or use Zhang’s calibration method to obtain the distortion coefficient; The data recognition module also includes cutting the videos shot from different angles by other cameras and inputting them into the data modeling module to obtain a set of distorted image samples; When correcting a part of the distorted image through the data modeling module, points with clear geometric features are detected based on image samples, and a rectangular coordinate system is established based on the points with clear geometric features; A polynomial correction model is used for correction, including the following: Using the radial distortion correction model: x corr =x+k1×r 2 +k2×r 4 +k3×r 6 ; and corr =y+k1×r 2 +k2×r 4 +k3×r 6 ; Where x and y are the pixel coordinates in the original image, r is the distance from the pixel to the center of the image, and k1, k2, k3 are the radial distortion coefficients; Tangential distortion correction model: x corr =x+p1×r 2 +p2×r 4 +p3×r 6 ; and corr =y+p1×r 2 +p2×r 4 +p3×r 6 ; Among them, p1, p2, p3 are tangential distortion coefficients; When rendering the corrected virtual image, images of the part taken at different angles by other cameras are used for image comparison. After the images are compared to be the same object, the colors of the corresponding positions in the compared images are identified. After identification, the colors of the corresponding positions are extracted, and the extracted colors are rendered to the virtual image after the distortion part is corrected. When comparing images of the part taken at different angles by other cameras, the following method is used: First, obtain a virtual image of the virtual image after the distortion part is corrected at the same angle as other cameras; The virtual image is assumed to be I1 and the contrast image is assumed to be I2, and the size is the same, both M×N; For each pixel point (x, y), compare the pixel values ​​I1(x, y) and I2(x, y) of I1 and I2 at that position; And use the absolute difference formula to calculate: Absolute difference: D(x, y) = |I1(x, y) - I2(x, y)|; Where D(x, y) is the absolute difference; And further adopt the cumulative difference metric, for the entire image, the difference metric of each pixel is accumulated to obtain the total difference value: Total difference: According to the total difference value D total To judge the similarity between two images, set a threshold T. If D total If it is less than T, the two images are considered similar; Otherwise, they are considered dissimilar; In Java, use the BufferedImage class to get the pixel color value of the image; After obtaining the pixel color value, the collected pixel color value is rendered onto the virtual image after the distortion part is corrected through the data modeling module; The virtual-reality combination module uses strong reality AR technology to merge virtual elements with the real world, allowing users to see virtual images in the real environment, thereby constructing a complete and undistorted image, and transmitting the image to the touch screen for display.