Image splicing method and device, mobile equipment, storage medium and product

Abstract

The invention discloses an image splicing method and device, mobile equipment, a storage medium and a product. The method comprises the following steps: acquiring pose information of a preset number of fisheye cameras in a world coordinate system and fisheye images shot by the preset number of fisheye cameras; converting a first pixel point on the target image into a world coordinate system according to preset target image view information and target image size information, and obtaining a first space point of the first pixel point in the world coordinate system; converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point; converting the second spatial point into the fisheye image by using the internal reference of the ideal camera and the corresponding relationship between the ideal image and the fisheye image to obtain a second pixel point; and according to a mapping relation between the first pixel points and the second pixel points, splicing the fisheye images to obtain a target image. According to the image splicing method provided by the invention, the calculation amount is saved, the splicing process is simplified, and the splicing effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image stitching method, apparatus, mobile device, storage medium and product. Background Art

[0002] The camera enables the vehicle to have environmental perception capabilities. By stitching images collected by multiple cameras, a panoramic view of the vehicle's surroundings is generated, which can assist users in observing the vehicle's blind spots during driving, parking and other scenarios to avoid safety accidents.

[0003] Currently, image stitching algorithms typically require dedistorting the image first, then performing a projection transformation, and then stitching the images together to create a panoramic image. This process is complex and computationally intensive. Furthermore, the size of the dedistorted image affects the field of view, which in turn affects the stitching quality of the entire overhead view. Summary of the Invention

[0004] The present application provides an image stitching method, apparatus, mobile device, storage medium and product, aiming to effectively solve technical problems such as complex process, large amount of calculation and poor stitching effect in the image stitching process.

[0005] According to a first aspect of the present application, the present application provides an image stitching method, the method comprising:

[0006] Obtaining pose information of a preset number of fisheye cameras in a world coordinate system and fisheye images captured by the preset number of fisheye cameras, wherein the pose information is obtained by jointly calibrating the fisheye cameras using original fisheye calibration images captured by the fisheye cameras;

[0007] Converting a first pixel point on the target image to a world coordinate system according to preset target image field of view information and target image size information to obtain a first spatial point of the first pixel point in the world coordinate system;

[0008] Converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point;

[0009] Converting the second spatial point into the fisheye image using an intrinsic parameter of an ideal camera and a correspondence between an ideal image and the fisheye image to obtain a second pixel point;

[0010] The fisheye images are spliced ​​according to a mapping relationship between the first pixel points and the second pixel points to obtain a target image.

[0011] Furthermore, the step of obtaining the position information of a preset number of fisheye cameras in the world coordinate system includes:

[0012] Perform corner detection on the original fisheye calibration image taken by each fisheye camera to obtain the image coordinates of the corner points;

[0013] The image coordinates of the corner points are used to calculate the pose information of each fisheye camera in the world coordinate system.

[0014] Furthermore, the step of converting the second spatial point into the fisheye image using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain the second pixel point includes:

[0015] The product of the intrinsic parameter of the ideal camera, the correspondence between the ideal image and the fisheye image, and the coordinate value of the second spatial point is calculated, and the calculation result is used as the coordinate value of the second pixel point.

[0016] Furthermore, before the step of converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point, the method further includes:

[0017] Determining a first area and a second area in the target image based on the size information in the mobile device information, wherein the first area is an independent area within the field of view of the fisheye cameras, and the second area is an area where the fields of view of the fisheye cameras intersect with each other, wherein the mobile device is configured with the preset number of fisheye cameras;

[0018] The step of converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point includes:

[0019] Converting the first spatial point in the second area into a camera coordinate system using the pose information of the first fisheye camera and the pose information of the second fisheye camera, respectively, to obtain a third spatial point and a fourth spatial point;

[0020] The step of converting the second spatial point into the fisheye image using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain the second pixel point includes:

[0021] Converting the third spatial point into the first fisheye image using the intrinsic parameters of the first ideal camera and a first correspondence relationship to obtain a third pixel point, wherein the first correspondence relationship refers to a correspondence relationship between the first ideal image and the first fisheye image;

[0022] Converting the fourth spatial point into the second fisheye image using the intrinsic parameters of the second ideal camera and a second correspondence relationship to obtain a fourth pixel point, wherein the second correspondence relationship refers to a correspondence relationship between the second ideal image and the second fisheye image;

[0023] According to the respective weights of the first fisheye image and the second fisheye image in the second area, the pixel value corresponding to the third pixel point and the pixel value corresponding to the fourth pixel point are weighted to obtain the pixel value of the first pixel point.

[0024] Furthermore, the step of performing weighted calculation on the pixel value corresponding to the third pixel and the pixel value corresponding to the fourth pixel based on the weighted proportions of the first fisheye image and the second fisheye image in the second area to obtain the pixel value of the first pixel includes:

[0025] When the angle of the first pixel point relative to the vertical direction in the second area is greater than or equal to the first angle threshold and less than the third angle threshold, the first weight of the third pixel point and the second weight of the fourth pixel point are calculated based on the angle of the first pixel point relative to the vertical direction in the second area, the first angle threshold and the third angle threshold; wherein the third angle threshold is the difference between the preset angle and the second angle threshold.

[0026] Furthermore, the step of performing weighted calculation on the pixel value corresponding to the third pixel and the pixel value corresponding to the fourth pixel based on the weighted proportions of the first fisheye image and the second fisheye image in the second area to obtain the pixel value of the first pixel further includes:

[0027] When the angle of the first pixel point relative to the vertical direction in the second area is less than the first angle threshold, using the pixel value corresponding to the third pixel point as the pixel value of the first pixel point;

[0028] When the angle of the first pixel point relative to the vertical direction in the second area is greater than the third angle threshold, the pixel value corresponding to the fourth pixel point is used as the pixel value of the first pixel point.

[0029] According to a second aspect of the present application, the present application further provides an image stitching device, the device comprising:

[0030] a pose and image acquisition module, configured to acquire pose information of a preset number of fisheye cameras in a world coordinate system and fisheye images captured by the preset number of fisheye cameras, wherein the pose information is obtained by jointly calibrating the fisheye cameras using original fisheye calibration images captured by the fisheye cameras;

[0031] A first coordinate conversion module is configured to convert a first pixel point on a target image into a world coordinate system according to preset target image field of view information and target image size information, and obtain a first spatial point of the first pixel point in the world coordinate system;

[0032] A second coordinate conversion module, configured to convert the first spatial point into a camera coordinate system according to the posture information to obtain a second spatial point;

[0033] a third coordinate conversion module, configured to convert the second spatial point into the fisheye image by using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image, to obtain a second pixel point;

[0034] An image stitching module is used to stitch the fisheye images according to a mapping relationship between the first pixel points and the second pixel points to obtain a target image.

[0035] According to a third aspect of the present application, the present application further provides a mobile device, comprising the above-mentioned image stitching device and a preset number of fisheye cameras.

[0036] Furthermore, the mobile device is a vehicle.

[0037] According to a fourth aspect of the present application, the present application further provides a storage medium, wherein a plurality of instructions are stored in the storage medium, and the instructions are suitable for being loaded by a processor to execute the steps of any of the methods described above.

[0038] According to a fifth aspect of the present application, the present application further provides a computer program product, which implements the steps of any of the above methods when the program code contained in the computer program product is executed by a processor in an electronic device.

[0039] By using one or more of the above-mentioned embodiments in the present application, at least the following technical effects can be achieved: the acquired posture information is directly obtained by jointly calibrating a preset number of fisheye cameras using the original fisheye calibration image, and correspondingly, the second spatial point in the camera coordinate system is directly mapped to the fisheye image, without dedistorting the fisheye image in the middle, thereby simplifying the entire stitching process, reducing the computational complexity of the dedistortion processing and the computational complexity of the process of mapping the second spatial point to the pixel point on the dedistorted image (ideal image) and then mapping the pixel point on the dedistorted image to the fisheye image. In addition, there is no process of setting the dedistorted image size, which will not affect the field of view, thereby improving the stitching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings will make the technical solutions and other beneficial effects of the present application apparent.

[0041] Figure 1 Shown is a flow chart of the image stitching method provided in an embodiment of the present application;

[0042] Figure 2Shown is a schematic diagram of the fusion area and the non-fusion area provided in the embodiment of the present application;

[0043] Figure 3 Shown is an enlarged schematic diagram of the fusion area A provided in an embodiment of the present application;

[0044] Figure 4 Shown is a schematic diagram of a fisheye camera in a vehicle provided by an embodiment of the present application;

[0045] Figure 5 The figure shows a schematic diagram of the original fisheye calibration image provided in an embodiment of the present application;

[0046] Figure 6 Shown is a schematic diagram of a target image provided in an embodiment of the present application;

[0047] Figure 7 FIG. 1 is a schematic structural diagram of an image stitching device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making any creative work are within the scope of protection of this application.

[0049] Exemplary embodiments will be described in detail herein, wherein the terminology used is for the purpose of describing specific embodiments only and is not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0051] To facilitate understanding, the terms appearing in the embodiments of this application are explained below.

[0052] The world coordinate system refers to the absolute coordinate system of the objective three-dimensional world, also known as the objective coordinate system. Its function is to describe the position of an object in the real world. It can be a coordinate system established with any real object as the origin.

[0053] The camera coordinate system (also called the camera coordinate system) uses the camera's optical center as its origin, with the X and Y axes parallel to the image's X and Y axes, respectively, and the Z axis coinciding with the optical axis. Its purpose is to describe the position of an object from the camera's perspective.

[0054] The pixel coordinate system of an image refers to a coordinate system with the vertices of the image as the coordinate origin, which uses pixels as units.

[0055] The existing image stitching algorithm first performs distortion correction processing on the image, calibrates the camera's extrinsic parameters based on the dedistorted image, and then uses the calibrated extrinsic parameters and the mapping relationship between the pixels in the panoramic image and the pixels in the dedistorted image to convert and stitch the dedistorted image to obtain a panoramic image. In the above image stitching process, dedistortion is required before subsequent conversion and stitching. The process is complex and computationally intensive. In addition, if the fisheye image is dedistorted before extrinsic calibration, the size of the dedistorted image needs to be pre-set, and then extrinsic calibration is performed based on the generated ideal image of the preset size. In this process, the preset dedistorted image will affect the field of view of the panorama, and the edges will be severely stretched, affecting the accuracy of corner detection and the stitching effect.

[0056] In view of the problems of existing image stitching algorithms such as complex process, large amount of calculation and poor stitching effect, this application provides an image stitching method, such as Figure 1 As shown, the image stitching method includes the following steps:

[0057] S101 , obtaining position information of a preset number of fisheye cameras in a world coordinate system and fisheye images captured by the preset number of fisheye cameras.

[0058] The posture information is obtained by jointly calibrating the fisheye camera using the original fisheye calibration image taken by the fisheye camera.

[0059] The preset number of fisheye cameras refers to at least two fisheye cameras, and the fisheye cameras are arranged adjacent to each other according to a preset distance or method. The preset distance or method refers to a setting method or distance that ensures that the fields of view of the two adjacent fisheye cameras intersect, that is, there is overlapping information between the images captured by the two adjacent fisheye cameras.

[0060] The pose information for each fisheye camera is calculated by jointly calibrating the fisheye cameras using the original fisheye calibration images captured by the fisheye cameras. This pose information is recalculated when the fisheye camera's pose changes due to factors such as turbulence and vibration. At other times, it is pre-stored as an external parameter for the fisheye camera. Acquiring the pose information for each fisheye camera in this step involves obtaining the pose information corresponding to each fisheye camera from the storage module.

[0061] In addition, the pose information refers to the pose of the fisheye camera in the world coordinate system. During the joint calibration of a preset number of fisheye cameras, corner detection is performed directly on the original fisheye calibration image, rather than dedistorting the original fisheye calibration image and then performing corner detection on the dedistorted calibration image.

[0062] S102 : Convert a first pixel point on the target image into a world coordinate system according to preset target image field of view information and target image size information, and obtain a first spatial point of the first pixel point in the world coordinate system.

[0063] The preset target image field of view information refers to the field of view range of the image finally stitched together, including the field of view width and field of view length of the image finally stitched together.

[0064] The preset target image size information refers to the size of the final stitched image, including the width and height of the image.

[0065] According to the ratio between the target image field of view and the target image size, the actual size represented by each pixel on the image is determined. Based on the actual size, the world coordinates of any pixel on the target image are determined in the world coordinate system. The pixel points on the target image are mapped to the world coordinate system to obtain the first spatial point.

[0066] S103: Convert the first spatial point into a camera coordinate system according to the posture information to obtain a second spatial point.

[0067] In this step, the first spatial point is converted into the respective camera coordinate system using the posture information of each fisheye camera to obtain the corresponding second spatial point.

[0068] S104 , converting the second spatial point into the fisheye image using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain a second pixel point.

[0069] Among them, since the images taken by the fisheye camera have a strong distortion effect, and the target image finally displayed needs to conform to the user's visual habits, an ideal camera and an ideal image are introduced in the stitching process. The ideal image obtained by the ideal camera (that is, the dedistorted image) will not produce a distortion effect, which corresponds to the fisheye image taken by the fisheye camera.

[0070] The correspondence between the ideal image and the fisheye image refers to a correspondence between the pixels on the fisheye image and the pixels on the ideal image.

[0071] In this step, the second spatial point can be directly converted to the fisheye image using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain the second pixel. This eliminates the need to first map the second spatial point to a pixel on the ideal image and then map the pixel on the ideal image to a pixel on the fisheye image. In other words, when mapping the second spatial point to the fisheye image, there is no need to transition from mapping to the ideal image.

[0072] It should be noted that there is a difference between the target image and the ideal image. The ideal image is the dedistorted image of the fisheye image taken by each fisheye camera, while the target image is the image obtained by stitching all the fisheye images.

[0073] S105 : splicing the fisheye images according to a mapping relationship between the first pixel points and the second pixel points to obtain a target image.

[0074] In this step, according to the mapping relationship between the first pixel point on the target image and the second pixel point on the fisheye image, each pixel point on each fisheye image is mapped to the target image, thereby achieving the splicing of a preset number of fisheye images.

[0075] The image stitching method provided in the embodiment of the present application obtains the posture information by directly using the original fisheye calibration image to jointly calibrate a preset number of fisheye cameras. Correspondingly, the second spatial point in the camera coordinate system is directly mapped to the fisheye image, and the fisheye image is not subjected to dedistortion processing in the middle, thereby simplifying the entire stitching process, reducing the computational amount of the dedistortion processing and the computational amount of the process of mapping the second spatial point to the pixel point on the dedistorted image (ideal image) and then mapping the pixel point on the dedistorted image to the fisheye image. Although the dedistortion process is omitted, the target image finally obtained by stitching is also an ideal image, which does not affect the stitching effect. In addition, there is no process of setting the dedistorted image size, which will not affect the field of view, thereby improving the stitching effect.

[0076] In some embodiments of the present application, S101 includes the following sub-steps:

[0077] S201 , performing corner point detection on the original fisheye calibration image captured by each fisheye camera to obtain image coordinates of the corner points.

[0078] S202 : Calculate and obtain the pose information of each fisheye camera in the world coordinate system using the image coordinates of the corner points.

[0079] More specifically, an arbitrary position in the real world is first selected as the origin of the world coordinate system, thereby determining the world coordinate system. Schematically, in the process of stitching the panoramas taken by the fisheye cameras on each vehicle, the world coordinate system is established with the center of the vehicle's rear axle as the origin.

[0080] After determining the world coordinate system, corner point detection is performed based on the fisheye camera's internal parameters and the original fisheye calibration image to obtain the image coordinates of the corner points. The original fisheye calibration image is an image formed when the fisheye camera captures the calibration plate at different positions, angles, and distances. The original fisheye calibration image can cover the entire field of view of the fisheye camera. Corner points are specific, easily identifiable points on the calibration plate, such as the intersection of squares on a checkerboard calibration plate. In this embodiment, the world coordinate information of the corner points is calculated using vehicle body data (including information such as wheelbase) and site information.

[0081] After obtaining the 2D image coordinates of the corner points, a camera calibration algorithm (such as the calibration function in the OpenCV library) is used to match the 2D image coordinates of the corner points with the actual coordinates of the corner points on the calibration plate. The rotation matrix, translation vector and other posture information of each fisheye camera are calculated based on the matching relationship.

[0082] To further improve the accuracy of the pose information, the pose information obtained above is optimized to obtain and store the final pose information. The specific optimization algorithm can be based on least squares optimization, nonlinear optimization algorithm, iterative closest point optimization, bundle adjustment optimization, etc., which are not limited to this.

[0083] The above steps are used to achieve joint calibration of a preset number of fisheye cameras and obtain the position and pose information of each fisheye camera.

[0084] In some other embodiments of the present application, the second pixel point in S104 is specifically calculated according to the following formula:

[0085] [u′,v′,1]=K1*K*[X c ,Y c ,Z c ];

[0086]

[0087] Where [u′,v′,1] is the coordinate of the second pixel in the fisheye image, [X c ,Y c ,Z c ] is the coordinate of the second spatial point, K1 is the correspondence between the ideal image and the fisheye image, where k1, k2, k3, k4, k5, k6 are radial distortion coefficients. K is the intrinsic parameter of the ideal camera, f x and f y is the focal length of the ideal camera, c x and c y The coordinates of the main point.

[0088] In some other embodiments of the present application, before the step of converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point, the method further includes:

[0089] A first area and a second area are determined in the target image based on the size information in the mobile device information, wherein the first area is an independent area within the field of view of the fisheye camera, and the second area is an area where the field of view of the fisheye camera intersects with each other, wherein the preset number of fisheye cameras is configured on the mobile device.

[0090] That is, based on the position information of the fisheye cameras installed on the mobile device, the intersecting areas and the non-intersecting areas (ie, the fused areas and the non-fused areas) on the fisheye images taken by the two adjacent fisheye cameras are determined.

[0091] Schematically, in the vehicle application scenario, assuming that there are 4 fisheye cameras, the 4 fisheye images need to be stitched together to obtain a bird's-eye view of the vehicle (i.e., the target image). On this basis, according to the conventional position of the fisheye cameras and the length and width information of the vehicle, the target image is divided into the first area and the second area, as shown in the following example: Figure 2 As shown, the first area (i.e., the non-fusion area) includes the non-fusion area B, the non-fusion area D, the non-fusion area E, and the non-fusion area G; the rest is the second area (i.e., the fusion area), including the fusion area A, the fusion area C, the fusion area F, and the fusion area H.

[0092] After the first area and the second area are determined, a mapping relationship between the first pixel point and the second pixel point is determined in different ways.

[0093] Since the first region is a non-fused region, each first pixel in the first region only has a corresponding second pixel in one fisheye image. Therefore, we only need to map each first pixel to the fisheye image corresponding to the first region. In other words, based on the correspondence between the position of the first region in the target image and the position of the fisheye camera, we use the pose information of the corresponding fisheye camera to convert the first spatial point corresponding to the first pixel in the first region into a second spatial point in the camera coordinate system. Then, we convert the second spatial point into the second pixel in the fisheye image.

[0094] Since the second area is the fusion area, the first pixel in the second area will have a corresponding second pixel in both fisheye images. In order to smoothly fuse the two fisheye images, it is necessary to perform weight calculation on the pixel values ​​of the two second pixels to determine the weight value of the first pixel. Specifically:

[0095] The first spatial point in the second area is converted into a camera coordinate system using the pose information of the first fisheye camera and the pose information of the second fisheye camera respectively, thereby obtaining a third spatial point and a fourth spatial point accordingly.

[0096] That is, first determine the two adjacent fisheye cameras corresponding to the second area (i.e., the first fisheye camera and the second fisheye camera). Then, using the pose information of the two fisheye cameras, convert the first spatial point to the respective camera coordinate systems, thereby obtaining the third spatial point corresponding to the first fisheye camera and the fourth spatial point corresponding to the second fisheye camera. It should be noted that the second spatial point, the third spatial point, and the fourth spatial point are all points mapped from the first spatial point to the camera coordinate system.

[0097] Furthermore, the third spatial point is converted into the first fisheye image using the intrinsic parameters of the first ideal camera and the first corresponding relationship to obtain a third pixel point, where the first corresponding relationship refers to the corresponding relationship between the first ideal image and the first fisheye image.

[0098] The fourth spatial point is converted into the second fisheye image using the intrinsic parameters of the second ideal camera and a second corresponding relationship to obtain a fourth pixel point, where the second corresponding relationship refers to a corresponding relationship between the second ideal image and the second fisheye image.

[0099] That is, based on the third spatial point and the fourth spatial point, the ideal images corresponding to the two fisheye cameras and the correspondence between the ideal images and the fisheye images are used to calculate the pixel points on each fisheye image, that is, the two second pixel points. In order to facilitate subsequent calculation and distinction, they are respectively marked as the third pixel point and the fourth pixel point.

[0100] It should be noted that the first ideal camera, the first corresponding relationship, and the third pixel point all correspond to the first fisheye camera, and the second ideal camera, the second corresponding relationship, and the fourth pixel point all correspond to the second fisheye camera.

[0101] According to the respective weights of the first fisheye image and the second fisheye image in the second area, the pixel value corresponding to the third pixel point and the pixel value corresponding to the fourth pixel point are weighted to obtain the pixel value of the first pixel point.

[0102] That is, after the third pixel point and the fourth pixel point are obtained by calculation, the pixel values ​​of these two pixel points are weighted to obtain the pixel value corresponding to the first pixel point on the target image.

[0103] Furthermore, the step of performing weighted calculation on the pixel value corresponding to the third pixel and the pixel value corresponding to the fourth pixel based on the weighted proportions of the first fisheye image and the second fisheye image in the second area to obtain the pixel value of the first pixel includes:

[0104] The angle θ of the first pixel point relative to the vertical direction in the second area is greater than or equal to a first angle threshold θ f and is less than a third angle threshold, a first weight of a third pixel and a second weight of a fourth pixel are calculated based on the angle of the first pixel relative to the vertical direction in the second area, the first angle threshold, and the third angle threshold; wherein the third angle threshold is the difference between the preset angle and the second angle threshold θ r The difference between

[0105] When the angle of the first pixel point relative to the vertical direction in the second area is smaller than the first angle threshold, the pixel value corresponding to the third pixel point is used as the pixel value of the first pixel point.

[0106] When the angle of the first pixel point relative to the vertical direction in the second area is greater than the third angle threshold, the pixel value corresponding to the fourth pixel point is used as the pixel value of the first pixel point.

[0107] Specifically, Figure 2 Taking the fusion area A as an example, any first pixel point p2 in the fusion area A can obtain the pixel point p′ corresponding to the first fisheye image. 2f and pixel p′ in the second fisheye image 2l .

[0108] According to the respective weights of the first fisheye image and the second fisheye image in the second area, the pixel value corresponding to the third pixel point and the pixel value corresponding to the fourth pixel point are weighted to obtain the pixel value of the first pixel point.

[0109] The first weight w1 of the third pixel, i.e., the weight of the first fisheye image in the second area, and the second weight w2=1-w1 of the fourth pixel, i.e., the weight of the second fisheye image in the second area, are calculated by the following formula:

[0110]

[0111] Where θ is the angle of the first pixel in the fusion area relative to the vertical direction, θ f and θ r are the first angle threshold and the second angle threshold respectively, and the third angle threshold is Schematically, θ o ,θ f and θ r The positional relationship between the fusion areas is as follows: Figure 3 As shown, the first angle threshold θ f and the second angle threshold θ r The size of θ determines o The size of the first angle threshold θ f and the second angle threshold θ r is a known quantity, set according to experience, Figure 3 Middle θ o The corresponding area is the actual fusion area, the fusion area A is the framed large-scale fusion area, and the first angle threshold θ f Corresponding to the first fisheye image, the second angle threshold θ r Corresponding to the second fisheye image.

[0112] After obtaining the first weight w1 and the second weight w2, the pixel value of the first pixel point on the target image is calculated by the following formula:

[0113] C=w1*C f +(1-w1)*C r ;

[0114] Where C is the pixel value of the first pixel point p2, C f is the pixel value of the third pixel, C r is the pixel value of the fourth pixel.

[0115] In some embodiments of the present application, taking panoramic image stitching in a vehicle as an example, S102 specifically includes the following steps:

[0116] First, determine the image size of the target image and the distance to the front, back, left, and right of the vehicle (i.e., the field of view), and establish a world coordinate system with the center of the vehicle's rear axle as the origin.

[0117] Then, the size of the vehicle, the field of view of the target image, and the image size of the target image are used to calculate the actual size represented by a single pixel point in the X and Y directions of the target image within the field of view.

[0118] Assuming that the width of the entire field of view of the target image is W and the length is H, and the image width and height of the target image are w and h respectively, the actual size represented by each pixel in the X direction and Y direction can be calculated as dx = W / w and dy = H / h. Based on the actual size represented by a single pixel point, the world coordinate P corresponding to the first spatial point of the pixel point p1 in the world coordinate system can be obtained. 1w (X w ,Y w ,Z w ), where Z w =0.

[0119] Specifically, the world coordinate system is set with the front of the car as the X-axis, the left door perpendicular to the outside as the Y-axis, the center of the rear axle as the origin, and the Z-axis perpendicular to the ground upward. Assuming that the pixel coordinates of any point in the target image are (u, v), the corresponding world coordinates are calculated based on the actual size represented by each pixel using the following formula to complete the conversion between pixel coordinates and world coordinates:

[0120] X w =-(v-v0)*dy+0.5*L;

[0121] Y w =-(u-u0)*dx;

[0122] Where L is the axis length of the vehicle, and (u0, v0) is the pixel coordinate corresponding to the center point of the target image.

[0123] In some embodiments of the present application, S103 specifically uses the pose information of each fisheye camera (including the rotation matrix R and the translation vector T) and the coordinates of the first spatial point to calculate the coordinates of the second spatial point:

[0124]

[0125] In the formula, (X c ,Y c ,Z c ) is the second spatial point P c1c The coordinates of (X w ,Y w ,Z w ) is the first spatial point P1w 's coordinates.

[0126] In another embodiment of the present application, the image stitching method includes the following steps:

[0127] S301: Jointly calibrate the four fisheye cameras on the vehicle using the original fisheye calibration image to obtain the corresponding pose information of each fisheye camera.

[0128] Among them, the positions of the four fisheye cameras on the vehicle are as follows Figure 4 As shown in the figure, they are named as front camera, rear camera, left camera and right camera according to their position characteristics in the vehicle. The original fisheye calibration images corresponding to each fisheye camera are shown in Figure 5 shown.

[0129] S302, determining the actual size represented by a single pixel on the target image according to the image size and field of view of the target image, mapping the first pixel on the target image to the world coordinate system according to the actual size represented by the single pixel to obtain a first spatial point.

[0130] S303 : Map the first spatial point to the respective camera coordinate systems using the respective pose information of the four fisheye cameras to obtain a second spatial point.

[0131] S304 : Map the second spatial point onto the fisheye image using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain a second pixel point.

[0132] Wherein, different second pixel point acquisition methods are adopted according to whether the first pixel point is in the fusion area or the non-fusion area.

[0133] For the non-fused area, the only second pixel point can be obtained by using the ideal camera intrinsic parameters related to the fisheye camera corresponding to the non-fused area and the correspondence between the ideal image and the fisheye image.

[0134] For the fused area, the third pixel in the first fisheye image and the fourth pixel in the second fisheye image are first calculated using the same method as for the unfused area. Then, using the weighted proportions of the first and second fisheye images in the unfused area, the pixel values ​​of the third and fourth pixels are weighted to obtain the final pixel value, achieving smooth fusion.

[0135] S305: According to the mapping relationship between the first pixel point on the target image and the second pixel point on the fisheye image, the four fisheye images are spliced ​​to obtain the following: Figure 6 Top view shown.

[0136] The above splicing process is Figure 4The target image after stitching is implemented in the main controller of Figure 4 The information is displayed on the display screen in the vehicle, which allows the driver to observe the information around the vehicle, avoid visual blind spots, and improve driving safety.

[0137] Based on any of the above embodiments, another embodiment of the present application further provides an image stitching device, Figure 7 This is a schematic diagram of the structure of the image stitching device provided in this application, such as Figure 7 As shown, the image stitching device includes:

[0138] The pose and image acquisition module 701 is used to obtain the pose information of a preset number of fisheye cameras in the world coordinate system and the fisheye images taken by the preset number of fisheye cameras, wherein the pose information is obtained by jointly calibrating the fisheye cameras using the original fisheye calibration images taken by the fisheye cameras.

[0139] The first coordinate conversion module 702 is used to convert the first pixel point on the target image into the world coordinate system according to preset target image field of view information and target image size information, and obtain the first spatial point of the first pixel point in the world coordinate system.

[0140] The second coordinate conversion module 703 is configured to convert the first spatial point into a camera coordinate system according to the posture information to obtain a second spatial point.

[0141] The third coordinate conversion module 704 is configured to convert the second spatial point into the fisheye image by using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain a second pixel point.

[0142] The image stitching module 705 is configured to stitch the fisheye images according to a mapping relationship between the first pixel points and the second pixel points to obtain a target image.

[0143] The image stitching device corresponds to the above-mentioned image stitching method, which will not be described in detail here.

[0144] Based on any of the above embodiments, another embodiment of the present application further provides a mobile device, which is equipped with the above-mentioned preset number of fisheye cameras and image stitching device. The image is acquired by an actual camera, and the image stitching device converts the image acquired by the actual camera into an image under virtual camera coordinates, so that the user can directly and clearly observe the image under a specific perspective. The mobile device can specifically be a product such as a vehicle, a robot, or a drone that is equipped with at least two fisheye cameras and requires image stitching.

[0145] Based on any of the above embodiments, another embodiment of the present application further provides an electronic device, which may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may call logic instructions in the memory to execute the above image stitching method.

[0146] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0147] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0148] On the other hand, an embodiment of the present application further provides a storage medium on which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to execute the image stitching method provided in the above embodiments.

[0149] On the other hand, an embodiment of the present application further provides a computer program product. When the program code contained in the computer program product is executed by a processor in an electronic device, the image stitching method provided in the above embodiments is implemented.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0152] In summary, although the present application has been disclosed as above with preferred embodiments, the above preferred embodiments are not intended to limit the present application. Ordinary technicians in this field can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims.

Claims

1. An image stitching method, characterized in that: include: Obtaining pose information of a preset number of fisheye cameras in a world coordinate system and fisheye images captured by the preset number of fisheye cameras, wherein the pose information is obtained by jointly calibrating the fisheye cameras using original fisheye calibration images captured by the fisheye cameras; Converting a first pixel point on the target image to a world coordinate system according to preset target image field of view information and target image size information to obtain a first spatial point of the first pixel point in the world coordinate system; Converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point; Converting the second spatial point into the fisheye image using an intrinsic parameter of an ideal camera and a correspondence between an ideal image and the fisheye image to obtain a second pixel point; The fisheye images are spliced ​​according to a mapping relationship between the first pixel points and the second pixel points to obtain a target image.

2. The image stitching method according to claim 1, wherein: The step of obtaining the position information of a preset number of fisheye cameras in a world coordinate system includes: Perform corner detection on the original fisheye calibration image taken by each fisheye camera to obtain the image coordinates of the corner points; The image coordinates of the corner points are used to calculate the pose information of each fisheye camera in the world coordinate system.

3. The image stitching method according to claim 1, wherein: The step of converting the second spatial point into the fisheye image using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain the second pixel point includes: The product of the intrinsic parameter of the ideal camera, the correspondence between the ideal image and the fisheye image, and the coordinate value of the second spatial point is calculated, and the calculation result is used as the coordinate value of the second pixel point.

4. The image stitching method according to claim 1, wherein: Before the step of converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point, the method further includes: Determining a first area and a second area in the target image based on the size information in the mobile device information, wherein the first area is an independent area within the field of view of the fisheye cameras, and the second area is an area where the fields of view of the fisheye cameras intersect with each other, wherein the mobile device is configured with the preset number of fisheye cameras; The step of converting the first spatial point into a camera coordinate system according to the pose information to obtain a second spatial point includes: Converting the first spatial point in the second area into a camera coordinate system using the pose information of the first fisheye camera and the pose information of the second fisheye camera, respectively, to obtain a third spatial point and a fourth spatial point; The step of converting the second spatial point into the fisheye image using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image to obtain the second pixel point includes: Converting the third spatial point into the first fisheye image using the intrinsic parameters of the first ideal camera and a first correspondence relationship to obtain a third pixel point, wherein the first correspondence relationship refers to a correspondence relationship between the first ideal image and the first fisheye image; Converting the fourth spatial point into the second fisheye image using the intrinsic parameters of the second ideal camera and a second correspondence relationship to obtain a fourth pixel point, wherein the second correspondence relationship refers to a correspondence relationship between the second ideal image and the second fisheye image; According to the respective weights of the first fisheye image and the second fisheye image in the second area, the pixel value corresponding to the third pixel point and the pixel value corresponding to the fourth pixel point are weighted to obtain the pixel value of the first pixel point.

5. The image stitching method according to claim 4, wherein: The step of performing weighted calculation on the pixel value corresponding to the third pixel and the pixel value corresponding to the fourth pixel according to the weighted proportions of the first fisheye image and the second fisheye image in the second area to obtain the pixel value of the first pixel includes: When the angle of the first pixel point relative to the vertical direction in the second area is greater than or equal to the first angle threshold and less than the third angle threshold, the first weight of the third pixel point and the second weight of the fourth pixel point are calculated based on the angle of the first pixel point relative to the vertical direction in the second area, the first angle threshold and the third angle threshold; wherein the third angle threshold is the difference between the preset angle and the second angle threshold.

6. The image stitching method according to claim 5, wherein: The step of performing weighted calculation on the pixel value corresponding to the third pixel and the pixel value corresponding to the fourth pixel based on the weighted proportions of the first fisheye image and the second fisheye image in the second area to obtain the pixel value of the first pixel further includes: When the angle of the first pixel point relative to the vertical direction in the second area is less than the first angle threshold, using the pixel value corresponding to the third pixel point as the pixel value of the first pixel point; When the angle of the first pixel point relative to the vertical direction in the second area is greater than the third angle threshold, the pixel value corresponding to the fourth pixel point is used as the pixel value of the first pixel point.

7. An image stitching device, characterized in that: The device comprises: a pose and image acquisition module, configured to acquire pose information of a preset number of fisheye cameras in a world coordinate system and fisheye images captured by the preset number of fisheye cameras, wherein the pose information is obtained by jointly calibrating the fisheye cameras using original fisheye calibration images captured by the fisheye cameras; A first coordinate conversion module is configured to convert a first pixel point on a target image into a world coordinate system according to preset target image field of view information and target image size information, and obtain a first spatial point of the first pixel point in the world coordinate system; A second coordinate conversion module, configured to convert the first spatial point into a camera coordinate system according to the posture information to obtain a second spatial point; a third coordinate conversion module, configured to convert the second spatial point into the fisheye image by using the intrinsic parameters of the ideal camera and the correspondence between the ideal image and the fisheye image, to obtain a second pixel point; An image stitching module is used to stitch the fisheye images according to a mapping relationship between the first pixel points and the second pixel points to obtain a target image.

8. A mobile device, characterized in that: The device comprises the image stitching device as claimed in claim 7 and a preset number of fisheye cameras.

9. The mobile device according to claim 8, wherein The mobile device is a vehicle.

10. A storage medium, characterized in that: The storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the steps of the method according to any one of claims 1 to 6.

11. A computer program product, characterized in that When the program code contained in the computer program product is executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 6 are implemented.

Images