Panoramic image processing method, device, related equipment and vehicle

Abstract

The present application relates to a panoramic image processing method, device, related equipment and vehicle, which constructs a sparse disparity map by taking overlapping areas of images taken by multiple shooting devices on a vehicle; performs segmentation based on the sparse disparity map to obtain a target layer image, which is used to indicate the image of the area where the target object is located in the overlapping area; performs ground plane projection on the target layer image to obtain a target projection image; and generates a panoramic image based on the target projection image. By constructing a sparse view for the overlapping areas of images taken by multiple shooting devices, and performing a series of operations based on the sparse disparity map to obtain a panoramic image, the panoramic image obtained by stitching does not have problems such as ghosting and artifacts, and the stitching effect is good, which improves the user experience.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a panoramic image processing method, device, related equipment and vehicle. Background Art

[0002] With the advent of the information age, cars are becoming more and more intelligent, and the panoramic image function of the vehicle is one of the most practical functions. This function can help users view the images around the vehicle through the vehicle display screen during parking, making parking more intuitive and convenient.

[0003] However, panoramic images are often stitched together by combining images captured by multiple cameras installed in the car. The existing panoramic image stitching solutions will produce double images and artifacts in the overlapping areas of the cameras, resulting in poor panoramic image stitching and affecting user experience. Summary of the invention

[0004] The embodiments of the present application provide a panoramic image processing method, device, electronic device, storage medium, computer program product and vehicle, which construct a sparse view of the overlapping areas of images captured by multiple shooting devices, and obtain a panoramic image by performing a series of operations based on the sparse disparity map, so that the panoramic image obtained by stitching does not have problems such as ghosting and artifacts, has a good stitching effect, and improves the user experience.

[0005] The present application provides a panoramic image processing method, including:

[0006] Constructing a sparse disparity map for overlapping areas of images captured by multiple cameras;

[0007] Perform segmentation based on the sparse disparity map to obtain a target layer image, wherein the target layer image is used to indicate an image of an area where a target object in the overlapping area is located;

[0008] Performing ground plane projection on the target layer image to obtain a target projection image;

[0009] A panoramic image is generated according to the target projection image.

[0010] Accordingly, an embodiment of the present application provides a panoramic image processing device, including:

[0011] An image construction unit, configured to construct a sparse disparity map for an overlapping area of ​​images captured by multiple shooting devices;

[0012] An image segmentation unit, configured to perform segmentation based on the sparse disparity map to obtain a target layer image, wherein the target layer image is used to indicate an image of an area where a target object in the overlapping area is located;

[0013] A projection unit, used for performing ground plane projection on the target layer image to obtain a target projection image;

[0014] The image generating unit is used to generate a panoramic image according to the target projection image.

[0015] In addition, an embodiment of the present application also provides an electronic device, including one or more processors and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the panoramic image processing method provided in the embodiment of the present application.

[0016] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium stores a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute any panoramic image processing method provided in the embodiment of the present application.

[0017] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements any panoramic image processing method provided in the embodiment of the present application.

[0018] In addition, an embodiment of the present application also provides a vehicle, including the above-mentioned electronic device, or the above-mentioned storage medium, or the above-mentioned computer program product.

[0019] In the embodiment of the present application, a sparse disparity map is constructed for the overlapping area of ​​images captured by multiple shooting devices; segmentation is performed based on the sparse disparity map to obtain a target layer image, which is used to indicate the image of the area where the target object is located in the overlapping area; the target layer image is projected on the ground plane to obtain a target projection image; and a panoramic image is generated based on the target projection image. A sparse view is constructed for the overlapping area of ​​images captured by multiple shooting devices, and a series of operations are performed based on the sparse disparity map to obtain a panoramic image, so that the panoramic image obtained by stitching does not have problems such as ghosting and artifacts, and the stitching effect is good, which improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a panoramic image processing method provided by an embodiment of the present application;

[0022] Figure 2This is a schematic diagram of a spatial layout of a camera device on a vehicle provided by an embodiment of the present application;

[0023] Figure 3 is another schematic diagram of the spatial layout of a camera device on a vehicle provided by an embodiment of the present application;

[0024] Figure 4 is a calibration schematic diagram of a photographing device provided by an embodiment of the present application;

[0025] Figure 5 This is a schematic diagram of four-corner grid division of a target layer image provided by an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of a triangular mesh plane provided by an embodiment of the present application;

[0027] Figure 7 is a flow chart of a panoramic image processing method provided by an embodiment of the present application;

[0028] Figure 8 is a structural schematic diagram of a panoramic image processing device provided by an embodiment of the present application;

[0029] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0031] The embodiments of the present application provide a panoramic image processing method, device, electronic device, storage medium, computer program product and vehicle. The panoramic image processing device can be integrated in an electronic device, which can be a server or a terminal.

[0032] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0033] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0034] In addition, the term "plurality" in the embodiments of the present application refers to two or more than two. The terms "first" and "second" in the embodiments of the present application are used to distinguish descriptions and should not be understood as implying relative importance.

[0035] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0036] See also Figure 1 , Figure 1 : is a flowchart of a panoramic image processing method provided by an embodiment of the present application. The panoramic image processing method may include the following steps S101 to S104:

[0037] S101 . Construct a sparse disparity map for an overlapping area of ​​images captured by multiple camera devices.

[0038] The photographing device refers to a photographing device installed on a vehicle for acquiring a panoramic image of the surroundings of the vehicle.

[0039] Since the camera devices are installed at different positions on the vehicle, there may be an overlapping area between the two original images taken by two adjacent camera devices. The original image refers to the image taken by the camera device. There are many ways to obtain the original image. For example, the original images taken by multiple camera devices on the vehicle can be obtained in real time, or the original images taken by multiple camera devices on the vehicle can be obtained when the vehicle panoramic imaging function is triggered. The vehicle panoramic imaging function can be triggered when the user turns on the reversing assist, blind spot monitoring, lane keeping and other functions.

[0040] There are many ways to determine the overlapping area. For example, feature point algorithms such as scale-invariant feature transform (SIFT) and speeded up robust features (SURF) can be used to extract feature points from the original images of two adjacent shooting devices; the extracted feature points can be matched using nearest neighbor matching or other more powerful feature matching algorithms; and the basic matrix or homography matrix of the two adjacent shooting devices is calculated based on the matched feature points. The overlapping area of ​​the original images taken by the two adjacent shooting devices is inferred using the basic matrix or homography matrix. It is also possible to use one shooting device as the main shooting device and another shooting device to laser scan or project a texture pattern (such as a grid matrix). Under the main shooting device, the overlapping area of ​​the shooting device is determined by analyzing the laser point cloud or projected pattern received from another shooting device using a point cloud registration algorithm or a pattern matching algorithm. The overlapping area of ​​the two adjacent shooting devices of the vehicle is obtained by combining the feature points detected in the main shooting device and the information provided in the laser point cloud or the projected pattern. It can be adjusted according to the actual situation, and the embodiment of the present application is not limited.

[0041] There are many types of shooting devices, for example, the shooting device can be a common camera or a camera with a wide-angle lens.

[0042] The number of the camera on the vehicle can be various, for example, the number of the camera on the vehicle can be 4, or the number of the camera on the vehicle can be 6.

[0043] There are many possible locations for installing the camera on a vehicle. Figure 2 As shown, Figure 2 : is a schematic diagram of a spatial layout of camera devices on a vehicle, wherein the number of camera devices is 4, and the camera devices are installed in the front, rear, left, and right directions of the vehicle. Figure 3 As shown, Figure 3 This is another schematic diagram of the spatial layout of camera devices on a vehicle, wherein the number of camera devices is 6 and the camera devices are installed in six directions of the vehicle, namely, front, rear, left front, left rear, right front, and left rear.

[0044] It is understandable that the number, type and installation position of the camera device on the vehicle can be adjusted according to the actual situation, and the embodiment of the present application does not limit it. Figure 2 The 4 cameras shown are explained as follows: Figure 2As shown, the shooting device of the embodiment of the present application includes a panoramic left-view camera, a panoramic front-view camera, a panoramic right-view camera, and a panoramic rear-view camera, but this is not intended to limit the embodiment of the present application.

[0045] It should be noted that before performing panoramic image processing, each camera device needs to be calibrated and the parameters of each camera device (such as the intrinsic matrix and the translation matrix) need to be determined. The calibration process of the camera device can be adjusted according to the actual situation, and the embodiment of the present application does not limit it.

[0046] For example, Figure 4 As shown, Figure 4 1 is a schematic diagram of calibrating a shooting device provided in an embodiment of the present application, wherein the shooting device is a panoramic camera (including a panoramic left-view camera, a panoramic front-view camera, a panoramic right-view camera, and a panoramic rear-view camera). Figure 4 As shown, a calibration plate can be laid on the ground plane near the panoramic camera, a mapping equation between the two planes can be established, and the parameters of the panoramic camera can be determined according to the mapping equation. The two planes refer to the ground plane on which the pre-designed calibration pattern is laid, and the plane where the image captured by the panoramic camera is located (e.g., the panoramic left view captured by the panoramic left view camera, and the panoramic front view captured by the panoramic front view camera).

[0047] Specifically, the key parameters of the panoramic camera are calibrated first. For example, the internal parameter matrix K of the panoramic camera is calibrated. The internal parameter matrix K of the panoramic camera can be determined by using the Zhang Zhengyou calibration model and the Levenberg-Marquardt (LM) algorithm to determine the internal parameter matrix of the corresponding panoramic camera (e.g., the internal parameter matrix K of the panoramic left view camera). l , the intrinsic parameter matrix K of the panoramic right view camera r ).

[0048] Furthermore, the image captured by the panoramic camera is obtained, and a conversion equation between the two-dimensional coordinates and the three-dimensional world coordinates of the pixel points in the image captured by the panoramic camera is established. The two-dimensional coordinates are used to represent the coordinate information of the pixel point in the pixel coordinate system, and the three-dimensional world coordinates are used to represent the coordinate information of the pixel point in the image coordinate system.

[0049] Specifically, the conversion equation between the two-dimensional coordinates and the three-dimensional world coordinates of any pixel point in the image captured by the panoramic camera is established according to the following formula (1):

[0050] (1).

[0051] Among them, f x is the horizontal focal length of the panoramic camera, f yis the vertical focal length of the panoramic camera, R is the rotation matrix of the panoramic camera, T is the translation matrix of the panoramic camera, and K is the internal parameter matrix of the panoramic camera. (u, v) is the two-dimensional coordinate of the pixel point, (x w ,y w , z w ) is the 3D world coordinate of the pixel, and (u0, v0) represents the 2D coordinate of the origin of the image coordinate system in the pixel coordinate system. Since the origin of the pixel coordinate system does not coincide with the origin of the image coordinate system, it can be assumed that the coordinates of the origin of the image coordinate system in the pixel coordinate system are (u0, v0).

[0052] It should be noted that the two-dimensional coordinates (u, v) of the pixel point and the three-dimensional world coordinates (x w ,y w , z w ) are a pair of matching points. That is, if the two-dimensional coordinates of a pixel point are known, the three-dimensional world coordinates of the pixel point can be solved. In formula (1), the rotation matrix R of the panoramic camera and the translation matrix T of the panoramic camera are unknown.

[0053] After calibrating the key parameters of the panoramic camera and establishing the conversion equations between the two-dimensional coordinates and three-dimensional world coordinates of the pixel points in the image captured by the panoramic camera, a mapping equation is established between the ground plane and the plane where the image captured by the panoramic camera is located.

[0054] Specifically, the mapping equation between the two-dimensional coordinates (u, v) of any pixel point in the plane where the image captured by the panoramic camera is located and the ground plane is established according to the following formula (2):

[0055] (2).

[0056] Among them, H is the homography matrix of the panoramic camera, h 11 to h 33 are the 9 unknowns of the homography matrix. w ,y w , 0) is the three-dimensional world coordinate used to represent the ground plane.

[0057] It is understandable that the homography matrix H of the panoramic camera is unknown, and there are many ways to solve the homography matrix H of the panoramic camera. For example, the homography matrix H of the panoramic camera can be solved by a direct linear transform (DLT) algorithm, or by a least squares method.

[0058] For example, the homography matrix H of the panoramic camera is solved according to the following formula (3):

[0059] (3).

[0060] Wherein, formula (3) is the DLT algorithm. In formula (3), by dividing both sides of the equation in formula (2) by an unknown quantity h 33 , the number of unknowns in the homography matrix can be reduced. Reducing the 9 unknowns to 8 unknowns facilitates the subsequent acquisition of at least 4 pixels with known coordinate information in the image captured by the panoramic camera, determining the 2D coordinates and 3D world coordinates of the 4 pixels, and solving the equation based on the 2D coordinates and 3D world coordinates of the 4 pixels and formula (3) to calculate the homography matrix H of the panoramic camera.

[0061] Furthermore, after determining the homography matrix H of the panoramic camera, the rotation matrix R and the translation matrix T of the panoramic camera can be solved. There are many ways to solve the rotation matrix R and the translation matrix T of the panoramic camera. For example, the singular value decomposition method and formula (1) can be used to solve the rotation matrix R and the translation matrix T of the panoramic camera. The least squares method and formula (1) can also be used to solve the rotation matrix R and the translation matrix T of the panoramic camera.

[0062] The sparse disparity map refers to a map formed by collecting disparity values ​​corresponding to feature points in the overlapping area of ​​images captured by multiple capturing devices.

[0063] The process of constructing a sparse disparity map includes: performing feature point matching on the overlapping area to obtain at least one feature point pair; and constructing a sparse disparity map according to the feature point pair.

[0064] Among them, the feature point pair refers to the feature points that match the two original images in the overlapping area. There are many ways to determine the feature point pair, which can be adjusted according to the actual situation, and the embodiments of the present application are not limited. For example, the Oriented FAST feature points and BRIEF descriptors can be found from the two original images, and then the feature point matching algorithm (such as Fast Library for Approximate Nearest Neighbors, FLANN) can be used to perform feature matching in the two original images with the Oriented FAST feature points and BRIEF descriptors to obtain feature point pairs. For another example, the original images can be aligned, and the Oriented FAST feature points and BRIEF descriptors can be found from the two aligned images, and then the feature point matching algorithm (such as Fast Library for Approximate Nearest Neighbors, FLANN) can be used to perform feature matching in the two aligned images with the Oriented FAST feature points and BRIEF descriptors to obtain feature point pairs. Among them, the alignment process includes row alignment and / or column alignment. Since the installation of the on-board panoramic camera simulates the horizontal arrangement of human eyes, the horizontal parallax is larger and the vertical parallax is smaller, so the row alignment method has higher accuracy in subsequent algorithm processing. The present application preferably adopts row alignment.

[0065] In some embodiments, based on the feature point pairs, the process of constructing a sparse disparity map may include: obtaining a first transformation matrix of the overlapping area based on the intrinsic parameter matrix of the shooting device corresponding to the feature point pairs, wherein the imaging planes where the feature point pairs are located are not coplanar; based on the first transformation matrix, aligning the feature point pairs in rows; and constructing a sparse disparity map based on the feature point pairs after the row alignment.

[0066] The first transformation matrix refers to a matrix used for row alignment. The first transformation matrix can be obtained in a variety of ways, which can be adjusted according to actual conditions, and the embodiment of the present application does not limit it.

[0067] For example, a mapping relationship between the intrinsic parameter matrix of the shooting device and the first transformation matrix is ​​constructed in advance, and the first transformation matrix is ​​directly obtained based on the intrinsic parameter matrix of the shooting device. For another example, the first transformation matrix is ​​obtained by performing calculation based on the intrinsic parameter matrix of the shooting device.

[0068] The feature point pair includes two feature points, each feature point includes feature information, and the feature information is used to characterize the coordinate information of the feature point in the original image or the target layer image. The process of constructing a sparse disparity map can also be: constructing a sparse disparity map by calculating the two-dimensional coordinate difference of two feature points in the feature point pair (or the feature point pair after row alignment). It can also be calculating the three-dimensional world coordinate difference of two feature points in the feature point pair (or the feature point pair after row alignment) to construct a sparse disparity map.

[0069] For example, the panoramic front view camera and the panoramic right view camera are used as the shooting device for explanation. f The internal parameter matrix K f and panoramic front view camera C f The rotation matrix R f , construct the panoramic front-view virtual camera Cv f Based on the panoramic right view camera C r The internal parameter matrix K r and panoramic right view camera C r The rotation matrix R r , construct the panoramic right view virtual camera Cv r . Determine the panoramic front-view virtual camera Cv f Optical Center Ov f and panoramic right view virtual camera Cv r Optical center Ov r Based on the optical center Ov f And light heart Ov r , determine the panoramic front-view virtual camera Cv f and panoramic right view virtual camera Cv r The rotation matrix R n And the internal parameter matrix K n Based on the rotation matrix R n And the internal parameter matrix K n , determine the panoramic front view camera C f To the panoramic front view virtual camera Cv f The first transformation matrix Tv f , and determine the panoramic right view camera C r To the panoramic right view virtual camera Cv r The first transformation matrix Tv r Based on the first transformation matrix Tv f For panoramic front view camera C f The original image captured is img f Perform row alignment and transform based on the first transformation matrix Tv r Panoramic right view camera C r The original image captured is img r Align rows.

[0070] The internal parameter matrix K of the panoramic front view camera is expressed as follows: f :

[0071] (4).

[0072] Among them, K f is the internal parameter matrix of the panoramic front view camera, f x_f is the horizontal focal length of the panoramic front view camera, f y_f is the vertical focal length of the panoramic front-view camera, (u0 f , v0 f ) represents the two-dimensional coordinates of the origin of the image coordinate system of the panoramic front view camera in the pixel coordinate system of the panoramic front view camera.

[0073] The internal parameter matrix K of the panoramic right view camera is expressed as follows: r :

[0074] (5).

[0075] Among them, K r is the internal parameter matrix of the panoramic right view camera, f x_r is the horizontal focal length of the panoramic right view camera, f y_r is the vertical focal length of the panoramic right-view camera, (u0 r , v0 r ) represents the two-dimensional coordinates of the origin of the image coordinate system of the panoramic right view camera in the pixel coordinate system of the panoramic right view camera.

[0076] Among them, according to the following formula (6), the panoramic front view virtual camera Cv is determined f Optical center Ov f and panoramic right view virtual camera Cv r Optical center Ov r :

[0077] (6).

[0078] Among them, according to the following formula (7), the panoramic front view virtual camera Cv is determined f and panoramic right view virtual camera Cv r The rotation matrix R n :

[0079] (7).

[0080] Among them, r 1 、r 2 and r 3 Intermediate parameters.

[0081] Determine the panoramic front-view virtual camera Cv according to the following formula (8): f and panoramic right view virtual camera Cv r The internal parameter matrix K n :

[0082] (8).

[0083] According to the following formula (9), the panoramic front view camera C is determined f To the panoramic front view virtual camera Cv f The first transformation matrix Tv f , and determine the panoramic right view camera C r To the panoramic right view virtual camera Cv r The first transformation matrix Tv r :

[0084] (9).

[0085] According to the following formula (10), based on the first transformation matrix Tv f For panoramic front view camera C f The original image captured is img f To align rows:

[0086] (10).

[0087] According to the following formula (11), based on the first transformation matrix Tv r Panoramic right view camera C r The original image captured is img r To align rows:

[0088] (11).

[0089] Based on the first transformation matrix Tv f For panoramic front view camera C f The original image captured is img f Perform row alignment to obtain the image img after the panoramic front view camera performs row alignment vf , based on the first transformation matrix Tv r Panoramic right view camera C r The original image captured is img r Perform row alignment to obtain the image img after the panoramic right view camera performs row alignment vr . From image img vf Find the Oriented FAST feature points and BRIEF descriptors from the image img vrFind the OrientedFAST feature points and BRIEF descriptors in the image img, and then use the FLANN algorithm to use the Oriented FAST feature points and BRIEF descriptors in the image img vf and image img vr Perform feature matching between them, and obtain the coordinates of the n pairs of matched feature points: and Among them, xf 1 With xr 1 is a pair of feature points, and the others are similar. Based on the matched feature points and , after row alignment, the image img vr and image img vf The disparity of each pair of feature points is calculated.

[0090] The disparity of each pair of feature points is calculated according to the following formula (12):

[0091] (12).

[0092] Among them, d fri represents the disparity of the i-th pair of feature points, xf i Represents image img vf The i-th feature point in xr i represents img vr The i-th feature point in .

[0093] The depth map is determined according to the following formula (13):

[0094] (13).

[0095] Among them, Z fri represents the depth value of the i-th pair of feature points, , d fri represents the disparity of the ith pair of feature points, T is the panoramic front-view virtual camera Cv f and panoramic right view virtual camera Cv r Corresponding to the component of the optical center on the X-axis.

[0096] S102 . Segmentation is performed based on the sparse disparity map to obtain a target layer image, where the target layer image is used to indicate an image of an area where a target object is located in the overlapping area.

[0097] There may be multiple target objects, for example, pedestrians, obstacles, etc.

[0098] Segmentation based on the sparse disparity map can obtain only the target layer image. It is also possible to obtain the target layer image and the ground plane layer image, where the ground plane layer image is used to indicate the image where the ground plane is located in the overlapping area.

[0099] Specifically, there are multiple ways to obtain the target layer image by performing segmentation based on the sparse disparity map.

[0100] For example, the target detection algorithm can be used to identify the object in the overlapping area to obtain the target object and the location information of the target object in the overlapping area. Based on the location information and the correspondence between the sparse disparity map and the overlapping area, the target layer image is segmented from the sparse disparity map. The deep learning target detector (Faster Regions with CNN features, Faster R-CNN) can also be used to identify the object in the overlapping area to obtain the target object and the location information of the target object in the overlapping area; based on the location information and the correspondence between the sparse disparity map and the overlapping area, the target layer image is segmented from the sparse disparity map. Among them, the location information is used to characterize the location information of the target object in the overlapping area.

[0101] Specifically, there are many ways to obtain the target layer image and the ground plane layer image by segmentation based on the sparse disparity map.

[0102] In some embodiments, based on the above content of obtaining the target layer image, the target layer image can be segmented from the sparse disparity map having a corresponding relationship with the overlapping area through the position information of the target object, and the image of the remaining area of ​​the overlapping area is used as the ground plane image. The ground plane area can also be identified from the overlapping area through image processing technology, and the image corresponding to the ground plane area can be segmented from the overlapping area to obtain the ground plane image.

[0103] In some embodiments, a first depth map may be constructed according to the sparse disparity map; and segmentation may be performed based on the first depth map to obtain a target layer image and a ground plane layer image.

[0104] The first depth map refers to a map with depth information constructed based on the sparse disparity map.

[0105] By collecting image information through multiple shooting devices, the positioning accuracy of the vehicle during driving can be improved. At the same time, by combining image processing and computer vision technology, more accurate navigation information can be provided.

[0106] S103, performing ground plane projection on the target layer image to obtain a target projection image.

[0107] The target projection image refers to a planar image after the target layer image is projected onto the ground plane.

[0108] In some optional embodiments, a mesh image can be obtained by converting the target layer image into a triangular mesh image; a third transformation matrix between the mesh plane and the ground plane of the mesh image is determined, and based on the third transformation matrix, the mesh image is projected onto the ground plane to obtain a target projection image.

[0109] The grid image refers to a grid-shaped image obtained by gridding the target layer image.

[0110] There are many ways to obtain a grid image. For example, by obtaining preset grid size information, the target layer image can be directly grid-divided based on the preset grid size information to obtain a grid image.

[0111] There may be multiple preset grid size information. For example, the preset grid is a triangular grid, and the triangular grid is an isosceles right triangle grid, and the waist length is 1 pixel. For another example, the preset grid is a square grid, and the size information of the square grid is r*l, where r is the length of the square grid and l is the width of the square grid.

[0112] The third transformation matrix refers to a transformation matrix for projecting the grid image onto the ground plane.

[0113] In this embodiment, the process of performing triangular mesh image conversion on the target layer image to obtain the mesh image may be: performing triangular mesh image conversion processing on the target layer image based on the first depth map to obtain the mesh image.

[0114] Specifically, based on the first depth map, the target layer image is converted into a triangular mesh image to obtain a mesh image, and the process includes: obtaining size information of the target layer image; based on the size information and predetermined mesh size information, performing four-corner mesh division on the target layer image to obtain multiple four-corner meshes; setting diagonals for each four-corner mesh, and determining target depth values ​​of mesh vertices based on depth values ​​in the first depth map; and setting three-dimensional information of mesh vertices based on the target depth values ​​to obtain a mesh image of the target layer image.

[0115] There may be multiple types of size information of the target layer image. For example, the size information of the target layer image may refer to the size information of the detection box where the target object is located in the target layer image, or may refer to the size information of the target layer image.

[0116] There are many ways to set diagonal lines for each quadrilateral grid. For example, a diagonal line from the upper left corner to the lower right corner is set. Another example is a diagonal line from the lower left corner to the upper right corner is set.

[0117] There are many ways to determine the target depth value. For example, the depth value of the 3*3 area of ​​the mesh vertex of the triangular mesh is obtained, and the median of the depth values ​​of the 3*3 neighborhood of the mesh vertex is used as the target depth value. For another example, the depth value of the 4*4 neighborhood of the mesh vertex of the triangular mesh is obtained, and the average of the depth values ​​of the 4*4 neighborhood of the mesh vertex is calculated, and the average is used as the target depth value.

[0118] By fitting three-dimensional pedestrian images in space through triangular grids and then projecting them onto the screen through a series of 2D and 3D views, blind spots and monitoring dead angles are eliminated, pedestrians can be fully displayed, and the user experience is improved.

[0119] For example, Figure 5 As shown, Figure 5 The target layer image provided in the embodiment of the present application is divided into four corner grids to obtain a schematic diagram of multiple four corner grids. Assuming that the preset grid is a four corner grid, the size information of the four corner grid is r*l, where r is the length of the four corner grid and l is the width of the four corner grid. The size information of the target layer image is width * ×height * , where width is the width of the target layer image and height is the length of the target layer image. Then the number of quadrilateral grids can be determined as .

[0120] Set the diagonal line from the upper left corner to the lower right corner and divide the four corners into grids to get A triangle mesh, each triangle mesh vertex is bound to a depth value vector.

[0121] In this embodiment, the process of determining the third transformation matrix between the grid plane of the grid image and the ground plane may be: obtaining the plane information of the plane where the ground plane layer image is located; based on the three-dimensional information of each grid vertex in the grid image and the plane information of the ground plane, determining the third transformation matrix between each triangular mesh in the grid image and the ground plane.

[0122] Wherein, based on the third transformation matrix corresponding to each triangular mesh, each triangular mesh in the mesh image may be projected onto the ground plane to obtain a target projection image.

[0123] Specifically, based on the three-dimensional information of each mesh vertex in the mesh image, the plane equation of each triangular mesh is determined. Based on the plane equation of the triangular mesh, the normal vector of the triangular mesh plane corresponding to the triangular mesh is determined. According to the normal vector of the triangular mesh plane and the normal vector of the ground plane, the rotation axis from the triangular mesh plane to the ground plane is determined. The rotation angle is calculated based on the rotation axis. The rotation matrix and the translation matrix are determined based on the rotation axis and the rotation angle, and the homography matrix corresponding to the triangular mesh is determined based on the rotation matrix and the translation matrix. The triangular mesh is projected onto the ground plane based on the homography matrix corresponding to the triangular mesh.

[0124] Exemplary, combined Figure 6 Explain the above content. Figure 6 A schematic diagram of a triangular mesh provided in an embodiment of the present application, wherein the coordinates of the three vertices of the triangular mesh are a (x1, y1, z1), b (x2, y2, z2), and c (x3, y3, z3), respectively.

[0125] The plane equation of the triangular mesh is determined according to the following formula (14):

[0126] (14).

[0127] Among them, A=(y3-y1)×(z3-z1)-(z2-z1)×(y3-y1);

[0128] B = (x3-x1) × (z2-z1) - (x2-x1) × (z3-z1);

[0129] C=(x2-x1)×(y3-y1)-(x3-x1)×(y2-y1);

[0130] D=-(A×x1+B×y1+C×z1).

[0131] At this time, the normal vector F of the triangular mesh plane is (A, B, C). Since the equation of the ground plane is Y=0, its normal vector is (0,1,0).

[0132] According to the following formula (15), the rotation axis from the triangular mesh plane to the ground plane is determined as (-C, 0, B):

[0133] (15).

[0134] The rotation angle cos(θ) is calculated based on the rotation axis according to the following formula (16):

[0135] (16).

[0136] The rotation matrix R is determined based on the rotation axis and the rotation angle according to the following formula (17):

[0137] (17).

[0138] Wherein, the translation matrix t is determined based on the rotation axis according to the following formula (18):

[0139] (18).

[0140] According to the following formula (19), the homography matrix corresponding to the triangular mesh is determined based on the rotation matrix and the translation matrix:

[0141] (19).

[0142] Here, m is used to refer to the mth triangle mesh.

[0143] S104: Generate a panoramic image according to the target projection image.

[0144] The panoramic image refers to an image that needs to be output after images taken by multiple shooting devices are stitched together and synthesized.

[0145] In some embodiments, based on the target projection image, the process of generating a panoramic image may include: based on a second transformation matrix, transforming the ground plane layer image, the target projection image, and the non-overlapping area in the captured image to obtain a processed image, the second transformation matrix is ​​used to indicate the transformation matrix of the shooting device that captures the image; binding the processed image to a predetermined bowl mold, and rendering a panoramic image based on the bound predetermined bowl mold.

[0146] The second transformation matrix is ​​used to indicate the transformation matrix of the shooting device that shoots the original image, that is, the homography matrix corresponding to the shooting device.

[0147] The processed image refers to an image obtained after solving the homography matrix corresponding to the shooting device, and the image includes the contents of the overlapping area and the non-overlapping area.

[0148] Among them, the predetermined bowl mold refers to a 3D model with a shape similar to a bowl.

[0149] The process of rendering a panoramic image is: rendering the predetermined bowl model to the screen coordinate system, and finally outputting the 2D view and 3D view of the panoramic surround image (Around View Monitor, AVM). Among them, the 2D view is also the bird's-eye view in the panoramic image, which is a flat image and a two-dimensional image. The 3D view is also the predetermined bowl model surround image, that is, the spatial image displayed on the two-dimensional image on the screen, which is a three-dimensional image.

[0150] Binding the processed image as a texture to the predetermined bowl mold includes binding the ground plane image as a texture to the predetermined bowl mold, and binding the target projection image as a texture to the predetermined bowl mold.

[0151] The panoramic image obtained through the above steps can help the driver obtain a wider field of view, especially in scenes such as reversing, turning on narrow roads, and U-turns, which can greatly reduce the safety risks caused by blind spots. At the same time, it can be integrated with other in-vehicle intelligent systems, such as automatic driving, intelligent cruise control, and automatic parking, to enhance the vehicle's intelligence level and driving experience.

[0152] In an embodiment of the present application, a sparse disparity map is constructed for the overlapping area of ​​images captured by multiple shooting devices; segmentation is performed based on the sparse disparity map to obtain a target layer image, which is used to indicate the image of the area where the target object is located in the overlapping area; the target layer image is projected on the ground plane to obtain a target projection image; and a panoramic image is generated based on the target projection image. Through the above steps, a sparse view is constructed for the overlapping area of ​​images captured by multiple shooting devices, and a series of operations are performed based on the sparse disparity map to obtain a panoramic image, so that the panoramic image obtained by stitching does not have problems such as ghosting and artifacts, and the stitching effect is good, which improves the user experience.

[0153] In some optional embodiments, when image processing resources meet preset requirements, interpolation processing is performed based on the sparse disparity map to obtain a dense disparity map, and a second depth map is constructed based on the dense disparity map;

[0154] Based on the second depth map, construct a three-dimensional point cloud stereo image for the overlapping area, and obtain a top-view projection image of the three-dimensional point cloud stereo image;

[0155] A panoramic image is rendered based on the overhead projection image and the image of the non-overlapping area of ​​the captured image.

[0156] The image processing resources include but are not limited to hardware resources, network resources, storage resources, etc. The preset requirement is that the vehicle image processing resources are relatively abundant, and other methods can be used to make the panoramic image processing more streamlined.

[0157] The second depth map refers to a map with depth information constructed based on the dense disparity map.

[0158] The process of rendering a panoramic image based on the overhead projection image and the image of the corresponding non-overlapping area is as follows: output the overhead projection image to the 2D bird's-eye view, and activate the 3D model for the 3D view to render the panoramic image. Output the overhead projection image to the 2D bird's-eye view, and any pixel point iPoind (x, y, z) of the three-dimensional point cloud stereo image, the corresponding 2D bird's-eye view coordinate is iPoind (x, z), that is, the x-axis remains unchanged, and the z-axis becomes the y-axis; for the same x value and z value, take the pixel RGB value corresponding to the maximum y value coordinate point and render it to the screen.

[0159] For example, the overlapping area is determined based on the panoramic front view camera and the panoramic right view camera as an example to explain the above content. fr , the second depth map is imgDepth fr As the z-axis coordinate, the image img vf and image img vr The column coordinates of are used as the x-axis coordinates, image img vf and image img vr The row coordinates of are used as the y-axis coordinates, image img vf and image img vr The RGB pixel values ​​of the pixels are used as the pixel values ​​of the three-dimensional coordinates (x, y, z), thereby constructing a three-dimensional point cloud stereo image. Reconstructing three-dimensional point cloud stereo images for different overlapping areas can reduce the amount of calculation for point cloud reconstruction and the computing power occupied by the platform, thereby improving operating efficiency.

[0160] Assume that there are two types of vehicle operation platforms, and the vehicle can run on platform 1 or platform 2. The image processing resources of platform 1 are more than those of platform 2. If the vehicle runs on platform 1, after obtaining the sparse disparity map, the step of segmenting based on the sparse disparity map to obtain the target layer image is not performed; and the following steps are performed: interpolation processing is performed based on the sparse disparity map to obtain a dense disparity map, and a second depth map is constructed based on the dense disparity map; based on the second depth map, a three-dimensional point cloud stereo image for the overlapping area is constructed, and a top-down projection image of the three-dimensional point cloud stereo image is obtained; a panoramic image is rendered based on the top-down projection image and the image of the corresponding non-overlapping area. If the vehicle runs on platform 2, segmentation is performed based on the sparse disparity map to obtain a target layer image, and the target layer image is used to indicate the image of the area where the target object is located in the overlapping area; the target layer image is projected on the ground plane to obtain a target projection image; and a panoramic image is generated based on the target projection image.

[0161] Furthermore, before rendering the panoramic image, a global tone mapping adjustment may be performed on the processed image.

[0162] There are many ways to adjust the global tone mapping. For example, the brightness difference of the processed YUV images captured by multiple shooting devices is calculated respectively, and then the brightness uniformity algorithm and the illumination uniformity algorithm are used to adjust the tones of the multiple images to be globally consistent.

[0163] Global tone mapping adjustment uses an algorithm to adjust images taken by multiple cameras to consistent brightness, color, and hue, ultimately achieving seamless stitching, making the user feel like the image is taken by a single camera rather than a stitching of images taken by multiple cameras.

[0164] Specifically, Figure 7 This is a flowchart of an embodiment of a panoramic image processing method provided by the present application. The vehicle in the panoramic image processing method can be applied to platform 1 or platform 2. The computing power of platform 1 is greater than that of platform 2. The vehicle is equipped with 4 cameras. Figure 7 It can be seen that the embodiment of the panoramic image processing method includes:

[0165] S201, obtaining and inputting fisheye views corresponding to four cameras.

[0166] S202: Perform fisheye distortion removal on the view of each camera and calibrate the camera's intrinsic parameter matrix.

[0167] S203, laying out a calibration plate under the view of each camera, and solving the homography transformation matrix H of each view by bird's-eye view projection transformation.

[0168] Among them, the bird's-eye view refers to the observation angle from a high place. The bird's-eye view provides an overall, overlooking visual perception, which allows you to see a wider area and more details.

[0169] S204 , decomposing the homography transformation matrix H of each view by using the SVD singular value decomposition method to obtain the rotation matrix R and translation matrix t of each view.

[0170] S205: Construct a transformation matrix of the virtual camera to achieve row alignment of adjacent camera views.

[0171] S206: Use radar to identify pedestrians or obstacles at close range.

[0172] S207: If there are pedestrians or obstacles, extract feature points from the aligned views and perform feature point matching.

[0173] S208 . Construct a sparse disparity map based on the matched feature points (ie, feature point pairs).

[0174] S209: Determine the platform used by the vehicle.

[0175] S210: If the vehicle is applied to platform No. 2, construct a depth map (ie, a first depth map) based on the sparse disparity map.

[0176] S211 . Construct a triangular mesh image of a moving pedestrian layer according to the first depth map.

[0177] S212, calculating the transformation matrix from the plane where the triangular mesh image is located to the ground plane.

[0178] S213: Project the triangular mesh image onto the ground plane.

[0179] S214. Bind the ground plane image as a texture to the bowl model.

[0180] S220 , performing global tone mapping adjustment on each image.

[0181] S221: Output a panoramic image.

[0182] S215: If the vehicle is applied to platform No. 1, the sparse disparity map needs to be interpolated to obtain a dense disparity map.

[0183] S216: construct a depth map (ie, a second depth map) according to the dense disparity map.

[0184] S217: construct a three-dimensional point cloud image for pedestrians or obstacles in the overlapping area of ​​adjacent cameras.

[0185] S218. Output the point cloud top-down projection graphics to a 2D bird's-eye view.

[0186] S219, 3D view rendering activates human model or object model animation (i.e., animation of pedestrians or obstacles).

[0187] S220 , performing global tone mapping adjustment on each image.

[0188] S221: Output a panoramic image.

[0189] Through the above steps, when there are moving pedestrians or obstacles in the foreground content of the image that cross the field of view of adjacent cameras, strict alignment of parallax objects can be achieved, eliminating the problems of ghosting and artifacts in the stitched panoramic image. At the same time, by fitting the spatial three-dimensional pedestrian image through the triangular mesh, and then projecting it to the screen through a series of 2D views and 3D views, blind spots and monitoring blind spots are eliminated, pedestrians can be fully displayed, and the user experience is improved.

[0190] In order to better implement the panoramic image processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the panoramic image processing method. The meanings of the terms are the same as those in the panoramic image processing method, and the specific implementation details can refer to the description in the method embodiment.

[0191] For example, Figure 8 As shown, the panoramic image processing device may include an image construction unit 301, an image segmentation unit 302, a projection unit 303 and an image generation unit 304, as follows:

[0192] An image construction unit 301 is used to construct a sparse disparity map for overlapping areas of images captured by multiple shooting devices;

[0193] An image segmentation unit 302 is used to perform segmentation based on the sparse disparity map to obtain a target layer image, where the target layer image is used to indicate an image of an area where a target object is located in the overlapping area;

[0194] The projection unit 303 is used to perform ground plane projection on the target layer image to obtain a target projection image;

[0195] The image generating unit 304 is used to generate a panoramic image according to the target projection image.

[0196] In one embodiment of the present application, the image construction unit 301 includes:

[0197] A feature point pair acquisition subunit is used to match feature points in the overlapping area to obtain at least one feature point pair;

[0198] The sparse disparity map construction subunit is used to construct a sparse disparity map according to feature point pairs.

[0199] In one embodiment of the present application, the sparse disparity map construction subunit includes:

[0200] A first transformation module, used to obtain a first transformation matrix of the overlapping area based on an intrinsic parameter matrix of the shooting device corresponding to the feature point pair, wherein the imaging planes where the feature point pair is located are not coplanar;

[0201] A row alignment module, used for performing row alignment on feature point pairs based on a first transformation matrix;

[0202] The sparse disparity map construction module is used to construct a sparse disparity map based on the feature point pairs after row alignment.

[0203] In one embodiment of the present application, the image segmentation unit 302 includes:

[0204] The image segmentation subunit is used to perform segmentation based on the sparse disparity map to obtain a target layer image and a ground plane layer image, wherein the ground plane layer image is used to indicate an image where a ground plane is located in an overlapping area.

[0205] In one embodiment of the present application, the image generation unit 304 includes:

[0206] A second transformation subunit is used to transform the ground plane layer image, the target projection image and the non-overlapping area in the captured image based on a second transformation matrix to obtain a processed image, wherein the second transformation matrix is ​​used to indicate the transformation matrix of the shooting device;

[0207] The image generation subunit is used to bind the processed image to a predetermined bowl mold, and render a panoramic image based on the bound predetermined bowl mold.

[0208] In one embodiment of the present application, the above-mentioned image segmentation subunit includes:

[0209] A first depth map construction module, configured to construct a first depth map according to the sparse disparity map;

[0210] The image segmentation module is used to perform segmentation based on the first depth map to obtain a target layer image and a ground plane layer image.

[0211] In one embodiment of the present application, the projection unit 303 includes:

[0212] A mesh image conversion subunit is used to convert the target layer image into a triangular mesh image to obtain a mesh image;

[0213] The projection subunit is used to determine a third transformation matrix between a grid plane of the grid image and a ground plane, and based on the third transformation matrix, project the grid image onto the ground plane to obtain a target projection image.

[0214] In one embodiment of the present application, the grid image conversion subunit includes:

[0215] The mesh image conversion module is used to perform triangular mesh image conversion processing on the target layer image based on the first depth map to obtain a mesh image.

[0216] In one embodiment of the present application, the grid image conversion module includes:

[0217] A size information acquisition submodule is used to obtain the size information of the target layer image;

[0218] A four-corner grid division submodule is used to perform four-corner grid division processing on the target layer image based on the size information and the predetermined grid size information to obtain a plurality of four-corner grids;

[0219] A target depth value determination submodule, configured to set a diagonal line for each quadrilateral grid and determine a target depth value of a grid vertex based on the depth value in the first depth map;

[0220] The mesh image determination submodule is used to set the three-dimensional information of the mesh vertices based on the target depth value to obtain the mesh image of the target layer image.

[0221] In one embodiment of the present application, the projection subunit includes:

[0222] A plane information acquisition module is used to acquire the plane information of the plane where the ground plane layer image is located;

[0223] The third transformation module is used to determine a third transformation matrix between each triangular mesh in the mesh image and the ground plane based on the three-dimensional information of each mesh vertex in the mesh image and the plane information of the ground plane.

[0224] In one embodiment of the present application, the panoramic image processing device further includes:

[0225] A second depth map determining unit is configured to perform interpolation processing based on the sparse disparity map to obtain a dense disparity map when image processing resources meet preset requirements, and construct a second depth map based on the dense disparity map;

[0226] A top-view projection image determining unit, configured to construct a three-dimensional point cloud stereo image for the overlapping area based on the second depth map, and obtain a top-view projection image of the three-dimensional point cloud stereo image;

[0227] The panoramic image determination unit is used to render a panoramic image according to the overhead projection image and the image of the non-overlapping area of ​​the captured image.

[0228] In the embodiment of the present application, the image construction unit 301 constructs a sparse disparity map for the overlapping area of ​​images taken by multiple shooting devices; the image segmentation unit 302 performs segmentation based on the sparse disparity map to obtain a target layer image, and the target layer image is used to indicate the image of the area where the target object is located in the overlapping area; the projection unit 303 performs ground plane projection on the target layer image to obtain a target projection image; the image generation unit 304 generates a panoramic image based on the target projection image. Through the above steps, a sparse view is constructed for the overlapping area of ​​images taken by multiple shooting devices, and a series of operations are performed based on the sparse disparity map to obtain a panoramic image, so that the panoramic image obtained by stitching does not have problems such as ghosting and artifacts, and the stitching effect is good, which improves the user experience.

[0229] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0230] The embodiment of the present application also provides an electronic device, wherein the operating system of the electronic device includes a first operating system and a second operating system. Fig. 9 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0231] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0232] The processor 401 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing computer programs and / or modules stored in the memory 402, and calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0233] The memory 402 can be used to store computer programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0234] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0235] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0236] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 402 according to the following instructions, and the processor 401 will run the computer programs stored in the memory 402, thereby realizing various functions, such as:

[0237] Constructing a sparse disparity map for overlapping areas of images captured by multiple cameras;

[0238] Segmentation is performed based on the sparse disparity map to obtain a target layer image, where the target layer image is used to indicate an image of an area where a target object is located in the overlapping area;

[0239] Perform ground plane projection on the target layer image to obtain a target projection image;

[0240] Generate a panoramic image based on the target projection image.

[0241] It can be seen that the electronic device provided by the embodiment of the present application constructs a sparse disparity map for the overlapping area of ​​images taken by multiple shooting devices; performs segmentation based on the sparse disparity map to obtain a target layer image, and the target layer image is used to indicate the image of the area where the target object is located in the overlapping area; performs ground plane projection on the target layer image to obtain a target projection image; and generates a panoramic image based on the target projection image. Through the above steps, a sparse view is constructed for the overlapping area of ​​images taken by multiple shooting devices, and a series of operations are performed based on the sparse disparity map to obtain a panoramic image, so that the panoramic image obtained by stitching does not have problems such as ghosting and artifacts, and the stitching effect is good, which improves the user experience.

[0242] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the panoramic image processing method above, which will not be elaborated here.

[0243] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a storage medium and loaded and executed by a processor.

[0244] To this end, an embodiment of the present application provides a storage medium in which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any panoramic image processing method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0245] Constructing a sparse disparity map for overlapping areas of images captured by multiple cameras;

[0246] Segmentation is performed based on the sparse disparity map to obtain a target layer image, where the target layer image is used to indicate an image of an area where a target object is located in the overlapping area;

[0247] Perform ground plane projection on the target layer image to obtain a target projection image;

[0248] Generate a panoramic image based on the target projection image.

[0249] It can be seen that the storage medium provided in the embodiment of the present application constructs a sparse disparity map for the overlapping area of ​​images taken by multiple shooting devices; performs segmentation based on the sparse disparity map to obtain a target layer image, and the target layer image is used to indicate the image of the area where the target object is located in the overlapping area; performs ground plane projection on the target layer image to obtain a target projection image; and generates a panoramic image based on the target projection image. Through the above steps, a sparse view is constructed for the overlapping area of ​​images taken by multiple shooting devices, and a series of operations are performed based on the sparse disparity map to obtain a panoramic image, so that the panoramic image obtained by stitching does not have problems such as ghosting and artifacts, and the stitching effect is good, which improves the user experience.

[0250] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0251] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0252] Since the computer program stored in the storage medium can execute the steps in any panoramic image processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any panoramic image processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0253] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned panoramic image processing method.

[0254] The embodiment of the present application also provides a vehicle, which includes the above-mentioned panoramic image processing device, or the above-mentioned electronic device, or the above-mentioned computer program product. The specific structure of the vehicle is not limited in this application. The specific implementation methods and corresponding beneficial effects of the above-mentioned operations of the electronic device are also applicable to the vehicle. For details, please refer to the detailed description of the panoramic image processing method above, which will not be repeated here.

[0255] The above is a detailed introduction to a panoramic image processing method, device, electronic device, storage medium and vehicle provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A panoramic image processing method, characterized in that: include: Performing feature point matching on overlapping areas of images captured by multiple camera devices to obtain at least one feature point pair; Based on the intrinsic parameter matrix of the shooting device corresponding to the feature point pair, obtaining a first transformation matrix of the overlapping area, wherein the imaging planes where the feature point pair are located are not coplanar; Based on the first transformation matrix, aligning the feature point pairs in rows; Construct a sparse disparity map based on the feature point pairs after row alignment; Based on the sparse disparity map, construct a first depth map; Perform segmentation based on the first depth map to obtain a target layer image, wherein the target layer image is used to indicate an image of an area where a target object in the overlapping area is located, and the target object is distinguished from a ground plane in the overlapping area; Performing ground plane projection on the target layer image to obtain a target projection image; A panoramic image is generated according to the target projection image.

2. The panoramic image processing method according to claim 1, characterized in that: The performing segmentation based on the first depth map to obtain a target layer image includes: Segmentation is performed based on the first depth map to obtain a target layer image and a ground plane layer image, wherein the ground plane layer image is used to indicate an image where a ground plane in the overlapping area is located.

3. The panoramic image processing method according to claim 2, characterized in that: The step of generating a panoramic image according to the target projection image comprises: Based on a second transformation matrix, transform the ground plane layer image, the target projection image and the non-overlapping area in the captured image to obtain a processed image, wherein the second transformation matrix is ​​used to indicate the transformation matrix of the shooting device; The processed image is bound to a predetermined bowl mold, and a panoramic image is rendered based on the bound predetermined bowl mold.

4. The panoramic image processing method according to claim 2, characterized in that: The step of performing ground plane projection on the target layer image to obtain a target projection image includes: Performing triangular mesh image conversion on the target layer image to obtain a mesh image; A third transformation matrix between a grid plane of the grid image and a ground plane is determined, and based on the third transformation matrix, the grid image is projected onto the ground plane to obtain a target projected image.

5. The panoramic image processing method according to claim 4, characterized in that: The step of converting the target layer image into a triangular mesh image to obtain a mesh image includes: Based on the first depth map, the target layer image is converted into a triangular mesh image to obtain a mesh image.

6. The panoramic image processing method according to claim 5, characterized in that: The step of performing triangular mesh image conversion processing on the target layer image based on the first depth map to obtain a mesh image includes: Acquire size information of the target layer image; Based on the size information and the predetermined grid size information, the target layer image is subjected to a four-corner grid division process to obtain a plurality of four-corner grids; Setting a diagonal line for each of the four-corner grids, and determining target depth values ​​of grid vertices based on the depth values ​​in the first depth map; The three-dimensional information of the mesh vertices is set based on the target depth value to obtain a mesh image of the target layer image.

7. The panoramic image processing method according to claim 4, characterized in that: The determining of a third transformation matrix between a grid plane of the grid image and a ground plane comprises: Acquire the plane information of the plane where the ground plane layer image is located; Based on the three-dimensional information of each mesh vertex in the mesh image and the plane information of the ground plane, a third transformation matrix between each triangular mesh in the mesh image and the ground plane is determined.

8. The panoramic image processing method according to claim 1, characterized in that: The method further comprises: When the image processing resources meet the preset requirements, interpolation processing is performed based on the sparse disparity map to obtain a dense disparity map, and a second depth map is constructed based on the dense disparity map; Based on the second depth map, construct a three-dimensional point cloud stereo image for the overlapping area, and obtain a top-view projection image of the three-dimensional point cloud stereo image; A panoramic image is rendered according to the overhead projection image and the image of the non-overlapping area of ​​the captured image.

9. A panoramic image processing device, characterized in that: include: An image construction unit, configured to perform feature point matching on overlapping areas of images captured by multiple capturing devices to obtain at least one feature point pair; Based on the intrinsic parameter matrix of the shooting device corresponding to the feature point pair, a first transformation matrix of the overlapping area is obtained, wherein the imaging planes where the feature point pair are located are not coplanar; based on the first transformation matrix, the feature point pair is row-aligned; based on the feature point pair after the row alignment, a sparse disparity map is constructed; An image segmentation unit, configured to construct a first depth map based on the sparse disparity map; perform segmentation based on the first depth map to obtain a target layer image, wherein the target layer image is used to indicate an image of an area where a target object in the overlapping area is located, and the target object is distinguished from a ground plane in the overlapping area; A projection unit, used for performing ground plane projection on the target layer image to obtain a target projection image; The image generating unit is used to generate a panoramic image according to the target projection image.

10. An electronic device, characterized in that: The method comprises one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the panoramic image processing method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: It comprises a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the panoramic image processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program or an instruction, and when the computer program or the instruction is executed by a processor, the panoramic image processing method described in any one of claims 1 to 8 is implemented.

13. A vehicle, characterized in that: The vehicle includes the panoramic image processing device as claimed in claim 9, or the electronic device as claimed in claim 10.

Images