Image splicing method and device, vehicle, storage medium and program product

Abstract

The invention discloses an image splicing method and device, a vehicle, a storage medium and a program product. The method comprises the following steps: acquiring original images acquired by at least one image acquisition device on a vehicle, and determining a top view image of each original image; determining an image stitching angle range if an obstacle is detected in the set search angle range of the vehicle surroundings; and according to the image splicing angle range, splicing the top view images to obtain a spliced vehicle panoramic top view. By utilizing the method, the obstacle is detected in the set search angle range, the image splicing angle range is determined when the obstacle exists, and the top view images are spliced according to the image splicing angle range to obtain the panoramic top view of the vehicle, so that each image splicing angle range can avoid the obstacle, and the vehicle panoramic top view can be obtained. The problems of ghosting and dislocation in the panoramic top view of the vehicle caused by obstacles in the splicing range are solved, and the definition and accuracy of the panoramic top view of the vehicle are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image stitching method, device, vehicle, storage medium and program product. Background Art

[0002] In today's automotive industry, most high-end and mid-range models are fully equipped with cutting-edge advanced driver assistance systems. Among them, 360-degree panoramic imaging technology, as an important part of these advanced functions, has been widely used, greatly improving the convenience and safety of driving.

[0003] The core of the 360-degree panoramic imaging system is its bird's-eye view function, which utilizes four ultra-wide-angle cameras installed around the vehicle. Through these four cameras, the system can capture images from four different directions in real time. With the help of advanced view transformation models, these images are accurately projected into a bird's-eye view, and each bird's-eye view is stitched together to present the driver with a full, all-round view of the vehicle's surroundings.

[0004] However, in the bird's-eye view, although the ground scene can be restored more accurately, the three-dimensional objects in the scene are generally deformed and distorted. Especially in the overlapping area of ​​adjacent images, the presence of three-dimensional objects will cause ghosting and dislocation in the stitched panoramic view, which greatly affects the clarity and accuracy of the visual effect. Summary of the invention

[0005] The present invention provides an image stitching method, device, vehicle, storage medium and program product to solve the problem of double image and dislocation of the view caused by the presence of three-dimensional obstacles in the stitching area of ​​the bird's-eye view.

[0006] In a first aspect, an embodiment of the present invention provides an image stitching method, the method comprising:

[0007] Acquire original images captured by at least one image acquisition device on the vehicle, and determine a top view image of each of the original images;

[0008] If an obstacle is detected within a set search angle range of the vehicle's surroundings, an image stitching angle range is determined;

[0009] The overhead images are stitched together according to the image stitching angle range to obtain a stitched panoramic overhead view of the vehicle.

[0010] In a second aspect, an embodiment of the present invention provides an image stitching device, the device comprising:

[0011] An image acquisition module, used to acquire original images acquired by at least one image acquisition device on the vehicle, and determine a top view image of each of the original images;

[0012] A stitching range determination module, used to determine the image stitching angle range if an obstacle is detected within a set search angle range of the vehicle's surroundings;

[0013] The panoramic image determination module is used to stitch the top view images according to the image stitching angle range to obtain a stitched panoramic top view of the vehicle.

[0014] In a third aspect, an embodiment of the present invention provides a vehicle, the vehicle comprising:

[0015] at least one controller;

[0016] and a memory communicatively coupled to the at least one controller;

[0017] The memory stores a computer program executable by the at least one controller, and the computer program is executed by the at least one controller so that the at least one controller can execute the image stitching method described in any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a controller to implement the image stitching method described in any embodiment of the present invention when executed.

[0019] In a fifth aspect, an embodiment of the present invention further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a controller, the image stitching method described in any embodiment of the present invention is implemented.

[0020] The technical solution of the embodiment of the present invention is to obtain the original images collected by at least one image acquisition device on the vehicle and determine the top view image of each original image; if an obstacle is detected in the set search angle range of the vehicle's surrounding environment, the image stitching angle range is determined; according to the image stitching angle range, each top view image is stitched to obtain a stitched panoramic top view of the vehicle. By detecting obstacles in the set search angle range, the image stitching angle range is determined when there is an obstacle in the set search angle range, and the top view images are stitched according to the image stitching angle range to obtain a panoramic top view of the vehicle, ensuring that each image stitching angle range can accurately avoid obstacles, solving the problem of ghosting and misalignment in the panoramic top view of the vehicle caused by obstacles in the stitching range, improving the clarity and accuracy of the visual effect, and thus improving the safety of vehicle driving.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flowchart of an image stitching method provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a coverage area of ​​an image stitching method provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of the structure of an image stitching device provided by an embodiment of the present invention;

[0026] Figure 4 A schematic structural diagram of a vehicle that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that the 360-degree panoramic view is obtained by directly stitching the bird's-eye views obtained by multiple cameras. When there are three-dimensional obstacles in the overlapping area of ​​adjacent images, the panoramic view will experience ghosting and misalignment, which greatly affects the clarity and accuracy of the visual effect.

[0030] Based on this, an embodiment of the present invention provides an image stitching method. Figure 1 A flowchart of an image stitching method provided in an embodiment of the present invention. The embodiment of the present invention is applicable to the situation where there are obstacles in the stitching area of ​​the bird's-eye view. The method can be executed by an image stitching device, which can be implemented in the form of software and / or hardware, and optionally, by a vehicle.

[0031] like Figure 1 As shown, the image stitching method provided by the embodiment of the present invention may specifically include:

[0032] S101, acquiring original images captured by at least one image capture device on a vehicle, and determining a top view image of each original image.

[0033] Among them, the original image can be considered as a real-time image directly captured by the image acquisition device; the bird's-eye view image can be understood as an image that can present an object and its surrounding environment from a bird's-eye view.

[0034] In this embodiment, at least one original image captured in real time is obtained from an image acquisition device installed on the vehicle, and each original image is converted into a corresponding overhead image that looks like it was taken from a bird's-eye view of the vehicle through image processing techniques such as image correction, cropping, scaling, and perspective transformation.

[0035] Exemplarily, the vehicle is equipped with four fisheye cameras, such as ADAS surround-view cameras, on the front, rear, left and right sides to capture the vehicle's surroundings. Four real-time original images are acquired from the four fisheye cameras respectively, and the overhead view image corresponding to each original image is determined by image processing technology.

[0036] S102: If an obstacle is detected within the set search angle range of the vehicle's surrounding environment, determine an image stitching angle range.

[0037] Among them, obstacles can be understood as three-dimensional objects existing in the real world, such as buildings, steps and flower pots, etc. Setting the search angle range can be understood as the range of angles set to search for obstacles. The image stitching angle range can be understood as the angle range for stitching two images.

[0038] In this embodiment, original images of different orientations of the environment around the vehicle are collected by multiple vehicle-mounted image acquisition devices, and the corresponding overhead image is determined according to each original image. All the overhead images are spliced ​​in pairs to obtain a complete image of the area around the vehicle. Since there are overlapping acquisition areas between adjacent image acquisition devices, the determined overhead images have overlapping areas. Therefore, before image splicing, it is necessary to use a detection method, such as ultrasonic detection, infrared detection, laser detection or electromagnetic wave detection, to detect the environment around the vehicle, or use image recognition technology, such as template matching, support vector machine, deep learning or reinforcement learning, to detect the overhead image, and determine whether there is an obstacle in the environment around the vehicle within the set search angle range. If an obstacle is detected, the splicing angle is adjusted according to the position, angle and other information of the obstacle, so as to determine the image splicing angle range. If it is detected that there is no obstacle, the image splicing is directly performed within the set search angle range.

[0039] It should be noted that the search angle range can be set based on historical experience, the overlapping area of ​​the overhead view images and / or the performance parameters of the detection equipment. The search angle range can be the overlapping area when the overhead view images are stitched, or it can be other reasonable ranges.

[0040] S103 . Stitching the bird's-eye view images according to the image stitching angle range to obtain a stitched panoramic bird's-eye view of the vehicle.

[0041] The vehicle panoramic bird's-eye view can be considered as an image presenting the environment in all directions around the vehicle from a bird's-eye view.

[0042] In this embodiment, when an obstacle is detected in the set search angle range, the image stitching angle range is determined, and image fusion techniques such as pyramid fusion, Poisson fusion, IHS fusion and Alpha fusion are used to stitch the top-view images in pairs within the image stitching angle range to obtain a stitched panoramic top-view of the vehicle.

[0043] An image stitching method provided by an embodiment of the present invention obtains original images captured by at least one image acquisition device on a vehicle, and determines a top view image of each original image; if an obstacle is detected in a set search angle range of the vehicle's surrounding environment, the image stitching angle range is determined; and each top view image is stitched according to the image stitching angle range to obtain a stitched panoramic top view of the vehicle. The technical solution of the embodiment of the present invention detects obstacles in a set search angle range, determines an image stitching angle range when an obstacle exists in the set search angle range, and stitches each of the top view images according to the image stitching angle range to obtain a panoramic top view of the vehicle, thereby ensuring that each image stitching angle range can accurately avoid obstacles, solving the problem of ghosting and misalignment in the panoramic top view of the vehicle caused by obstacles in the stitching range, improving the clarity and accuracy of the visual effect, and thereby improving the safety of vehicle driving.

[0044] As a first optional embodiment of the embodiment of the present invention, based on the above embodiment, the step of determining the overhead image of each original image may be further specified as the following steps:

[0045] a1) Correcting each of the original images to obtain a corrected image.

[0046] The corrected image can be understood as an image obtained by correcting the original image.

[0047] In this embodiment, due to the characteristics, position and angle of the image acquisition device and other factors, the collected original images may be distorted and deformed. At this time, by obtaining camera parameters, such as distortion parameters, intrinsic parameters and extrinsic parameters, and using distortion correction models such as an isometric model, Kannala-Brandt model, Mei model or unified omnidirectional camera model, each original image is corrected to obtain a corrected image.

[0048] For example, when the image acquisition device is a fisheye camera, the original image captured may be distorted due to factors such as the installation angle and position of the fisheye camera or the optical characteristics of the lens itself.

[0049] b1) transforming each of the corrected images from a plane coordinate system into a three-dimensional space coordinate system to obtain a first transformed image of each of the corrected images in the three-dimensional space coordinate system.

[0050] The first converted image can be understood as an image in a three-dimensional coordinate system.

[0051] In this embodiment, the coordinates of each pixel point in each corrected image in the plane coordinate system are obtained, and then the pixel point is converted from the plane coordinate system to the three-dimensional space coordinate system through a coordinate conversion method, such as a perspective projection matrix, an inverse perspective projection matrix, etc., to obtain the first converted image corresponding to each corrected image in the three-dimensional space coordinate system.

[0052] Exemplarily, assume that in a determined corrected image, the coordinates of a certain pixel point in a plane coordinate system are (u, v). First, obtain the EOL calibration parameters calibrated by the vehicle end of line process (End of Line, EOL), including the internal parameters of the camera (such as focal length, principal point coordinates, etc.) and external parameters (such as the position and posture of the camera in the world's three-dimensional space coordinate system); then, the pinhole imaging model can be used to describe the geometric relationship between the world point coordinates (X, Y, Z) of a certain point in the world in the three-dimensional space coordinate system and the pixel point coordinates (u, v) of the point in the two-dimensional plane coordinate system according to the EOL calibration parameters; determine the inverse perspective projection matrix according to the imaging relationship described by the pinhole imaging model, and obtain the conversion relationship between the coordinates (u, v) of the pixel point in the corrected image in the plane coordinate system and the world point coordinates (X, Y, Z) of the pixel point in the three-dimensional space coordinate system through the inverse perspective projection transformation.

[0053] Exemplarily, the imaging relationship described by the pinhole imaging model may be:

[0054]

[0055] Among them, z c is the coordinate value of the z-axis in the camera coordinate system; A is a 3×3 scaling matrix, which can be determined by the camera intrinsic parameters; R is the rotation matrix, and t is the translation matrix. R and t can be determined by the camera intrinsic parameters respectively; M 3×4 is a 3×4 matrix. is the rotation matrix after scaling, is the scaled translation matrix; since the required top view image does not focus on three-dimensional obstacles, only the case of Z = 0 is considered. When Z = 0, a 3×3 matrix M is obtained. 3×3 .

[0056] Exemplarily, the determined inverse perspective projection matrix may be:

[0057]

[0058] c1) converting each of the first converted images from the three-dimensional space coordinate system to a bird's-eye view coordinate system to obtain a second converted image of each of the first converted images in the bird's-eye view coordinate system.

[0059] The second converted image may be understood as an image in a bird's-eye view coordinate system.

[0060] In this embodiment, the coordinates of each point in each first transformed image in the three-dimensional space coordinate system are transformed into the bird's-eye view coordinate system by a coordinate transformation method to obtain a second transformed image of each first transformed image in the bird's-eye view coordinate system.

[0061] Exemplarily, the setting of the three-dimensional space coordinate system can be that the origin of the three-dimensional space coordinate system is located at the projection of the vehicle's geometric center on the ground, the x-axis is perpendicular to the vehicle body to the right, the y-axis is parallel to the vehicle body upward, and the z-axis is 0; the setting of the bird's-eye view coordinate system can be that the center of the bird's-eye view coordinate system corresponds to the vehicle's geometric center, the origin of the bird's-eye view coordinate system is located at the upper left corner of the second transformed image, the x-axis is to the right, and the y-axis is downward.

[0062] Exemplarily, the coordinates of the world point in the three-dimensional space coordinate system can be transformed according to the size and scale factor of the set second conversion image, wherein the scale factor can be set according to the actual transformation needs, and the coordinate transformation method can adopt a linear change method, which can be expressed as:

[0063]

[0064] Among them, BEV x BEV is the x-axis coordinate value in the bird's-eye view coordinate system; y The y-axis coordinate value in the bird's-eye view coordinate system; scale x is the scale factor relative to the x-axis coordinate value; scale y BEV is the scale factor relative to the y-axis coordinate value; width BEV is the image width of the second converted image; height is the image height (length) of the second converted image.

[0065] d1) determining each of the second converted images as the top-view image.

[0066] In this embodiment, each of the second converted images in the bird's-eye view coordinate system is determined as a top view image.

[0067] The above technical scheme of this embodiment provides an implementation method for obtaining a bird's-eye view image based on an original image, by correcting each original image to obtain a corrected image, converting the corrected image from a plane coordinate system to a three-dimensional space coordinate system, and then converting the three-dimensional space coordinate system to a bird's-eye view coordinate system to obtain a second converted image in the bird's-eye view coordinate system, and determining the second converted image as a bird's-eye view image, accurately determining the bird's-eye view image of each original image, and ensuring the clarity and accuracy of the single-view image.

[0068] As a second optional embodiment of the embodiment of the present invention, based on the above embodiment, if an obstacle is detected in the set search angle range of the vehicle surrounding environment, the image stitching angle range is determined, which can be specifically optimized into the following steps:

[0069] a2) In the vehicle surrounding environment with the vehicle body as the center, the four corners of the vehicle body are respectively used as target origins, and ultrasonic detection is performed with each of the target origins as a vertex.

[0070] The target origin may be understood as the origin of a coordinate system for performing detection and setting a detection angle.

[0071] In this embodiment, ultrasonic detection technology is used to detect the vehicle's surrounding environment with the vehicle body as the center, and the four corners of the vehicle body are respectively used as target origins, and ultrasonic detection is performed with each of the target origins as a vertex.

[0072] It should be noted that ultrasonic detection with the target origin as the vertex can be performed by installing the ultrasonic device at the target origin, or by performing coordinate transformation on the information returned by the ultrasonic wave so that the information returned by the ultrasonic wave is displayed with the target origin as the origin, as if ultrasonic detection is performed with the target origin as the vertex.

[0073] b2) determining whether there is an obstacle within the set search angle range according to the ultrasonic detection data.

[0074] The ultrasonic detection data may be data acquired by an ultrasonic sensor after detection, such as the distance, direction, and approximate area of ​​an obstacle.

[0075] In this embodiment, ultrasonic detection is performed on the vehicle's surroundings with each of the target origins as vertices to obtain ultrasonic detection data, and it is determined whether there are obstacles within the set search angle range based on the ultrasonic detection data.

[0076] Exemplarily, an ultrasonic sensor is installed at each target origin, and four coordinate systems are established with each target origin as the origin, the direction perpendicular to the vehicle body pointing in the opposite direction of the vehicle as the x-axis, and the direction parallel to the vehicle body pointing in the opposite direction of the vehicle as the y-axis. The ultrasonic detection data returned by each ultrasonic sensor is obtained, and a search angle range is set in the coordinate system. It is determined whether there is an obstacle within the set search angle range based on the ultrasonic detection data.

[0077] Exemplarily, an ultrasonic sensor is installed on the front, rear, left and right of the vehicle, respectively. Each ultrasonic sensor detects the surrounding environment in front, rear, left and right of the vehicle with the initial coordinate system in each ultrasonic sensor and returns a reflected signal. Then, four coordinate systems are established with the origin of each target as the origin, the direction perpendicular to the vehicle body pointing in the opposite direction of the vehicle as the x-axis, and the direction parallel to the vehicle body pointing in the opposite direction of the vehicle as the y-axis. The reflected signal returned by each ultrasonic sensor is transformed in coordinates, and the reflected signal is mapped into the coordinate system for processing to obtain ultrasonic detection data. At this time, a search angle range is set in the coordinate system, and whether there is an obstacle within the set search angle range is determined based on the data after coordinate transformation corresponding to the ultrasonic detection data.

[0078] c2) If there is an obstacle, determining the image stitching angle range according to the ultrasonic detection data.

[0079] In this embodiment, if it is determined that there is an obstacle in the set search angle range, a suitable image stitching angle range is determined based on the ultrasonic detection data.

[0080] The above technical scheme of this embodiment obtains ultrasonic detection data through ultrasonic detection of the vehicle's surrounding environment, and determines whether there are obstacles within the set search angle range based on the ultrasonic detection data. If there are obstacles, the image stitching angle range is determined, thereby avoiding the problem of ghosting and misalignment in the panoramic bird's-eye view of the vehicle caused by the presence of obstacles in the image stitching angle range.

[0081] As a third optional embodiment of the embodiment of the present invention, based on the above embodiment, if an obstacle is detected within the set search angle range of the vehicle surrounding environment, the image stitching angle range is determined, which can be further specifically the following steps:

[0082] a3) determining an image search area corresponding to a set search angle range in each of the bird's-eye view images, and performing image recognition on each of the image search areas.

[0083] The image search area may be understood as an area for determining obstacles.

[0084] In this embodiment, the set search angle range in each overhead image is first determined, and then the corresponding image search area is determined according to the set search angle range. Then, image recognition technology, such as deep learning, support vector machine, etc., is used to perform image recognition on the image search area to determine whether there are obstacles in the image search area.

[0085] As one implementation manner, based on the above optional embodiment, the step of determining the image search area corresponding to the set search angle range in each of the overhead images may be further concretized as follows:

[0086] a31) Taking the vehicle as the center of the bird's-eye view, determine the orientation information of each of the bird's-eye view images relative to the center of the bird's-eye view.

[0087] The top view can be understood as a combined image of each top view image, including a top view image of a single perspective and a top view image of two overlapping perspectives. The orientation information can be understood as the position information in front of, behind, left of, or right of an object.

[0088] In this embodiment, the vehicle position is placed at the center of a bird's-eye view, and the orientation information of each bird's-eye view image relative to the center of the bird's-eye view is determined, that is, whether each bird's-eye view image is located in front of or behind or on the left or right side of the center of the bird's-eye view, to obtain a complete bird's-eye view of the vehicle.

[0089] a32) determining a splicing coordinate system corresponding to the top view according to the orientation information.

[0090] The stitching coordinate system may be understood as a coordinate system used when performing image stitching.

[0091] In this embodiment, the adjacent top-view images of each top-view image in the top view and the splicing coordinate system when the top-view images are spliced ​​in pairs are determined according to the orientation information.

[0092] Exemplarily, the spliced ​​coordinate system can be a four-coordinate system with the four corners of the vehicle body as the origin of the coordinate system, the direction perpendicular to the vehicle body pointing in the opposite direction of the vehicle as the x-axis, and the direction parallel to the vehicle body pointing in the opposite direction of the vehicle as the y-axis.

[0093] a33) determining the coverage area corresponding to the set search angle range in each of the stitching coordinate systems, and obtaining the coverage coordinate points contained in the coverage area.

[0094] The coverage area can be understood as the area surrounded by the boundary of the bird's-eye view in each stitching coordinate system and the setting search angle range; the coverage coordinate points can be understood as the coordinates of all pixel points contained in the coverage area in the bird's-eye view image.

[0095] In this embodiment, a set search angle range is determined in the stitching coordinate system, each coverage area in each overhead image is determined according to the set search angle range, and all coverage coordinate points contained in each coverage area are obtained.

[0096] For example, an ADAS surround view camera is installed at the front, rear, left, and right of the vehicle. The stitched coordinate system is constructed with the four corners of the vehicle as the origin of the coordinate system, the direction perpendicular to the vehicle body pointing to the opposite direction of the vehicle as the x-axis, and the direction parallel to the vehicle body pointing to the opposite direction of the vehicle as the y-axis. The search angle range is set to where degree gap is the set angle value.

[0097] According to the above description, Figure 2 Schematic diagram of the coverage area of ​​an image stitching method provided by an embodiment of the present invention. Figure 2 As shown, the four corners of the vehicle body are respectively used as the origin of the coordinate system (0,0), the x-axis is the 0 degree side, and the y-axis is the 90 degree side, then the search angle range is set The corresponding coverage area in each of the stitching coordinate systems is as follows Figure 2 shown.

[0098] a34) determining the area formed by each of the coverage coordinate points in the corresponding overhead image as the image search area.

[0099] In this embodiment, each coverage coordinate point is mapped to each overhead image, a plurality of regions are formed in each overhead image, and the formed regions are determined as image search regions.

[0100] The above technical solution of this embodiment provides an implementation method for determining the image search area before performing image recognition, so that the image search area can be accurately determined according to the set search angle range, thereby improving the effectiveness of image recognition.

[0101] b3) If there is an obstacle in the image recognition result of the image search area, it is determined that there is an obstacle in the set search angle range, and the image stitching angle range is determined according to the ultrasonic detection data.

[0102] In this embodiment, if there is an obstacle in the image recognition result of the image search area, it is determined that there is an obstacle in the set search angle range. At this time, ultrasonic detection data in the set search angle range is determined, and the image stitching angle range is determined based on the ultrasonic detection data.

[0103] The above technical scheme of this embodiment determines the image search area corresponding to the set search angle range, and performs image recognition on each of the image search areas to determine whether there is an obstacle within the set search angle range. If there is an obstacle, the image stitching angle range is determined based on the ultrasonic detection data, thereby avoiding the problem of ghosting and misalignment in the panoramic bird's-eye view of the vehicle caused by the presence of obstacles in the image stitching angle range.

[0104] As one implementation of the second optional embodiment or the third optional embodiment of the present invention, the step of determining the image search area corresponding to the set search angle range in each of the overhead images may be further implemented as the following steps:

[0105] a4) determining an angle lower limit value and an angle upper limit value according to the ultrasonic detection data.

[0106] The lower angle limit can be understood as the angle value closest to the starting angle value of the obstacle area starting from the 0 degree side in the coordinate system; the upper angle limit can be understood as the angle value closest to the ending angle value of the obstacle area.

[0107] In this embodiment, the lower angle limit value and the upper angle limit value are determined according to the azimuth information of the obstacle contained in the ultrasonic detection data.

[0108] As one implementation manner, the step of determining the angle lower limit value and the angle upper limit value according to the ultrasonic detection data may be further concretized into the following steps:

[0109] a41) Determine the 0 degree side and the 90 degree side corresponding to each of the ultrasonic detection data.

[0110] The 0-degree side can be understood as the straight line that coincides with the x-axis in the two-dimensional plane coordinate system, located in the first quadrant and horizontally to the right. The 90-degree side can be understood as the straight line that is aligned after rotating 90 degrees counterclockwise from the 0-degree side (positive direction of the x-axis), that is, the straight line that coincides with the positive direction (upward) of the y-axis.

[0111] In this embodiment, if an obstacle is detected in at least one set search angle range, the coordinate system corresponding to the ultrasonic detection data in the set search angle range can be determined based on each ultrasonic detection data, and the 0 degree edge and 90 degree edge corresponding to each ultrasonic detection data can be determined based on the coordinate system.

[0112] a42) Starting from the 0 degree edge, determine the minimum angle value without obstacles based on the corresponding ultrasonic detection data.

[0113] Among them, the minimum angle value can be understood as the maximum angle between the 0-degree edge and the starting angle of the obstacle area. For example, when the accuracy requirement for the angle value is 1 degree, there are no obstacles in the range from 0-degree edge to 40-degree edge, and there are obstacles in the range from 0-degree edge to 41-degree edge, then 40 degrees is the minimum angle value; when the accuracy requirement for the angle value is 0.5 degrees, there are no obstacles in the range from 0-degree edge to 40-degree edge, there are no obstacles in the range from 0-degree edge to 40.5-degree edge, and there are obstacles in the range from 0-degree edge to 41-degree edge, then 40.5 degrees is the minimum angle value.

[0114] In this embodiment, starting from the 0 degree edge in each coordinate system, the minimum angle value of the search angle range without obstacles in each coordinate system is determined based on the azimuth coordinate information, occupied range information and accuracy requirement information of the obstacles contained in the corresponding ultrasonic detection data.

[0115] a43) Starting from the 90-degree side, determine the maximum angle value without obstacles based on the corresponding ultrasonic detection data.

[0116] Among them, the maximum angle value can be understood as the minimum angle between the 90-degree edge and the starting angle of the obstacle area. For example, when the accuracy requirement of the angle value is 1 degree, there are no obstacles in the range from the 90-degree edge to the 60-degree edge, and there are obstacles in the range from the 0-degree edge to the 59-degree edge, then 60 degrees is the maximum angle value.

[0117] In this embodiment, starting from the 90-degree side in each coordinate system, the maximum angle value of the search angle range without obstacles in each coordinate system is determined based on the azimuth coordinate information, occupied range information, and accuracy requirement information of the obstacles contained in each ultrasonic detection data.

[0118] a44) Determine the minimum angle value as the angle lower limit value, and determine the maximum angle value as the angle upper limit value.

[0119] In this embodiment, the minimum angle value is determined as the angle lower limit value, and the maximum angle value is determined as the angle upper limit value, that is, the range between the angle lower limit value and the angle upper limit value is the range where obstacles exist.

[0120] The above technical scheme of this embodiment determines the lower and upper angle limits in each coordinate system where obstacles exist in the set search angle range based on the corresponding angle information and obstacle information in the ultrasonic detection data, so as to accurately locate the range where there are no obstacles, thereby providing a basis for the subsequent determination of the image stitching angle range.

[0121] b4) determining the image stitching angle range according to the angle lower limit value and the angle upper limit value in combination with a set angle range selection condition.

[0122] The angle range selection condition may be understood as a series of rules for determining which operation or processing method to take according to the size of the angle range. The angle range may be determined by the difference between two angles or the absolute value of the difference between two angles.

[0123] In this embodiment, the angle lower limit value and the angle upper limit value are determined according to the ultrasonic data, and the angle range of each image stitching is determined in combination with a given angle range selection condition.

[0124] The above technical solution of this embodiment improves the accuracy of selecting the image stitching angle range by determining the image stitching angle range according to the angle lower limit value and the angle upper limit value in combination with the set angle range selection condition.

[0125] As an optional implementation, the image stitching angle range may be determined according to the angle lower limit value and the angle upper limit value in combination with a set angle range selection condition, and further concretized into the following steps:

[0126] b41) If the absolute value of the angle difference between the angle lower limit and 45 degrees is smaller than the absolute value of the angle difference between the angle upper limit and 45 degrees, and the angle difference between the angle lower limit and the set angle threshold is greater than 0 degrees, then the image stitching angle range is constructed based on the angle difference and the angle lower limit.

[0127] The angle threshold can be understood as a preset angle value, which is used to determine the search angle range (that is, the degree gap ), angle range selection conditions and image stitching angle range.

[0128] In this embodiment, the determination of the image stitching angle range in a certain coordinate system is described as an example. First, the angle range selection condition is judged according to the angle lower limit value, angle upper limit value and angle threshold value determined in the coordinate system. If the absolute value of the angle difference between the angle lower limit value and the 45-degree angle is less than the absolute value of the angle difference between the angle upper limit value and the 45-degree angle, and the angle difference between the angle lower limit value and the set angle threshold value is greater than 0 degrees, then the lower limit of the image stitching angle range is the difference between the angle lower limit value and the angle threshold value, and the upper limit of the image stitching angle range is the angle lower limit value.

[0129] b42) If the absolute value of the angle difference between the angle lower limit and 45 degrees is smaller than the absolute value of the angle difference between the angle upper limit and 45 degrees, and the angle difference between the angle lower limit and the set angle threshold is less than or equal to 0 degrees, then the image stitching angle range is constructed based on 0 degrees and the angle lower limit.

[0130] In this embodiment, the determination of the image stitching angle range in a certain coordinate system is still described as an example. If the absolute value of the angle difference between the angle lower limit value and the 45-degree angle is less than the absolute value of the angle difference between the angle upper limit value and the 45-degree angle, and the angle difference between the angle lower limit value and the set angle threshold value is less than or equal to 0 degrees, then the lower limit of the image stitching angle range is 0 degrees, and the upper limit of the image stitching angle range is the angle lower limit value.

[0131] b43) If the absolute value of the angle difference between the angle lower limit and 45 degrees is greater than or equal to the absolute value of the angle difference between the angle upper limit and 45 degrees, and the angle sum of the angle upper limit and the set angle threshold is less than 90 degrees, then the image stitching angle range is constructed based on the angle upper limit and the angle sum.

[0132] In this embodiment, the determination of the image stitching angle range in a certain coordinate system is still described as an example. If the absolute value of the angle difference between the angle lower limit value and the 45-degree angle is greater than or equal to the absolute value of the angle difference between the angle upper limit value and the 45-degree angle, and the angle sum of the angle upper limit value and the set angle threshold is less than 90 degrees, then the lower limit of the image stitching angle range is the angle upper limit value, and the upper limit of the image stitching angle range is the angle sum of the angle upper limit value and the set angle threshold.

[0133] b44) If the absolute value of the angle difference between the angle lower limit and the 45-degree angle is greater than or equal to the absolute value of the angle difference between the angle upper limit and the 45-degree angle, and the sum of the angles of the angle upper limit and the set angle threshold is greater than or equal to 90 degrees, then the image stitching angle range is constructed based on the angle upper limit and the 90-degree angle.

[0134] In this embodiment, the determination of the image stitching angle range in a certain coordinate system is still described as an example. If the absolute value of the angle difference between the angle lower limit value and the 45-degree angle is greater than or equal to the absolute value of the angle difference between the angle upper limit value and the 45-degree angle, and the sum of the angle of the angle upper limit value and the set angle threshold is greater than or equal to 90 degrees, then the lower limit of the image stitching angle range is the angle upper limit value, and the upper limit of the image stitching angle range is 90 degrees.

[0135] The above technical solution of this embodiment determines the image stitching angle range by giving four set angle range selection conditions, thereby improving the accuracy of determining the image stitching angle range, so that each image stitching angle range can avoid obstacles and make the obstacles fully presented in the single-view view, thereby providing a smoother and more natural visual experience. Firstly, by setting the angle range selection condition, the angle range of each image stitching can fully consider the overlapping area of ​​each overhead image; and when the area range of the existing obstacle is large, a larger image stitching angle range is selected for stitching, so that the stitched vehicle panoramic overhead view is smooth and natural; when the area range of the existing obstacle is small, a larger range is selected from the range consisting of the angle upper limit value and the 90-degree angle and the range consisting of the 0-degree angle and the angle lower limit value, and a suitable range smaller than the angle upper limit value and the 90-degree angle, and also smaller than the 0-degree angle and the angle lower limit value is selected from the larger range as the image stitching angle range for stitching, so that the image stitching angle range can ensure the smoothness and naturalness requirements of the vehicle panoramic overhead view, while saving the computing resources required for stitching, and retaining a larger range of single-view images, so as to further improve the clarity and accuracy of the vehicle panoramic overhead view.

[0136] As a fourth optional embodiment of the embodiment of the present invention, on the basis of the above embodiment, each of the top view images may be stitched according to the image stitching angle range to obtain a stitched panoramic top view of the vehicle, which is further specifically the following steps:

[0137] a5) determining image fusion areas of each pair of the overhead images according to the image stitching angle range.

[0138] The image area to be fused can be understood as an overlapping area in two images that needs to be spliced ​​using a fusion technology.

[0139] In this embodiment, each overhead view image is first centered on the vehicle body, and the image is filled according to the direction information of each overhead view relative to the vehicle. Based on the image stitching angle range, the image fusion area of ​​two adjacent overhead view images in each overhead view image can be determined based on information such as pixel point coordinates within the image stitching angle range.

[0140] b5) for each adjacent pairwise overhead view images, forming image fusion content according to the image content in the image to be fused area in the pairwise overhead view images.

[0141] The image fusion content can be understood as the image content after image fusion through image fusion technology.

[0142] In this embodiment, for every two adjacent overhead images, image fusion content can be formed according to the image contents in the image to be fused areas in the pairwise overhead images through image fusion techniques such as pyramid fusion, Poisson fusion, IHS fusion and Alpha fusion.

[0143] Exemplarily, taking the image to be fused area of ​​the left-shot top-view image and the image to be fused area of ​​the front-shot image as examples, the image fusion content can be calculated by the following formula:

[0144] p i =s left ×p i_left +s front ×p i_front ;

[0145] Among them, p i is the fused pixel value of the i-th point within the image stitching angle range; p i_left is the pixel value of the i-th point in the image to be fused area of ​​the left overhead image; p i_front is the pixel value of the i-th point in the image to be fused area of ​​the forward-view image; s left is the proportional coefficient of the pixel value of the i-th point in the image to be fused area relative to the left-view top-view image; s front is the proportional coefficient of the pixel value of the i-th point in the image to be fused area relative to the forward-view image; within the image stitching angle range, from the position closest to the left-view image to the position closest to the forward-view image, s left Changes uniformly from 1 to 0, s front It changes evenly from 0 to 1, that is, the closer the image stitching angle range is to the left-shot top-view image, the smaller the image stitching angle is. left The larger the value, the front The smaller the value, the closer the image stitching angle range is to the forward-looking image. left The smaller the value, the front The larger the value.

[0146] c5) splicing the image fusion contents and the image contents in the non-fusion areas of the bird's-eye view images according to pixel coordinates to form the panoramic bird's-eye view of the vehicle.

[0147] The image non-image fusion area can be understood as an area in the overhead image where image fusion is not required.

[0148] In this embodiment, the formed image fusion contents and the image non-fusion area contents in each overhead view image are spliced ​​according to pixel coordinates to form the vehicle panoramic overhead view.

[0149] The above technical scheme of the present embodiment determines the area to be fused of each image according to the image stitching angle range, adopts image fusion technology to form image fusion content, and stitches the image fusion content and the image content in the non-fusion area of ​​each overhead view image according to pixel coordinates to form the panoramic overhead view of the vehicle, thereby reducing the fusion traces within the image stitching angle range, ensuring the smoothness and fineness of image stitching, and improving the clarity and accuracy of the panoramic overhead view of the vehicle.

[0150] Figure 3 FIG. 1 is a schematic diagram of the structure of an image splicing device provided by an embodiment of the present invention. Figure 3 As shown, the device may include: an image acquisition module 31, a stitching range determination module 32 and a panoramic image determination module 33, wherein:

[0151] An image acquisition module 31 is used to acquire original images acquired by at least one image acquisition device on the vehicle, and determine a top view image of each of the original images;

[0152] A stitching range determination module 32, for determining an image stitching angle range if an obstacle is detected within a set search angle range of the vehicle's surroundings;

[0153] The panoramic image determination module 33 is used to stitch the top view images according to the image stitching angle range to obtain a stitched panoramic top view of the vehicle.

[0154] An image stitching device provided by an embodiment of the present invention acquires original images captured by at least one image acquisition device on a vehicle, and determines a top-view image of each original image; if an obstacle is detected in a set search angle range of the vehicle's surrounding environment, an image stitching angle range is determined; and each top-view image is stitched according to the image stitching angle range to obtain a stitched panoramic top-view of the vehicle. The technical solution of an embodiment of the present invention detects obstacles in a set search angle range, determines an image stitching angle range when an obstacle exists in the set search angle range, and stitches each of the top-view images according to the image stitching angle range to obtain a panoramic top-view of the vehicle, thereby ensuring that each image stitching angle range can accurately avoid obstacles, solving the problem of ghosting and misalignment in the panoramic top-view of the vehicle caused by obstacles in the stitching range, improving the clarity and accuracy of the visual effect, and thereby improving the safety of vehicle driving.

[0155] Furthermore, the image acquisition module 31 can be specifically used for:

[0156] Correcting each of the original images to obtain a corrected image;

[0157] Convert each of the corrected images from a plane coordinate system to a three-dimensional space coordinate system to obtain a first converted image of each of the original images in the three-dimensional space coordinate system;

[0158] Convert each of the first converted images from the three-dimensional space coordinate system to a bird's-eye view coordinate system to obtain a second converted image of each of the first converted images in the bird's-eye view coordinate system;

[0159] Each of the second converted images is determined as the top-view image.

[0160] Furthermore, the splicing range determination module 32 may specifically include:

[0161] In the vehicle surrounding environment with the vehicle body as the center, the four corners of the vehicle body are respectively used as target origins, and ultrasonic detection is performed with each of the target origins as a vertex;

[0162] A detection data judgment unit, used to determine whether there is an obstacle within the set search angle range according to the ultrasonic detection data;

[0163] An image recognition unit, used to determine an image search area corresponding to a set search angle range in each of the overhead images, and perform image recognition on each of the image search areas;

[0164] An angle range determination unit is used to determine the image stitching angle range according to the ultrasonic detection data if there is an obstacle; if there is an obstacle in the image recognition result of the image search area, determine that there is an obstacle in the set search angle range, and determine the image stitching angle range according to the ultrasonic detection data.

[0165] Furthermore, the image recognition unit can be specifically used for:

[0166] Taking the vehicle as the center of the bird's-eye view, determining the orientation information of each bird's-eye view image relative to the center of the bird's-eye view;

[0167] Determine a splicing coordinate system corresponding to the top view according to the orientation information;

[0168] Determine the coverage area corresponding to the set search angle range in each of the stitching coordinate systems, and obtain the coverage coordinate points included in the coverage area;

[0169] The area formed by each of the coverage coordinate points in the corresponding overhead image is determined as the image search area.

[0170] Furthermore, the angle range determination unit may specifically include:

[0171] An angle determination subunit, used to determine an angle lower limit value and an angle upper limit value according to the ultrasonic detection data;

[0172] The range determination subunit is used to determine the image stitching angle range according to the angle lower limit value and the angle upper limit value in combination with the set angle range selection condition.

[0173] Furthermore, the angle determination subunit may be specifically used for:

[0174] Determine the 0 degree side and the 90 degree side corresponding to each of the ultrasonic data;

[0175] Starting from the 0 degree edge, determining a minimum angle value without obstacles based on corresponding ultrasonic data;

[0176] Starting from the 90-degree edge, determining a maximum angle value without obstacles based on corresponding ultrasonic data;

[0177] The minimum angle value is determined as a lower angle limit value, and the maximum angle value is determined as an upper angle limit value.

[0178] Furthermore, the scope determination subunit may be specifically used for:

[0179] If the absolute value of the angle difference between the angle lower limit and the 45-degree angle is less than the absolute value of the angle difference between the angle upper limit and the 45-degree angle, and the angle difference between the angle lower limit and the set angle threshold is greater than 0 degrees, then the image stitching angle range is formed based on the angle difference and the angle lower limit;

[0180] If the absolute value of the angle difference between the angle lower limit and the 45-degree angle is less than the absolute value of the angle difference between the angle upper limit and the 45-degree angle, and the angle difference between the angle lower limit and the set angle threshold is less than or equal to 0 degrees, then the image stitching angle range is formed based on the 0-degree angle and the angle lower limit;

[0181] If the absolute value of the angle difference between the angle lower limit and the 45-degree angle is greater than or equal to the absolute value of the angle difference between the angle upper limit and the 45-degree angle, and the angle sum of the angle upper limit and the set angle threshold is less than 90 degrees, then the image stitching angle range is formed based on the angle upper limit and the angle sum;

[0182] If the absolute value of the angle difference between the angle lower limit and the 45-degree angle is greater than or equal to the absolute value of the angle difference between the angle upper limit and the 45-degree angle, and the sum of the angles of the angle upper limit and the set angle threshold is greater than or equal to 90 degrees, then the image stitching angle range is constructed based on the angle upper limit and the 90-degree angle.

[0183] The panoramic image determination module 33 may be specifically used for:

[0184] According to the image stitching angle range, determining image fusion areas of each pair of the overhead images in each of the overhead images;

[0185] For each adjacent pairwise overhead view image, forming image fusion content according to image content in the image to-be-fused area in the pairwise overhead view image;

[0186] The image fusion contents and the image contents in the non-fusion areas of the overhead images are spliced ​​according to pixel coordinates to form the panoramic overhead view of the vehicle.

[0187] The image stitching device provided in the embodiment of the present invention can execute the image stitching method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0188] Figure 4 A schematic diagram of a vehicle 40 is shown that can be used to implement an embodiment of the present invention. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit implementations of the inventions described and / or claimed herein.

[0189] like Figure 4 As shown, the vehicle 40 includes at least one controller 41, and a memory connected to the at least one controller 41 in communication, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one controller, and the controller 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the vehicle 40 can also be stored. The controller 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0190] A number of components in the vehicle 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the vehicle 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0191] The controller 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the controller 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The controller 41 performs the various methods and processes described above, such as an image stitching method.

[0192] In some embodiments, the image stitching method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the vehicle 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the controller 41, one or more steps of the image stitching method described above may be performed. Alternatively, in other embodiments, the controller 41 may be configured to perform the image stitching method in any other suitable manner (e.g., by means of firmware).

[0193] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable controller, which can be a special purpose or general purpose programmable controller that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0195] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a vehicle having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the vehicle. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0197] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0198] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0199] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0200] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image stitching method, characterized in that: include: Acquire original images captured by at least one image acquisition device on the vehicle, and determine a top view image of each of the original images; If an obstacle is detected within a set search angle range of the vehicle's surroundings, an image stitching angle range is determined; The overhead images are stitched together according to the image stitching angle range to obtain a stitched panoramic overhead view of the vehicle.

2. The method according to claim 1, characterized in that The step of determining the overhead image of each of the original images comprises: Correcting each of the original images to obtain a corrected image; Convert each of the corrected images from a plane coordinate system to a three-dimensional space coordinate system to obtain a first converted image of each of the corrected images in the three-dimensional space coordinate system; Convert each of the first converted images from the three-dimensional space coordinate system to a bird's-eye view coordinate system to obtain a second converted image of each of the first converted images in the bird's-eye view coordinate system; Each of the second converted images is determined as the top-view image.

3. The method according to claim 1, characterized in that: If an obstacle is detected within a set search angle range of the vehicle's surrounding environment, determining the image stitching angle range includes: In the vehicle surrounding environment with the vehicle body as the center, the four corners of the vehicle body are respectively used as target origins, and ultrasonic detection is performed with each of the target origins as a vertex; Determine whether there is an obstacle within the set search angle range according to the ultrasonic detection data; If there is an obstacle, the image stitching angle range is determined according to the ultrasonic detection data.

4. The method according to claim 1, characterized in that If an obstacle is detected within a set search angle range of the vehicle's surrounding environment, determining the image stitching angle range includes: Determining an image search area corresponding to a set search angle range in each of the overhead images, and performing image recognition on each of the image search areas; If there is an obstacle in the image recognition result of the image search area, it is determined that there is an obstacle in the set search angle range, and the image stitching angle range is determined according to the ultrasonic detection data.

5. The method according to claim 4, characterized in that The step of determining the image search area corresponding to the set search angle range in each of the overhead images includes: Taking the vehicle as the center of the bird's-eye view, determining the orientation information of each bird's-eye view image relative to the center of the bird's-eye view; Determine a splicing coordinate system corresponding to the top view according to the orientation information; Determine the coverage area corresponding to the set search angle range in each of the stitching coordinate systems, and obtain the coverage coordinate points included in the coverage area; The area formed by each of the coverage coordinate points in the corresponding overhead image is determined as the image search area.

6. The method according to claim 3 or 4, characterized in that: The determining the image stitching angle range according to the ultrasonic detection data includes: Determine an angle lower limit value and an angle upper limit value according to the ultrasonic detection data; The image stitching angle range is determined according to the angle lower limit value and the angle upper limit value in combination with a set angle range selection condition.

7. The method according to claim 6, characterized in that The step of determining the angle lower limit value and the angle upper limit value according to the ultrasonic detection data comprises: Determine the 0 degree side and the 90 degree side corresponding to each of the ultrasonic data; Starting from the 0 degree edge, determining a minimum angle value without obstacles based on corresponding ultrasonic data; Starting from the 90-degree edge, determining a maximum angle value without obstacles based on corresponding ultrasonic data; The minimum angle value is determined as a lower angle limit value, and the maximum angle value is determined as an upper angle limit value.

8. The method according to claim 6, characterized in that The determining the image stitching angle range according to the angle lower limit value and the angle upper limit value in combination with a set angle range selection condition includes: If the absolute value of the angle difference between the angle lower limit and 45 degrees is smaller than the absolute value of the angle difference between the angle upper limit and 45 degrees, and the angle difference between the angle lower limit and the set angle threshold is greater than 0 degrees, the image stitching angle range is constructed based on the angle difference and the angle lower limit.

9. The method according to claim 6, characterized in that The determining the image stitching angle range according to the angle lower limit value and the angle upper limit value in combination with a set angle range selection condition includes: If the absolute value of the angle difference between the angle lower limit and 45 degrees is smaller than the absolute value of the angle difference between the angle upper limit and 45 degrees, and the angle difference between the angle lower limit and the set angle threshold is less than or equal to 0 degrees, then the image stitching angle range is constructed based on 0 degrees and the angle lower limit.

10. The method according to claim 6, characterized in that The determining the image stitching angle range according to the angle lower limit value and the angle upper limit value in combination with a set angle range selection condition includes: If the absolute value of the angle difference between the angle lower limit and 45 degrees is greater than or equal to the absolute value of the angle difference between the angle upper limit and 45 degrees, and the angle sum of the angle upper limit and the set angle threshold is less than 90 degrees, then the image stitching angle range is constructed based on the angle upper limit and the angle sum.

11. The method according to claim 6, characterized in that The determining the image stitching angle range according to the angle lower limit value and the angle upper limit value in combination with a set angle range selection condition includes: If the absolute value of the angle difference between the angle lower limit and the 45-degree angle is greater than or equal to the absolute value of the angle difference between the angle upper limit and the 45-degree angle, and the sum of the angles of the angle upper limit and the set angle threshold is greater than or equal to 90 degrees, then the image stitching angle range is constructed based on the angle upper limit and the 90-degree angle.

12. The method according to claim 1, characterized in that The step of stitching the top view images according to the image stitching angle range to obtain a stitched panoramic top view of the vehicle includes: According to the image stitching angle range, determining image fusion areas of each pair of the overhead images in each of the overhead images; For each adjacent pairwise overhead view image, forming image fusion content according to image content in the image to-be-fused area in the pairwise overhead view image; The image fusion contents and the image contents in the non-fusion areas of the overhead images are spliced ​​according to pixel coordinates to form the panoramic overhead view of the vehicle.

13. An image stitching device, characterized in that: include: An image acquisition module, used to acquire original images acquired by at least one image acquisition device on the vehicle, and determine a top view image of each of the original images; A stitching range determination module, used to determine the image stitching angle range if an obstacle is detected within a set search angle range of the vehicle's surroundings; The panoramic image determination module is used to stitch the top view images according to the image stitching angle range to obtain a stitched panoramic top view of the vehicle.

14. A vehicle, characterized in that: The vehicle comprises: at least one controller; and a memory communicatively coupled to the at least one controller; The memory stores a computer program executable by the at least one controller, and the computer program is executed by the at least one controller so that the at least one controller can execute the image stitching method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a controller to implement the image stitching method according to any one of claims 1 to 12 when executed.

16. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a controller, implements the image stitching method according to any one of claims 1 to 12.

Images