External parameter calibration method, device and equipment for vehicle-mounted panoramic image

Abstract

The invention relates to the technical field of auxiliary driving, in particular to an external parameter calibration method, device and equipment for a vehicle-mounted panoramic image. The invention discloses a vehicle-mounted panoramic image external parameter calibration method, and the method comprises the steps: obtaining a frame of original image of a target object at a preset time interval in the vertical advancing process of a vehicle based on the target object; determining a first number of calibration images from the plurality of frames of obtained original images; respectively determining a first angular point coordinate of a target angular point of the target object in an image coordinate system and a second angular point coordinate of the target angular point of the target object in a world coordinate system in the calibration image; and executing external parameter calibration according to the first angular point coordinate and the second angular point coordinate in each calibration image. The vehicle-mounted panoramic image external parameter calibration method provided by the invention has the advantages of low site requirement, more flexible dynamic calibration and higher accuracy, and is suitable for dynamic calibration of external parameters of panoramic images after sales of platforms of different vehicle types.

Description

Technical Field

[0001] The present application relates to the field of assisted driving technology, and in particular to a method, device and equipment for calibrating external parameters of a vehicle-mounted panoramic image. Background Art

[0002] In-vehicle panoramic imaging technology is a technology that displays a 360-degree bird's-eye view of the vehicle in the vehicle system. Multiple cameras deployed on the vehicle body simultaneously capture images of the vehicle's surroundings, and stitch the captured images together through image algorithms to present a panoramic bird's-eye view of the vehicle's surroundings, improving the driver's ability to perceive the surrounding environment. In order to implement the image processing algorithm for panoramic images and accurately complete the stitching of multiple images, it is necessary to calibrate the external parameters of the camera.

[0003] When calibrating the camera's external parameters, the traditional external parameter calibration method needs to be performed in a fixed calibration site. At the same time, the site space, target placement and ambient light requirements are high. Considering the site and target costs, it is not conducive to the external parameter calibration of vehicle-mounted panoramic images. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and equipment for calibrating external parameters of vehicle-mounted panoramic images, which are used to solve the problem that the external parameter calibration method in the prior art has high requirements on site space, target placement and ambient light, and considering the site and target costs, it is not conducive to the execution of external parameter calibration of vehicle-mounted panoramic images.

[0005] In a first aspect, an embodiment of the present invention provides an external parameter calibration method for a vehicle-mounted panoramic image, characterized in that the method comprises: When the vehicle moves vertically based on the target object, a frame of original image of the target object is acquired at a preset time interval; Determining a first number of calibration images from the acquired frames of original images; In the calibration image, respectively, determining the first corner point coordinates of the target corner point of the target object in the image coordinate system and the second corner point coordinates in the world coordinate system; External parameter calibration is performed according to the first corner point coordinates and the second corner point coordinates in each of the calibration images.

[0006] Optionally, determining the coordinates of a first corner point of a target corner point of the target object in an image coordinate system includes: Establishing an image coordinate system based on the calibration image; Extracting a first edge expansion map of the target object from the calibration image by using an image processing algorithm, wherein the first edge expansion map is used to represent an edge contour of the target object; In the first edge expansion map, the vertex coordinates of each vertex of the target object in the image coordinate system are obtained by using a vertex extraction algorithm; Determining a second number of target corner points from all vertices of the target object; The vertex coordinates of each of the target corner points are determined as the first corner point coordinates.

[0007] Optionally, determining the coordinates of a second corner point of the target corner point of the target object in the world coordinate system includes: Establishing a world coordinate system based on the center of the rear axle of the vehicle, wherein the direction perpendicular to the rear axle of the vehicle is the X-axis, and the direction of the rear axle of the vehicle is the Y-axis; splicing original images acquired when the front of the vehicle and the position of the target object coincide with each other through universal external references to obtain a universal bird's-eye view of the vehicle; In the general bird's-eye view, determining a second number of target corner points from all vertices of the target object; Determine the Y-axis coordinate of each target corner point according to the positional relationship between each target corner point in the bird's-eye view and a preset calculation coefficient scale; Determine the X-axis coordinate of each target corner point according to the acquired original image of each frame, the vehicle speed when acquiring the original image of each frame, and the first length of the vehicle, wherein the first length of the vehicle is the sum of the vehicle wheelbase and the front overhang length; The coordinates of the second corner point are determined according to the Y-axis coordinates and the X-axis coordinates of each of the target corner points.

[0008] Optionally, determining the Y-axis coordinate of each target corner point according to the positional relationship between the target corner points in the bird's-eye view and a preset calculation coefficient scale includes: Acquire a second edge expansion map of the target object from the general bird's-eye view map by using an image processing algorithm, wherein the second edge expansion map is used to represent the contour of the target object; In the second edge expansion map, determining the positional relationship of each of the target corner points, wherein the positional relationship includes the relative positional relationship between each of the target corner points and the X-axis of the world coordinate system; Determine the Y-axis coordinate of each of the target corner points according to the positional relationship between the target corner points and a preset calculation coefficient scale; The calculation coefficient scale is used to represent the corresponding relationship between pixels in the panoramic image and the length in the real world.

[0009] Optionally, determining the X-axis coordinate of each target corner point according to each frame of the original image acquired, the vehicle speed when each frame of the original image is acquired, and the first length of the vehicle includes: Determining a first distance traveled by the vehicle when acquiring the calibration image according to the number of first images of the original image acquired before acquiring the calibration image and the real-time speed of the vehicle when each of the original images is acquired; When it is determined that the front of the vehicle coincides with the position of the target object, determining a second distance traveled when the front of the vehicle coincides with the position of the target object according to the number of second images of the acquired original images and the real-time speed of the vehicle when each of the original images is acquired; The X-axis coordinate of each of the target corner points is determined according to the first distance, the second distance and the first length.

[0010] Optionally, determining that the front of the vehicle coincides with the position of the target object includes: Obtaining the perimeter of the target object in each frame of the original image; The time at which the frame of the original image with the longest circumference of the target object is acquired among the frames of the original image is determined as the time at which the front of the vehicle coincides with the position of the target object.

[0011] Optionally, determining a target corner point of the target object includes: According to the distance between each vertex in the target object and the optical axis of the camera, a second number of the target corner points are determined from all the vertices of the target object.

[0012] Optionally, the image processing algorithm includes: a mask algorithm, a grayscale algorithm, a dilation algorithm, and an edge detection algorithm; The vertex extraction algorithm includes: a contour detection algorithm and a polygon fitting algorithm.

[0013] In a second aspect, an embodiment of the present invention provides an external parameter calibration device for a vehicle-mounted panoramic image, the device comprising: An acquisition module, which acquires a frame of original image of the target object at a preset time interval during the process of the vehicle moving vertically based on the target object; A first determination module determines a first number of calibration images from the acquired frames of original images; A second determination module determines, in the calibration image, first corner point coordinates of a target corner point of the target object in the image coordinate system and second corner point coordinates in the world coordinate system; The calibration module performs external parameter calibration according to the first corner point coordinates and the second corner point coordinates in each calibration image.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including: at least one processor; and at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the method as described in any one of the first aspects.

[0015] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any method described in the first aspect.

[0016] In an embodiment of the present invention, when a vehicle dynamically drives past a target object, an original image of the target object is obtained and processed through an image recognition algorithm to determine the first corner point coordinates of the target corner point in the image coordinate system and the second corner point coordinates in the world coordinate system, thereby realizing external parameter calibration of a panoramic image. This method has low requirements on the site, and the dynamic calibration is more flexible and more accurate, ensuring that objects in the bird's-eye view of the panoramic image appear in the correct proportion and direction. This method is suitable for after-sales dynamic calibration of external parameters of panoramic images on different vehicle platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 The figure is a flow chart of an external parameter calibration method for a vehicle-mounted panoramic image provided by an embodiment of the present invention; Figure 2 FIG. 1 is a schematic diagram of an external parameter calibration scene of a vehicle-mounted panoramic image provided by an embodiment of the present invention; Figure 3 FIG. 1 is a schematic diagram of a calibration image provided by an embodiment of the present invention; Figure 4 Shown is a schematic diagram of a first edge expansion map provided by an embodiment of the present invention; Figure 5 Shown is a schematic diagram of a general bird's-eye view provided by an embodiment of the present invention; Figure 6 Shown is a schematic diagram of a second edge expansion map provided by an embodiment of the present invention; Figure 7FIG. 1 is a schematic diagram of another external parameter calibration scene of a vehicle-mounted panoramic image provided by an embodiment of the present invention; Figure 8 FIG. 1 is a schematic diagram of the structure of an external parameter calibration device for vehicle-mounted panoramic images provided by an embodiment of the present invention; Fig. 9 Shown is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0020] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0021] like Figure 1 FIG. 1 is a flow chart of an external parameter calibration method for a vehicle-mounted panoramic image provided by an embodiment of the present invention. Figure 1 The specific steps of the method include: S101, when the vehicle is moving vertically based on the target object, a frame of original image of the target object is acquired at every preset time interval.

[0022] Specifically, the vehicle needs to align the front of the vehicle parallel to the target object and move in a straight line perpendicular to the target object. During the moving process, a frame of the original image of the target object needs to be acquired through the camera at every preset time interval, and the vehicle speed at that moment is recorded accordingly. Generally, the preset time is set to 0.1 seconds.

[0023] In a specific embodiment, the zebra crossing area of ​​the city road can be used as the target object, that is, the front of the vehicle is controlled to be parallel to the center area of ​​the zebra crossing and move forward, and the original image of the zebra crossing is obtained through the camera at each preset time interval.

[0024] S102: Determine a first number of calibration images from the acquired frames of original images.

[0025] S103, determining the first corner point coordinates of the target corner point of the target object in the image coordinate system and the second corner point coordinates in the world coordinate system in the calibration image.

[0026] Specifically, when determining the coordinates of the first corner point of the target object in the image coordinate system, it is necessary to establish an image coordinate system in the determined calibration image. Generally, the upper left corner of the calibration image is taken as the origin, the right is the X axis of the image coordinate system, and the downward is the Y axis of the image coordinate system.

[0027] The calibration image is processed by an image processing algorithm to process the calibration image into a first expansion image of the target object.

[0028] Among them, the image processing algorithm includes a mask algorithm, a grayscale algorithm, a dilation algorithm and an edge detection algorithm. The mask algorithm is used to remove the area other than the target object in the calibration image to obtain a mask view, eliminating most of the environmental interference; the grayscale algorithm is used to continue to process the mask view to obtain a grayscale image with only brightness information, avoiding the interference that may be caused by color information and providing sufficient information for subsequent edge detection; the grayscale image is processed by the dilation algorithm to obtain an expansion image, so as to expand the edge jaggedness and internal gaps of the target object into a whole; the expansion image is processed by the edge detection algorithm to extract the edge of the target object and obtain an edge image composed only of edges; the edge image is processed by the dilation algorithm again to make the frame line of the edge image thicker and reduce the value error of the subsequent vertex extraction algorithm to obtain the first edge dilation image. The first edge dilation image is used to represent the edge contour of the target object.

[0029] In the first edge expansion map, vertex coordinates of each vertex of the target object in the image coordinate system are obtained by vertex extraction algorithm. A second number of target corner points are determined from all vertices of the target object, and the vertex coordinates of the determined target corner points are determined as first corner point coordinates.

[0030] Among them, the vertex extraction algorithm specifically includes a contour detection algorithm and a polygon fitting algorithm.

[0031] When determining the target corner points from all vertices, a second number of target corner points are determined from all vertices according to the distance between each vertex in the target object and the optical axis of the camera. That is, the second number of vertices closest to the optical axis of the camera are determined as target corner points.

[0032] When determining the coordinates of the second corner point of the target corner point of the target object in the world coordinate system, it is necessary to establish a world coordinate system based on the center of the rear axle of the vehicle. The center position of the rear axle of the vehicle is determined as the origin of the world coordinate system, the direction perpendicular to the rear axle of the vehicle is the X-axis, the direction of the rear axle of the vehicle is the Y-axis, and the direction perpendicular to the plane formed by the X-axis and the Y-axis is the Z-axis.

[0033] When the front of the vehicle coincides with the position of the target object during the vehicle's movement, the original images acquired at this time are stitched together using universal external parameters to obtain a universal bird's-eye view of the vehicle. In the on-board panoramic imaging system, a universal external parameter corresponding to the standard installation of the camera of the vehicle model is usually written. The universal external parameter can be used to output an inaccurate universal bird's-eye view through panoramic image stitching when the vehicle is not calibrated or the calibration parameters are lost.

[0034] In the general bird's-eye view, a second number of target corner points are also determined from all vertices of the target object, and the target corner points determined in the general bird's-eye view should be the same as the target corner points determined in the calibration image.

[0035] In the general bird's-eye view, the second number of target corner points is also determined from all the vertices according to the distance between each vertex in the target object and the optical axis of the camera. Since the center area of ​​the target object is close to the center of the optical axis of the camera in the general bird's-eye view, the distortion is small, and the difference between the external parameters after the general external parameters and the actual calibration is not much. Therefore, according to the distance between each vertex of each target object in the general bird's-eye view and the optical axis of the camera, the second number of target corner points is determined from all the vertices, that is, the second number of vertices closest to the optical axis of the camera are determined as the target corner points.

[0036] According to the positional relationship of each target corner point in the bird's-eye view and the preset calculation coefficient scale, the Y-axis coordinate of each target corner point is determined.

[0037] The X-axis coordinates of each target corner point are determined based on each frame of the original image obtained, the vehicle speed when each frame of the original image is obtained, and the first length of the vehicle, wherein the first length of the vehicle is the sum of the vehicle wheelbase and the front suspension length.

[0038] In the ideal case where the ground is flat, the Z-axis coordinate of the world coordinate system is determined to be 0.

[0039] The coordinates of the second corner point are determined according to the Y-axis coordinates and the X-axis coordinates of each target corner point.

[0040] S104, performing extrinsic calibration according to the first corner point coordinates and the second corner point coordinates in each calibration image.

[0041] Specifically, the first corner point coordinates and the second corner point coordinates of the obtained target corner point are used as a set of target corner point coordinates. The multiple sets of target corner point coordinates obtained under multiple frames of calibration images are respectively imported into the preset calibration algorithm, and the camera external parameters are calculated in combination with the read camera internal parameters, thereby realizing the external parameter calibration of the vehicle-mounted panoramic image.

[0042] In an embodiment of the present invention, when a vehicle dynamically drives past a target object, an original image of the target object is obtained and processed through an image recognition algorithm to determine the first corner point coordinates of the target corner point in the image coordinate system and the second corner point coordinates in the world coordinate system, thereby realizing external parameter calibration of a panoramic image. This method has low requirements on the site, and the dynamic calibration is more flexible and more accurate, ensuring that objects in the bird's-eye view of the panoramic image appear in the correct proportion and direction. This method is suitable for after-sales dynamic calibration of external parameters of panoramic images on different vehicle platforms.

[0043] Optionally, in some embodiments, when executing S103, determining the Y-axis coordinate of each target corner point according to the positional relationship of each target corner point in the bird's-eye view and a preset calculation coefficient scale specifically includes: The general bird's-eye view image is processed by an image processing algorithm to be processed into a second edge expansion image for representing the contour of the target object.

[0044] In the second edge expansion map, the positional relationship of each target corner point is determined, that is, the relative positional relationship between each target corner point is determined, and accordingly, the relative positional relationship between each target corner point and the X-axis of the world coordinate system is determined.

[0045] According to the relative position relationship between each target corner point and the preset calculation coefficient, the distance between each target corner point and the X axis of the world coordinate system is converted from the pixel representation in the image to the distance representation in the real world, so as to determine the Y axis coordinate of each target corner point. Among them, the calculation coefficient scale is used to represent the corresponding relationship between the pixel in the panoramic image and the length in the real world, that is, the calculation coefficient scale can be used to realize the mapping of a distance in the panoramic image from the pixel in the image to the distance in the real world.

[0046] Optionally, in some embodiments, when executing S103, determining the X-axis coordinate of each target corner point according to each frame of the original image acquired, the vehicle speed when each frame of the original image is acquired, and the first length of the vehicle specifically includes: According to a first number of original images acquired before the calibration image is acquired and the real-time speed of the vehicle when each original image is acquired, a first distance traveled by the vehicle since the start of calibration, that is, since the vehicle starts driving, to the time when the calibration image is acquired is determined.

[0047] Among them, since the preset time interval of the original images is determined, and the preset time interval is short and no significant vehicle speed change occurs, the vehicle's travel distance within the preset time interval can be determined according to the vehicle speed when a frame of the original image is collected, that is, the vehicle's travel distance during the period between the collection of two consecutive frames of original images is determined. According to the first number of original images obtained, the vehicle's travel distances within all time intervals are summed to determine the first distance traveled by the vehicle when the calibration image is obtained.

[0048] The calculation formula of the first distance is: Wherein, S1 is the first distance, n1 is the first number of original images acquired, dt is the preset time interval, and v is the real-time speed of the vehicle.

[0049] When it is determined that the front of the vehicle coincides with the position of the target object, the second distance traveled by the vehicle since the start of calibration to the time when it is determined that the front of the vehicle coincides with the position of the target object is determined based on the number of second images of the original image acquired at this time and the real-time speed of the vehicle when each original image is acquired.

[0050] The calculation formula of the second distance is: Wherein, S2 is the second distance, n2 is the second number of original images acquired, dt is the preset time interval, and v is the real-time speed of the vehicle.

[0051] After the first distance, the second distance and the first length are determined respectively, the X-axis coordinates of each target corner point are determined accordingly. That is, the distance between the rear axle of the vehicle (the Y-axis of the world coordinate system) and the target object is represented by the sum of the second distance subtracted from the first distance and the first length, that is, the X-axis coordinates of each target corner point in the target object.

[0052] When executing the above S103 and calculating the second distance, it is also necessary to determine whether the front of the vehicle coincides with the position of the target object based on the collected original image.

[0053] Specifically, each frame of the acquired original image is processed by an image processing algorithm to obtain an edge expansion map that is convenient for identifying the boundary of the target object. In each frame of the edge expansion map, the perimeter of the target object is determined.

[0054] Since the vehicle is moving perpendicular to the target object, based on the principle that the closer the target object is, the smaller the target object is. As the vehicle gets closer to the target object, the circumference of the target object becomes larger. When the front of the vehicle just overlaps with the position of the target object, the circumference of the target object reaches its maximum value. As the vehicle continues to move forward, the circumference of the target object becomes smaller because the target object is partially covered by the front of the vehicle.

[0055] Therefore, the acquisition time of the original image frame with the longest circumference of the target object among the original image frames is determined as the time when the front of the vehicle coincides with the position of the target object, and the second distance is calculated.

[0056] The following is a specific embodiment provided by the present invention, which takes a zebra crossing as an example to illustrate the external parameter calibration method of the above-mentioned vehicle-mounted panoramic image.

[0057] like Figure 2 FIG. 1 is a schematic diagram of a scene for extrinsic calibration of a vehicle-mounted panoramic image provided by an embodiment of the present invention. Figure 2, the vehicle will align its front end parallel to the center of the zebra crossing and move forward at a speed of v. A world coordinate system is established based on the center position of the vehicle's rear axle, with the vertical direction of the vehicle's rear axle as the X-axis and the direction of the vehicle's rear axle as the Y-axis. The preset time interval is set to 0.1 seconds, that is, during the vehicle's driving process, the zebra crossing is photographed once through the camera every 0.1 seconds to obtain a frame of the original image of the zebra crossing.

[0058] According to the distance from the zebra crossing, three original images are determined from the acquired frames of original images as calibration images. Figure 3 FIG. 1 is a schematic diagram of a calibration image provided by an embodiment of the present invention. Figure 3 , is one of the calibration images determined. Figure 3 The calibration image shown is processed to obtain the first corner point coordinates of the target corner point of the zebra crossing in the image coordinate system and the second corner point coordinates in the world coordinate system as an example for explanation.

[0059] When determining the coordinates of the first corner point, you need to Figure 3 The image coordinate system is established in . The image coordinate system is established with the upper left corner of the calibration image as the origin, the right as the X axis, and the downward as the Y axis. The mask algorithm, grayscale image processing algorithm, dilation algorithm, and edge detection algorithm are respectively executed on the calibration image to obtain the following image: Figure 4 The first edge expansion diagram is shown in Figure 1. Figure 4 , which is a schematic diagram of a first edge expansion map provided by an embodiment of the present invention.

[0060] In the first edge expansion graph, multiple vertex extraction algorithms such as contour detection algorithm and polygon fitting algorithm are executed to determine the vertex coordinates of each zebra crossing in the image coordinate system. According to the distance between each vertex and the camera optical axis, the 6 vertices closest to the camera optical axis are selected. Figure 4 The p1, p2, p3, p4, p5, and p6 shown are determined as target corner points. In the embodiment of the present invention, the six vertices close to one side of the vehicle in the three quadrilaterals with the longest perimeters are determined as target corner points. The vertex coordinates of the six target corner points are determined in advance as the first corner point coordinates.

[0061] When determining the coordinates of the second corner point, the original image taken when entering the zebra crossing is spliced ​​using the universal external reference to obtain a universal bird's-eye view of the vehicle. Figure 5 FIG. 1 is a schematic diagram of a general bird's-eye view provided by an embodiment of the present invention. Figure 5In the general bird's-eye view, the left and right areas of the zebra crossing are far away from the center of the camera optical axis, so the distortion is large; while the center area of ​​the zebra crossing is close to the center of the camera optical axis, so the distortion is small. Therefore, the three zebra crossings in the middle are considered accurate, and the six vertices of the three zebra crossings close to the vehicle side are determined as the target corner points, which correspond to Figure 4 The target corner points p1-p6 are shown.

[0062] Specifically, when calculating the Y-axis coordinate of the target corner point, Figure 5 The general bird's-eye view shown in the figure is used to perform image processing algorithms to obtain a second edge map. Figure 6 FIG. 1 is a schematic diagram of a second edge expansion diagram provided by an embodiment of the present invention. Figure 6 , the positional relationship between each target corner point is determined in the second edge expansion map.

[0063] Since the width of each zebra crossing is known to be d1=d3=d5, and the distance interval between each zebra crossing is known to be d2=d4, the Y-axis coordinates of the six target corner points can be determined according to the X-axis of the world coordinate system. The world coordinate system Y-axis coordinate corresponding to the target corner point p1 is d1+d2+(d3 / 2); the world coordinate system Y-axis coordinate corresponding to the target corner point p2 is d2+(d3 / 2); the world coordinate system Y-axis coordinate corresponding to the target corner point p3 is (d3 / 2); the world coordinate system Y-axis coordinate corresponding to the target corner point p4 is (-d3 / 2); the world coordinate system Y-axis coordinate corresponding to the target corner point p5 is –[(d3 / 2)+d4]; the world coordinate system Y-axis coordinate corresponding to the target corner point p6 is –[(d3 / 2)+d4+d5].

[0064] Furthermore, the pixel height of the panoramic image bird's-eye view displayed on the car computer is defined as 846pix, and the corresponding field of view length displayed in the real world is 9.75m. The calculation coefficient scale=9.75 / 846=0.011525m / pix, that is, each pixel on the panoramic image bird's-eye view represents a distance of 0.011525 meters in the real world.

[0065] according to Figure 6 The zebra crossing width is 33pix, the zebra crossing distance is 52pix, and the calculation coefficient scale is used to convert the length of each Y-axis coordinate into the length in the real world, completing the calculation of the Y-axis coordinate of each target corner point.

[0066] like Figure 7 FIG. 1 is a schematic diagram of another scene of extrinsic calibration of a vehicle-mounted panoramic image provided by an embodiment of the present invention. When calculating the X-axis coordinate of the target corner point, Figure 7As shown, when the front of the vehicle just reaches the zebra crossing, the corresponding X-axis coordinates of the zebra crossing are all the first length, that is, the sum of the vehicle wheelbase of 2.7 meters and the front overhang length of 0.75 meters, which is 3.45 meters in total.

[0067] When the front of the vehicle just reaches the zebra crossing, the second distance traveled by the vehicle is , the vehicle obtains Figure 3 When the calibration image is shown, the first distance traveled by the vehicle is So we can determine the Figure 3 In the calibration image shown, the X-axis coordinates of the target corner points p1-p6 are all 3.450+(S2–S1).

[0068] Based on the Y-axis coordinates and X-axis coordinates of the target corner points p1-p6 determined above, the coordinates of the second corner points of the six target corner points in the world coordinate system are determined. For example, the coordinates of point p1 in the world coordinate system are (3.450+(S2–S1), d1+d2+(d3 / 2), 0), and the coordinates of the other target corner points in the world coordinate system are similar.

[0069] By the same method, the image coordinates and world coordinates of the target corner points in the other two calibration images are obtained, and 18 image coordinates and world coordinates are obtained. The 18 image coordinates, 18 world coordinates and the read internal parameters are input into the calibration external parameter algorithm to obtain the calibration external parameters. The cameras of the other three perspectives are processed by the same method to obtain the corresponding calibration external acquisition and store them locally, completing the external parameter calibration of the vehicle-mounted panoramic image.

[0070] Corresponding to the above-mentioned extrinsic parameter calibration method for vehicle-mounted panoramic images, the present application embodiment further provides an extrinsic parameter calibration device for vehicle-mounted panoramic images. Figure 8 , which is a structural diagram of an external parameter calibration device for a vehicle-mounted panoramic image provided in an embodiment of the present application. The external parameter calibration device for a vehicle-mounted panoramic image may include: an acquisition module 801, a first determination module 802, a second determination module 803 and a calibration module 804.

[0071] The acquisition module 801 acquires a frame of the original image of the target object at every preset time interval during the process of the vehicle moving vertically based on the target object.

[0072] The first determination module 802 determines a first number of calibration images from the acquired frames of original images.

[0073] The second determination module 803 determines, in the calibration image, first corner point coordinates of the target corner point of the target object in the image coordinate system and second corner point coordinates in the world coordinate system.

[0074] The calibration module 804 performs external parameter calibration according to the first corner point coordinates and the second corner point coordinates in each calibration image.

[0075] Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present specification. Fig. 9 As shown, the electronic device may include at least one processor; and at least one memory communicatively connected to the processing unit, wherein: the memory stores program instructions that can be executed by the processing unit, and the processor calls the program instructions to execute the external parameter calibration method of the vehicle-mounted panoramic image provided in this embodiment.

[0076] The electronic device can be a device capable of intelligently communicating with a user, such as a cloud server, and the specific form of the electronic device is not limited in the embodiments of this specification. It can be understood that the electronic device here is the machine mentioned in the method embodiment.

[0077] Fig. 9 A block diagram of an exemplary electronic device suitable for implementing the embodiments of the present description is shown. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of this specification.

[0078] like Fig. 9 As shown, the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors 910, a communication interface 920, a memory 930, and a communication bus 940 connecting different system components (including the memory 930, the communication interface 920 and the processor 910).

[0079] Communication bus 940 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnection (PCI) bus.

[0080] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0081] The memory 930 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 930 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of each embodiment of the present specification.

[0082] A program / utility having a set (at least one) of program modules may be stored in memory 930, such program modules including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described in this specification.

[0083] The processor 910 executes various functional applications and data processing by running the programs stored in the memory 930, such as implementing the external parameter calibration method of the vehicle-mounted panoramic image provided in the embodiment shown in this specification.

[0084] An embodiment of the present specification provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the external parameter calibration method of the vehicle-mounted panoramic image provided by the embodiment shown in the present specification.

[0085] The above-mentioned non-transitory computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ReadOnly Memory; hereinafter referred to as: ROM), an erasable programmable read-only memory (Erasable Programmable ReadOnly Memory; hereinafter referred to as: EPROM) or flash memory, optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0086] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0087] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0088] Computer program code for performing the operations of this specification may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0089] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0090] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this specification, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0091] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of this specification includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of this specification belong.

[0092] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0093] It should be noted that the terminals involved in the embodiments of this specification may include but are not limited to personal computers (Personal Computer; hereinafter referred to as: PC), personal digital assistants (Personal Digital Assistant; hereinafter referred to as: PDA), wireless handheld devices, tablet computers (Tablet Computer), mobile phones, MP3 players, MP4 players, etc.

[0094] In the embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0095] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0096] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the method described in each embodiment of this specification.

[0097] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A method for calibrating external parameters of vehicle-mounted panoramic images, characterized in that: The method comprises: When the vehicle moves vertically based on the target object, a frame of original image of the target object is acquired at a preset time interval; Determining a first number of calibration images from the acquired frames of original images; In the calibration image, respectively, determining the first corner point coordinates of the target corner point of the target object in the image coordinate system and the second corner point coordinates in the world coordinate system; External parameter calibration is performed according to the first corner point coordinates and the second corner point coordinates in each of the calibration images.

2. The method according to claim 1, characterized in that Determining the coordinates of a first corner point of a target corner point of the target object in an image coordinate system includes: Establishing an image coordinate system based on the calibration image; Extracting a first edge expansion map of the target object from the calibration image by using an image processing algorithm, wherein the first edge expansion map is used to represent an edge contour of the target object; In the first edge expansion map, the vertex coordinates of each vertex of the target object in the image coordinate system are obtained by using a vertex extraction algorithm; Determining a second number of target corner points from all vertices of the target object; The vertex coordinates of each of the target corner points are determined as the first corner point coordinates.

3. The method according to claim 1, characterized in that Determining the coordinates of a second corner point of the target corner point of the target object in the world coordinate system includes: Establishing a world coordinate system based on the center of the rear axle of the vehicle, wherein the direction perpendicular to the rear axle of the vehicle is the X-axis, and the direction of the rear axle of the vehicle is the Y-axis; splicing original images acquired when the front of the vehicle and the position of the target object coincide with each other through universal external references to obtain a universal bird's-eye view of the vehicle; In the general bird's-eye view, determining a second number of target corner points from all vertices of the target object; Determine the Y-axis coordinate of each target corner point according to the positional relationship between each target corner point in the bird's-eye view and a preset calculation coefficient scale; Determine the X-axis coordinate of each target corner point according to the acquired original image of each frame, the vehicle speed when acquiring the original image of each frame, and the first length of the vehicle, wherein the first length of the vehicle is the sum of the vehicle wheelbase and the front overhang length; The coordinates of the second corner point are determined according to the Y-axis coordinates and the X-axis coordinates of each of the target corner points.

4. The method according to claim 3, characterized in that The step of determining the Y-axis coordinate of each target corner point according to the positional relationship between the target corner points in the bird's-eye view and a preset calculation coefficient scale includes: Acquire a second edge expansion map of the target object from the general bird's-eye view map by using an image processing algorithm, wherein the second edge expansion map is used to represent the contour of the target object; In the second edge expansion map, determining the positional relationship of each of the target corner points, wherein the positional relationship includes the relative positional relationship between each of the target corner points and the X-axis of the world coordinate system; Determine the Y-axis coordinate of each of the target corner points according to the positional relationship between the target corner points and a preset calculation coefficient scale; The calculation coefficient scale is used to represent the corresponding relationship between pixels in the panoramic image and the length in the real world.

5. The method according to claim 3, characterized in that: Determining the X-axis coordinate of each target corner point according to the acquired original image of each frame, the vehicle speed when acquiring the original image of each frame, and the first length of the vehicle includes: Determining a first distance traveled by the vehicle when acquiring the calibration image according to the number of first images of the original image acquired before acquiring the calibration image and the real-time speed of the vehicle when each of the original images is acquired; When it is determined that the front of the vehicle coincides with the position of the target object, determining a second distance traveled when the front of the vehicle coincides with the position of the target object according to the number of second images of the acquired original images and the real-time speed of the vehicle when each of the original images is acquired; The X-axis coordinate of each of the target corner points is determined according to the first distance, the second distance and the first length.

6. The method according to claim 3, characterized in that The determining that the front of the vehicle coincides with the position of the target object includes: Obtaining the perimeter of the target object in each frame of the original image; The time at which the frame of the original image with the longest circumference of the target object is acquired among the frames of the original image is determined as the time at which the front of the vehicle coincides with the position of the target object.

7. The method according to any one of claims 2 to 3, characterized in that: Determining a target corner point of the target object includes: According to the distance between each vertex in the target object and the optical axis of the camera, a second number of the target corner points are determined from all the vertices of the target object.

8. The method according to any one of claims 2 to 4, characterized in that: The image processing algorithms include: mask algorithm, grayscale algorithm, expansion algorithm, edge detection algorithm; The vertex extraction algorithm includes: a contour detection algorithm and a polygon fitting algorithm.

9. An external parameter calibration device for vehicle-mounted panoramic images, characterized in that: The device comprises: An acquisition module, which acquires a frame of original image of the target object at a preset time interval during the process of the vehicle moving vertically based on the target object; A first determination module determines a first number of calibration images from the acquired frames of original images; A second determination module determines, in the calibration image, first corner point coordinates of a target corner point of the target object in the image coordinate system and second corner point coordinates in the world coordinate system; The calibration module performs external parameter calibration according to the first corner point coordinates and the second corner point coordinates in each calibration image.

10. An electronic device, characterized in that: include: at least one processor; as well as at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 8 by calling the program instructions.

11. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images