Image acquisition device pose calibration method, device, apparatus and storage medium

Abstract

The application provides an image acquisition device posture calibration method and device, equipment and a storage medium. The method comprises the following steps: acquiring an environment image comprising a plurality of structural lines; determining a plurality of intersection points formed by a plurality of reference line segments and intersection position information; projecting the intersection points to a preset Gaussian sphere to obtain a projection point; configuring a plurality of matching point pairs of the projection point; determining an optimal Manhattan world assumption; determining Manhattan coordinate axis attributes according to the position relationship between the Manhattan projection point, the Manhattan first matching point, the Manhattan second matching point and each reference line segment in the optimal Manhattan world assumption; obtaining a Manhattan world coordinate; determining a rotation matrix of the image acquisition device to the Manhattan world based on the coordinate and a preset image acquisition device world coordinate; and calibrating the posture of the image acquisition device. The rotation matrix between the Manhattan world and the image acquisition device can be estimated based on a single image to calibrate the posture of the image acquisition device, which is simple, accurate and fast.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of vehicle technology, and in particular to a method, apparatus, device and storage medium for calibrating the posture of an image acquisition device. Background Art

[0002] Smart driving vehicles are equipped with a variety of sensors, such as lidar, cameras, millimeter-wave radar, and ultrasonic radar. Among these sensors, cameras are one of the most important, and the system relies heavily on them to abstract and model information about the surrounding environment.

[0003] In camera applications, camera extrinsic calibration is a critical step. The accuracy of the calibration results and the stability of the algorithm directly impact the accuracy of the camera's results. The position and attitude of the camera center relative to the vehicle center are called the camera-to-vehicle extrinsic parameters. During vehicle use, the camera's attitude relative to the vehicle body changes, resulting in significant errors between the previously calibrated parameters and the actual relationship. This ultimately leads to significant errors in perception and positioning results, and in severe cases, system performance degradation. Therefore, a simple and accurate camera attitude calibration method is urgently needed to improve the accuracy of perception and positioning results. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, embodiments of the present invention provide a method, apparatus, device and storage medium for calibrating the posture of an image acquisition device to solve the above-mentioned technical problems.

[0005] An embodiment of the present invention provides a method for calibrating the posture of an image acquisition device, the method comprising:

[0006] Acquiring an environmental image including a plurality of structural lines, wherein the environmental image is acquired by an image acquisition device;

[0007] determining a plurality of intersections formed by a plurality of reference line segments and intersection position information of the intersections, wherein the plurality of reference line segments include at least a portion of the structural line;

[0008] Projecting the intersection point onto a preset Gaussian sphere according to the intersection point position information to obtain a projection point of the intersection point, and configuring multiple matching point pairs of the projection point on the preset Gaussian sphere, the matching point pairs including a first matching point and a second matching point, generating a first vector according to the projection point and a center of the preset Gaussian sphere, generating a second vector according to the first matching point and the center of the preset Gaussian sphere, and generating a third vector according to the first matching point and the center of the preset Gaussian sphere, wherein the first vector, the second vector, and the third vector are perpendicular to each other;

[0009] generating a candidate combination of the intersection point based on the projection point, the first matching point and the second matching point of the intersection point, to obtain a candidate combination set of the intersection point;

[0010] determining a preferred Manhattan world hypothesis, the preferred Manhattan world hypothesis being a candidate combination of the candidate combination set of the intersection point;

[0011] determining Manhattan coordinate axis attributes according to the position relationship between the Manhattan projection point, the Manhattan first matching point and the Manhattan second matching point in the preferred Manhattan world hypothesis and each reference line segment, to obtain a projection Manhattan world coordinate of the Manhattan projection point, a first Manhattan world coordinate of the Manhattan first matching point and a second Manhattan world coordinate of the Manhattan second matching point;

[0012] determining a rotation matrix of the image acquisition device to the Manhattan world based on the projection Manhattan world coordinate, the first Manhattan world coordinate and the second Manhattan world coordinate and a preset image acquisition device world coordinate, to perform pose calibration on the image acquisition device.

[0013] In an embodiment of the present application, after obtaining an environment image including a plurality of structural lines, determining a plurality of intersection points of the extension lines of at least part of the structural lines, and intersection position information of the intersection points, the image acquisition device pose calibration method further comprises:

[0014] obtaining a region of interest in the environment image, and performing image de-distortion operation on the image of interest in the region of interest;

[0015] extracting reference line segments in the image of interest after image de-distortion operation;

[0016] recording endpoint coordinate information of at least part of the reference line segments, to determine the intersection position information through the endpoint coordinate information.

[0017] In an embodiment of the present application, the determination manner of the number of reference line segments comprises:

[0018] obtaining a preset noise parameter and a preset confidence;

[0019] determining a first reference parameter according to the preset noise parameter, and determining a second reference parameter according to the preset confidence;

[0020] determining the number of reference line segments based on the first reference parameter and the second reference parameter.

[0021] In one embodiment of the present invention, the image acquisition device posture calibration method is applied to a vehicle. The image acquisition device is disposed on the vehicle. Before acquiring an environment image including a plurality of structural lines, the image acquisition device posture calibration method includes:

[0022] Controlling the longitudinal direction of the vehicle to be parallel to the longitudinal direction of the space where the vehicle is located;

[0023] The environment image is collected by the image collection device.

[0024] In one embodiment of the present invention, the method for determining the preferred Manhattan world hypothesis includes:

[0025] Expanding the hemisphere of the preset Gaussian sphere to obtain a preset grid, and dividing each of the intersection points into subgrids of the preset grid;

[0026] Determining a response value of the intersection point based on the segment lengths, segment angles, and preset control parameters of the two reference line segments forming the intersection point, determining the response value as the response value of the sub-grid where the intersection point is located, and obtaining a response value of each sub-grid;

[0027] Determine the combined response value of each candidate combination in the candidate combination set of each intersection point according to the response values ​​of the sub-grids corresponding to the projection point, the first matching point, and the second matching point in the candidate combination in the preset grid;

[0028] The candidate combination with the largest combined response value is determined as the preferred Manhattan world hypothesis.

[0029] In one embodiment of the present invention, unfolding the hemisphere of the preset Gaussian sphere to obtain a preset grid, and dividing each of the intersection points into sub-grids of the preset grid includes:

[0030] Determine the grid width and grid height according to the preset noise parameters;

[0031] Expanding the hemisphere of the preset Gaussian sphere, dividing it into a plurality of sub-grids according to the grid width and the grid height, and obtaining the preset grid;

[0032] Determine the Gaussian spherical coordinates of each intersection projected onto the preset Gaussian sphere according to the intersection position information, and convert the Gaussian spherical coordinates into polar coordinates;

[0033] The intersection points are divided into sub-grids of the preset grid based on the polar coordinates.

[0034] In one embodiment of the present invention, determining the response value of the intersection point according to the segment lengths, segment angles, and preset control parameters of the two reference line segments forming the intersection point includes:

[0035]

[0036] Wherein, score is the response value, l1 is the segment length of one of the reference segments, l2 is the segment length of another of the reference segments, θ is the segment angle, factor is a preset control parameter, and n is the number of reference segments.

[0037] In one embodiment of the present invention, before determining the response value of the intersection point based on the segment lengths, segment angles, and preset control parameters of the two reference line segments forming the intersection point, the image acquisition device posture calibration method includes:

[0038] Obtaining the segment angle between the two reference line segments forming each intersection point;

[0039] If the line segment angle is greater than a preset angle threshold, the intersection point is filtered out;

[0040] The response value of the intersection point is determined according to the segment lengths, segment angles, and preset control parameters of the two reference line segments that form the intersection point after screening.

[0041] In one embodiment of the present invention, determining the Manhattan coordinate axis attributes according to the positional relationship between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the preferred Manhattan world hypothesis includes:

[0042] Converting the Manhattan projection point, the first Manhattan matching point, and the second Manhattan matching point to a preset normalized image coordinate system to obtain a normalized projection point, a normalized first matching point, and a normalized second matching point;

[0043] Determining positional relationships between at least a portion of the reference line segments projected to the preset normalized image coordinate system and the normalized projection point, the normalized first matching point, and the normalized second matching point;

[0044] Clustering the reference line segments based on the positional relationship to obtain categories of the reference line segments;

[0045] The average slope distribution of the reference line segments in each of the categories is determined to obtain Manhattan coordinate axis properties.

[0046] The present invention also provides an image acquisition device posture calibration device, the image acquisition device posture calibration device comprising:

[0047] An acquisition module, configured to acquire an environment image including a plurality of structural lines, wherein the environment image is acquired by an image acquisition device;

[0048] an intersection determination module, configured to determine a plurality of intersections formed by a plurality of reference line segments, and intersection position information of the intersections, wherein the plurality of reference line segments include at least a portion of the structural lines;

[0049] a matching point pair determination module, configured to project the intersection point onto a preset Gaussian sphere based on the intersection point position information to obtain a projection point of the intersection point, and configure multiple matching point pairs of the projection points on the preset Gaussian sphere, wherein the matching point pairs include a first matching point and a second matching point, generate a first vector based on the projection point and the center of the preset Gaussian sphere, generate a second vector based on the first matching point and the center of the preset Gaussian sphere, and generate a third vector based on the first matching point and the center of the preset Gaussian sphere, wherein the first vector, the second vector, and the third vector are perpendicular to each other;

[0050] a candidate combination set determination module, configured to generate candidate combinations of the intersection point based on the projection point of the intersection point, the first matching point, and the second matching point, to obtain a candidate combination set of the intersection point;

[0051] a preferred Manhattan world hypothesis determining module, configured to determine a preferred Manhattan world hypothesis, wherein the preferred Manhattan world hypothesis is a candidate combination in the set of candidate combinations of each intersection point;

[0052] a Manhattan coordinate axis attribute determination module, configured to determine Manhattan coordinate axis attributes based on the positional relationship between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the preferred Manhattan world hypothesis, and obtain the projected Manhattan world coordinates of the Manhattan projection point, the first Manhattan world coordinates of the first Manhattan matching point, and the second Manhattan world coordinates of the second Manhattan matching point;

[0053] A rotation matrix determination module is used to determine the rotation matrix of the image acquisition device to the Manhattan world based on the projected Manhattan world coordinates, the first Manhattan world coordinates, the second Manhattan world coordinates, and the preset image acquisition device world coordinates, so as to perform posture calibration on the image acquisition device.

[0054] An embodiment of the present invention provides an electronic device, comprising:

[0055] one or more processors;

[0056] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the image acquisition device posture calibration method described in any of the above embodiments.

[0057] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor of a computer, the computer executes the image acquisition device posture calibration method in any one of the above embodiments.

[0058] The embodiment of the present application has the following beneficial effects: the image acquisition device posture calibration method, device, equipment and storage medium in the embodiment of the present application, the method constructs a Manhattan world by structural lines in an environment image collected by an image acquisition device, and can estimate a rotation matrix between the Manhattan world and the image acquisition device based on a single image, so that the posture of the image acquisition device is calibrated, and the method is simple, accurate and fast.

[0059] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0060] The drawings incorporated into the specification and constituting a part of the specification show the embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0061] Figure 1 is a schematic diagram of an implementation environment of the image acquisition device posture calibration method shown by an exemplary embodiment of the present application;

[0062] Figure 2 is a flowchart of the image acquisition device posture calibration method shown by an exemplary embodiment of the present application;

[0063] Figure 3 is a schematic diagram of the Manhattan hypothesis shown by an exemplary embodiment of the present application;

[0064] Figure 4 is a Manhattan hypothesis generation schematic diagram shown by an exemplary embodiment of the present application;

[0065] Figure 5 is a schematic diagram of a determination manner of a preset grid and a determination manner of a response value of a sub-grid shown by an exemplary embodiment of the present application;

[0066] Figure 6 is a schematic diagram of a grid response value table shown by an exemplary embodiment of the present application;

[0067] Figure 7 is a schematic diagram of a rotation calibration process according to a preferred Manhattan hypothesis shown by an exemplary embodiment of the present application;

[0068] Figure 8 is a schematic diagram of a three-axis distribution shown in an exemplary embodiment of the present application;

[0069] Figure 9 is a schematic diagram of a specific method of a method for calibrating the posture of an image acquisition device shown in an exemplary embodiment of the present application;

[0070] Figure 10 is a block diagram of an image acquisition device posture calibration device shown in an exemplary embodiment of the present application;

[0071] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0072] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0073] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0074] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0075] The Manhattan world hypothesis is an assumption about the scene statistics of urban and indoor scenes. It assumes that such urban or indoor room scenes are built on a Cartesian grid, where surfaces in the world are aligned with one of three cardinal directions, with all planes either parallel or perpendicular to each other. This assumption is based on the observation that most indoor and urban scenes are designed on a three-dimensional Manhattan grid, and in most cases, man-made structures can be abstracted as blocks stacked together in accordance with the three cardinal directions.

[0076] Structural line, which can be understood as the line in space aligned with the three main directions of the building. For the scene of artificial buildings, the plane containing several dominant directions, and the straight line along the dominant direction can roughly outline the structure of the building, so it can be called structural line.

[0077] Gauss sphere, which maps the normal vector of each point on the surface of the object to a unit sphere, which is called Gaussian reference sphere (Gaussian Reference Sphere), Gauss sphere. The mapping process can be to translate the starting point of the normal to the center of the Gaussian reference sphere, and each normal vector will have an intersection point with the surface of the Gaussian reference sphere.

[0078] Figure 1 is a schematic diagram of an implementation environment shown by an exemplary embodiment of the present application. As shown in Figure 1 , the server 101 communicates with the vehicle terminal arranged inside the vehicle 102 through the network. The implementation devices of the vehicle terminal include but are not limited to personal computers, notebook computers, smart phones, tablet computers, portable wearable devices and any terminal device supporting the software generated by the image acquisition device pose calibration method. The vehicle terminal is equipped with networked terminal software, such as PC website, mobile phone website, Android APP, iOS APP, WeChat applet, smart watch APP, smart car APP and other smart hardware applications, etc. The server 101 can be realized by an independent server or a server cluster composed of multiple servers, such as TSP (Telematics Service Provider, automobile remote service provider) server. The server / server cluster can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms, etc. Basic cloud computing services, which are not limited here. Through the image acquisition device of the vehicle, the environmental image is acquired, the structural line image of the environmental image is detected, a plurality of straight line segments are obtained, a plurality of line segment pairs are formed, a certain number of line segment pairs are selected to calculate the intersection point coordinates of the intersection line of each line segment pair, the center of the image acquisition device is taken as the origin o, the Gauss sphere is established, and the intersection points are projected on the Gauss sphere to obtain a plurality of projection points p1, and the normal vector of the Gauss sphere surface is obtained The vertical circle of the vector, on which a plurality of vertical points {p2} are collected. The vector And The intersection of the normal vector of the plane and the Gaussian sphere {p3}, and finally multiple three-dimensional point groups {p1, p2, p3} are obtained. Half of the Gaussian sphere is expanded into a rectangular grid, and each intersection point is divided into the corresponding grid to obtain the response value of the grid. Based on the grid with response value and the above three-dimensional point groups, the response value sum of each three-dimensional point group is determined, and the three-dimensional point group with the largest response value sum is used as the optimal Manhattan world hypothesis. Then the three points in the three-dimensional point group of the optimal Manhattan world hypothesis are converted into the normalized image coordinate system to obtain 2d{p best1 , p best2 , p best3}, traverse each straight line segment obtained above and calculate {p best1 , p best2 , p best3 The line segments connecting each point and the midpoint of the line segment are clustered by angle threshold determination. The Manhattan coordinate axis attributes x, y, and z of each cluster are determined by counting the horizontal and vertical slopes of each cluster, and then the correspondence of the three axes of the Manhattan world is determined. Then the rotation p of the camera to the Manhattan world is calculated. m =R mc p c , where Rmc represents the rotation matrix from the camera to the Manhattan. The above process can be executed independently by the server, independently by the vehicle terminal, or interactively by the server and the vehicle terminal.

[0079] Smart vehicles are equipped with a variety of sensors, including lidar, cameras, millimeter-wave radar, and ultrasonic radar. Among these sensors, cameras are among the most important, and the system relies heavily on them to abstract and model information about the surrounding environment. In camera applications, extrinsic calibration is a critical step. The accuracy of the calibration results and the stability of the algorithm directly impact the accuracy of the camera's operating results. The position and attitude of the camera center relative to the center of the vehicle are called the camera-to-vehicle extrinsic parameters. During vehicle use, the camera's position relative to the vehicle body changes, resulting in large errors between the previously calibrated parameters and the actual relationship. This ultimately leads to large errors in the perception and positioning results, and in severe cases, system performance degradation. Therefore, a simple and accurate camera attitude calibration method is urgently needed to improve the accuracy of perception and positioning results.

[0080] Many natural and man-made scenes have structurally regular lines that are either parallel or perpendicular to each other, such as floors, ceilings, and walls. If the rotation relationship between the Manhattan world and the vehicle body is known, and the camera's rotation with respect to the Manhattan world is solved, the rotation between the vehicle body and the camera can be calculated. The rotation between the Manhattan world and the vehicle body can be approximated by the identity matrix, achieved by positioning the vehicle parallel to the longitudinal lines of the Manhattan world.

[0081] To solve these problems, the embodiments of the present application respectively propose a method for calibrating the posture of an image acquisition device, a device for calibrating the posture of an image acquisition device, an electronic device, a computer-readable storage medium, and a computer program product. These embodiments will be described in detail below.

[0082] See also Figure 2 , Figure 2 This is a flow chart of an image acquisition device posture calibration method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 The implementation environment shown is specifically executed by the server 101 or the vehicle-mounted terminal in the implementation environment, or by the server and the vehicle-mounted terminal together. It should be understood that the method can also be applied to other exemplary implementation environments and specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0083] like Figure 2 As shown, in an exemplary embodiment, the image acquisition device posture calibration method includes at least steps S201 to S207, which are described in detail as follows:

[0084] Step S201: Acquire an environment image including a plurality of structural lines.

[0085] Wherein, the environmental image is collected by an image collection device.

[0086] When the image acquisition device attitude calibration method is applied to a vehicle, the vehicle can be placed in an environment with abundant structural lines, with the ground as flat as possible. The image acquisition device is set on the vehicle. Before acquiring an image of the environment including multiple structural lines, the image acquisition device attitude calibration method includes the following steps:

[0087] The longitudinal direction of the vehicle is controlled to be parallel to the longitudinal direction of the space in which the vehicle is located;

[0088] The environment image is collected by an image acquisition device.

[0089] The image acquisition device may be a device with an image capture function such as a vehicle's driving recorder. The location of the image acquisition device is not limited here and may be set by those skilled in the art as needed.

[0090] The vehicle longitudinal direction is the plane of the side where the vehicle door is located, or it can be the front-to-rear direction. The spatial longitudinal direction is the longitudinal direction defined by those skilled in the art as the direction of the space in which the vehicle is located. It can be a direction parallel to certain structural lines in that space.

[0091] In an embodiment, when the image acquisition device posture calibration method is applied to other devices or equipment, the device or equipment to which the method is applied needs to be arranged in a position parallel to the boundary in a space with a certain boundary in the photographed image, for example, in the case of a vehicle, the vehicle is parked in a parking space in a garage, beside a lane line, beside a wall, etc.

[0092] In an embodiment, after step S201 and before step S202, the image acquisition device posture calibration method comprises:

[0093] Obtaining a region of interest in the environment image, and performing image de-distortion on the image of interest in the region of interest;

[0094] Extracting a reference line segment in the image of interest after the image de-distortion operation;

[0095] Recording the endpoint coordinate information of at least part of the reference line segment to determine the intersection position information through the endpoint coordinate information.

[0096] The endpoint coordinate information of the reference line segment includes the endpoint coordinates of the two endpoints, and the way of determining the intersection position information through the endpoint coordinate information can be:

[0097] Converting the two endpoint coordinates of a reference line segment into homogeneous coordinates to calculate the cross product to obtain the expression of the reference line segment;

[0098] Determining an intersection point based on the expressions of two reference line segments, and the intersection position information of the intersection point.

[0099] The region of interest can be the upper part of the environment image or other parts set by those skilled in the art. The upper part of the original fisheye image (environment image) can be de-distorted through the intrinsic parameters of the image acquisition device to realize the image de-distortion operation on the image of interest in the region of interest.

[0100] In an embodiment, the number of reference line segments is determined as follows:

[0101] Obtaining a preset noise parameter and a preset confidence;

[0102] Determining a first reference parameter according to the preset noise parameter and a second reference parameter according to the preset confidence;

[0103] Determining the number of reference line segments based on the first reference parameter and the second reference parameter.

[0104] The specific determination method of the number of reference line segments is as follows:

[0105] m = log(1-coe) / log(1-noise) Formula (1);

[0106] Where m is the number of reference segments, coe is the preset confidence, and noise is the preset noise parameter.

[0107] At this time, the intersection points are calculated by combining two reference line segments based on the number of reference line segments. The number of intersection points is less than or equal to the intersection threshold. The intersection threshold is calculated as follows:

[0108]

[0109] Where J is the intersection threshold and m is the number of reference line segments.

[0110] By determining the number of reference line segments in the above manner, the amount of calculation can be reduced and resource waste can be minimized while ensuring the accuracy of the subsequent rotation matrix determination.

[0111] Step S202: determining a plurality of intersections formed by a plurality of reference line segments and intersection position information of the intersections.

[0112] The plurality of reference line segments are at least a portion of structural lines in the environment image. In other words, a portion of the plurality of structural lines in the environment image is selected as the reference line segments.

[0113] Please refer to the above embodiments for the method of determining the number of reference line segments and the method of determining the intersection position information, which will not be described in detail here.

[0114] Step S203 : projecting the intersection point onto a preset Gaussian sphere according to the intersection point position information to obtain a projection point of the intersection point and multiple matching point pairs of the projection points configured on the preset Gaussian sphere.

[0115] Among them, the matching point pair includes a first matching point and a second matching point, a first vector is generated based on the projection point and the center of a preset Gaussian sphere, a second vector is generated based on the first matching point and the center of the preset Gaussian sphere, and a third vector is generated based on the first matching point and the center of the preset Gaussian sphere. The first vector, the second vector and the third vector are perpendicular to each other.

[0116] A method for generating matching point pairs is as follows:

[0117] Project all intersection points onto the preset Gaussian sphere to obtain the set {p1}, and use each point in {p1} and the center of the preset Gaussian sphere as the first axis;

[0118] For each point in {p1}, a perfect circle on the Gaussian sphere can be obtained based on the orthogonal relationship. Points are selected on the perfect circle according to a certain angular resolution to obtain the first matching point, and finally the projection point set {p2} is determined. When the angular resolution is 0.5, the number of projection points is Take each point in {p2} and the center of the preset Gaussian sphere as the second axis;

[0119] The position of the third axis can be uniquely determined according to the orthogonal relationship of the first axis and the second axis, to obtain a second matching point, and when the angle resolution is 0.5, the number of points {p3} of the axis on the Gaussian sphere is

[0120] A specific matching point pair generation manner is as follows:

[0121] A certain number of reference line segments are selected to obtain n pairs of line segments, and the coordinates {p1} of the intersection points of each pair of line segments in the distance are calculated.

[0122] Taking the center o of the camera (taking the image acquisition device as the camera as an example) as the origin, a Gaussian sphere s (a preset Gaussian sphere) with a length of unit 1 (other unit lengths can also be selected) is established, and the projection points {p1} of all the intersection points on the Gaussian sphere are calculated.

[0123] Referring to Figure 3 , Figure 3 is a schematic diagram of the Manhattan assumption shown in an example embodiment of the present application, and the vector is perpendicular to the circle, and 720 points {p2} (a first matching point set) are collected on the circle according to a resolution of 0.5 degrees (other angles can also be used, which is an example here).

[0124] The normal vector of the plane where the vectors and are located is calculated, and the intersection point {p3} (i.e., a second matching point set) of the normal vector and the Gaussian sphere surface is calculated, and finally the number of combinations (a candidate combination of an intersection point) of {p1, p2, p3} is 720n (taking 0.5 degrees as an example), that is, a candidate combination set of the intersection point can be obtained.

[0125] In step S204, a candidate combination of an intersection point is generated based on the projection point of the intersection point, a first matching point and a second matching point, to obtain a candidate combination set of the intersection point.

[0126] As shown in the above embodiment, a first matching point of an intersection point, a second matching point of the first matching point, and the first matching point form a candidate combination of the intersection point, and the second matching point in the candidate combination corresponds to the first matching point one by one. The second matching point is a point determined according to the projection point and the first matching point.

[0127] Referring to Figure 4 , Figure 4 is a Manhattan assumption generation schematic diagram shown in an example embodiment of the present application, that is, a generation process schematic diagram of a candidate combination set of an intersection point, as shown in Figure 4 , the process includes:

[0128] In step S401, the number m of selected line segments is calculated according to a noise factor and uncertainty.

[0129] Wherein the noise factor is the preset noise parameter, the uncertainty is the preset reliability, the number of line segments is the number of reference line segments. The specific determination method can be referred to the formula (1).

[0130] Step S402, randomly selecting a fixed number m of line segments.

[0131] That is, randomly selecting m structural lines as reference line segments from the plurality of structural lines.

[0132] Step S403, calculating all

[0133] That is, combining the reference line segments obtained in the above step S402 two by two, calculating the intersection points, and the number of intersection points is

[0134] Step S404, projecting all intersection points to a unit Gaussian sphere to obtain Point set {p1}.

[0135] The unit Gaussian sphere is a preset Gaussian sphere, and the projection points of the intersection points are obtained by projection, and the projection points of the intersection points form a set {p1}.

[0136] Step S405, determining all Gaussian projection point set {p2}.

[0137] Selecting points with an angle resolution of 0.5 degrees, the number of first matching points is {p2} is the set of first matching points of each projection point.

[0138] Step S406, determining all Gaussian projection point set {p3}.

[0139] Selecting points with an angle resolution of 0.5 degrees, obtaining the first matching points, and according to the orthogonal relationship, the corresponding second matching points are obtained based on the projection points and the first matching points, and the number of first matching points is The number of second matching points is {p3} is the set of second matching points of each projection point.

[0140] Step S407, series Manhattan hypothesis combination.

[0141] Through the above method, each of the candidate combinations of the candidate combination set of each of the intersection points can be obtained.

[0142] Step S205, determining the preferred Manhattan world hypothesis.

[0143] The Manhattan world hypothesis is preferably determined from a candidate combination in a candidate combination set of each intersection point.

[0144] In an embodiment, the Manhattan world hypothesis is preferably determined in the following manner:

[0145] A half-sphere of a preset Gaussian sphere is unfolded to obtain a preset grid, and each intersection point is divided into a sub-grid of the preset grid.

[0146] A response value of each intersection point is determined according to a line segment length, a line segment angle of two reference line segments forming the intersection point, and a preset control parameter, the response value is determined as a response value of a sub-grid where the intersection point is located, and a response value of each sub-grid is obtained.

[0147] A combination response value of each candidate combination in the candidate combination set of each intersection point is determined according to a response value of a corresponding sub-grid of the preset grid of a projection point, a first matching point and a second matching point in the candidate combination.

[0148] The candidate combination with the maximum combination response value is determined as the preferred Manhattan world hypothesis.

[0149] In an embodiment, the Manhattan world hypothesis is preferably determined from a candidate combination in a candidate combination set of each intersection point.

[0150] A grid width and a grid height are determined according to a preset noise parameter.

[0151] A half-sphere of a preset Gaussian sphere is unfolded, and a plurality of sub-grids are divided according to the grid width and the grid height, thereby obtaining the preset grid.

[0152] A Gaussian sphere coordinate of each intersection point projected to the preset Gaussian sphere is determined according to intersection position information, and the Gaussian sphere coordinate is converted into a polar coordinate.

[0153] Each intersection point is divided into each sub-grid of the preset grid based on the polar coordinate.

[0154] In an embodiment, the response value of each intersection point is determined according to a line segment length, a line segment angle of two reference line segments forming the intersection point, and a preset control parameter.

[0155]

[0156] In the formula, score represents the response value, l1 represents a line segment length of one reference line segment, l2 represents a line segment length of another reference line segment, θ represents the line segment angle, factor represents the preset control parameter, and n represents the number of reference line segments.

[0157] In an embodiment, before the response value of each intersection point is determined according to a line segment length, a line segment angle of two reference line segments forming the intersection point, and a preset control parameter, the image acquisition device pose calibration method comprises:

[0158] Get the segment angle between the two reference line segments forming each intersection point;

[0159] If the angle between the line segments is greater than the preset angle threshold, the intersection point will be filtered out;

[0160] The response value of the intersection point is determined according to the segment lengths, segment angles and preset control parameters of the two reference line segments that form the intersection point after screening.

[0161] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram showing a method for determining a preset grid and a method for determining a response value of a sub-grid, as shown in an exemplary embodiment of the present application. Figure 5 As shown:

[0162] Step S501 , calculating the coordinates x, y, z of the intersection points of all the straight lines when they are combined in pairs.

[0163] That is, calculate the intersection points between reference line segments.

[0164] Step S502 , calculating the Gaussian hemisphere width (number of columns), w=360 / 0.5.

[0165] That is, the width (number of columns) of the Gaussian hemisphere equivalent grid is calculated, w = 360 / 0.5 formula (4), where w is the width.

[0166] Step S503, calculate the Gaussian hemisphere height (number of rows), h=90 / 0.5.

[0167] That is, the height (number of rows) of the Gaussian hemisphere equivalent grid is calculated, h = 90 / 0.5 formula (5), where h is the height.

[0168] Step S504: Generate a Gaussian sphere equivalent grid.

[0169] That is, generate a preset grid.

[0170] Step S505: The intersection coordinates are converted to the lower coordinates of the Gaussian sphere center.

[0171] In one embodiment, the intersection coordinates are projected onto Gaussian sphere coordinates, and the calculation formula is:

[0172]

[0173] Among them, x1, y1, z1 are the Gaussian sphere coordinates of the intersection point, the initial coordinates of x, y, z (intersection position information), x c ,y c The image center coordinates in the preset normalized image coordinate system.

[0174] Step S506: Convert the central coordinates to polar coordinates.

[0175] The conversion from the Gaussian sphere coordinates to the polar coordinates is calculated as follows:

[0176]

[0177] wherein, and p is the polar coordinate of the intersection point, x1, y1, z1 are the Gaussian sphere coordinates of the intersection point.

[0178] In step S507, the intersection point polar coordinates are converted to the grid and the grid response value is updated.

[0179] The calculation formula of the response value can be referred to the above embodiment.

[0180] In step S508, it is determined whether the angle between the two straight lines constituting the intersection point is greater than a threshold value, if yes, it is skipped, and the step S505 is performed for the new intersection point. If not, the step S509 is performed.

[0181] wherein, the response value of the intersection point of the two line segments whose line segment angle exceeds a certain threshold value (preset angle threshold) is directly ignored.

[0182] In step S509, the equivalent grid response value table is obtained.

[0183] That is, the response value of the intersection point is determined as the response value of the sub-grid of the preset grid.

[0184] Through the above method, the response value of each sub-grid can be obtained, and the combination response value can be obtained by looking up the table for the Manhattan hypothesis (to-be-selected combination) subsequently.

[0185] In an embodiment, referring to Figure 6 , Figure 6 is a schematic diagram of the grid response value table shown in an exemplary embodiment of the present application.

[0186] By selecting a certain number of intersection points to generate the Manhattan hypothesis, the response value of each group of hypotheses is calculated by traversing all Manhattan hypotheses, and the optimal Manhattan hypothesis is the one with the maximum response value.

[0187] In step S206, the Manhattan coordinate axis attribute is determined according to the positional relationship between the Manhattan projection point, the Manhattan first matching point and the Manhattan second matching point in the Manhattan world hypothesis and each reference line segment, and the projection Manhattan world coordinates of the Manhattan projection point, the first Manhattan world coordinates of the Manhattan first matching point and the second Manhattan world coordinates of the Manhattan second matching point are obtained.

[0188] The projection point, the first matching point and the second matching point in the to-be-selected combination of the Manhattan world hypothesis are determined as the Manhattan projection point, the Manhattan first matching point and the Manhattan second matching point.

[0189] In one embodiment, determining the Manhattan coordinate axis attributes according to the positional relationship between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the Manhattan world hypothesis includes:

[0190] Converting the Manhattan projection point, the first Manhattan matching point, and the second Manhattan matching point to a preset normalized image coordinate system to obtain a normalized projection point, a normalized first matching point, and a normalized second matching point;

[0191] Determining positional relationships between at least a portion of the reference line segments projected to a preset normalized image coordinate system and the normalized projection point, the normalized first matching point, and the normalized second matching point;

[0192] Clustering each reference line segment based on the positional relationship to obtain the category of the reference line segment;

[0193] Determine the average slope distribution of the reference line segments in each category and obtain the Manhattan coordinate axis properties.

[0194] Step S207 , determining a rotation matrix from the image acquisition device to the Manhattan world based on the projected Manhattan world coordinates, the first Manhattan world coordinates, the second Manhattan world coordinates, and the preset image acquisition device world coordinates, so as to perform posture calibration on the image acquisition device.

[0195] The preset normalized image coordinate system may be the image coordinate system of the image acquisition device. In this case, there is no need to project the reference line segment, as the reference line segment itself is in the image coordinate system.

[0196] See also Figure 7 , Figure 7 This is an exemplary embodiment of the present application showing a process of performing rotation calibration based on the preferred Manhattan hypothesis, which may specifically include the following steps:

[0197] Step S701: transform the combined coordinates of the Manhattan world hypothesis.

[0198] The combination of all previously generated Manhattan world hypotheses is transformed from the image coordinate system to the Gaussian sphere Cartesian coordinate system to the Gaussian sphere polar coordinate system.

[0199] Step S702: Calculate the total score of the response value of each group of hypotheses.

[0200] Query the response value scores of the grids where each Manhattan world hypothesis is located and calculate the sum of the scores of each group of hypotheses (the response value of the candidate combination). For example, the Gaussian spherical coordinates of each point in each Manhattan world hypothesis (candidate combination) can be converted to polar coordinates (refer to the method of formula (7)), and then the angle information of the point can be obtained. By querying the response value of the sub-grid corresponding to the point in the preset grid, the response value of the point is used as the response value of the point. After obtaining the response value of each point in a Manhattan world hypothesis (candidate combination), the sum of the response values ​​is taken to obtain the response value of the candidate combination.

[0201] Step S703: Determine the optimal Manhattan world hypothesis combination.

[0202] Traverse to find the Manhattan world hypothesis combination with the highest score, that is, the preferred Manhattan world hypothesis.

[0203] Step S704: optimal Manhattan world hypothesis combined coordinate transformation.

[0204] The three points of the optimal combination (corresponding to the three axes) are converted from the Gaussian sphere coordinate system back to the image coordinate system (the preset normalized coordinate system), and the geometric relationship between each line segment and the three hypothetical points is calculated to determine the category to which the line belongs. Finally, the line segment cluster that meets the conditions is obtained.

[0205] Step S705: line segment clustering.

[0206] The geometric relationship between each line segment and the three hypothetical points is calculated to determine the category to which the line belongs, and finally the line segment cluster that meets the conditions is obtained.

[0207] Step S706: Count the slope distribution of each category.

[0208] Statistics of the average slope distribution of each category, d y / d x The axis to which the smallest class belongs is the x-axis, d x / d y The axis to which the smallest class belongs is the y-axis, and the one in between is the z-axis.

[0209] Step S707: Determine the coordinate axis corresponding to each category.

[0210] Step S708: Determine the three-axis distribution of the optimal Manhattan world hypothesis.

[0211] The relationship between all classes and axes has been determined, and the hypothetical points corresponding to each class have been determined, such as Figure 8 As shown, Figure 8 It is a schematic diagram of a three-axis distribution shown in an exemplary embodiment of the present application, that is, the three orthogonal axis distributions (Manhattan coordinate axis attributes) of the final Manhattan world hypothesis.

[0212] Step S709: Calculate the rotation from the camera to the Manhattan world.

[0213] Calculate the rotation from the camera to the vehicle body (the image acquisition device, that is, the camera here, is located on the vehicle). The rotation matrix is ​​determined as follows:

[0214] p m =R mc p c Formula (8),

[0215] Among them, R mc represents the rotation matrix from the camera to Manhattan, p c is the preset camera coordinate, p m The Manhattan world coordinate is composed of the projected Manhattan world coordinate, the first Manhattan world coordinate, and the second Manhattan world coordinate.

[0216] like Figure 9 As shown, Figure 9 This is a specific method of the image acquisition device posture calibration method shown in an exemplary embodiment of the present application. It adopts a camera extrinsic parameter calibration method based on the Manhattan world hypothesis, constructs a Manhattan world according to the more common structured lines in the scene, and then calculates the rotation part between the camera and the Manhattan world. When this method is used in a vehicle, by constraining the parking posture of the vehicle, the rotation part of the vehicle and the Manhattan world is approximated to the unit matrix, and finally the rotation part of the camera and the vehicle is obtained. The principle of this method is relatively simple, the workload of the data preparation process is small, the results are relatively accurate, and it has good application prospects for scenes with rich structured lines. Figure 9 As shown, taking a camera as an example, the specific method includes:

[0217] Step S901: collecting structural line images.

[0218] That is, collecting environmental images.

[0219] Step S902: image preprocessing.

[0220] Dedistort the upper part of the original fisheye image based on the camera's internal parameters.

[0221] Step S903: detecting line segments in the image.

[0222] Extract m line segments that meet certain conditions from the dedistorted image.

[0223] Step S904: Calculate relevant information of all line segments.

[0224] Statistics record the endpoints, length, expression and rotation angle of each line.

[0225] Step S905: Generate a Manhattan world hypothesis.

[0226] Select a certain number of n pairs of line segments, calculate the coordinates of the intersection of each pair of line segments at a distance {p1}, take the camera center o as the origin, establish a Gaussian sphere s with a length of unit 1, and find the projection point {p1} of all the above intersection points on the Gaussian sphere, and obtain The vertical circle of the vector, on which 720 points {p2} are collected at a resolution of 0.5 degrees, and the vector is calculated and The intersection point of the normal vector of the plane and the Gaussian sphere is {p3}, and the final number of combinations of {p1, p2, p3} is 720n.

[0227] Step S906: Gaussian hemisphere expansion grid.

[0228] Use the statistical combination method to exhaustively enumerate the situations of two line segment pairs and calculate the intersection point of each pair {p i}, and its intersection {p i} is projected onto the above-mentioned unit Gaussian sphere s, and the Gaussian hemisphere is expanded into a rectangular grid g of (w = 360 / 0.5, h = 90 / 0.5) according to the angular resolution.

[0229] Step S907: Calculate the response value of each grid.

[0230] All the intersection points p obtained in the above steps are i Divide the grid into corresponding grids according to the angle, and update the grid response value score. The response value calculation formula is: Where θ is the angle between the two straight lines, and factor is the control parameter.

[0231] Step S908: Determine the optimal Manhattan world hypothesis.

[0232] Substitute all {p1, p2, p3} combinations in step S905, calculate the sum of the response values ​​of each combination, and the combination with the largest response value constitutes the Manhattan world hypothesis.

[0233] Step S909: clustering and determining the Manhattan three-axis distribution.

[0234] Convert the three Gaussian sphere points on the Manhattan world hypothesis obtained above to the normalized plane to obtain the corresponding 2d p best1 , p best2 , p best3 , traverse all line segments and calculate {p best1 , p best2 , p best3The line segments connecting each point and the midpoint of the line segment are clustered by determining the angle threshold, and the Manhattan coordinate axis attributes x, y, and z of each cluster are determined by counting the horizontal and vertical slopes of each cluster, and then the correspondence of the three axes of the optimal Manhattan world hypothesis in step S908 is determined.

[0235] Step S910: Calculate the rotation matrix from the camera to the Manhattan world.

[0236] Calculate the camera's rotation p to the Manhattan world m =R mc p c , where R mc Represents the rotation matrix from camera to Manhattan.

[0237] By constructing a Manhattan world using regular lines in natural or artificial scenes, the rotation relationship between the Manhattan world and the camera can be estimated by extracting a single image. This approach eliminates the need for complex data collection and preprocessing, nor does it require additional calibration tools. Calibration is fast and highly accurate.

[0238] Figure 10 This is a block diagram of an image acquisition device posture calibration device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The device may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0239] like Figure 10 As shown, the exemplary image acquisition device posture calibration device 1000 includes:

[0240] An acquisition module 1001 is configured to acquire an environment image including a plurality of structural lines, wherein the environment image is acquired by an image acquisition device;

[0241] An intersection determination module 1002 is configured to determine a plurality of intersections formed by a plurality of reference line segments and intersection position information of the intersections, wherein the plurality of reference line segments include at least a portion of structural lines;

[0242] A matching point pair determination module 1003 is configured to project the intersection point onto a preset Gaussian sphere based on the intersection point position information, thereby obtaining a projection point of the intersection point and multiple matching point pairs configured with the projection points on the preset Gaussian sphere, wherein the matching point pairs include a first matching point and a second matching point, generate a first vector based on the projection point and the center of the preset Gaussian sphere, generate a second vector based on the first matching point and the center of the preset Gaussian sphere, and generate a third vector based on the first matching point and the center of the preset Gaussian sphere, wherein the first vector, the second vector, and the third vector are perpendicular to each other.

[0243] A candidate combination set determination module 1004 is configured to generate candidate combinations of intersection points based on a projection point of an intersection point, a first matching point, and a second matching point, thereby obtaining a candidate combination set of intersection points;

[0244] A preferred Manhattan world hypothesis determining module 1005 is used to determine a preferred Manhattan world hypothesis, where the preferred Manhattan world hypothesis is a candidate combination in the candidate combination set of each intersection point;

[0245] a Manhattan coordinate axis attribute determination module 1006 for determining Manhattan coordinate axis attributes based on the positional relationships between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the preferred Manhattan world hypothesis, to obtain the projected Manhattan world coordinates of the Manhattan projection point, the first Manhattan world coordinates of the first Manhattan matching point, and the second Manhattan world coordinates of the second Manhattan matching point;

[0246] 1007, used to determine the rotation matrix of the image acquisition device to the Manhattan world based on the projected Manhattan world coordinates, the first Manhattan world coordinates, the second Manhattan world coordinates, and the preset image acquisition device world coordinates, so as to perform attitude calibration on the image acquisition device.

[0247] It should be noted that the image acquisition device posture calibration device provided in the above embodiment is the same as the above embodiment. Figure 2 The provided method for calibrating the posture of an image acquisition device is based on the same concept. The specific manner in which each module and unit performs operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the image acquisition device posture calibration device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0248] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the image acquisition device posture calibration method provided in the above-mentioned embodiments.

[0249] Figure 11 The following is a schematic diagram showing the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application. Figure 11 The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0250] like Figure 11As shown, the computer system 1100 includes a central processing unit (CPU) 1101 which can perform various suitable actions and processes in accordance with programs stored in a read-only memory (ROM) 1102 or loaded from the storage section 1108 into a random access memory (RAM) 1103, such as performing the methods described in the above embodiments. Various programs and data required for the operation of the system are also stored in the RAM 1103. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0251] Connected to the I / O interface 1105 are an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as necessary. A removable recording medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1110 as necessary, so that a computer program read therefrom is installed in the storage section 1108 as necessary.

[0252] In particular, in accordance with embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing computer programs for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1109, and / or installed from the removable recording medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, various functions defined in the system of the present application are performed.

[0253] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which the computer-readable computer program is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit the program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0254] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by special-purpose hardware-based systems, which perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0255] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0256] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor of a computer, the computer executes the image acquisition device pose calibration method as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0257] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image acquisition device pose calibration method provided in the above embodiments.

[0258] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought of the present application should be covered by the claims of the present application.

Claims

1. A method for calibrating the posture of an image acquisition device, characterized in that: The image acquisition device posture calibration method comprises: Acquiring an environmental image including a plurality of structural lines, wherein the environmental image is acquired by an image acquisition device; determining a plurality of intersections formed by a plurality of reference line segments and intersection position information of the intersections, wherein the plurality of reference line segments include at least a portion of the structural line; Projecting the intersection point onto a preset Gaussian sphere according to the intersection point position information to obtain a projection point of the intersection point, and configuring multiple matching point pairs of the projection point on the preset Gaussian sphere, the matching point pairs including a first matching point and a second matching point, generating a first vector according to the projection point and a center of the preset Gaussian sphere, generating a second vector according to the first matching point and the center of the preset Gaussian sphere, and generating a third vector according to the second matching point and the center of the preset Gaussian sphere, wherein the first vector, the second vector, and the third vector are perpendicular to each other; generating a candidate combination of the intersection point based on the projection point of an intersection point, a first matching point, and a second matching point, to obtain a candidate combination set of the intersection point; Determining a preferred Manhattan world hypothesis, where the preferred Manhattan world hypothesis is a candidate combination in the set of candidate combinations of each intersection point; Determining Manhattan coordinate axis attributes based on the positional relationships between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the preferred Manhattan world hypothesis to obtain the projected Manhattan world coordinates of the Manhattan projection point, the first Manhattan world coordinates of the first Manhattan matching point, and the second Manhattan world coordinates of the second Manhattan matching point, wherein determining the Manhattan coordinate axis attributes based on the positional relationships between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the preferred Manhattan world hypothesis includes: converting the Manhattan projection point, the first Manhattan matching point, and the second Manhattan matching point into a preset normalized image coordinate system to obtain normalized projection points, normalized first matching points, and normalized second matching points; determining the positional relationships between at least a portion of the reference line segments projected into the preset normalized image coordinate system and the normalized projection points, the normalized first matching points, and the normalized second matching points, respectively; clustering the reference line segments based on the positional relationships to obtain categories of the reference line segments; and determining the average slope distribution of the reference line segments in each category to obtain Manhattan coordinate axis attributes; A rotation matrix from the image acquisition device to the Manhattan world is determined based on the projected Manhattan world coordinates, the first Manhattan world coordinates, the second Manhattan world coordinates, and the preset image acquisition device world coordinates to perform posture calibration on the image acquisition device.

2. The method for calibrating the posture of an image acquisition device according to claim 1, wherein: After acquiring an environmental image including a plurality of structural lines, and before determining a plurality of intersection points of extension lines of at least a portion of the structural lines and intersection position information of the intersection points, the image acquisition device posture calibration method further includes: Acquire a region of interest in the environment image, and perform an image dedistortion operation on the image of interest in the region of interest; extracting reference line segments from the image of interest after the image dedistortion operation; The endpoint coordinate information of at least part of the reference line segment is recorded to determine the intersection position information through the endpoint coordinate information.

3. The image acquisition device posture calibration method according to claim 2, wherein: The method for determining the number of the reference line segments includes: Obtaining preset noise parameters and preset confidence; Determine a first reference parameter according to the preset noise parameter, and determine a second reference parameter according to the preset confidence; The number of the reference line segments is determined based on the first reference parameter and the second reference parameter.

4. The method for calibrating the posture of an image acquisition device according to claim 1, wherein: The image acquisition device posture calibration method is applied to a vehicle. The image acquisition device is provided on the vehicle. Before acquiring an environment image including a plurality of structural lines, the image acquisition device posture calibration method includes: Controlling the longitudinal direction of the vehicle to be parallel to the longitudinal direction of the space where the vehicle is located; The environment image is collected by the image collection device.

5. The method for calibrating the posture of an image acquisition device according to any one of claims 1 to 4, wherein: Methods for determining the preferred Manhattan world hypothesis include: Expanding the hemisphere of the preset Gaussian sphere to obtain a preset grid, and dividing each of the intersection points into subgrids of the preset grid; Determining a response value of the intersection point based on the segment lengths, segment angles, and preset control parameters of the two reference line segments forming the intersection point, determining the response value as the response value of the sub-grid where the intersection point is located, and obtaining a response value of each sub-grid; Determine the combined response value of each candidate combination in the candidate combination set of each intersection point according to the response values ​​of the sub-grids corresponding to the projection point, the first matching point, and the second matching point in the candidate combination in the preset grid; The candidate combination with the largest combined response value is determined as the preferred Manhattan world hypothesis.

6. The method for calibrating the posture of an image acquisition device according to claim 5, wherein: Expanding the hemisphere of the preset Gaussian sphere to obtain a preset grid, and dividing each of the intersection points into subgrids of the preset grid includes: Determine the grid width and grid height according to the preset noise parameters; Expanding the hemisphere of the preset Gaussian sphere, dividing it into a plurality of sub-grids according to the grid width and the grid height, and obtaining the preset grid; Determine the Gaussian spherical coordinates of each intersection projected onto the preset Gaussian sphere according to the intersection position information, and convert the Gaussian spherical coordinates into polar coordinates; The intersection points are divided into sub-grids of the preset grid based on the polar coordinates.

7. The method for calibrating the posture of an image acquisition device according to claim 5, wherein: Determining the response value of the intersection point according to the segment lengths, segment angles, and preset control parameters of the two reference line segments forming the intersection point includes: Wherein, score is the response value, l1 is the segment length of one of the reference segments, l2 is the segment length of another of the reference segments, θ is the segment angle, factor is a preset control parameter, and n is the number of reference segments.

8. The method for calibrating the posture of an image acquisition device according to claim 5, wherein: Before determining the response value of the intersection point according to the segment lengths, segment angles, and preset control parameters of the two reference line segments forming the intersection point, the image acquisition device posture calibration method includes: Obtaining the segment angle between the two reference line segments forming each intersection point; If the line segment angle is greater than a preset angle threshold, the intersection point is filtered out; The response value of the intersection point is determined according to the segment lengths, segment angles, and preset control parameters of the two reference line segments that form the intersection point after screening.

9. An image acquisition device posture calibration device, characterized in that: The image acquisition device posture calibration device comprises: An acquisition module, configured to acquire an environment image including a plurality of structural lines, wherein the environment image is acquired by an image acquisition device; an intersection determination module, configured to determine a plurality of intersections formed by a plurality of reference line segments, and intersection position information of the intersections, wherein the plurality of reference line segments include at least a portion of the structural lines; a matching point pair determination module, configured to project the intersection point onto a preset Gaussian sphere based on the intersection point position information to obtain a projection point of the intersection point, and configure multiple matching point pairs of the projection points on the preset Gaussian sphere, wherein the matching point pairs include a first matching point and a second matching point, generate a first vector based on the projection point and the center of the preset Gaussian sphere, generate a second vector based on the first matching point and the center of the preset Gaussian sphere, and generate a third vector based on the second matching point and the center of the preset Gaussian sphere, wherein the first vector, the second vector, and the third vector are perpendicular to each other; a candidate combination set determination module, configured to generate candidate combinations of the intersection point based on the projection point of the intersection point, the first matching point, and the second matching point, to obtain a candidate combination set of the intersection point; a preferred Manhattan world hypothesis determining module, configured to determine a preferred Manhattan world hypothesis, wherein the preferred Manhattan world hypothesis is a candidate combination in the set of candidate combinations of each intersection point; a Manhattan coordinate axis attribute determination module, configured to determine Manhattan coordinate axis attributes based on the positional relationships between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the preferred Manhattan world hypothesis, and obtain the projected Manhattan world coordinates of the Manhattan projection point, the first Manhattan world coordinates of the first Manhattan matching point, and the second Manhattan world coordinates of the second Manhattan matching point. Determining the Manhattan coordinate axis attributes based on the positional relationships between the Manhattan projection point, the first Manhattan matching point, the second Manhattan matching point, and each reference line segment in the preferred Manhattan world hypothesis comprises: converting the Manhattan projection point, the first Manhattan matching point, and the second Manhattan matching point into a preset normalized image coordinate system to obtain normalized projection points, normalized first matching points, and normalized second matching points; determining the positional relationships between at least a portion of the reference line segments projected into the preset normalized image coordinate system and the normalized projection points, the normalized first matching points, and the normalized second matching points, respectively; clustering the reference line segments based on the positional relationships to obtain categories of the reference line segments; and determining the average slope distribution of the reference line segments in each category to obtain Manhattan coordinate axis attributes. A rotation matrix determination module is used to determine the rotation matrix of the image acquisition device to the Manhattan world based on the projected Manhattan world coordinates, the first Manhattan world coordinates, the second Manhattan world coordinates, and the preset image acquisition device world coordinates, so as to perform posture calibration on the image acquisition device.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the image acquisition device posture calibration method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the image acquisition device posture calibration method according to any one of claims 1 to 8.

Images