Camera alignment method and camera alignment and monitoring method

Abstract

The invention provides a camera alignment method and a camera alignment and monitoring method. The method comprises the following steps: for a single camera, firstly, correcting a course angle and a pitch angle of the camera by acquiring a connecting line of an optical center and a vanishing point of the camera and combining the direction of a map lane line, and projecting a lane boundary into a three-dimensional space through lane boundary extraction and known height information in an image, so as to align the lane boundary with a map lane boundary; therefore, the horizontal position of the camera is corrected. For a multi-camera system, the poses of multiple cameras are optimized by calculating relative transformation information of a camera view overlapping area and combining lane line and stop line matching information of a single camera. Through an automatic algorithm, under the condition that the position error is 5m and the angle error is 10 degrees, automatic alignment of the camera pose can be rapidly completed, and the problem that a large amount of manpower is consumed in a scheme of carrying out camera pose alignment based on a position initial value and an angle initial value provided by a construction party at present is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular, to a camera alignment method, a camera alignment and monitoring method, a camera alignment device, a computer-readable storage medium, an electronic device, and a vehicle. Background Art

[0002] In V2X projects, cameras are widely used in communication technology between vehicles and the environment due to their advantages such as high cost-effectiveness, high recognition accuracy and good portability. One of the core links of the project is to use cameras to identify targets such as vehicles and pedestrians from images and convert their position information in two-dimensional space into coordinates in three-dimensional space. This process relies on the precise position and orientation information of the camera to achieve accurate mapping from two-dimensional images to three-dimensional space. In short, the precise "pose" of the camera is crucial to complete this conversion.

[0003] The general process of obtaining the camera's "pose" is: the first thing to do is to fix the camera device on a pole and calibrate the internal parameters. Next, based on the rough installation position provided by the construction party, the engineer will use a special parameter adjustment tool to manually adjust the position and angle of the camera to gradually become more precise. The purpose of this process is to enable the actual road elements captured by the camera, such as lane lines, stop lines, and zebra crossings, to accurately coincide with the corresponding road elements in the map.

[0004] The above approach faces two major challenges:

[0005] First: The initial position and angle values ​​provided by the construction party have large deviations. The initial position of the camera can only provide the cabinet position or the current road center (the center of multiple lanes), resulting in the reported camera position error of the order of 5m and the angle installation error of the order of 5-10°.

[0006] Second, the cost of labeling is high: Currently, for a large intersection (no less than 20 cameras), the manual calibration efficiency is 1 intersection / person / day. When the number of intersections involved in the project gradually increases, a large number of engineers will need to be hired to perform manual calibration and complete the subsequent adjustments after the camera position changes. Summary of the invention

[0007] The main purpose of the present application is to provide a camera alignment method, a camera alignment and monitoring method, a camera alignment device, a computer-readable storage medium, an electronic device and a vehicle, so as to at least solve the problem that the current solution of camera posture alignment based on the initial position value and the initial angle value provided by the construction party consumes a lot of manpower.

[0008] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a camera alignment method is provided, comprising: for a single camera, obtaining a line connecting the optical center of the single camera and a vanishing point, and correcting the heading angle and pitch angle of the single camera by aligning the line with the direction of a lane line on a map; extracting an image lane boundary from an image, and projecting the image lane boundary into a 3D space using a known height and the corrected heading angle and pitch angle of the single camera, and aligning the image lane boundary with the map lane boundary, so as to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space; for a multi-camera system, obtaining relative transformation information between cameras based on an area where fields of view overlap, and simultaneously aligning the postures of the multi-cameras according to the relative transformation information and the lane line matching information and stop line matching information of the single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation.

[0009] Optionally, before obtaining the line connecting the optical center of the single camera and the vanishing point, the method further includes: if the single camera is a same-direction camera, and the lane lines at the distant points have the same orientation, indicating that there are straight lanes before and after the intersection, all detection lines are used to calculate the vanishing point; if the lane lines at the distant points have different orientations, indicating that the intersection contains broken lanes, the vanishing point is calculated using the nearby detection lines in the lower half of the image, wherein the camera field of view of the same-direction camera is oriented in the same direction as the lane direction; if the single camera is a same-direction camera or a reverse camera, based on the candidate lane line detection results, according to the degree of discreteness of the detection points and the number of detection points, a preset detection line with the best linearity and the longest length is selected to calculate the vanishing point, wherein the camera field of view of the reverse camera is oriented in the opposite direction to the lane direction.

[0010] Optionally, a preset detection line with the best linearity and the longest length is selected to calculate the vanishing point, including: determining the preset detection line with the best linearity and the longest length as a pre-selected lane line; obtaining the intersection of any two of the pre-selected lane lines, and obtaining the sum of the distances from any intersection to all the pre-selected lane lines; and determining the vanishing point as the intersection with the smallest sum of the distances.

[0011] Optionally, before obtaining the line connecting the optical center of a single camera and the vanishing point, the method further includes: for a same-direction camera, if there are straight lanes before and after the intersection, the average value of the lane direction of the distant point and the lane direction of the near point is determined as the direction of the lane line on the map; for a same-direction camera, if the intersection contains a zigzag lane, the direction of the lane where the current rough position is located is determined as the direction of the lane line on the map; for a reverse camera, the difference between the average value of the lane direction of the distant point and the lane direction of the near point and 180° is determined as the direction of the lane line on the map.

[0012] Optionally, the heading angle and pitch angle of the single camera are corrected by aligning the connecting line with the direction of the map lane line, including: acquiring a standard camera coordinate system, a correction camera coordinate system, a standard world coordinate system and a correction world coordinate system, wherein the correction camera coordinate system is obtained after correction of the standard camera coordinate system, and the correction world coordinate system is obtained after correction of the standard world coordinate system, and the positive direction of the x-axis of the correction world coordinate system is the direction of the map lane line; acquiring a rotation matrix converted from the correction world coordinate system to the correction camera coordinate system; and determining the heading angle and pitch angle of the single camera according to the rotation matrix.

[0013] Optionally, extracting image lane boundaries in an image includes: for a same-direction camera, sorting the detected lane lines from right to left, if the current line is a solid line and the left adjacent line is a dotted line, then the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the left adjacent line is a solid line, then the left adjacent line corresponds to the left boundary of the actual road; for a reverse camera, sorting the detected lane lines from left to right, if the current line is a solid line and the right adjacent line is a dotted line, then the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the right adjacent line is a solid line, then the right adjacent line corresponds to the left boundary of the actual road.

[0014] Optionally, the method also includes: extracting the map lane boundary before correcting the horizontal position of the single camera; wherein, extracting the map lane boundary is specifically implemented as follows: for the same-direction camera, extending a preset distance in the forward direction of the current lane, and continuously searching laterally for the leftmost and rightmost boundary lanes during the extension process to extract the map lane boundary; for the reverse camera, extending a preset distance in the reverse direction of the current lane, and continuously searching laterally for the leftmost and rightmost boundary lanes during the extension process to extract the map lane boundary; if the extension is less than the preset extension distance, re-starting the extension search for the map lane boundary from the adjacent lane.

[0015] Optionally, an image lane boundary is extracted from the image, and the image lane boundary is projected into 3D space and aligned with the map lane boundary using the known height and the corrected heading angle and pitch angle of the single camera to correct the horizontal position of the single camera, including: using the known height and the corrected heading angle and pitch angle of the single camera, projecting the image lane boundary into 3D space by the principle of similar triangles; performing point-to-line matching processing and residual vector calculation after projection to correct the horizontal position of the single camera.

[0016] Optionally, after performing point-to-line matching processing and residual vector calculation after projection to correct the horizontal position of the single camera, the method further includes: applying a full amount of lane line and stop line detection results and corresponding map elements for matching, and optimizing the position and posture of the single camera; wherein the matching process includes a segmentation step, a projection step, a matching residual calculation step, and a matching result screening step; and the optimization process includes a line segment endpoint coordinate calculation step and an optimized residual construction step.

[0017] Optionally, for a multi-camera system, relative transformation information between cameras is calculated based on the area where the fields of view overlap, including: performing feature extraction processing on the area where the fields of view overlap to obtain key points, and using feature descriptions to describe local image information around the key points; using the feature descriptions to match feature points in two camera images to obtain matching results; and calculating a transformation matrix between the two camera images based on the matching results.

[0018] According to another aspect of the present application, a method for camera alignment and monitoring is provided, comprising: for a single camera, obtaining a line connecting the optical center of the single camera and a vanishing point, and correcting the heading angle and pitch angle of the single camera by aligning the line with the direction of a lane line on a map; extracting an image lane boundary from an image, and projecting the image lane boundary into a 3D space using a known height and the corrected heading angle and pitch angle of the single camera, and aligning the image lane boundary with the map lane boundary, so as to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space; for a multi-camera system, obtaining relative transformation information between cameras based on an area where fields of view overlap, and obtaining the relative transformation information based on the relative transformation information and the lane line matching information, stop line matching information, and the single camera lane matching information in the multi-camera system. The relative transformation information is used to align the posture of multiple cameras at the same time, wherein the relative transformation information includes relative position and angle transformation; after the posture of multiple cameras in the single camera and the multi-camera system is calibrated, the semantic elements in the map are projected into the image for each camera, and matched with the corresponding elements detected on the image. If the number of matches is less than the number threshold, a first alarm message is generated, and the first alarm message indicates that the position and posture of the current camera have changed; if the number of matches is greater than or equal to the number threshold, if the matching residual about the matching distance in the image coordinate system continues to be large, a second alarm message is generated; if the matching residual is within the preset residual range, the position and posture of the camera are optimized again according to the matching residual, and compared with the initial position and initial posture. If the position deviation or angle deviation continues to increase, a third alarm message is generated.

[0019] Optionally, the method further includes: for the installation method of multiple cameras on the same pole, detecting whether the optimized heights of the multiple cameras are consistent, and if not, generating a fourth alarm message.

[0020] According to another aspect of the present application, a camera alignment device is provided, comprising: a first alignment unit, for obtaining, for a single camera, a line connecting the optical center of the single camera and a vanishing point, and correcting the heading angle and pitch angle of the single camera by aligning the line with the direction of a lane line on a map; extracting an image lane boundary from an image, and projecting the image lane boundary into a 3D space using a known height and the corrected heading angle and pitch angle of the single camera, and aligning the image lane boundary with the map lane boundary, so as to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space; and a second alignment unit, for a multi-camera system, calculating relative transformation information between cameras based on an area where fields of view overlap, and simultaneously aligning the postures of the multi-cameras according to the relative transformation information and the lane line matching information and the stop line matching information of the single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation.

[0021] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0022] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0023] According to another aspect of the present application, a vehicle is provided, comprising: a camera installed on the vehicle, wherein the camera is aligned using any one of the camera alignment methods.

[0024] By applying the technical solution of the present application, the posture of single-camera and multi-camera systems is accurately calibrated and optimized through a vision-based algorithm. For a single camera, the camera's heading angle and pitch angle are first corrected by obtaining the line connecting the camera's optical center and the vanishing point, combined with the direction of the map lane line, and the lane boundary is projected into the three-dimensional space through the lane boundary extraction and known height information in the image, and aligned with the map lane boundary, thereby correcting the horizontal position of the camera. The vanishing point is a point in the imaging plane that is reflected by a set of parallel lines that are not parallel to the imaging plane intersecting at an infinite point in three-dimensional space. For a multi-camera system, the posture of multiple cameras is optimized by calculating the relative transformation information of the overlapping area of ​​the camera's field of view, and combining the lane line and stop line matching information of a single camera, thereby improving the overall accuracy and consistency of the system. The present application uses an automated algorithm to quickly complete the automatic alignment of the camera posture with a position error of 5m and an angle error of 10 degrees, effectively solving the problem that the current solution for camera posture alignment based on the initial position value and angle initial value provided by the construction party consumes a lot of manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 A hardware structure block diagram of a mobile terminal for executing a camera alignment method provided in an embodiment of the present application is shown;

[0027] Figure 2 A schematic flow chart of a camera alignment method provided according to an embodiment of the present application is shown;

[0028] Figure 3 A schematic diagram showing a vanishing point definition in a camera alignment method provided according to an embodiment of the present application is shown;

[0029] Figure 4 A schematic diagram of a coordinate system in a camera alignment method provided according to an embodiment of the present application is shown;

[0030] Figure 5 A schematic diagram of determining a lane boundary in a camera alignment method provided according to an embodiment of the present application is shown;

[0031] Figure 6 A schematic diagram showing correction based on the similar triangle principle in a camera alignment method provided in an embodiment of the present application is shown;

[0032] Figure 7 A schematic diagram of point-to-line matching in a camera alignment method provided according to an embodiment of the present application is shown;

[0033] Figure 8 A schematic flow chart of a camera alignment and monitoring method provided according to an embodiment of the present application is shown;

[0034] Fig. 9 A structural block diagram of a camera alignment device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0039] XFeat (eXtreme Feature Engineering): A feature engineering automation framework that aims to optimize the performance of machine learning models by automating feature selection, generation, and transformation. It can automatically select the most relevant features, perform feature crossover and transformation, reduce manual intervention, and improve efficiency when processing high-dimensional data.

[0040] IPM (Interior Point Method): A type of algorithm used to solve linear and nonlinear optimization problems. It avoids the extreme point search of traditional exterior point methods by iterating within the feasible region. The interior point method ensures the feasibility of the solution by introducing a barrier function and gradually approaches the optimal solution. It has good computational efficiency and theoretical performance, and is particularly suitable for large-scale optimization problems. It is widely used in operations research, control theory and other fields.

[0041] KD-Tree (k-dimensional tree): A spatial partitioning data structure for multidimensional data, mainly used for efficient neighbor search, range query and other operations. It organizes multidimensional data by recursively dividing data points into multiple hyper-rectangular regions, thereby accelerating spatial query operations.

[0042] As introduced in the background technology, the initial position and angle errors of the camera provided by the construction party in the prior art are large, and usually only the cabinet position or the center of the road can be provided, resulting in a position error of up to 5 meters and an angle error of 5-10°. In addition, manual calibration is inefficient, especially in large intersections (at least 20 cameras), where only a calibration efficiency of 1 intersection / person / day can be achieved. As the scale of the project expands, a large number of engineering personnel are required for calibration and adjustment, which significantly increases the cost. In order to solve the problem that the current solution for camera pose alignment based on the initial position value and angle value provided by the construction party consumes a lot of manpower, the embodiments of the present application provide a camera alignment method, a camera alignment and monitoring method, a camera alignment device, a computer-readable storage medium, an electronic device, and a vehicle.

[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0044] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a camera alignment method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1More or fewer components as shown, or with Figure 1 Different configurations shown.

[0045] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the camera alignment method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The specific example of the above-mentioned network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] In this embodiment, a method for camera alignment running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] Figure 2 FIG. 1 is a flow chart of a method for aligning a camera according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0048] Step S201, for a single camera, obtain a line connecting the optical center of the single camera and a vanishing point, and align the line with the direction of the lane line on the map to correct the heading angle and pitch angle of the single camera; extract the image lane boundary from the image, and project the image lane boundary into the 3D space using the known height and the corrected heading angle and pitch angle of the single camera and align it with the lane boundary on the map to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space;

[0049] Specifically, for the alignment of a single camera, first obtain the line connecting the optical center of the single camera and the vanishing point. The vanishing point is the point in the imaging plane where all parallel lines that are not parallel to the imaging plane intersect, and is usually located at a specific position in the image. By aligning the line connecting the optical center of the camera and the vanishing point with the direction of the lane line on the map, the heading angle and pitch angle of the single camera can be effectively corrected. This method relies on the geometric relationship between the lane line and the vanishing point, and can achieve high-precision angle calibration.

[0050] Furthermore, the lane lines in the image are extracted, and combined with the known height of the camera and the heading and pitch angles calculated by the vanishing point, the lane boundaries are projected into the 3D space using the projection of the image plane to the 3D space (such as the IPM algorithm). By comparing with the lane boundaries in the map, the matching residual is calculated to correct the horizontal position of the camera. This process can automatically and quickly achieve accurate alignment of the camera position without relying on traditional manual calibration methods.

[0051] Step S202: For a multi-camera system, relative transformation information between cameras is calculated based on the overlapping area of ​​field of view, and the postures of the multi-cameras are simultaneously aligned according to the relative transformation information and lane line matching information and stop line matching information of a single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation.

[0052] Specifically, for multi-camera alignment, the relative transformation information between cameras, including position and angle transformation, is calculated by analyzing the overlapping areas of view between each camera. In this process, feature extraction and matching algorithms (such as the XFeat algorithm, etc.) are used to extract and match feature points in the common view area between cameras, and further calculate the relative transformation information between cameras. This information can not only be used to correct the posture of a single camera, but also integrate the lane line and stop line matching information of multiple cameras through optimization algorithms to coordinate the posture of the entire multi-camera system. By optimizing the posture of each camera relative to the other cameras, it is possible to ensure that the position and orientation of all cameras in the multi-camera system remain consistent, thereby improving the accuracy and robustness of the entire system.

[0053] Through this embodiment, the problem of large deviations in the initial position and angle of the camera is successfully solved through an automated visual calibration algorithm, and accurate automatic alignment of single-camera and multi-camera systems is achieved. A single camera quickly corrects the orientation and position of the camera by aligning the vanishing point with the lane line, reducing the complexity and cost of manual calibration. In a multi-camera system, the coordination accuracy and consistency of the entire system are improved by matching the relative transformation information between cameras with lane lines and stop lines. Ultimately, this technology significantly improves the camera positioning accuracy in V2X applications, reduces the need for manual calibration, and maintains high stability and reliability in complex environments, effectively solving the problem that the current solution for camera posture alignment based on the initial position and angle values ​​provided by the construction party consumes a lot of manpower.

[0054] In the specific implementation process, before the above step S201, the method further includes the following steps:

[0055] Step S301: If the single camera is a same-direction camera, and the lane lines at the distant points have the same orientation, indicating that there are straight lanes before and after the intersection, all detection lines are used to calculate the vanishing point; if the lane lines at the distant points have different orientations, indicating that the intersection contains a broken lane, the detection lines in the near lower half of the image are used to calculate the vanishing point, wherein the camera field of view of the same-direction camera has the same orientation as the lane direction;

[0056] Specifically, Figure 3 FIG. 1 is a schematic diagram of the definition of the vanishing point in the camera alignment method according to an embodiment of the present application. Figure 3 As shown, the vanishing point is the intersection of all parallel lines in space that are not parallel to the imaging plane at infinity. It appears as a specific point in the image, and the line connecting the camera's optical center and the vanishing point is parallel to these parallel lines in three-dimensional space. For the application scenarios of this application, lane lines, as typical straight lines, can be used as a comparison reference for vanishing points. By aligning the line connecting the camera's optical center and the vanishing point with the map lane direction in three-dimensional space, the camera's pitch angle and heading angle can be accurately calculated.

[0057] Furthermore, by judging whether the intersection is a straight lane or a zigzag lane based on the direction of the lane line, the detection lines used to calculate the vanishing point can be reasonably selected. This judgment method based on the direction of the lane line and the shape of the intersection can more accurately adapt to different scenarios and ensure the accuracy of the vanishing point calculation. In the case of a camera in the same direction (the camera field of view is in the same direction as the lane), if the lane lines are in the same direction and are straight lanes, using all detection lines to calculate the vanishing point can provide more comprehensive reference information and enhance the accuracy of the calculation; for the case of a zigzag lane, using the detection lines in the lower half of the image can eliminate interference factors and provide a more stable vanishing point calculation.

[0058] Step S302: If the single camera is a forward-facing camera or a reverse-facing camera, based on the candidate lane line detection results, according to the degree of dispersion of the detection points and the number of detection points, select the preset number of detection lines with the best linearity and the longest length to calculate the vanishing point. Among them, the camera field of view of the reverse-facing camera is opposite to the lane direction.

[0059] Specifically, according to the detection results of the candidate lane lines, select appropriate detection lines for vanishing point calculation. By considering the degree of dispersion and the number of detection points, preferentially select the detection lines with the best linearity and the longest length, which can effectively improve the stability and accuracy of vanishing point calculation. For the reverse-facing camera (the camera field of view is opposite to the lane direction), since the lane lines in the field of view are often continuous, 5 detection lines with the best linearity and the longest length can be directly selected according to the degree of dispersion of the detection points and the number of detection points to calculate the vanishing point.

[0060] As a possible implementation, the above step 302 can be implemented through the following steps:

[0061] Step S3021: Determine the preset number of detection lines with the best linearity and the longest length as the preselected lane lines;

[0062] Specifically, first select 5 detection lines with the best linearity and the longest length as the preselected lane lines. This selection ensures that the geometric shape represented by the selected lane lines most conforms to the actual road characteristics and ensures the accuracy of vanishing point calculation.

[0063] Step S3022: Obtain the intersection points of any two preselected lane lines, and obtain the sum of the distances from any intersection point to all the preselected lane lines;

[0064] Specifically, for all the preselected lane lines, use a double loop for pairing. For each pair of preselected lane lines line[i] and line[j] (where i < j), calculate the intersection point of these two lines. The intersection point is the point where these two lines meet on the image plane and represents a possible vanishing point candidate. For each intersection point, calculate the distance from this intersection point to all other preselected lane lines. These distances measure the geometric relationship between the intersection point and each lane line and reflect the possibility of whether the intersection point is suitable as the vanishing point.

[0065] Step S3023: Determine the vanishing point determined by the intersection point with the minimum sum of distances.

[0066] Specifically, by calculating the sum of the distances from all intersection points to the preselected lane lines, select the intersection point with the minimum sum of distances as the final vanishing point. This method ensures that the selected vanishing point best conforms to the geometric relationship of all lane lines, thereby improving the accuracy and precision of the vanishing point.

[0067] As a possible implementation, before obtaining the line connecting the optical center of the single camera and the vanishing point in step 201, it is necessary to first determine the direction of the lane line on the map. In order to accurately correct the heading angle and pitch angle of the single camera, preprocessing can be performed through the following steps:

[0068] Step S401: for the same-direction camera, if there are straight lanes before and after the intersection, the average of the lane direction of the distant point and the lane direction of the near point is determined as the map lane line direction;

[0069] Step S402: for the same-direction camera, if the intersection contains a broken-line lane, the direction of the lane where the current rough position is located is determined as the direction of the lane line on the map;

[0070] Specifically, for cameras in the same direction, if there are straight lanes before and after the intersection, the direction of the map lane line can be determined by calculating the average of the directions of the lanes where the distant point and the near point are located. Since the lanes are parallel and in the same direction, taking the average of the two can accurately obtain the direction of the lane. If the intersection contains a broken lane, the direction of the map lane line can be estimated by determining the direction of the lane where the current rough position is located. At this time, the lanes at the intersection may turn, but by locating the current position of the camera and calculating the direction of the lane it is in, the direction of the map lane line can be inferred more accurately.

[0071] Step S403: for the reverse camera, the difference between the average of the lane directions of the distant point and the lane directions of the near point and 180° is determined as the map lane line direction.

[0072] Specifically, for the reverse camera, it is necessary to calculate the average of the lane directions of the distant point and the lane directions of the near point, and combine them with the 180° difference to determine the direction of the map lane line. Since the field of view of the reverse camera is opposite to the lane direction, it is necessary to calculate the direction of the map lane line by taking the complementary angle of its direction.

[0073] As a possible implementation method, it is necessary to obtain the pitch angle and heading angle of the camera by calculating the conversion matrix between the coordinate systems. The above step S201 can be implemented by the following steps:

[0074] Step S501, obtaining a standard camera coordinate system, a calibrated camera coordinate system, a standard world coordinate system, and a calibrated world coordinate system, wherein the calibrated camera coordinate system is obtained after calibrating the standard camera coordinate system, the calibrated world coordinate system is obtained after calibrating the standard world coordinate system, and the positive direction of the x-axis of the calibrated world coordinate system is the direction of the lane line on the map;

[0075] Specifically, the calibrated camera coordinate system is obtained by calibration based on the standard camera coordinate system, with the purpose of correcting errors caused by camera position and angle deviation. The calibrated world coordinate system is obtained by calibration based on the standard world coordinate system, with the purpose of being consistent with the direction of the map lane line, so that the positive direction of the x-axis of the coordinate system is consistent with the direction of the map lane line, which is convenient for subsequent angle correction.

[0076] Furthermore, the coordinate system is defined first: standard camera coordinate system (ccs): x-axis facing right, y-axis facing down, z-axis facing forward; corrected camera coordinate system (corrected_ccs): x-axis facing forward, y-axis facing left, z-axis facing up; standard world coordinate system (wcs): x-axis pointing east, y-axis pointing north, z-axis facing up. Corrected world coordinate system (corrected_wcs): "map lane line direction" is the positive direction of x-axis, y-axis direction is derived by the right-hand rule, and z-axis is facing up.

[0077] Step S502, obtaining a rotation matrix for converting from a calibrated world coordinate system to a calibrated camera coordinate system;

[0078] Specifically, the rotation matrix R between the corrected camera coordinate system (corrected_ccs) and the corrected world coordinate system (corrected_wcs) wc It can be expressed as rotation around the x-axis, y-axis and z-axis in sequence.

[0079] Furthermore, the calculation formula of the rotation matrix is: R wc =R θ R β R α ; The rotation matrix is: Among them, α, β, and θ are the rotation angles around the x, y, and z axes respectively.

[0080] Step S503: determining the heading angle and pitch angle of the single camera according to the rotation matrix.

[0081] Specifically, the first column of the rotation matrix represents the direction vector of the camera: Since the position of the vanishing point on the image has been calculated, the direction of the line connecting the optical center and the vanishing point can be obtained after conversion to the normalized coordinate system. This direction is parallel to r1, from which the pitch angle φ and heading angle of the corrected camera coordinate system corrected_ccs can be obtained

[0082] Further, Figure 4 is a schematic diagram of a coordinate system provided in an embodiment of the present application, such as Figure 4 As shown, the conversion relationship between the corrected camera coordinate system (corrected_ccs) and the standard camera coordinate system (ccs) is: Since the corrected world coordinate system (corrected_wcs) takes the "lane line orientation on the map" as the positive x-axis direction, the camera's heading angle needs to compensate for the "lane line orientation on the map" to get the final camera orientation. Finally, the calculated rotation from the standard camera coordinate system (ccs) to the standard world coordinate system (wcs) is used as the camera orientation to complete the camera posture correction.

[0083] After completing the angle correction, the orientation of the single camera is close to correct. However, in order to further correct the large deviation in position (especially when the deviation exceeds the width of a lane), position correction based on road boundary information is required. This process extracts the boundary information in the detection results and projects the detected boundary information onto the 3D ground in combination with the angle and height priors. By comparing and matching with the boundaries in the map, the residual vector of the position is calculated, and the position is corrected based on this residual vector.

[0084] In the specific implementation process, in order to further accurately determine the lane boundary position of the single camera, the above step S201 can also be implemented by the following steps:

[0085] Step S601: for the same-direction camera, the detected lane lines are sorted from right to left. If the current line is a solid line and the left adjacent line is a dotted line, the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the left adjacent line is a solid line, the left adjacent line corresponds to the left boundary of the actual road;

[0086] Step S602, for the reverse camera, sort the detected lane lines from left to right. If the current line is a solid line and the right adjacent line is a dotted line, the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the right adjacent line is a solid line, the right adjacent line corresponds to the left boundary of the actual road.

[0087] Specifically, since there is currently no determination of lane direction, if the solid line detection results are directly matched, mismatching may occur at the boundary of the two-way lane. Therefore, it is necessary to distinguish between the forward and reverse cameras, and extract the left and right boundary information of the lane according to the camera orientation, where the current line is the current lane line, the left adjacent line is the left adjacent lane line, and the right adjacent line is the right adjacent lane line. Figure 5 FIG. 1 is a schematic diagram of determining lane boundaries according to the present application. Figure 5 As shown, Figure 5 The upper middle part is a schematic diagram of the same-direction camera judging lane boundaries. Figure 5 The lower middle part is a schematic diagram of the reverse camera judging lane boundaries.

[0088] Furthermore, for the same-direction camera, the detected lane lines need to be sorted from right to left, and the boundary information is extracted according to the following rules: if the current lane line is a solid line and the left adjacent lane line is a dotted line, then the current lane line corresponds to the right boundary of the actual road; if the current lane line is a dotted line and the left adjacent lane line is a solid line, then the left adjacent lane line corresponds to the left boundary of the actual road.

[0089] Furthermore, for the reverse camera, the detected lane lines need to be sorted from left to right, and the boundary information is extracted according to the following rules: if the current lane line is a solid line and the right adjacent lane line is a dotted line, then the current lane line corresponds to the right boundary of the actual road; if the current lane line is a dotted line and the right adjacent lane line is a solid line, then the right adjacent lane line corresponds to the left boundary of the actual road.

[0090] As a possible implementation, step 201 further includes: extracting the map lane boundary before correcting the horizontal position of the single camera to ensure positioning accuracy. The specific implementation steps of extracting the map lane boundary are as follows:

[0091] Step S701, for the same-direction camera, the current lane is extended forward by a preset distance, and the leftmost and rightmost boundary lanes are continuously searched horizontally during the extension process to extract the map lane boundary;

[0092] Step S702, for the reverse camera, extending the current lane in the reverse direction by a preset distance, and continuously searching for the leftmost and rightmost boundary lanes in the horizontal direction during the extension process to extract the map lane boundary;

[0093] Step S703: If the extension is less than the preset extension distance, the map lane boundary is searched again from the adjacent lane.

[0094] Specifically, based on the rough position of the current camera, the leftmost and rightmost boundary information of the current lane is extracted. These boundary information needs to meet the following conditions: the lane itself should be a drivable lane and a non-turning lane; the lane has no adjacent lanes further out, or the adjacent lanes are all non-drivable lanes or turning lanes.

[0095] Furthermore, for the same-direction camera, the lane can be extended 200 meters in the forward direction, and the leftmost and rightmost lane boundaries can be searched laterally step by step. Through this process, the boundary information of the current lane can be extracted as part of the map lane boundary; for the reverse camera, the lane can be extended 200 meters in the reverse direction, and the leftmost and rightmost lane boundaries can be searched laterally step by step. This method ensures that the reverse camera can also extract accurate lane boundary information; if the extension of the current lane fails to reach 200 meters (for example, the lane has entered the turning area or other reasons have limited the extension), the extension search will be restarted from the adjacent lane to ensure that the complete lane boundary information is extracted.

[0096] As a possible implementation, the above step S201 can be implemented by the following steps:

[0097] Step S801, using the known height and the calibrated heading angle and pitch angle of the single camera, project the image lane boundary into the 3D space through the principle of similar triangles;

[0098] Specifically, Figure 6 is a schematic diagram of correction according to the similar triangle principle provided in an embodiment of the present application, such as Figure 6 As shown in the figure, the similar triangle principle can accurately convert the two-dimensional lane boundaries in the image into corresponding points in three-dimensional space by establishing a geometric relationship between the camera's perspective and the actual ground. Using the known camera height and the corrected heading and pitch angles, the lane boundaries in the image are projected into three-dimensional space using the similar triangle principle. This process facilitates the subsequent matching of lane boundaries and stop lines with map data and the calculation of position residuals, thereby correcting the camera position.

[0099] Step S802: After projection, point-to-line matching processing and residual vector calculation are performed to correct the horizontal position of the single camera.

[0100] Specifically, Figure 7 This is a schematic diagram of point-to-line matching provided by this application, such as Figure 7 As shown in the figure, when matching lane boundaries and stop lines, the following method is used to ensure accurate matching: the midpoints of lane lines and stop lines are aggregated, and an efficient search structure is established through the KD-Tree algorithm. KD-Tree can quickly find the line segment closest to the current monitoring point, ensuring the efficiency and accuracy of the matching process. Since the initial lateral position error may be large (for example, due to the camera deviating from the center line of the road), the maximum lateral distance deviation from the point to the line is allowed to be 5 meters. This screening condition can effectively tolerate large initial errors and still ensure the effectiveness of matching. The matching accuracy is ensured by screening the longitudinal projection of the point to the line. If the longitudinal distance between the projection point and the line segment exceeds 1 meter, it is considered that there is a high possibility of an incorrect match, so the matching results that do not meet the conditions need to be eliminated.

[0101] Furthermore, after matching the lane boundary and the parking line, the residual vector can be calculated based on the deviation between the detection point and the map segment, and the camera position can be corrected by these vectors. w , the starting point of the map line segment is map s , the end point is map e , then the direction vector l of the map segment w It can be calculated by the difference between the starting point and the end point of the map segment: w =(map e-map s ).normalized(), where normalized() means normalizing the line segment direction vector. The residual vector residual can be obtained by the cross product method: residual = l w ×[l w ×(P w -map s )], by averaging the residual vectors of multiple detection points, a correction vector correction can be obtained, which is used to adjust the camera position: Among them, num represents the number of detection points, and the weighted average of the residual vectors of all detection points is used to calculate the final correction vector. Finally, according to the correction vector, the position of the camera in the standard world coordinate system is adjusted to obtain the position of the calibrated camera: P w-corrected =P w -correction.

[0102] After completing the rough alignment, the angle and position of the camera are already at relatively accurate initial values ​​(for example, the angle error is 0.5° and the position error is 0.5m). Next, fine alignment can be performed through more precise steps. The goal of fine alignment is to further optimize the position and orientation of the camera to improve positioning accuracy.

[0103] As a possible implementation, after performing point-to-line matching processing and residual vector calculation after projection to correct the horizontal position of the single camera in the above step 201, the method further includes:

[0104] Use the full amount of lane line and stop line detection results and the corresponding map elements for matching, and optimize the position and posture of a single camera;

[0105] The matching process includes a segmentation step, a projection step, a matching residual calculation step and a matching result screening step; the optimization process includes a line segment endpoint coordinate calculation step and an optimization residual construction step.

[0106] Specifically, the matching process in the precise alignment includes: dividing the lane lines and stop lines in the map into multiple 20cm segments; projecting the two endpoints of all map segments onto the image plane (ICS) according to the camera pose and intrinsic parameters, and calculating the center of the segment; traversing all lane line and stop line detection points in the image, finding the map projection segment with the closest distance in the corresponding category as the matching result, and calculating the distance from the point to the segment on the ICS as the matching residual; counting the matching residuals of all matching results, calculating the mean and standard deviation; calculating the adaptive matching threshold: threshold = mean + 3 * stddev, this threshold is set based on the distribution of most matching points, usually the matching results with residuals less than the threshold are considered reasonable, and those exceeding the threshold may be mismatched; screening all matching results. If the residual of a matching result exceeds the adaptive matching threshold, the matching result is considered unacceptable and should be discarded. This ensures that only high-quality matching results are considered in the final optimization.

[0107] Furthermore, during the fine alignment process, the optimization process mainly involves further optimization through matching results and accurate estimation of the camera pose.

[0108] The optimization process includes: for each matched detection point, first convert it from the image coordinate system (ICS) to the normalized camera coordinate system (NCCS) according to the camera's intrinsic parameters: P det-nccs =project(P det ), where P det is the coordinate of the detected lane line or stop line, and project is the projection transformation from the image coordinate system to the NCCS. Then based on the camera's pose T c w , project the two endpoints of the lane segment detected on the map into the standard camera coordinate system (CCS) and calculate the two endpoints P of the matching segment a-ccs and P b-ccs Coordinates in NCCS. The coordinates of the two endpoints in the standard camera coordinate system (CCS) are expressed as: a-ccs =(T c w ) -1 *P a-wcs , P b-ccs =(T c w ) -1 *P b-wcs Among them, Pa-wcs and P b-wcs is the coordinate of the endpoint of the map segment, T c wIt is the transformation matrix from the standard camera coordinate system to the standard world coordinate system. The coordinates of the two endpoints in the normalized camera coordinate system (NCCS) are expressed as: Direction of map line segments in NCCS: line nccs =(P b-nccs -P a-nccs ).normalized(), the direction vector is used for subsequent calculation of matching residuals. Construct optimized residuals: error = (P det-nccs -P a-nccs )×line nccs , add all the calculated residuals to the optimizer (such as Ceres optimizer), and optimize the camera pose T by minimizing the residuals c w , the Ceres optimizer will adjust the camera pose parameters according to the size of the residual, gradually approaching the optimal solution. After the optimization process is completed, the optimizer will output the optimized camera pose T c w , this pose can be used as the final camera position and orientation for subsequent system calculations.

[0109] When a single camera optimizes its posture independently, the result may not meet the consistency of the area with overlapping fields of view due to the influence of positioning accuracy. Therefore, based on the precise alignment of the camera posture, the association between cameras is completed by aligning the areas with overlapping fields of view between the cameras, and the camera systems with multiple areas with overlapping fields of view are optimized together.

[0110] In a specific implementation process, for a multi-camera system in step S202, obtaining relative transformation information between cameras based on the area where the fields of view overlap may be achieved by the following steps:

[0111] Step S901, performing feature extraction processing on the area where the fields of view overlap, obtaining key points, and using feature description to describe the local image information around the key points;

[0112] Specifically, by analyzing the shooting parameters and positional relationship of the two cameras, the area where their fields of view overlap is determined. This area of ​​overlapping fields of view is the part that can be captured by both cameras and usually contains some common objects or scene features. Using the camera's intrinsic and extrinsic parameter matrices, the correspondence between the two camera image planes can be calculated to determine the range of the area where the fields of view overlap. In the determined area of ​​overlapping fields of view, the images of the two cameras are processed using the advanced XFeat feature extraction algorithm. These algorithms can detect key points in the image, such as corner points, edge points, etc., and generate a unique feature description for each key point. The feature description usually contains local image information around the key point, such as color, texture, gradient, etc., so that key points in different images can be matched by comparing feature descriptions.

[0113] Step S902, applying feature description to match feature points in two camera images to obtain a matching result;

[0114] Specifically, the similarity between features is characterized by calculating the Euclidean distance between feature descriptions, matching the feature points in the two camera images, and finding the corresponding key point pairs in the overlapping area of ​​the field of view. In order to improve the accuracy of matching, the RANSAC (random sampling consensus) algorithm is used to remove incorrect matching point pairs.

[0115] Step S903: Calculate the transformation matrix between the two camera images according to the matching result.

[0116] Specifically, after feature matching is completed, the transformation matrix between the two camera images can be calculated based on the coordinate relationship of the matching point pairs. This transformation matrix can transform the image of one camera into the coordinate system of another camera, so that they can be seamlessly spliced ​​in the area where the field of view overlaps. Output the transformation matrix between the two cameras Describes the relative pose between two cameras.

[0117] Furthermore, the common view system is optimized. The common view system is a system composed of cameras with overlapping fields of view. The core goal is to ensure the consistency of multiple cameras in the common view area by jointly optimizing their poses, while minimizing the reprojection error and relative pose error. Among them, the reprojection error of a single camera is consistent with the precise alignment part. The relative residual between the newly added cameras includes: establishing a relative pose transformation between two cameras with "common view relative pose" During the optimization process, the camera poses of the two cameras It needs to change gradually to meet the following conditions: the reprojection error of each lane line and stop line is minimized; the relative position, orientation and The deviation is the smallest. Finally, the poses of each camera under the overall optimized structure are output.

[0118] According to another aspect of the present application, a camera alignment and monitoring method is provided. Figure 8 FIG. 1 is a flow chart of a method for camera alignment and monitoring according to an embodiment of the present application. Figure 8 As shown, the following steps are included:

[0119] Step S1101: for a single camera, a line connecting the optical center of the single camera and a vanishing point is obtained, and the heading angle and pitch angle of the single camera are corrected by aligning the line with the direction of the lane line on the map; the image lane boundary is extracted from the image, and the image lane boundary is projected into the 3D space using the known height and the corrected heading angle and pitch angle of the single camera and aligned with the map lane boundary to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space;

[0120] Specifically, by extracting the line connecting the optical center of a single camera and the vanishing point, and combining the map lane line orientation to correct the heading and pitch angles of the single camera, the camera can be quickly and accurately aligned with a large initial error. By extracting the lane boundary from the image and using the known height and the corrected heading and pitch angles, the lane boundary is further projected into 3D space and aligned with the map lane boundary, effectively correcting the horizontal position of the camera. This method allows the camera's posture to be automatically and quickly accurately aligned even when there are large deviations in the initial position and angle values, reducing the labor cost and error in the traditional manual calibration process.

[0121] Step S1102: for a multi-camera system, relative transformation information between cameras is calculated based on the overlapping area of ​​the field of view, and the postures of the multi-cameras are simultaneously aligned according to the relative transformation information and lane line matching information and stop line matching information of a single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation;

[0122] Specifically, in a multi-camera system, the relative transformation information between cameras is calculated based on the overlapping area of ​​the field of view, and combined with the matching results of the lane lines and stop lines of a single camera, the postures of multiple cameras can be accurately aligned. Through relative transformation information, including the adjustment of relative position and angle, the overall synergy of the multi-camera system is optimized, and the positioning accuracy of each camera is improved. This solution can not only ensure the individual accuracy of each camera, but also achieve consistent correction between multiple cameras, ensuring the accuracy and stability of the multi-camera system, and reducing the inconsistency and positioning errors caused by the posture errors between cameras.

[0123] Step S1103, after the position and posture of multiple cameras in a single-camera or multi-camera system are calibrated, for each camera, a semantic element in the map is projected into the image, and matched with the corresponding element detected on the image. If the number of matches is less than a number threshold, a first alarm message is generated, and the first alarm message indicates that the position and posture of the current camera have changed;

[0124] Specifically, when powered on for the first time, the detection features on the image are matched with the map elements based on the installation position and orientation information, and the optimized position, posture, and initial matching residuals of various map elements are recorded. The camera matching is continuously run. The camera device projects the semantic elements such as lane lines, stop lines, zebra crossings, poles, etc. in the map into the image through the initial position and posture, and matches them with the corresponding elements detected in the image. If the number of matches remains low for a certain period of time, it is determined that the position and posture of the current device have changed, and an early warning message is reported.

[0125] Step S1104: if the number of matches is greater than or equal to the number threshold, and if the matching residual about the matching distance in the image coordinate system is continuously large, then generating a second warning message;

[0126] Specifically, if the match is successful, the matching distance in the image coordinate system is checked within a certain period of time. If the matching residual is continuously larger than the initial matching residual, an early warning message is reported to indicate a possible abnormality.

[0127] Step S1105: If the matching residual is within the preset residual range, the position and posture of the camera are optimized again according to the matching residual, and compared with the initial position and initial posture. If the position error or angle deviation continues to increase, a third alarm information is generated.

[0128] Specifically, if the matching residual is within the threshold range, the position and posture of the camera device are optimized according to the matching residual and compared with the initial position and posture. If the position error or angle deviation continues to be large, an early warning message is reported, indicating that the device may be faulty or unstable.

[0129] Through this embodiment, the accuracy of camera positioning and the consistency of the system are greatly improved by automatically correcting the posture of single-camera and multi-camera systems. Specifically, the single camera calibrates the heading angle and pitch angle, and corrects the horizontal position through lane boundary projection, ensuring fast and accurate posture alignment even in the case of large initial errors. The multi-camera system calculates the relative transformation information between cameras through the overlapping area of ​​​​the field of view, optimizes the synergy between multiple cameras, ensures system stability and reduces errors. In addition, through continuous matching monitoring and dynamic optimization, the state changes and potential problems of the camera are detected in real time, and warning information is generated in time, thereby improving the reliability and adaptability of the system, reducing manual intervention and errors, and significantly improving the overall performance and automation level.

[0130] In the specific implementation process, the camera aiming and monitoring method also includes: for the installation method of multiple cameras on the same pole, detecting whether the optimized heights of the multiple cameras are consistent, and if not, generating a fourth alarm information.

[0131] Specifically, by setting an alarm mechanism for camera installation height consistency, the camera installation status can be monitored in real time, improper installation or adjustment errors can be discovered in time, and the performance of the entire system can be avoided from being affected by inconsistent heights, thereby improving the reliability of the system.

[0132] The embodiment of the present application also provides a device for camera alignment. It should be noted that the device for camera alignment of the embodiment of the present application can be used to execute the method for camera alignment provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0133] The following is an introduction to the camera alignment device provided in the embodiments of the present application.

[0134] Fig. 9 is a structural block diagram of a camera alignment device according to an embodiment of the present application. Fig. 9 As shown, the device comprises: a first alignment unit 10 and a second alignment unit 20.

[0135] The first alignment unit 10 is used for obtaining, for a single camera, a line connecting the optical center of the single camera and a vanishing point, and correcting the heading angle and pitch angle of the single camera by aligning the line with the direction of the lane line on the map; extracting the image lane boundary in the image, and projecting the image lane boundary into the 3D space using the known height and the corrected heading angle and pitch angle of the single camera and aligning it with the lane boundary on the map to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space;

[0136] Specifically, by extracting the line connecting the optical center of a single camera and the vanishing point, and combining the map lane line orientation to correct the heading and pitch angles of the single camera, the camera can be quickly and accurately aligned with a large initial error. By extracting the lane boundary from the image and using the known height and the corrected heading and pitch angles, the lane boundary is further projected into 3D space and aligned with the map lane boundary, effectively correcting the horizontal position of the camera. This method allows the camera's posture to be automatically and quickly accurately aligned even when there are large deviations in the initial position and angle values, reducing the labor cost and error in the traditional manual calibration process.

[0137] The second alignment unit 20 is used to calculate the relative transformation information between cameras based on the overlapping area of ​​the field of view for the multi-camera system, and align the postures of the multiple cameras at the same time according to the relative transformation information and the lane line matching information and stop line matching information of the single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation.

[0138] Specifically, in a multi-camera system, the relative transformation information between cameras is calculated based on the overlapping area of ​​the field of view, and combined with the matching results of the lane lines and stop lines of a single camera, the postures of multiple cameras can be accurately aligned. Through relative transformation information, including the adjustment of relative position and angle, the overall synergy of the multi-camera system is optimized, and the positioning accuracy of each camera is improved. This solution can not only ensure the individual accuracy of each camera, but also achieve consistent correction between multiple cameras, ensuring the accuracy and stability of the multi-camera system, and reducing the inconsistency and positioning errors caused by the posture errors between cameras.

[0139] In this embodiment, a device for camera alignment is provided. The device includes: a first alignment unit and a second alignment unit. The first alignment unit is used to obtain the line connecting the optical center of the single camera and the vanishing point for a single camera, and to correct the heading angle and pitch angle of the single camera by aligning the line with the direction of the lane line on the map; extract the image lane boundary in the image, and use the known height and the corrected heading angle and pitch angle of the single camera to project the image lane boundary into the 3D space and align it with the map lane boundary to correct the horizontal position of the single camera; the second alignment unit is used for a multi-camera system, based on the area where the field of view overlaps, to calculate the relative transformation information between the cameras, and align the postures of the multiple cameras at the same time according to the relative transformation information and the lane line matching information and stop line matching information of the single camera in the multi-camera system. The device effectively solves the problem that the current solution for camera posture alignment based on the initial position value and angle initial value provided by the construction party consumes a lot of manpower.

[0140] As an optional solution, the device includes: a first computing unit and a second computing unit.

[0141] The first calculation unit is used to calculate the vanishing point using all detection lines if the single camera is a same-direction camera and the lane lines at the distant points have the same orientation, indicating that there are straight lanes before and after the intersection; if the lane lines at the distant points have different orientations, indicating that the intersection contains broken lanes, then the vanishing point is calculated using the nearby detection lines in the lower half of the image, wherein the camera field of view of the same-direction camera is oriented in the same direction as the lane direction; and the second calculation unit is used to calculate the vanishing point if the single camera is a same-direction camera or a reverse camera, based on the candidate lane line detection results, according to the degree of detection point dispersion and the number of detection points, selecting a preset detection line with the best linearity and the longest length, wherein the camera field of view of the reverse camera is oriented in the opposite direction to the lane direction.

[0142] As an optional solution, the second calculation unit includes: a pre-selection determination module, a distance acquisition module and a vanishing point determination module.

[0143] The pre-selection determination module is used to determine the preset detection line with the best linearity and the longest length as the pre-selected lane line; the distance acquisition module is used to obtain the intersection of any two pre-selected lane lines and the sum of the distances from any intersection to all the pre-selected lane lines; the vanishing point determination module is used to determine the vanishing point of the intersection with the smallest sum of distances.

[0144] As an optional solution, the device further includes: a first orientation determining unit, a second orientation determining unit and a third orientation determining unit.

[0145] The first direction determination unit is used for determining the direction of the lane of the map lane line for the same-direction camera. If there are straight lanes before and after the intersection, the average value of the lane direction of the distant point and the lane direction of the near point is determined as the direction of the lane line of the map; the second direction determination unit is used for determining the direction of the lane of the current rough position as the direction of the lane line of the map for the same-direction camera. If the intersection contains a zigzag lane, the third direction determination unit is used for determining the difference between the average value of the lane direction of the distant point and the lane direction of the near point and 180° as the direction of the lane line of the map for the reverse camera.

[0146] As an optional solution, the first calculation unit includes: a coordinate system determination module, a matrix acquisition module and an angle determination module.

[0147] A coordinate system determination module is used to obtain a standard camera coordinate system, a correction camera coordinate system, a standard world coordinate system and a correction world coordinate system, wherein the correction camera coordinate system is obtained after correction of the standard camera coordinate system, and the correction world coordinate system is obtained after correction of the standard world coordinate system, and the positive direction of the x-axis of the correction world coordinate system is the direction of the lane line on the map; a matrix acquisition module is used to obtain a rotation matrix converted from the correction world coordinate system to the correction camera coordinate system; an angle determination module is used to determine the heading angle and pitch angle of a single camera according to the rotation matrix.

[0148] As an optional solution, the first calculation unit includes: a first boundary module and a second boundary module.

[0149] The first boundary module is used to sort the detected lane lines from right to left for the same-direction camera. If the current line is a solid line and the left adjacent line is a dotted line, the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the left adjacent line is a solid line, the left adjacent line corresponds to the left boundary of the actual road; the second boundary module is used to sort the detected lane lines from left to right for the reverse camera. If the current line is a solid line and the right adjacent line is a dotted line, the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the right adjacent line is a solid line, the right adjacent line corresponds to the left boundary of the actual road.

[0150] As an optional solution, the first calculation unit includes: a first extension module, a second extension module and a third extension module.

[0151] The first extension module is used to extend the current lane in the forward direction by a preset distance for the same-direction camera, and continuously search the leftmost and rightmost boundary lanes in the horizontal direction during the extension process to extract the map lane boundary; the second extension module is used to extend the current lane in the reverse direction by a preset distance for the reverse camera, and continuously search the leftmost and rightmost boundary lanes in the horizontal direction during the extension process to extract the map lane boundary; the third extension module is used to restart the extension search for the map lane boundary from the adjacent lane if the extension is less than the preset extension distance.

[0152] As an optional solution, the first calculation unit includes: a projection module and a correction module.

[0153] The projection module is used to project the image lane boundary into 3D space by using the known height and the calibrated heading and pitch angles of the single camera through the principle of similar triangles. The matching module is used to perform point-to-line matching processing and residual vector calculation after projection to correct the horizontal position of the single camera.

[0154] As an optional solution, the first computing unit includes: an optimization module.

[0155] The optimization module is used to apply the full amount of lane line and stop line detection results and the corresponding map elements for matching, and optimize the position and posture of a single camera; the matching process includes a segmentation step, a projection step, a matching residual calculation step, and a matching result screening step; the optimization process includes a line segment endpoint coordinate calculation step and an optimization residual construction step.

[0156] As an optional solution, the second alignment unit includes: an extraction module, a matching module and a calculation module.

[0157] The extraction module is used to perform feature extraction processing on the area where the field of view overlaps, obtain key points, and use feature description to describe the local image information around the key points; the matching module is used to apply the feature description to match the feature points in the two camera images to obtain the matching results; the calculation module is used to calculate the transformation matrix between the two camera images based on the matching results.

[0158] The camera alignment device includes a processor and a memory, wherein the first alignment unit and the second alignment unit are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0159] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the current solution of camera pose alignment based on the initial position and angle values ​​provided by the construction party consumes a lot of manpower.

[0160] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0161] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, a device where the computer-readable storage medium is located is controlled to execute a camera alignment method.

[0162] An embodiment of the present invention provides an electronic device, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for performing any camera alignment or camera alignment and monitoring.

[0163] An embodiment of the present invention provides a vehicle, comprising: a camera installed on the vehicle, wherein the camera is aligned using any camera alignment method.

[0164] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0174] 1) The camera alignment method of the present application successfully solves the problem of large deviations in the initial position and angle of the camera through an automated visual calibration algorithm, and realizes accurate automatic alignment of single-camera and multi-camera systems. A single camera quickly corrects the orientation and position of the camera by aligning the vanishing point with the lane line, reducing the complexity and cost of manual calibration. In a multi-camera system, the coordination accuracy and consistency of the entire system are improved by matching the relative transformation information between cameras with lane lines and stop lines. Ultimately, this technology significantly improves the camera positioning accuracy in V2X applications, reduces the need for manual calibration, and maintains high stability and reliability in complex environments, effectively solving the problem that the current solution for camera posture alignment based on the initial position and angle values ​​provided by the construction party consumes a lot of manpower.

[0175] 2) The camera alignment and monitoring method of the present application greatly improves the accuracy of camera positioning and the consistency of the system by automatically correcting the posture of single-camera and multi-camera systems. Specifically, a single camera calibrates the heading and pitch angles, and corrects the horizontal position through lane boundary projection, ensuring fast and accurate posture alignment even in the case of large initial errors. The multi-camera system calculates the relative transformation information between cameras through the overlapping area of ​​​​the field of view, optimizes the synergy between multiple cameras, ensures system stability and reduces errors. In addition, through continuous matching monitoring and dynamic optimization, the state changes and potential problems of the camera are detected in real time, and warning information is generated in time, thereby improving the reliability and adaptability of the system, reducing manual intervention and errors, and significantly improving the overall performance and automation level.

[0176] 3) Camera alignment device. The device includes: a first alignment unit and a second alignment unit. The first alignment unit is used to obtain the line between the optical center and the vanishing point of a single camera, and to correct the heading angle and pitch angle of the single camera by aligning the line with the direction of the lane line on the map; extract the image lane boundary in the image, and use the known height and the corrected heading angle and pitch angle of the single camera to project the image lane boundary into the 3D space and align it with the map lane boundary to correct the horizontal position of the single camera; the second alignment unit is used for the multi-camera system, based on the area where the field of view overlaps, to calculate the relative transformation information between the cameras, and align the postures of the multiple cameras at the same time according to the relative transformation information and the lane line matching information and stop line matching information of the single camera in the multi-camera system. This device effectively solves the problem that the current solution of camera posture alignment based on the initial position value and angle initial value provided by the construction party consumes a lot of manpower.

[0177] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for camera alignment, characterized in that: include: For a single camera, obtain the line connecting the optical center of the single camera and the vanishing point, and align the line with the direction of the lane line on the map to correct the heading angle and pitch angle of the single camera; extract the image lane boundary from the image, and project the image lane boundary into the 3D space using the known height and the corrected heading angle and pitch angle of the single camera and align it with the map lane boundary to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space; For a multi-camera system, relative transformation information between cameras is calculated based on the area of ​​overlapping fields of view, and the postures of the multiple cameras are simultaneously aligned based on the relative transformation information and lane line matching information and stop line matching information of a single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation.

2. The method according to claim 1, characterized in that: Before obtaining the line connecting the optical center of the single camera and the vanishing point, the method further includes: If the single camera is a same-direction camera, and the lane lines at the distant points have the same orientation, indicating that there are straight lanes before and after the intersection, all detection lines are used to calculate the vanishing point; if the lane lines at the distant points have different orientations, indicating that the intersection contains a broken lane, the nearby detection lines in the lower half of the image are used to calculate the vanishing point, wherein the camera field of view of the same-direction camera has the same orientation as the lane direction; If the single camera is a same-direction camera or a reverse camera, based on the candidate lane line detection results, according to the discrete degree of detection points and the number of detection points, a preset detection line with the best linearity and the longest length is selected to calculate the vanishing point, wherein the camera field of view of the reverse camera is opposite to the lane direction.

3. The method according to claim 2, characterized in that Selecting a preset detection line with the best linearity and the longest length to calculate the vanishing point includes: Determine the detection line with the best linearity and the longest length as the pre-selected lane line; Obtaining an intersection point of any two of the pre-selected lane lines, and obtaining the sum of distances from any of the intersection points to all of the pre-selected lane lines; The intersection point where the sum of the distances is the smallest is determined as the vanishing point.

4. The method according to claim 1, characterized in that: Before obtaining the line connecting the optical center of the single camera and the vanishing point, the method further includes: For cameras in the same direction, if there are straight lanes before and after the intersection, the average of the lane directions of the distant point and the lane directions of the near point is determined as the lane line direction of the map; For the same-direction camera, if the intersection contains a broken lane, the direction of the lane where the current rough position is located is determined as the direction of the lane line on the map; For the reverse camera, the difference between the average of the lane directions of the distant point and the lane directions of the near point and 180° is determined as the lane line direction of the map.

5. The method according to claim 1, characterized in that Correcting the heading angle and pitch angle of the single camera by aligning the connecting line with the direction of the lane line on the map includes: Obtain a standard camera coordinate system, a calibrated camera coordinate system, a standard world coordinate system, and a calibrated world coordinate system, wherein the calibrated camera coordinate system is obtained after calibrating the standard camera coordinate system, the calibrated world coordinate system is obtained after calibrating the standard world coordinate system, and the positive direction of the x-axis of the calibrated world coordinate system is the direction of the lane line on the map; Obtaining a rotation matrix for transforming from the rectified world coordinate system to the rectified camera coordinate system; The heading angle and pitch angle of the single camera are determined according to the rotation matrix.

6. The method according to claim 1, characterized in that Extract image lane boundaries from the image, including: For the same-direction camera, the detected lane lines are sorted from right to left. If the current line is a solid line and the left adjacent line is a dotted line, the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the left adjacent line is a solid line, the left adjacent line corresponds to the left boundary of the actual road; For the reverse camera, the detected lane lines are sorted from left to right. If the current line is a solid line and the right adjacent line is a dotted line, the current line corresponds to the right boundary of the actual road; if the current line is a dotted line and the right adjacent line is a solid line, the right adjacent line corresponds to the left boundary of the actual road.

7. The method according to claim 1, characterized in that The method further comprises: extracting a map lane boundary before correcting the horizontal position of the single camera; The specific implementation of extracting the lane boundary of the map is as follows: For the same-direction camera, the current lane is extended forward by a preset distance, and during the extension process, the leftmost and rightmost boundary lanes are continuously searched horizontally to extract the map lane boundary; For the reverse camera, the current lane is extended in the reverse direction by a preset distance, and during the extension process, the leftmost and rightmost boundary lanes are continuously searched horizontally to extract the lane boundary of the map; If the extension is less than the preset extension distance, the search for the map lane boundary is restarted from the adjacent lane.

8. The method according to claim 1, characterized in that Extracting the image lane boundary from the image, and projecting the image lane boundary into the 3D space and aligning it with the map lane boundary using the known height and the calibrated heading angle and pitch angle of the single camera to calibrate the horizontal position of the single camera, including: Using the known height and the calibrated heading angle and pitch angle of the single camera, the image lane boundary is projected into the 3D space by the principle of similar triangles; After the projection, point-to-line matching processing and residual vector calculation are performed to correct the horizontal position of the single camera.

9. The method according to claim 8, characterized in that After performing point-to-line matching processing and residual vector calculation after projection to correct the horizontal position of the single camera, the method further includes: Applying the full amount of lane line and stop line detection results and corresponding map elements for matching, and optimizing the pose of the single camera; The matching process includes a segmentation step, a projection step, a matching residual calculation step and a matching result screening step; the optimization process includes a line segment endpoint coordinate calculation step and an optimization residual construction step.

10. The method according to claim 1, characterized in that For a multi-camera system, the relative transformation information between cameras is calculated based on the overlapping areas of the field of view, including: Performing feature extraction processing on the area where the fields of view overlap to obtain key points, and using feature description to describe local image information around the key points; Applying the feature description to match feature points in two camera images to obtain a matching result; A transformation matrix between the two camera images is calculated according to the matching result.

11. A method for camera aiming and monitoring, characterized in that: include: For a single camera, obtain the line connecting the optical center of the single camera and the vanishing point, and align the line with the direction of the lane line on the map to correct the heading angle and pitch angle of the single camera; extract the image lane boundary from the image, and project the image lane boundary into the 3D space using the known height and the corrected heading angle and pitch angle of the single camera and align it with the map lane boundary to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space; For a multi-camera system, relative transformation information between cameras is calculated based on the overlapping area of ​​field of view, and the postures of the multi-cameras are simultaneously aligned according to the relative transformation information and lane line matching information and stop line matching information of a single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation; After the single camera and the multi-camera system have been calibrated, for each camera, semantic elements in the map are projected into the image and matched with corresponding elements detected on the image. If the number of matches is less than a number threshold, a first warning message is generated, wherein the first warning message indicates that the position and posture of the current camera have changed. If the number of matches is greater than or equal to the number threshold, and if the matching residual about the matching distance in the image coordinate system continues to be large, generating a second warning message; If the matching residual is within a preset residual range, the position and posture of the camera are optimized again according to the matching residual, and compared with the initial position and initial posture. If the position deviation or angle deviation continues to increase, a third alarm message is generated.

12. The method according to claim 11, characterized in that The method further comprises: With respect to the installation method of multiple cameras on the same pole, it is detected whether the optimized heights of the multiple cameras are consistent. If not, a fourth alarm message is generated.

13. A camera aiming device, characterized in that: include: A first alignment unit is used for obtaining, for a single camera, a line connecting the optical center of the single camera and a vanishing point, and correcting the heading angle and pitch angle of the single camera by aligning the line with the direction of the lane line on the map; extracting the image lane boundary from the image, and projecting the image lane boundary into the 3D space using the known height and the corrected heading angle and pitch angle of the single camera and aligning it with the lane boundary on the map, so as to correct the horizontal position of the single camera, wherein the vanishing point is a point in the imaging plane where a set of parallel lines that are not parallel to the imaging plane intersect at an infinite point in the 3D space; The second alignment unit is used to calculate the relative transformation information between cameras for a multi-camera system based on the area of ​​overlapping fields of view, and simultaneously align the postures of the multiple cameras according to the relative transformation information and lane line matching information and stop line matching information of a single camera in the multi-camera system, wherein the relative transformation information includes relative position and angle transformation.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of claims 1 to 10, or the method of claim 11 or 12.

15. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 10, or claims 11 or 12.

16. A vehicle, characterized in that: include: A camera installed on a vehicle, wherein the camera is aligned using the camera alignment method described in any one of claims 1 to 10.

Images