Camera self-calibration method, device and equipment and storage medium

Abstract

The invention discloses a camera self-calibration method, device and equipment and a storage medium, relates to the technical field of automobiles, and discloses a camera self-calibration method which comprises the steps that when a target signboard is recognized, historical detection information and current information of the target signboard are acquired, the positioning information, the size information and the picture information in the historical detection information and the current information are compared to obtain an error comparison result, when the error comparison result is that the camera has an error, an internal reference matrix is calculated based on the current information, and the internal reference of the camera is updated based on the internal reference matrix to complete the camera self-calibration process. After a vehicle leaves a factory, the current information, collected in real time, of the target signboard is compared with the historical detection information of the target signboard, the defects of the vehicle-mounted camera are automatically recognized, the camera internal reference of the vehicle-mounted camera is accurately calibrated, and time and money cost are saved.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a camera self-calibration method, device, equipment and storage medium. Background Art

[0002] Cameras play an important role in intelligent driving assistance systems and are indispensable for tasks such as target recognition and environmental reconstruction. At the same time, their detection accuracy is also closely related to the overall performance of intelligent driving. Traditional car companies calibrate cameras during the offline electrical inspection stage at the factory. Once the calibration is completed, its internal and external parameters will not be automatically optimized and updated after delivery to customers. However, when the camera is installed on the vehicle, the camera's inherent parameters due to mechanical vibration caused by road bumps or temperature fluctuations caused by weather will change.

[0003] The existing technology generally uses visual recognition to extract feature points of images in a controlled environment, obtains camera intrinsic parameters through transformation matrix, and further forms self-calibration. However, this calibration solution is limited to professional calibration scenarios. It is impossible to accurately calibrate the camera intrinsic parameters of the vehicle-mounted camera after the vehicle is sold. Returning the camera to the factory for recalibration requires a lot of time and money.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a camera self-calibration method, device, equipment and storage medium, aiming to solve the technical problem of how to accurately calibrate the camera intrinsic parameters of a vehicle-mounted camera.

[0006] To achieve the above objectives, the present application proposes a camera self-calibration method, which includes:

[0007] When a target sign is identified, historical detection information and current information of the target sign are obtained;

[0008] Comparing the historical detection information with the positioning information, size information and picture information in the current information to obtain an error comparison result;

[0009] When the error comparison result shows that there is an error in the camera, an intrinsic parameter matrix is ​​calculated based on the current information, and the intrinsic parameters of the camera are updated based on the intrinsic parameter matrix to complete the camera self-calibration process.

[0010] In one embodiment, when the target sign is identified, before the step of acquiring the historical detection information and current information of the target sign, the method further includes:

[0011] Collecting data on the signboards, and when a target signboard is identified, obtaining historical image information and preset historical size information of the target signboard;

[0012] The historical positioning information of the target sign is obtained, and the historical image information, the preset historical size information and the historical positioning information are stored as historical detection information.

[0013] In one embodiment, the step of collecting data on the signboard and obtaining historical image information and preset historical size information of the target signboard when the target signboard is identified includes:

[0014] Set the preset number of collections;

[0015] Collect data on the signboard based on the preset number of collection times to obtain a plurality of historical picture information and a plurality of historical point cloud data of the target signboard;

[0016] Fusing the plurality of historical picture information and the plurality of historical point cloud data to obtain a plurality of historical size information of the target sign;

[0017] The plurality of historical size information are processed to obtain the preset historical size information.

[0018] In one embodiment, the step of fusing the plurality of historical image information and the plurality of historical point cloud data to obtain the plurality of historical size information of the target sign comprises:

[0019] Performing image recognition on the plurality of historical picture information to obtain historical picture features of the target sign;

[0020] Preprocessing the plurality of historical point cloud data to obtain historical point cloud features of the target sign;

[0021] Fusing the historical image features with the historical point cloud features to obtain fusion information;

[0022] A plurality of historical size information of the target sign is calculated based on the fusion information.

[0023] In one embodiment, the step of comparing the historical detection information with the positioning information, size information and picture information in the current information to obtain an error comparison result includes:

[0024] Comparing the historical detection information with the positioning information, size information and picture information in the current information;

[0025] When the positioning information and the picture information of the historical detection information are consistent with the current information, but the size information is inconsistent, it is determined that the error comparison result is that there is an error in the camera.

[0026] In one embodiment, after the step of comparing the historical detection information with the positioning information, size information and picture information in the current information, the method further includes:

[0027] When the positioning information and the picture information of the historical detection information are consistent with the current information, and the size information is consistent, it is determined that the error comparison result is that there is no error in the camera.

[0028] In one embodiment, the step of calculating an intrinsic parameter matrix based on the current information includes:

[0029] Acquire the corner point coordinates of the target sign based on the current information;

[0030] Calculating a homography matrix based on the corner point coordinates;

[0031] An intrinsic parameter matrix is ​​calculated based on the homography matrix.

[0032] In addition, to achieve the above purpose, the present application also proposes a camera self-calibration device, the camera self-calibration device comprising:

[0033] An information collection module, used to obtain historical detection information and current information of the target identification plate when the target identification plate is identified;

[0034] An error comparison module is used to compare the positioning information, size information and image information of the historical detection information with the current information to obtain an error comparison result;

[0035] The automatic calibration module is used to calculate an internal parameter matrix based on the current information when the error comparison result shows that there is an error in the camera, and to update the internal parameters of the camera based on the internal parameter matrix to complete the camera self-calibration process.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a camera self-calibration device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the camera self-calibration method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the camera self-calibration method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the camera self-calibration method described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] When the target sign is identified, historical detection information and current information of the target sign are obtained, and the positioning information, size information and image information in the historical detection information are compared with the current information to obtain an error comparison result. When the error comparison result shows that there is an error in the camera, an intrinsic parameter matrix is ​​calculated based on the current information, and the intrinsic parameters of the camera are updated based on the intrinsic parameter matrix to complete the camera self-calibration process. After the vehicle leaves the factory, the current information of the target sign collected in real time can be compared with the historical detection information of the target sign to automatically identify the defects of the vehicle-mounted camera and accurately calibrate the camera intrinsic parameters of the vehicle-mounted camera, saving time and money. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the camera self-calibration method of the present application;

[0044] Figure 2 A schematic diagram of the process flow provided for the second embodiment of the camera self-calibration method of the present application;

[0045] Figure 3 A schematic diagram of a brief process of the camera self-calibration method provided in the second embodiment of the present application;

[0046] Figure 4 A system structure diagram of a camera self-calibration method provided in Embodiment 2 of the present application;

[0047] Figure 5 This is a schematic diagram of the module structure of the camera self-calibration device according to an embodiment of the present application;

[0048] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the camera self-calibration method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a camera self-calibration system, etc. The following takes the camera self-calibration system as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the present application embodiment provides a camera self-calibration method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the camera self-calibration method of the present application.

[0054] In this embodiment, the camera self-calibration method includes steps S10 to S30:

[0055] Step S10, when a target sign is identified, obtaining historical detection information and current information of the target sign;

[0056] It should be noted that for new cars that roll off the factory assembly line, while the vehicle is driving, information about traffic signs identified on the road can be collected through perception sensors such as cameras and lidars, and the collected information about the traffic signs can be saved as a detection package. The traffic signs can specifically be signs such as speed limit and no parking.

[0057] It should be understood that when a vehicle recognizes a traffic sign with the same characteristics again on a certain road section, the sign will be used as a target sign, and historical detection information will be obtained based on the detection package of the target sign. The historical detection information contains the results of multiple data collection and updates of the target sign. While obtaining the historical detection information of the target sign, the current information of the target sign will be collected through perception sensors such as cameras and laser radars. The current information and historical detection information at least include the size information, image information and positioning information of the target sign, and may also include the timestamp of each information collection, weather and other environmental conditions information.

[0058] Step S20, comparing the historical detection information with the positioning information, size information and picture information in the current information to obtain an error comparison result;

[0059] In a specific implementation, the positioning information of the target signboard can be obtained through a panoramic camera or other positioning sensors installed on the vehicle, or through a global positioning system (GPS) receiver of the vehicle. The image information can be obtained by capturing the image of the target signboard in real time through a camera, and processing the image based on algorithms such as convolutional neural networks, specifically including the size, color and contour shape information of the target signboard. The size information is obtained by fusing the signboard information collected by multiple perception sensors including cameras, and this embodiment does not impose specific restrictions on this.

[0060] In a feasible implementation, step S20 may include steps A11 to A12:

[0061] Step A11, comparing the historical detection information with the positioning information, size information and picture information in the current information;

[0062] It should be noted that when the vehicle detects the target sign on a certain road section again, the historical detection information of the sign that is consistent with the positioning information of the target sign can be extracted from the permanently stored traffic sign data packet, and the historical detection information of the sign that is consistent with the picture information of the target sign can be extracted from the permanently stored traffic sign data packet. At the same time, the positioning information, picture information and size information of the target sign collected and processed by perception sensors such as cameras and lidars at the current moment can be obtained for subsequent information comparison.

[0063] Step A12, when the positioning information and the picture information of the historical detection information are consistent with the current information, but the size information is inconsistent, it is determined that the error comparison result is that there is an error in the camera.

[0064] It should be noted that when the historical detection information and the current information are consistent in positioning information and image information, but inconsistent in size information, this may indicate that the inherent mechanical vibration of the camera caused by road bumps or temperature fluctuations caused by weather causes the camera's internal parameters to change, thereby causing the size information of the same sign collected to change. In addition, when the historical detection information is inconsistent with the positioning information in the current information of the target sign, but the image information is consistent, it means that another sign with a similar appearance but a different position has been detected. When the historical detection information is inconsistent with the image information in the current information of the target sign, but the positioning information is consistent, it means that the appearance of the sign may have changed due to external factors such as being replaced or hit. At this time, the size information of the target sign collected will not have reference value and cannot be used as a basis for subsequent camera calibration of the vehicle camera.

[0065] It should be understood that when the historical detection information is consistent with the current information in terms of location information and image information, and size information, the error comparison result is determined to be that there is no error in the camera. At this time, there is no need to perform subsequent camera calibration operations on the vehicle camera.

[0066] Step S30, when the error comparison result shows that there is an error in the camera, an intrinsic parameter matrix is ​​calculated based on the current information, and the intrinsic parameters of the camera are updated based on the intrinsic parameter matrix to complete the camera self-calibration process.

[0067] It should be noted that camera calibration refers to the process of determining the internal parameters (hereinafter referred to as intrinsic parameters) and external parameters of the camera, which are used to map points in the three-dimensional world coordinate system to the two-dimensional image coordinate system. The intrinsic parameter matrix is ​​the transformation matrix after the perspective projection process and the pixel discretization process are combined in the process of coordinate conversion between three-dimensional space points and two-dimensional image plane points.

[0068] It should be understood that when a vehicle is driving, it will be affected by road vibrations and mechanical vibrations, which may cause the internal components of the camera to move or shift, thereby affecting the stability of the intrinsic parameters. Especially in bad road conditions, such as bumpy roads or uneven roads, the vibration will be more obvious. At this time, it is necessary to redetermine the intrinsic parameter matrix and update the intrinsic parameters of the vehicle camera based on the intrinsic parameter matrix to eliminate image distortion caused by lens characteristics and improve the usability of the image acquired by the camera.

[0069] Specifically, using the pinhole camera model for calibration, the three-dimensional coordinate point (X, Y, Z) in the world coordinates can be projected into the image as a two-dimensional pixel point (u, v). The formula for converting (X, Y, Z) to (u, v) is as follows:

[0070]

[0071] Among them, s is an unknown scalar, A is the camera intrinsic matrix of size 3 × 3. r1, r2, r3 are the columns of the rotation matrix, t is the translation vector, and the rotation matrix and translation vector are the camera external parameters.

[0072] During the calibration process, since the flat object identification plate is used as the calibration object, the component in the Z direction is always zero, and the simplified formula can be obtained as follows:

[0073]

[0074] In a feasible implementation manner, step S30 may include steps A21 to A23:

[0075] Step A21, obtaining the corner point coordinates of the target sign based on the current information;

[0076] It should be noted that the corner point coordinates refer to the vertex positions of the signboard on the image. A signboard usually has 8 corner points. Based on the current information of the target signboard, the image information can be obtained. Based on the image information, the corner point coordinates of the target signboard can be detected and identified. This process can be achieved through image processing technology, such as edge detection, feature point detection, etc.

[0077] Step A22, calculating a homography matrix based on the corner point coordinates;

[0078] It should be noted that the homography matrix is ​​a matrix that describes the mapping relationship between the image plane and the world coordinate system.

[0079] Specifically, we can use B = (X, Y, 1) T and b=(u,v,1) T They represent the position of a point on a plane object and its position in the image respectively. The relationship between B and b is defined by the homography matrix H as follows:

[0080] sb=HB with H=A[r1r2t]

[0081] H is a 3×3 matrix. For a planar object sign, after detecting the 8 corner points of the target sign, H can be expressed as H=[h1 h2 h3], and the formula is as follows:

[0082]

[0083] Step A23, calculating an intrinsic parameter matrix based on the homography matrix.

[0084] Specifically, since r1 and r2 are orthogonal, the internal parameter matrix A can be calculated based on the following two constraint formulas:

[0085]

[0086] The present embodiment provides a camera self-calibration method. When a target sign is identified, historical detection information and current information of the target sign are obtained, and the positioning information, size information and image information in the historical detection information are compared with the current information to obtain an error comparison result. When the error comparison result shows that there is an error in the camera, an internal parameter matrix is ​​calculated based on the current information, and the internal parameters of the camera are updated based on the internal parameter matrix to complete the camera self-calibration process. After the vehicle leaves the factory, the defects of the vehicle-mounted camera can be automatically identified by comparing the current information of the target sign collected in real time with the historical detection information of the target sign, and the camera internal parameters of the vehicle-mounted camera can be accurately calibrated, saving time and money.

[0087] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 Before step S10, the camera self-calibration method further includes steps S01 to S02:

[0088] Step S01, collecting data on the signboard, and when a target signboard is identified, obtaining historical image information and preset historical size information of the target signboard;

[0089] It should be noted that after the vehicle rolls off the factory, it will collect data on the signboards it recognizes on the road based on perception sensors, including image information and size information. In order to distinguish it from the current information used for subsequent error comparison, it will be used as historical image information and preset historical size information.

[0090] In a feasible implementation, step S01 may include steps A31 to A34:

[0091] Step A31, setting a preset number of acquisition times;

[0092] It should be understood that the preset number of acquisitions is a pre-set parameter used to determine how many times data needs to be collected for the same sign before the on-board camera is calibrated, and the sign information is updated based on the latest collected data. Specifically, the preset number of acquisitions can be set to 5 times. When a single sign is updated 5 times, it means that the vehicle has passed a certain road section 5 times after leaving the factory. The sign information will no longer be updated and will be permanently saved on the vehicle side. Its size information and positioning information will be used for error comparison later.

[0093] Step A32, collecting data on the signboard based on the preset number of collection times, to obtain a plurality of historical picture information and a plurality of historical point cloud data of the target signboard;

[0094] It should be noted that point cloud data is a data set consisting of a large number of discrete points, each of which contains the coordinate information of a certain position in space. These points can represent the surface features of physical objects or scenes in three-dimensional space. The spatial point cloud data of the signboard obtained by three-dimensional perception sensors such as lidar can preliminarily determine the size of the signboard.

[0095] Step A33, fusing the plurality of historical image information and the plurality of historical point cloud data to obtain a plurality of historical size information of the target sign;

[0096] It should be noted that each time the image information and point cloud data of the target sign are collected, the image information and point cloud data collected at this time will be sensed and fused to obtain more accurate size information. In the process of perception fusion, the size information is calculated by combining the sign information collected by the lidar and camera, which can reduce the limitations of a single sensor and avoid the accuracy of the size information being affected by occlusion or multipath reflection in the collected point cloud data.

[0097] Specifically, step A33 may include: performing image recognition on multiple historical picture information to obtain historical picture features of the target sign; preprocessing multiple historical point cloud data to obtain historical point cloud features of the target sign; fusing the historical picture features with the historical point cloud features to obtain fusion information; and calculating multiple historical size information of the target sign based on the fusion information.

[0098] It should be understood that a general detection architecture such as YOLO, YOLOv3, etc. can be used to identify the image information of the sign, and the structural information of the sign can be extracted from the image through the Canny algorithm and the edge line fitting algorithm, and the edge of the sign can be identified to obtain the image features of the target sign, including the gradient, contour, size, etc. of the image. The collected point cloud data needs to be preprocessed, including filtering to remove noise, downsampling to reduce the amount of data, segmentation to distinguish between the sign and other objects, etc., to obtain the point cloud features of the target sign. The historical image features obtained by image recognition and the point cloud features obtained by preprocessing can be fused through the matching algorithm, and the target sign information obtained by the laser radar and the camera can be matched and associated. Based on the physical characteristics of each sensor, the precise size of the target sign is calculated using a weighted method to obtain the size information.

[0099] Step A34, processing the multiple historical size information to obtain the preset historical size information.

[0100] It should be noted that each time the image information and point cloud data of the target sign are collected, the image information and point cloud data collected at this time will be sensed and fused to obtain historical size information. When the vehicle repeatedly passes a certain road section and sign, the latest historical size information obtained and the previously saved historical size information are averaged and updated and saved until the preset number of collections is reached, and the last historical size information updated and saved in sequence is used as the preset historical size information.

[0101] Step S02, obtaining the historical positioning information of the target sign, and storing the historical image information, the preset historical size information and the historical positioning information as historical detection information.

[0102] It should be understood that in the process of collecting target sign information multiple times, the positioning information and the image information are the basis for matching, that is, the historical positioning information and the historical image information of the target sign collected multiple times are the same. After the historical image information, the preset historical size information and the historical positioning information are stored as historical detection information, the historical detection information is permanently saved to the vehicle side and used as the basis for subsequent vehicle camera calibration.

[0103] For example, to help understand the implementation process of the camera self-calibration method application obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A brief flowchart of a camera self-calibration method is provided, specifically:

[0104] After the vehicle leaves the factory and the vehicle system is initialized, if the preset number of collections is 5, when the vehicle passes a certain road section and recognizes a certain signboard, the image information and point cloud data of the signboard will be collected based on the laser radar / camera signal perception, and image recognition, feature extraction and target detection will be performed on the image information. At the same time, point cloud processing, feature extraction and target detection will be performed on the point cloud data. Based on the results of the target detection of the two, the image information and point cloud data are fused, and the size information is calculated after correlation matching (when the signboard is repeatedly passed through the road section, the size information will also be updated based on the size information stored last time), and the data packet is stored in combination with the positioning information. If the number of times the signboard is recognized reaches 5 times, the data packet will be permanently stored and used as historical detection information. At this time, if a signboard is matched based on the positioning information in the historical detection information, and the image information in the current information of the signboard collected is consistent with the historical detection information, there is an error between the size information and the true value in the historical detection information, and the calculated error exceeds the error range, the size information will be considered inconsistent, and the error comparison result will be determined to be that there is an error in the camera, and the vehicle-mounted camera will be self-calibrated based on the current information.

[0105] In addition, please refer to Figure 4 , Figure 4 The system structure diagram of the camera self-calibration method application is provided, including an information collection module, an error comparison module and an automatic calibration module, which are used to successively complete the information collection, error comparison and automatic calibration functions. Specifically, the information collection module is used to complete the information perception, information fusion and data storage functions, the error comparison module is used to complete the positioning matching, true value acquisition and error calculation functions, and the automatic calibration is used to complete the internal parameter calculation and memory update functions.

[0106] This embodiment collects data on the signboard. When the target signboard is identified, the historical image information and preset historical size information of the target signboard are obtained, and the historical positioning information of the target signboard is obtained, and the historical image information, the preset historical size information and the historical positioning information are stored as historical detection information. In the process of collecting data, the signboard information collected by different sensors is integrated to calculate the size information, and the size information of the same signboard is repeatedly collected for updating, so as to ensure that the size information in the historical detection information has a high accuracy, which can be used as the basis for whether the subsequent vehicle-mounted camera performs camera self-calibration.

[0107] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the camera self-calibration method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0108] This application also provides a camera self-calibration device, please refer to Figure 5 , the camera self-calibration device comprises:

[0109] The information collection module 10 is used to obtain the historical detection information and current information of the target identification plate when the target identification plate is identified;

[0110] The error comparison module 20 is used to compare the positioning information, size information and picture information of the historical detection information with the current information to obtain an error comparison result;

[0111] The automatic calibration module 30 is used to calculate the intrinsic parameter matrix based on the current information when the error comparison result shows that there is an error in the camera, and update the intrinsic parameters of the camera based on the intrinsic parameter matrix to complete the camera self-calibration process. The camera self-calibration device provided in the present application adopts the camera self-calibration method in the above embodiment, which can solve the technical problem of how to accurately calibrate the camera intrinsic parameters of the vehicle-mounted camera. Compared with the prior art, the beneficial effects of the camera self-calibration device provided in the present application are the same as the beneficial effects of the camera self-calibration method provided in the above embodiment, and the other technical features in the camera self-calibration device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0112] The present application provides a camera self-calibration device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the camera self-calibration method in the above-mentioned embodiment 1.

[0113] Reference below Figure 6, which shows a schematic diagram of the structure of a camera self-calibration device suitable for implementing the embodiment of the present application. The camera self-calibration device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The camera self-calibration device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0114] like Figure 6 As shown, the camera self-calibration device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the camera self-calibration device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the camera self-calibration device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a camera self-calibration device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0115] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0116] The camera self-calibration device provided by the present application adopts the camera self-calibration method in the above embodiment, which can solve the technical problem of how to accurately calibrate the camera internal parameters of the vehicle-mounted camera. Compared with the prior art, the beneficial effects of the camera self-calibration device provided by the present application are the same as the beneficial effects of the camera self-calibration method provided by the above embodiment, and the other technical features in the camera self-calibration device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0117] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0118] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0119] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the camera self-calibration method in the above embodiment.

[0120] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0121] The computer-readable storage medium may be included in the camera self-calibration device; or may exist independently without being assembled into the camera self-calibration device.

[0122] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the camera self-calibration device, the camera self-calibration device: when a target sign is identified, obtains a detection package and current information of the target sign; compares the positioning information, size information and image information of the detection package with the current information to obtain an error comparison result; when the error comparison result shows that there is an error in the camera, calculates an intrinsic parameter matrix based on the current information, and updates the intrinsic parameters of the camera based on the intrinsic parameter matrix.

[0123] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0126] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned camera self-calibration method, and can solve the technical problem of how to accurately calibrate the camera intrinsic parameters of the vehicle-mounted camera. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the camera self-calibration method provided in the above-mentioned embodiment, and will not be repeated here.

[0127] The present application also provides a computer program product, including a computer program, which implements the steps of the camera self-calibration method as described above when executed by a processor.

[0128] The computer program product provided by the present application can solve the technical problem of how to accurately calibrate the camera intrinsic parameters of the vehicle-mounted camera. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the camera self-calibration method provided by the above embodiment, which will not be repeated here.

[0129] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A camera self-calibration method, characterized in that: The camera self-calibration method comprises: When a target identification plate is identified, a detection package and current information of the target identification plate are obtained; Compare the positioning information, size information and picture information of the detection package with the current information to obtain an error comparison result; When the error comparison result shows that there is an error in the camera, an intrinsic parameter matrix is ​​calculated based on the current information, and the intrinsic parameters of the camera are updated based on the intrinsic parameter matrix.

2. The camera self-calibration method according to claim 1, characterized in that: Before the step of obtaining the detection package and current information of the target identification plate when the target identification plate is identified, the step further includes: When the target sign is recognized for the first time, the image information and point cloud data of the target sign are obtained, and the image information is matched with the point cloud data to obtain the size information of the target sign; The positioning information of the target identification plate is obtained, and the image information, size information and positioning information are stored as a detection package.

3. The camera self-calibration method according to claim 2, characterized in that: The step of fusing the image information with the point cloud data to obtain the size information of the target sign includes: Performing image recognition on the image information to obtain image features of the target sign; Preprocessing the point cloud data to obtain point cloud features of the target sign; Matching the image features with the point cloud features to obtain matching information; The size information of the target identification plate is calculated based on the matching information.

4. The camera self-calibration method according to claim 2, characterized in that: After the step of obtaining the positioning information of the target identification plate and storing the image information, size information and positioning information as a detection package, the method further includes: When the target sign is recognized again, the size information of the target sign is obtained based on the detection package as the first size information, and the image information and point cloud data of the target sign are collected, and the image information and the point cloud data are merged to obtain the second size information; updating the size information in the detection package based on the first size information and the second size information, and recording the number of times the detection package is updated; When the update times reaches a preset number, the updating of the detection package is stopped.

5. The camera self-calibration method according to claim 2, wherein: The step of obtaining the image information and point cloud data of the target sign includes: Collecting image information of the target sign through a camera; The point cloud data of the target sign is collected through lidar.

6. The camera self-calibration method according to claim 1, characterized in that: The step of comparing the positioning information, size information and picture information of the detection package with the current information to obtain an error comparison result includes: Comparing the location information, size information and picture information of the detection package with the current information; When the positioning information and size information of the detection package are consistent with the current information, and the size information is inconsistent, the error comparison result is that there is an error in the camera.

7. The camera self-calibration method according to claim 1, wherein: The step of calculating the internal parameter matrix based on the current information comprises: Acquire the coordinates of the corner points of the sign based on the current information; Calculating a homography matrix based on the corner point coordinates; An intrinsic parameter matrix is ​​calculated based on the homography matrix.

8. A camera self-calibration device, characterized in that: The device comprises: An information collection module, used to obtain a detection package and current information of a target identification plate when the target identification plate is identified; An error comparison module is used to compare the positioning information, size information and picture information of the detection package with the current information to obtain an error comparison result; The automatic calibration module is used to calculate an internal parameter matrix based on the current information when the error comparison result shows that there is an error in the camera, and update the internal parameters of the camera based on the internal parameter matrix.

9. A camera self-calibration device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the camera self-calibration method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the camera self-calibration method according to any one of claims 1 to 7 are implemented.

Images