Calibration Method, Device and Electronic Device of a Device

Abstract

The present application discloses a calibration method, device and electronic device for a device. The method includes: obtaining a plurality of image information sensed by the device; determining a plurality of license plate image position information based on the plurality of image information, and determining straight trajectories of a plurality of vehicles; calculating a first vanishing point, a second vanishing point and a third vanishing point based on the plurality of license plate image position information and the plurality of straight trajectories; and automatically calibrating the device in combination with the first vanishing point, the second vanishing point and the third vanishing point. Through the technical solution provided by the embodiments of the present application, the automatic calibration of the device can be realized only based on the image information sensed by the device, without training and optimizing a deep learning model, nor calibrating data, thereby improving the versatility of the automatic calibration of the device.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly to a calibration method, device, and electronic device for a device. Background Art

[0002] Devices (such as cameras) can be used to perform various tasks such as capturing and monitoring, and play an important role in traffic scenarios.

[0003] In the application of traffic scenarios, it is often necessary to accurately measure information such as the position and speed of vehicles in the monitored traffic scenario. To achieve this function, it is necessary to calibrate the device (such as a camera) to associate the image coordinate system of the device with the actual world coordinate system, so as to realize the mutual conversion between coordinate systems. Among them, calibrating the device is to determine the calibration matrix of the device.

[0004] Traditional device calibration methods (such as camera calibration methods) need to rely on calibration objects with known geometric shapes, such as calibration plates or three-dimensional stereo targets, to achieve device calibration. However, this method requires the use of external calibration objects placed, and also requires the assistance of professional staff to complete the operation. The required equipment is numerous and the workload is huge, resulting in difficulty in widespread application of this method in traffic scenarios.

[0005] Therefore, it is particularly important to study an automatic calibration method for devices (such as camera automatic calibration methods) in traffic scenarios.

[0006] Existing device automatic calibration methods can be implemented based on deep learning models. However, deep learning models require a large amount of data for training and optimization, and the data needs to be calibrated, and the calibration work of the data is also relatively cumbersome, resulting in low generality of device automatic calibration. Summary of the Invention

[0007] This application provides a calibration method, device, and electronic device for a device to solve the problem of low generality of device automatic calibration. The specific implementation solutions are as follows:

[0008] In a first aspect, this application provides a calibration method for a device, the method comprising:

[0009] Obtain a plurality of image information sensed by the device;

[0010] Based on the plurality of image information, determine a plurality of license plate image position information and determine a straight-line trajectory of a plurality of vehicles;

[0011] Based on the plurality of license plate image position information and the plurality of straight-line trajectories, calculate a first vanishing point, a second vanishing point, and a third vanishing point;

[0012] Perform automatic calibration of the device in combination with the first vanishing point, the second vanishing point, and the third vanishing point.

[0013] Through the above application embodiments, the device is calibrated based on the acquired image information sensed by the device. Without the training iteration optimization of the deep learning model, it is only necessary to determine the license plate image position information based on the image information sensed by the device, then determine three vanishing points based on the license plate image position information, and then combine these three vanishing points to achieve automatic calibration of the device. Therefore, compared with the device automatic calibration that requires the training iteration optimization of the deep learning model in the prior art, less information is required, and there is no need to calibrate the data, reducing the time required for the training iteration optimization of the deep learning model and the time for data calibration, thereby improving the versatility of device automatic calibration.

[0014] In a possible implementation manner, the determining multiple license plate image position information based on the multiple image information includes:

[0015] Determine the target license plate detection model;

[0016] Process the multiple image information through the target license plate detection model respectively to obtain the corresponding multiple license plate image position information.

[0017] Through the above application embodiments, the determined license plate image position information is more accurate.

[0018] In a possible implementation manner, the determining the straight-line trajectories of multiple vehicles includes:

[0019] Based on the license plate image position information of the same vehicle at different times, determine the vehicle trajectory of the corresponding vehicle;

[0020] Perform straight-line trajectory fitting on the vehicle trajectory to obtain the straight-line trajectory corresponding to the corresponding vehicle.

[0021] Through the above application embodiments, first, according to the license plate image position information of the same vehicle at different times, the vehicle trajectory of the corresponding vehicle can be accurately determined, and then the straight-line trajectory fitting is performed on the vehicle trajectory, so as to accurately fit the straight-line trajectory of the vehicle, thereby making the obtained straight-line trajectory more accurate and conducive to improving the accuracy of the vanishing point calculated based on the straight-line trajectory.

[0022] In a possible implementation manner, the calculating the first vanishing point, the second vanishing point, and the third vanishing point based on the multiple license plate image position information and the multiple straight-line trajectories includes:

[0023] Construct a license plate information set including a plurality of the license plate image position information; and, construct a vehicle straight trajectory set including a plurality of the straight trajectories;

[0024] Based on the license plate information set and the vehicle straight trajectory set, calculate the first vanishing point, the second vanishing point, and the third vanishing point.

[0025] Through the above application embodiments, first construct a license plate information set including a plurality of license plate image position information and a vehicle straight trajectory set including a plurality of straight trajectories, so as to convert a plurality of license plate image position information into a set form for subsequent calculation, and convert a plurality of straight trajectories into a set form for subsequent calculation, in order to more quickly and accurately determine the required license plate image position information and the straight trajectory set, and avoid repeated calculations.

[0026] In a possible implementation manner, the license plate image position information includes a first image coordinate point and a second image coordinate point, then calculating the first vanishing point, the second vanishing point, and the third vanishing point based on the plurality of license plate image position information and the plurality of straight trajectories includes:

[0027] Based on the first image coordinate point and the second image coordinate point corresponding to each license plate respectively, calculate a first intersection point of the upper edge line and the lower edge line of each license plate, and a second intersection point of the left edge line and the right edge line; and

[0028] For the straight trajectory of each vehicle, calculate a third intersection point between every two of the straight trajectories;

[0029] Based on the plurality of first intersection points, the plurality of second intersection points, and the plurality of third intersection points, determine the first vanishing point, the second vanishing point, and the third vanishing point.

[0030] Through the above application embodiments, the determined first intersection point, second intersection point, and third intersection point are more accurate, which is beneficial to improving the accuracy of the first vanishing point, second vanishing point, and third vanishing point determined based on the first intersection point, second intersection point, and third intersection point.

[0031] In a possible implementation manner, the determining the first vanishing point, the second vanishing point, and the third vanishing point based on the plurality of first intersection points, the plurality of second intersection points, and the plurality of third intersection points includes:

[0032] Process the plurality of first intersection points, the plurality of second intersection points, and the plurality of third intersection points respectively through a clustering algorithm to obtain the first vanishing point, the second vanishing point, and the third vanishing point.

[0033] Through the above application embodiments, by separately processing multiple first intersection points, multiple second intersection points, and multiple third intersection points based on the clustering algorithm, abnormal interference points among the multiple first intersection points, multiple second intersection points, and multiple third intersection points are removed, thereby obtaining the most accurate first vanishing point, second vanishing point, and third vanishing point, which is beneficial to improving the accuracy of automatic calibration of the device based on the first vanishing point, second vanishing point, and third vanishing point.

[0034] In a possible implementation manner, the combining the first vanishing point, the second vanishing point, and the third vanishing point to perform automatic calibration on the device includes:

[0035] Obtain license plate standard size information; wherein, the license plate standard size information is the difference between the length value and the width value of the license plate;

[0036] Process the first vanishing point, the second vanishing point, the third vanishing point, and the license plate standard size information through a vanishing point calibration algorithm to determine the calibration matrix of the device.

[0037] Through the above application embodiments, since the license plate standard information is the difference between the length value and the width value of the license plate, the license plate standard size information is a known information. Combining with the accurately determined first vanishing point, second vanishing point, and third vanishing point, the automatic calibration of the device can be realized through the vanishing point calibration algorithm, and the calibration result is more accurate.

[0038] In a second aspect, the present application further provides a calibration device for a device, and the device includes:

[0039] An acquisition module, configured to acquire multiple image information sensed by the device;

[0040] A determination module, configured to determine multiple license plate image position information based on the multiple image information, and determine the straight-line trajectories of multiple vehicles;

[0041] A calculation module, configured to calculate a first vanishing point, a second vanishing point, and a third vanishing point based on the multiple license plate image position information and the multiple straight-line trajectories;

[0042] A processing module, configured to combine the first vanishing point, the second vanishing point, and the third vanishing point to perform automatic calibration on the device.

[0043] In a possible implementation manner, the determination module is specifically configured to determine a target license plate detection model; and process the multiple image information through the target license plate detection model to obtain the corresponding multiple license plate image position information.

[0044] In a possible implementation, the determining module is further configured to determine the vehicle trajectory of the corresponding vehicle based on the license plate image position information of the same vehicle at different times; perform a straight-line trajectory fitting on the vehicle trajectory to obtain the straight-line trajectory corresponding to the corresponding vehicle.

[0045] In a possible implementation, the calculating module is specifically configured to construct a license plate information set including a plurality of the license plate image position information; and construct a vehicle straight-line trajectory set including a plurality of the straight-line trajectories; calculate the first vanishing point, the second vanishing point, and the third vanishing point based on the license plate information set and the vehicle straight-line trajectory set.

[0046] In a possible implementation, the license plate image position information includes a first image coordinate point and a second image coordinate point, and the calculating module is further configured to calculate a first intersection point of the upper and lower sides of each license plate and a second intersection point of the left and right sides of each license plate based on the first image coordinate point and the second image coordinate point corresponding to each license plate; and calculate a third intersection point between every two of the straight-line trajectories for the straight-line trajectory of each vehicle; determine the first vanishing point, the second vanishing point, and the third vanishing point based on a plurality of the first intersection points, a plurality of the second intersection points, and a plurality of the third intersection points.

[0047] In a possible implementation, the calculating module is further configured to process a plurality of the first intersection points, a plurality of the second intersection points, and a plurality of the third intersection points respectively through a clustering algorithm to obtain the first vanishing point, the second vanishing point, and the third vanishing point.

[0048] In a possible implementation, the processing module is specifically configured to obtain license plate standard size information; wherein the license plate standard size information is the difference between the length value and the width value of the license plate; process the first vanishing point, the second vanishing point, the third vanishing point, and the license plate standard size information through a vanishing point calibration algorithm to determine the calibration matrix of the device.

[0049] In a third aspect, the present application provides an electronic device, including:

[0050] A memory for storing a computer program;

[0051] A processor for implementing the steps of the calibration method of a device as described above when executing the computer program stored on the memory.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the calibration method of a device as described above are implemented.

[0053] For the technical effects that can be achieved by each of the above second to fourth aspects and each aspect, please refer to the technical effects that can be achieved by the above-mentioned first aspect or various possible solutions in the first aspect, and will not be repeated here. Description of the Drawings

[0054] Figure 1 A schematic diagram of a side view traffic scene provided by an embodiment of the present application;

[0055] Figure 2 A flowchart of a calibration method for a device provided by an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the processing process of the calibration method for the device provided by an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a calibration device for a device provided by an embodiment of the present application;

[0058] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0059] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The specific operation methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of the present application, "a plurality of" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The connection between A and B can represent: two cases of direct connection between A and B and connection between A and B through C. In addition, in the description of the present application, terms such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0060] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0061] Existing device automatic calibration methods (such as camera automatic calibration methods) often require a large amount of data to train and optimize deep learning models, and the calibration work of the data is also relatively cumbersome, resulting in the lack of universality in the automatic calibration of devices.

[0062] Therefore, the present application proposes a calibration method for a device. Based on multiple image information sensed by the device, multiple license plate image position information is determined, and multiple straight trajectories of vehicles are determined. Then, based on the multiple license plate image position information and the multiple straight trajectories, a first vanishing point, a second vanishing point, and a third vanishing point are calculated. Furthermore, in combination with the first vanishing point, the second vanishing point, and the third vanishing point, the device is calibrated. In this way, only the image information sensed by the device is required to determine the vanishing point, and then the automatic calibration of the device can be achieved by combining the vanishing point. There is no need to use a large amount of data to train and optimize a deep learning model, thus avoiding cumbersome data calibration work and reducing the training and optimization time of the deep learning model, making the automatic calibration of the device more general.

[0063] The above device can be a camera. Thus, based on multiple image information sensed by the camera, the automatic calibration of the camera can be achieved.

[0064] The above vanishing points (such as the first vanishing point, the second vanishing point, and the third vanishing point) represent the intersection points where two parallel straight lines intersect after perspective transformation, which can be understood as the intersection points of two parallel lines in the real world but intersecting in the image.

[0065] Moreover, the above first vanishing point, second vanishing point, and third vanishing point correspond to three axes in the world coordinate system.

[0066] For example, the first vanishing point corresponds to the X-axis in the world coordinate system. That is, on the image plane, the intersection point where two straight lines parallel to the X-axis in the world coordinate system intersect after perspective transformation reflects the perspective information of the objects in the scene in the X-axis direction. That is to say, the first vanishing point corresponds to the axis corresponding to the parallel direction of the two straight lines corresponding to the first vanishing point in the world coordinate system. The first vanishing point can be expressed as: .

[0067] The second vanishing point corresponds to the Z-axis in the world coordinate system. That is, on the image plane, the intersection point where two straight lines parallel to the Z-axis in the world coordinate system intersect after perspective transformation reflects the perspective information of the objects in the scene in the Z-axis direction. That is to say, the second vanishing point corresponds to the axis corresponding to the parallel direction of the two straight lines corresponding to the second vanishing point in the world coordinate system. The second vanishing point can be expressed as: .

[0068] The third vanishing point corresponds to the Y-axis in the world coordinate system. That is, on the image plane, the intersection point where two straight lines parallel to the Y-axis in the world coordinate system intersect after perspective transformation reflects the perspective information of the objects in the scene in the Y-axis direction. That is to say, the third vanishing point corresponds to the axis corresponding to the parallel direction of the two straight lines corresponding to the third vanishing point in the world coordinate system. The third vanishing point can be expressed as: 。

[0069] Due to a large number of vehicles (i.e., vehicle targets) in the traffic scene, each vehicle has its own license plate, which is a unique information in the traffic scene with standard size information. As Figure 1 shown in the typical side view traffic scene, the vehicle has its own license plate. Therefore, the automatic calibration of the device can be achieved through the relevant information of the license plate.

[0070] The upper, lower, left, and right sides of the license plate form two sets of parallel lines, and these two sets of parallel lines are orthogonal, and are parallel to the X and Z axes of the world coordinate system. Therefore, based on the position information of the license plate image in the parking space, the first vanishing point corresponding to the X axis in the world coordinate system and the second vanishing point corresponding to the Y axis in the world coordinate system can be calculated.

[0071] In addition, in the traffic scene, there are a large number of vehicles moving in a nearly straight line. It can be considered that the moving direction of the vehicle is parallel to the Y axis of the world coordinate system. Therefore, based on the straight-line trajectory of the vehicle, the third vanishing point corresponding to the Y axis in the world coordinate system can be calculated.

[0072] Referring to Figure 2 shown in the flowchart of a calibration method for a device provided by an embodiment of the present application, the method includes:

[0073] S201, obtaining a plurality of image information sensed by the device.

[0074] Among them, the image information can be the image information of the traffic scene. That is, based on the traffic scene, the image information sensed by the device is obtained. The image information can be the image information within a time period T (T is a positive number), so as to facilitate collecting the position information of the license plate image within the time period T based on the image information.

[0075] S202, based on the plurality of image information, determining a plurality of license plate image position information and determining the straight-line trajectories of a plurality of vehicles.

[0076] Optionally, the specific determination process of determining a plurality of license plate image information based on the plurality of image information can be as follows:

[0077] First, obtain the target license plate detection model, and then by processing each image information through the target license plate detection model, one or more license plate image position information in each image information can be obtained, making the determined license plate image position information more accurate, which helps to improve the accuracy of device calibration based on the license plate image position information. Thus, through the target license plate detection model, the license plate information (i.e., the license plate image position information) in the traffic scene can be detected in real time and accurately, that is, the license plate information (i.e., the license plate image position information) in the image information sensed by the device can be detected in real time and accurately.

[0078] In the embodiments of the present application, the above-mentioned target license plate detection model may be a pre-trained license plate detection model, such as a Region-based Convolutional Neural Networks (R-CNN) model, a Single Shot MultiBox Detector (SSD) model, etc. This avoids the training of the license plate detection model, further reduces the calibration time of the device for automatic calibration, improves the efficiency of the device for automatic calibration, and further enhances the versatility of the device for automatic calibration.

[0079] The above-mentioned multiple license plate image position information may be the license plate image position information of different vehicles at different times, the license plate image position information of the same vehicle at different times, or the license plate image position information of different vehicles at the same time. Moreover, since the multiple image information obtained is the image information within the time period T, the collection of license plate information at different positions and different times within the time period T is realized.

[0080] The above-mentioned license plate image position information at least includes the first image coordinate point and the second image coordinate point of the license plate, and may also include information such as the license plate number, the identity document (ID) of the vehicle, and the time.

[0081] The above-mentioned first image coordinate point can be expressed as: ; The above-mentioned second image coordinate point can be expressed as: . Wherein, V represents the image coordinate system.

[0082] The first image coordinate point may be the upper left image coordinate point of the license plate, that is, the intersection point of the left side line and the upper side line of the license plate; the second image coordinate point may be the lower right image coordinate point of the license plate, that is, the intersection point of the right side line and the lower side line of the license plate. Thus, the third image coordinate point and the fourth image coordinate point of the corresponding license plate can be determined according to the first image coordinate point and the second image coordinate point. In this case, the third image coordinate point is the upper right image coordinate point of the license plate, that is, the intersection point of the right side line and the upper side line of the license plate. The fourth image coordinate point is the lower left image coordinate point of the license plate, that is, the intersection point of the left side line and the lower side line of the license plate.

[0083] Alternatively, the above first image coordinate point can be the lower left image coordinate point of the license plate, that is, the intersection point of the left side line and the lower side line of the license plate. The above second image coordinate point can be the upper right image coordinate point of the license plate, that is, the intersection point of the right side line and the upper side line of the license plate. Thus, according to the first image coordinate point and the second image coordinate point, the third image coordinate point and the fourth image coordinate point corresponding to the license plate can be determined. In this case, the third image coordinate point is the lower right image coordinate point of the license plate, that is, the intersection point of the right side line and the lower side line of the license plate. The fourth image coordinate point is the upper left image coordinate point of the license plate, that is, the intersection point of the left side line and the upper side line of the license plate.

[0084] Assume that the position information corresponding to the above first image coordinate point in the world coordinate system is , where W represents the world coordinate system, and the position information corresponding to the above second image coordinate point in the world coordinate system is , then there is . D represents a known license plate standard size information, and D can be the difference between the length value and the width value of the license plate. Thus, the conversion relationship between the image coordinate points of the license plate and the coordinate points in the world coordinate system can be determined through the known D, the first image coordinate point, and the second image coordinate point, so as to facilitate the determination of the calibration matrix of the device.

[0085] Optionally, in the embodiment of the present application, after obtaining multiple license plate image position information, the Zhang-Zhengyou calibration method can also be combined to achieve automatic calibration of the device.

[0086] Specifically, using the license plate information with known size (i.e., the license plate standard size information) as a calibration board unique to the traffic scene, and combining the multiple license plate image position information at different positions, the Zhang-Zhengyou calibration method can be used to achieve automatic calibration of the device, thereby making full use of the environmental information of the traffic scene, achieving automatic calibration of the device, and improving the flexibility of device calibration.

[0087] Optionally, the specific determination process of determining the straight-line trajectories of multiple vehicles based on the above multiple image information can be as follows:

[0088] First, based on the multiple image information, determine the vehicle trajectories of multiple vehicles.

[0089] In a possible implementation manner, the license plate position information is used as a specific manifestation of the vehicle trajectory information. Then the vehicle trajectory can be determined according to the license plate image position information. That is, after obtaining multiple license plate image position information based on the multiple image information, based on the license plate image position information of the same license plate at different times, determine the vehicle trajectory of the corresponding vehicle.

[0090] In another possible implementation manner, the above vehicle trajectory can be determined according to the position information of the vehicle, and the determination process can be as follows:

[0091] First, obtain the target vehicle detection model, and then process multiple image information through the target vehicle detection model respectively to obtain one or more vehicle information in each image information, so that the determined vehicle information is more accurate. Thus, through the target vehicle detection model, the vehicle information in the traffic scene can be detected in real time and accurately, that is, the vehicle information in the image information sensed by the device can be detected in real time and accurately.

[0092] Then, based on the position information of the vehicle in the vehicle information, determine the vehicle trajectory of the corresponding vehicle. Specifically, based on the position information of the same vehicle at different times, determine the vehicle trajectory of the corresponding vehicle.

[0093] In the embodiment of the present application, in addition to including the position information of the vehicle, the above vehicle information may further include information such as the ID and time of the vehicle, so as to facilitate determining the position information of the same vehicle at different times according to the ID. The position information of the above vehicle can be represented by the coordinate points of the detected vehicle identification frame.

[0094] The above target vehicle detection model can be a trained vehicle detection model, such as models like R-CNN, vehicle detection models based on attention mechanisms, etc., thus avoiding the training of the vehicle detection model, further reducing the calibration time of device automatic calibration, improving the efficiency of device automatic calibration, and further enhancing the versatility of device automatic calibration.

[0095] Furthermore, after determining the vehicle trajectories of multiple vehicles, perform linear trajectory fitting on the vehicle trajectories to obtain the respective linear trajectories corresponding to each vehicle, so that the determined linear trajectories are more accurate.

[0096] In the embodiment of the present application, the Random Sample Consensus (RANSAC) algorithm can be used to perform linear trajectory fitting on the vehicle trajectories to obtain the linear trajectories of the corresponding vehicles, but it is not limited thereto, and specific linear trajectory fitting methods can be selected according to specific application scenarios.

[0097] Optionally, after determining the vehicle trajectories of multiple vehicles, a vehicle trajectory set can be constructed based on the multiple vehicle trajectories, so as to better store the vehicle trajectories of multiple vehicles and facilitate subsequent better fitting of the respective linear trajectories corresponding to each vehicle based on the vehicle trajectories. Furthermore, the above performing linear trajectory fitting on the vehicle trajectories to obtain the respective linear trajectories corresponding to each vehicle can be: performing linear trajectory fitting on the multiple vehicle trajectories in the vehicle trajectory set to obtain the respective vehicle trajectories corresponding to each vehicle.

[0098] When the vehicle trajectory is determined based on the license plate image position information, the vehicle trajectories included in the constructed vehicle trajectory set are constituted by the license plate image position information.

[0099] When the vehicle trajectory is determined based on the position information of the vehicle, the vehicle trajectories included in the constructed vehicle trajectory set are constituted by the position information of the vehicle. This vehicle trajectory set can be expressed as: P V ={Pt t}. Where P v represents the vehicle trajectory set; t represents time, and Pt t represents the position information of the vehicle at the t-th moment, that is, the coordinate point at the t-th moment.

[0100] S203. Calculate the first vanishing point, the second vanishing point, and the third vanishing point based on multiple license plate image position information and multiple straight trajectories.

[0101] After obtaining multiple license plate image position information and multiple straight trajectories, calculate the first vanishing point, the second vanishing point, and the third vanishing point based on the multiple license plate image position information and the multiple straight trajectories.

[0102] Specifically, first, based on the first image coordinate point and the second image coordinate point in each license plate image position information, calculate the first intersection point of the upper and lower sides of the corresponding license plate, and the second intersection point of the left and right sides; use the first intersection point as the first vanishing point to be determined, and the second intersection point as the second vanishing point to be determined, so as to make the determined first vanishing point to be determined and the second vanishing point to be determined more accurate.

[0103] The specific calculation process of calculating the first intersection point of the upper and lower sides of the corresponding license plate and the second intersection point of the left and right sides based on the first image coordinate point and the second image coordinate point in the license plate image position information can be as follows:

[0104] Taking the first image coordinate point as the upper left image coordinate point of the license plate, that is, the intersection point of the left side and the upper side of the license plate; and the second image coordinate point as the lower right image coordinate point of the license plate, that is, the intersection point of the right side and the lower side of the license plate as an example, first determine the third image coordinate point and the fourth image coordinate point of the license plate based on the first image coordinate point and the second image coordinate point in the license plate image position information.

[0105] Then, based on the first image coordinate point and the fourth image coordinate point, fit the image straight line of the left side of the license plate; based on the second image coordinate point and the third image coordinate point, fit the image straight line of the right side of the license plate; based on the first image coordinate point and the third image coordinate point, fit the image straight line of the upper side of the license plate; based on the second image coordinate point and the fourth image coordinate point, fit the image straight line of the lower side of the license plate.

[0106] Based on the image lines of the upper edge and the lower edge of the license plate, calculate the first intersection point of the upper edge and the lower edge of the license plate; based on the image lines of the left edge and the right edge of the license plate, calculate the second intersection point of the left edge and the right edge of the license plate.

[0107] To better save and use the position information of multiple license plate images, first, based on the obtained position information of multiple license plate images, construct a license plate information set including the position information of multiple license plate images. Then, based on the position information of multiple license plate images in the license plate information set, calculate the first intersection point and the second intersection point.

[0108] In the embodiments of the present application, the above license plate information set may include the position information of license plate images of the same vehicle at different times, the position information of license plate images of different vehicles at the same time, and the position information of license plates of different vehicles at different times. Moreover, each piece of license plate image position information in the license plate information set includes a first image coordinate point and a second image coordinate point.

[0109] The above license plate information set may be as follows:

[0110]

[0111] Among them, P represents the license plate information set; L p represents the license plate position information.

[0112] Thus, through the above license plate information set, the license plate information at different times and different positions is collected, that is, the position information of multiple license plate images is collected, so as to better save the position information of multiple license plate images, and when calculating the vanishing point based on the license plate image position information, the required license plate image position information can be taken out more quickly and accurately.

[0113] Furthermore, after obtaining the position information of multiple license plate images, combining with the Zhang Zhengyou calibration method to realize the automatic calibration of the device can be: using the known size of the license plate information (i.e., the license plate standard size information) as a specific calibration board in the traffic scene, and then combining with the license plate information set, and adopting the Zhang Zhengyou calibration method, the automatic calibration of the device can be realized.

[0114] Next, based on multiple vehicle trajectories, calculate the third intersection point of the straight-line trajectories between two vehicles, that is, calculate the third intersection point between each two straight-line trajectories. And use this third intersection point as the third vanishing point to be determined, so that the calculated third vanishing point to be determined is more accurate.

[0115] The third intersection point between the above two straight-line trajectories can be calculated by the method of finding the intersection point of two straight lines.

[0116] To better preserve and use multiple straight-line trajectories, a vehicle straight-line trajectory set including multiple straight-line trajectories can be constructed based on the obtained multiple straight-line trajectories. Then, based on the multiple straight-line trajectories in the vehicle straight-line trajectory set, the third intersection point can be calculated.

[0117] The above vehicle straight-line trajectory set can be shown as follows:

[0118]

[0119] Among them, L V represents the vehicle straight-line trajectory set, and l V represents a straight-line trajectory.

[0120] Thus, through the vehicle straight-line trajectory set, it is convenient to better preserve the straight-line trajectories of the vehicle, and when calculating the third intersection point, the required straight-line trajectories can be retrieved more quickly and accurately.

[0121] In the embodiments of the present application, the calculation processes of calculating the first intersection point and the second intersection point and the calculation process of calculating the third intersection point can be executed simultaneously; or the first intersection point and the second intersection point can be calculated first, and then the third intersection point can be calculated; or the third intersection point can be calculated first, and then the first intersection point and the second intersection point can be calculated.

[0122] Furthermore, after calculating multiple first to-be-determined vanishing points, multiple second to-be-determined vanishing points, and multiple third to-be-determined vanishing points, through a clustering algorithm, the multiple first to-be-determined vanishing points, the multiple second to-be-determined vanishing points, and the multiple third to-be-determined vanishing points are processed respectively to remove the abnormal vanishing points among the multiple first to-be-determined vanishing points, the multiple second to-be-determined vanishing points, and the multiple third to-be-determined vanishing points. Thus, the most accurate first to-be-determined vanishing point can be determined from the multiple first to-be-determined vanishing points, the most accurate second to-be-determined vanishing point can be determined from the multiple second to-be-determined vanishing points, and the most accurate third to-be-determined vanishing point can be determined from the multiple third to-be-determined vanishing points.

[0123] Then, the determined most accurate first to-be-determined vanishing point is used as the first vanishing point, the most accurate second to-be-determined vanishing point is used as the second vanishing point, and the most accurate third to-be-determined vanishing point is used as the third vanishing point, so that the obtained first vanishing point, second vanishing point, and third vanishing point are more accurate.

[0124] To better preserve and use multiple first vanishing points to be determined, multiple second vanishing points to be determined, and multiple third vanishing points to be determined, first, based on the multiple first vanishing points to be determined, construct a first set of vanishing points to be determined that includes the multiple first vanishing points to be determined. Based on the multiple second vanishing points to be determined, construct a second set of vanishing points to be determined that includes the multiple second vanishing points to be determined. Based on the multiple third vanishing points to be determined, construct a third set of vanishing points to be determined that includes the multiple third vanishing points to be determined. The first set of vanishing points to be determined can be expressed as: ; The second set of vanishing points to be determined can be expressed as: ; The third set of vanishing points to be determined can be expressed as: .

[0125] Then, use a clustering algorithm to process the first set of vanishing points to be determined, the second set of vanishing points to be determined, and the third set of vanishing points to be determined, so as to eliminate the influence of abnormal interference points, and obtain the most accurate first vanishing point to be determined in the first set of vanishing points to be determined, the most accurate second vanishing point to be determined in the second set of vanishing points to be determined, and the most accurate third vanishing point to be determined in the third set of vanishing points to be determined. And use the most accurate first vanishing point to be determined in the first set of vanishing points to be determined as the first vanishing point, the most accurate second vanishing point to be determined in the second set of vanishing points to be determined as the second vanishing point, and the most accurate third vanishing point to be determined in the third set of vanishing points to be determined as the third vanishing point.

[0126] Through the above method, by using the first set of vanishing points to be determined, the second set of vanishing points to be determined, and the third set of vanishing points to be determined, multiple first vanishing points to be determined, multiple second vanishing points to be determined, and multiple third vanishing points to be determined are better preserved. Moreover, the clustering algorithm can better cluster the multiple first vanishing points to be determined, multiple second vanishing points to be determined, and multiple third vanishing points to be determined, further increasing the accuracy of the determined first vanishing point, second vanishing point, and third vanishing point.

[0127] S204. Combine the first vanishing point, the second vanishing point, and the third vanishing point to automatically calibrate the device.

[0128] Since the vanishing point calibration algorithm for realizing the automatic calibration of the device requires three vanishing points in a scene (such as a traffic scene) and a known prior size, and these three vanishing points correspond to three axes in the world coordinate system, the internal and external parameters of the device can be accurately obtained, that is, the automatic calibration of the device is realized.

[0129] Therefore, after obtaining the first vanishing point, the second vanishing point, and the third vanishing point, and these first vanishing point, second vanishing point, and third vanishing point correspond to three axes in the world coordinate system, first obtain the license plate standard size information (i.e., the aforementioned D).

[0130] Then, by combining the standard size information of the license plate and the first vanishing point, the second vanishing point, and the third vanishing point determined based on the first image coordinate point and the second image coordinate point of the license plate, and adopting the vanishing point calibration algorithm, the solution of the device calibration matrix can be realized, thereby realizing the calibration of the device and obtaining the calibration matrix of the device.

[0131] In summary, the device calibration method proposed in this application determines three vanishing points (i.e., the first vanishing point, the second vanishing point, and the third vanishing point) based on the image information sensed by the device (such as a camera). Thus, by combining these three vanishing points with the known standard size information of the license plate, the online automatic calibration of the device (such as a camera) can be realized.

[0132] In this way, the environmental information in the traffic scene is fully utilized. Without prior configuration information, there are no high requirements for the scene information and the carrier, and there is no need for manual intervention or iterative training of deep learning methods. The automatic calibration of the device (such as a camera) can be realized, making the automatic calibration of the device (such as a camera) more concise, faster, and more versatile, and improving the accuracy of the calibration result (i.e., the calibration accuracy), with excellent usability and feasibility.

[0133] The technical solution of this application will be further described below in combination with the specific application process.

[0134] As Figure 3 shown in the schematic diagram of the processing process of the device calibration method, taking a camera as an example, first in the data acquisition module, based on the traffic scene, multiple image information sensed by the camera is obtained. And the multiple image information is transmitted to the detection module for the detection module to detect the image information.

[0135] In the detection module, a target vehicle detection model and a target license plate detection model are obtained. Through the processing of the image information by the target vehicle detection model and the target license plate detection model respectively, the vehicle information and the license plate image position information in the traffic scene are detected in real time. And the license plate image position information is transmitted to the first set construction module for the first set construction module to construct a license plate information set based on the license plate image position information. And the vehicle information is transmitted to the second set construction module for the second set construction module to construct a vehicle straight line trajectory set based on the vehicle information.

[0136] In the first set construction module, the license plate image position information transmitted from the detection module is received, and then based on the license plate image position information, a license plate information set is constructed. And the license plate information set is transmitted to the vanishing point calculation module for the vanishing point calculation module to calculate the vanishing point based on the license plate information set.

[0137] In the second set construction module, vehicle information transmitted by the detection module is received. Then, based on the position information of the vehicle in the vehicle information, a vehicle trajectory set is constructed. Next, RANSAC is used to perform linear trajectory fitting on the position information of the same vehicle in the vehicle trajectory set, and based on the fitted linear trajectory, a vehicle linear trajectory set is constructed. And this vehicle linear trajectory set is transmitted to the vanishing point calculation module, so that the vanishing point calculation module calculates the vanishing point based on the vehicle linear trajectory set.

[0138] In the vanishing point calculation module, based on the first image coordinate points and the second image coordinate points corresponding to each license plate in the license plate information set, the first intersection point of the upper edge line and the lower edge line of the license plate and the second intersection point of the left edge line and the right edge line of the license plate are calculated, and the first intersection point is used as the first vanishing point to be determined, and the second intersection point is used as the second vanishing point to be determined. Then, based on multiple first vanishing points to be determined, a first vanishing point set to be determined is constructed, and based on multiple second vanishing points to be determined, a second vanishing point set to be determined is constructed.

[0139] In addition, in the vehicle linear trajectory set, the third intersection point of the linear trajectories between two vehicles is calculated, and the third intersection point is used as the third vanishing point to be determined. Then, based on multiple third vanishing points to be determined, a third vanishing point set to be determined is constructed. Then, through the processing of the first vanishing point set to be determined, the second vanishing point set to be determined, and the third vanishing point set to be determined by the clustering algorithm, the most accurate first vanishing point to be determined (i.e., the first vanishing point), the most accurate second vanishing point to be determined (i.e., the second vanishing point), and the most accurate third vanishing point to be determined (i.e., the third vanishing point) are obtained. Then, the first vanishing point, the second vanishing point, and the third vanishing point are transmitted to the calibration module, so that the calibration module calibrates the camera in combination with the first vanishing point, the second vanishing point, and the third vanishing point.

[0140] In the calibration module, the standard size information of the license plate is obtained. Then, in combination with the standard size information of the license plate and the received first vanishing point, second vanishing point, and third vanishing point, the camera is automatically calibrated through the vanishing point calibration algorithm.

[0141] Based on the same inventive concept, an apparatus calibration device for a device is also provided in an embodiment of the present application, as Figure 4 shown in the structural schematic diagram of an apparatus calibration device provided by the present application. The device includes:

[0142] An acquisition module 401, configured to acquire a plurality of image information sensed by the device;

[0143] A determination module 402, configured to determine a plurality of license plate image position information based on the plurality of image information, and determine the linear trajectories of a plurality of vehicles;

[0144] The calculation module 403 is configured to calculate a first vanishing point, a second vanishing point, and a third vanishing point based on the multiple license plate image position information and the multiple straight trajectories.

[0145] The processing module 404 is configured to automatically calibrate the device by combining the first vanishing point, the second vanishing point, and the third vanishing point.

[0146] In a possible implementation manner, the determination module 402 is specifically configured to determine a target license plate detection model; and process the multiple image information through the target license plate detection model to obtain the corresponding multiple license plate image position information.

[0147] In a possible implementation manner, the determination module 402 is further configured to determine the vehicle trajectory of the corresponding vehicle based on the license plate image position information of the same vehicle at different times; and perform straight trajectory fitting on the vehicle trajectory to obtain the straight trajectory corresponding to the corresponding vehicle.

[0148] In a possible implementation manner, the calculation module 403 is specifically configured to construct a license plate information set including the multiple license plate image position information; and construct a vehicle straight trajectory set including the multiple straight trajectories; and calculate the first vanishing point, the second vanishing point, and the third vanishing point based on the license plate information set and the vehicle straight trajectory set.

[0149] In a possible implementation manner, the license plate image position information includes a first image coordinate point and a second image coordinate point. The calculation module 403 is further configured to calculate a first intersection point of the upper and lower sides of each license plate and a second intersection point of the left and right sides of each license plate based on the first image coordinate point and the second image coordinate point corresponding to each license plate; and calculate a third intersection point between every two of the straight trajectories for each vehicle's straight trajectory; and determine the first vanishing point, the second vanishing point, and the third vanishing point based on the multiple first intersection points, the multiple second intersection points, and the multiple third intersection points.

[0150] In a possible implementation manner, the calculation module 403 is further configured to process the multiple first intersection points, the multiple second intersection points, and the multiple third intersection points respectively through a clustering algorithm to obtain the first vanishing point, the second vanishing point, and the third vanishing point.

[0151] In a possible implementation manner, the processing module 404 is specifically configured to obtain license plate standard size information; where the license plate standard size information is the difference between the length value and the width value of the license plate; and process the first vanishing point, the second vanishing point, the third vanishing point, and the license plate standard size information through a vanishing point calibration algorithm to determine the calibration matrix of the device.

[0152] Based on the same inventive concept, an electronic device is further provided in an embodiment of the present application. The above-mentioned electronic device can implement the functions of the calibration device of the foregoing device. Refer to Figure 5 , the above-mentioned electronic device includes:

[0153] At least one processor 501 and a memory 502 connected to the at least one processor 501. In the embodiment of the present application, the specific connection medium between the processor 501 and the memory 502 is not limited. Figure 5 In [reference], it is taken as an example that the processor 501 and the memory 502 are connected through a bus 500. The bus 500 is represented by a thick line in Figure 5 . The connection manners between other components are only for illustrative purposes and are not limited thereto. The bus 500 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5 it is only represented by a thick line in [reference], but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 501 can also be called a controller, and the name is not limited.

[0154] In the embodiment of the present application, the memory 502 stores instructions executable by the at least one processor 501. By executing the instructions stored in the memory 502, the at least one processor 501 can execute the calibration method of the device described above. The processor 501 can implement Figure 4 the functions of each module in the device shown in [reference].

[0155] Among them, the processor 501 is the control center of the device. It can connect various parts of the entire control device through various interfaces and lines. By running or executing the instructions stored in the memory 502 and calling the data stored in the memory 502, various functions of the device and process data, so as to monitor the device as a whole.

[0156] In a possible design, the processor 501 may include one or more processing units. The processor 501 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 can be implemented on the same chip. In some embodiments, they can also be implemented on separate chips respectively.

[0157] The processor 501 may be a general-purpose processor, such as a central processing unit (CPU for short), a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the calibration method of the device disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0158] The memory 502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 502 may include at least one type of storage medium, for example, it may include flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM for short), a static random access memory (SRAM for short), a programmable read-only memory (PROM for short), a read-only memory (ROM for short), an electrically erasable programmable read-only memory (EEPROM for short), a magnetic memory, a magnetic disk, an optical disk, and so on. The memory 502 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0159] By designing and programming the processor 501, the code corresponding to the calibration method of the device introduced in the foregoing embodiments can be solidified into the chip, so that the chip can execute Figure 2 the steps of the calibration method of the device shown in the embodiments when running. How to design and program the processor 501 is a well-known technology to those skilled in the art and will not be elaborated here.

[0160] Based on the same inventive concept, the embodiments of the present application also provide a storage medium, which stores computer instructions. When the computer instructions run on a computer, the computer is made to execute the calibration method of the device discussed above.

[0161] In some possible embodiments, various aspects of the calibration method of the device provided in this application can also be implemented in the form of a program product, which includes program code. When the program product runs on the device, the program code is used to cause the control device to execute the steps in the calibration method of the device according to various exemplary embodiments of this application described above in this specification.

[0162] Those skilled in the art should understand that the embodiments of this application can be provided as a method, a system, or a computer program product. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0163] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of multiple flows and / or blocks.

[0164] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including instruction means, and the instruction means implements the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of multiple flows and / or blocks.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of multiple flows and / or blocks.

[0166] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A method for calibrating a device, characterized in that: include: Acquire multiple image information perceived by the device; Based on the plurality of image information, a plurality of license plate image position information is determined, and a plurality of straight-line trajectories of vehicles are determined; wherein the license plate image position information includes a first image coordinate point and a second image coordinate point; Based on the first image coordinate point and the second image coordinate point corresponding to each license plate, calculate the first intersection point of the upper sideline and the lower sideline of each license plate, and the second intersection point of the left sideline and the right sideline; and Based on the straight line trajectories of the plurality of vehicles, calculating a third intersection point of the straight line trajectories between any two of the vehicles; Taking the first intersection point as a first vanishing point to be determined, taking the second intersection point as a second vanishing point to be determined, and taking the third intersection point as a third vanishing point to be determined; Processing the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined respectively by a clustering algorithm, removing abnormal vanishing points from the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined, and taking the most accurate first vanishing point to be determined determined from the plurality of first vanishing points to be determined as the first vanishing point, the most accurate second vanishing point to be determined determined from the plurality of second vanishing points to be determined as the second vanishing point, and the most accurate third vanishing point to be determined determined from the plurality of third vanishing points to be determined as the third vanishing point; wherein the third vanishing point is an intersection point between two straight line trajectories; The device is automatically calibrated in combination with the first vanishing point, the second vanishing point, and the third vanishing point.

2. The method according to claim 1, characterized in that The determining of a plurality of license plate image position information based on the plurality of image information comprises: Determine the target license plate detection model; The target license plate detection model is used to process the plurality of image information respectively to obtain the corresponding plurality of license plate image position information.

3. The method according to claim 1, characterized in that The determining of the straight-line trajectories of the plurality of vehicles comprises: Determining a vehicle trajectory of a corresponding vehicle based on the license plate image position information of the same vehicle at different times; A straight line trajectory is fitted for the vehicle trajectory to obtain the straight line trajectory corresponding to the corresponding vehicle.

4. The method according to claim 1, characterized in that Before automatically calibrating the device by combining the first vanishing point, the second vanishing point, and the third vanishing point, the method further includes: Constructing a license plate information set including a plurality of license plate image position information; and constructing a vehicle straight line trajectory set including a plurality of straight line trajectories; The first vanishing point, the second vanishing point, and the third vanishing point are calculated based on the license plate information set and the vehicle straight track set.

5. The method according to claim 1, characterized in that The automatically calibrating the device by combining the first vanishing point, the second vanishing point, and the third vanishing point includes: Obtaining standard size information of a license plate; wherein the standard size information of a license plate is the difference between a length value and a width value of the license plate; The first vanishing point, the second vanishing point, the third vanishing point and the standard size information of the license plate are processed by a vanishing point calibration algorithm to determine a calibration matrix of the device.

6. A calibration device for equipment, characterized in that: include: An acquisition module, used to acquire multiple image information perceived by the device; A determination module, used to determine a plurality of license plate image position information based on a plurality of the image information, and determine a plurality of straight-line trajectories of the vehicles; wherein the license plate image position information includes a first image coordinate point and a second image coordinate point; a calculation module, for calculating the first intersection point of the upper sideline and the lower sideline of each license plate, and the second intersection point of the left sideline and the right sideline based on the first image coordinate point and the second image coordinate point corresponding to each license plate; and, based on the straight line tracks of the plurality of vehicles, calculating the third intersection point of the straight line tracks between two vehicles; taking the first intersection point as the first vanishing point to be determined, taking the second intersection point as the second vanishing point to be determined, and taking the third intersection point as the third vanishing point to be determined; processing the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined respectively by a clustering algorithm, removing abnormal vanishing points from the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined, and taking the most accurate first vanishing point to be determined determined from the plurality of first vanishing points to be determined as the first vanishing point, the most accurate second vanishing point to be determined determined from the plurality of second vanishing points to be determined as the second vanishing point, and the most accurate third vanishing point to be determined determined from the plurality of third vanishing points to be determined as the third vanishing point; wherein the third vanishing point is the intersection point between two straight line tracks; A processing module is used to automatically calibrate the device based on the first vanishing point, the second vanishing point, and the third vanishing point.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 5 when executing the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images