Equipment calibration method and device and electronic equipment

Abstract

The invention discloses a device calibration method and apparatus, and an electronic device. The method comprises the steps of obtaining multiple pieces of image information sensed by a device; based on the multiple pieces of image information, multiple pieces of license plate image position information are determined, and linear tracks of multiple vehicles are determined; calculating a first vanishing point, a second vanishing point and a third vanishing point based on the plurality of license plate image position information and the plurality of linear tracks; and combining the first vanishing point, the second vanishing point and the third vanishing point to automatically calibrate the equipment. Through the technical scheme provided by the embodiment of the invention, automatic calibration of the equipment can be realized only based on the image information sensed by the equipment, training optimization of a deep learning model is not needed, data calibration is not needed, and thus the universality of automatic calibration of the equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a device calibration method, apparatus, and electronic device. Background Art

[0002] Devices (such as cameras) can be used to perform a variety of tasks such as capturing and monitoring, and play an important role in traffic scenarios.

[0003] In traffic scene applications, it is often necessary to accurately measure the position, speed and other information of vehicles in the monitored traffic scene. In order to achieve this function, the device (such as a camera) must be calibrated to associate the device's image coordinate system with the actual world coordinate system, thereby realizing the mutual conversion between the coordinate systems. Among them, calibrating the device is to determine the calibration matrix of the device.

[0004] Traditional equipment calibration methods (such as camera calibration methods) require the use of calibration objects of known geometric shapes, such as calibration plates or three-dimensional targets, to achieve equipment calibration. However, this method requires the use of external calibration objects and the assistance of professional staff to complete. It requires a lot of equipment and a huge workload, which makes it difficult to be widely used in traffic scenarios.

[0005] Therefore, it is particularly important to study an automatic equipment calibration method in traffic scenes (such as a camera automatic calibration method).

[0006] Existing automatic equipment calibration methods can realize automatic equipment calibration based on deep learning models. However, deep learning models require a large amount of data for training and optimization, and the data needs to be calibrated, and the data calibration work is also relatively cumbersome, resulting in low versatility of automatic equipment calibration. Summary of the invention

[0007] The present application provides a device calibration method, apparatus and electronic device to solve the problem of low universality of automatic calibration of devices. The specific implementation scheme is as follows: In a first aspect, the present application provides a method for calibrating a device, the method comprising: Acquire multiple image information perceived by the device; Based on the plurality of image information, determining the position information of the plurality of license plate images, and determining the linear trajectories of the plurality of vehicles; Calculating a first vanishing point, a second vanishing point, and a third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories; The device is automatically calibrated in combination with the first vanishing point, the second vanishing point, and the third vanishing point.

[0008] Through the above-mentioned application embodiments, the device is calibrated based on the image information perceived by the acquired device, without the need for iterative optimization of training of the deep learning model. It is only necessary to determine the license plate image position information based on the image information perceived by the device, and then determine three vanishing points based on the license plate image position information. Then, the device can be automatically calibrated in combination with the three vanishing points. Compared with the automatic calibration of the device in the prior art which requires iterative optimization of training of the deep learning model to be achieved, less information is required, and there is no need to calibrate the data, which reduces the time required for iterative optimization of training of the deep learning model and the time for data calibration, thereby improving the versatility of automatic calibration of the device.

[0009] In a possible implementation, determining multiple license plate image position information based on multiple image information includes: Determine the target license plate detection model; The target license plate detection model is used to process the plurality of image information respectively to obtain the corresponding plurality of license plate image position information.

[0010] Through the above application embodiments, the determined license plate image position information is made more accurate.

[0011] In a possible implementation, determining the straight-line trajectories of the plurality of vehicles includes: Determining a vehicle trajectory of a corresponding vehicle based on the license plate image position information of the same vehicle at different times; A straight line trajectory is fitted for the vehicle trajectory to obtain the straight line trajectory corresponding to the corresponding vehicle.

[0012] Through the above-mentioned application embodiment, the vehicle trajectory of the corresponding vehicle can be accurately determined based on the license plate image position information of the same vehicle at different times, and then a straight line trajectory is fitted to the vehicle trajectory, so as to accurately fit the straight line trajectory of the vehicle, thereby making the obtained straight line trajectory more accurate, which is beneficial to improving the accuracy of the vanishing point calculated based on the straight line trajectory.

[0013] In a possible implementation, the calculating the first vanishing point, the second vanishing point, and the third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories includes: Constructing a license plate information set including a plurality of license plate image position information; and constructing a vehicle straight line trajectory set including a plurality of straight line trajectories; The first vanishing point, the second vanishing point, and the third vanishing point are calculated based on the license plate information set and the vehicle straight track set.

[0014] Through the above-mentioned application embodiment, a license plate information set including multiple license plate image position information and a vehicle straight line trajectory set including multiple straight line trajectories are first constructed, so that the multiple license plate image position information is converted into a set form for subsequent calculations, and the multiple straight line trajectories are converted into a set form for subsequent calculations, so as to more quickly and accurately determine the required license plate image position information and straight line trajectory set, and avoid repeated calculations.

[0015] In a possible implementation, the license plate image position information includes a first image coordinate point and a second image coordinate point, and the calculating of the first vanishing point, the second vanishing point, and the third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories includes: Based on the first image coordinate point and the second image coordinate point corresponding to each license plate, calculate a first intersection point between an upper sideline and a lower sideline, and a second intersection point between a left sideline and a right sideline of each license plate; and For each of the straight line trajectories of each vehicle, calculating a third intersection point between every two of the straight line trajectories; The first vanishing point, the second vanishing point, and the third vanishing point are determined based on the plurality of the first intersection points, the plurality of the second intersection points, and the plurality of the third intersection points.

[0016] Through the above-mentioned application embodiments, the determined first intersection point, second intersection point, and third intersection point are made more accurate, thereby facilitating improving the accuracy of the first vanishing point, second vanishing point, and third vanishing point determined based on the first intersection point, second intersection point, and third intersection point.

[0017] In a possible implementation, the determining the first vanishing point, the second vanishing point, and the third vanishing point based on the plurality of first intersection points, the plurality of second intersection points, and the plurality of third intersection points includes: The first intersection points, the second intersection points, and the third intersection points are processed respectively by a clustering algorithm to obtain the first vanishing point, the second vanishing point, and the third vanishing point.

[0018] Through the above-mentioned application embodiment, based on the clustering algorithm, the multiple first intersection points, the multiple second intersection points, and the multiple third intersection points are processed separately, and the abnormal interference points in the multiple first intersection points, the multiple second intersection points, and the multiple third intersection points are removed, so as to obtain the most accurate first vanishing point, second vanishing point, and third vanishing point, which is beneficial to improve the accuracy of automatic calibration of the equipment based on the first vanishing point, the second vanishing point, and the third vanishing point.

[0019] In a possible implementation, the automatically calibrating the device in combination with the first vanishing point, the second vanishing point, and the third vanishing point includes: Obtaining standard size information of a license plate; wherein the standard size information of a license plate is the difference between a length value and a width value of the license plate; The first vanishing point, the second vanishing point, the third vanishing point and the standard size information of the license plate are processed by a vanishing point calibration algorithm to determine a calibration matrix of the device.

[0020] Through the above application embodiments, since the standard license plate information is the difference between the length value and the width value of the license plate, the standard license plate size information is known information. Combined with the accurately determined first vanishing point, second vanishing point, and third vanishing point, the vanishing point calibration algorithm can be used to realize automatic calibration of the device and make the calibration result more accurate.

[0021] In a second aspect, the present application further provides a device calibration apparatus, the apparatus comprising: An acquisition module, used to acquire multiple image information perceived by the device; A determination module, used to determine the position information of a plurality of license plate images and the linear tracks of a plurality of vehicles based on the plurality of image information; A calculation module, used for calculating a first vanishing point, a second vanishing point and a third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories; A processing module is used to automatically calibrate the device based on the first vanishing point, the second vanishing point, and the third vanishing point.

[0022] In a possible implementation, the determination module is specifically used to determine a target license plate detection model; and the plurality of image information are processed respectively by the target license plate detection model to obtain corresponding plurality of license plate image position information.

[0023] In a possible implementation, the determination module is further used to determine the vehicle trajectory of the corresponding vehicle based on the license plate image position information of the same vehicle at different times; and perform straight line trajectory fitting on the vehicle trajectory to obtain the straight line trajectory corresponding to the corresponding vehicle.

[0024] In a possible implementation, the calculation module is specifically used to construct a license plate information set including multiple license plate image position information; and to construct a vehicle straight line trajectory set including multiple straight line trajectories; based on the license plate information set and the vehicle straight line trajectory set, calculate the first vanishing point, the second vanishing point and the third vanishing point.

[0025] In a possible implementation, the license plate image position information includes a first image coordinate point and a second image coordinate point, and the calculation module is further used to calculate the first intersection point of the upper edge line and the lower edge line of each license plate, and the second intersection point of the left edge line and the right edge line based on the first image coordinate point and the second image coordinate point corresponding to each license plate; and for the straight line trajectory of each vehicle, calculate the third intersection point between every two straight line trajectories; based on multiple first intersection points, multiple second intersection points, and multiple third intersection points, determine the first vanishing point, the second vanishing point, and the third vanishing point.

[0026] In a possible implementation, the calculation module is further configured to process the plurality of first intersection points, the plurality of second intersection points, and the plurality of third intersection points respectively by using a clustering algorithm to obtain the first vanishing point, the second vanishing point, and the third vanishing point.

[0027] In a possible implementation, the processing module is specifically used to obtain standard license plate size information; wherein the standard license plate size information is the difference between a length value and a width value of the license plate; the first vanishing point, the second vanishing point, the third vanishing point and the standard license plate size information are processed by a vanishing point calibration algorithm to determine a calibration matrix of the device.

[0028] In a third aspect, the present application provides an electronic device, including: Memory, used to store computer programs; The processor is used to implement the above-mentioned device calibration method steps when executing the computer program stored in the memory.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the calibration method of a device described above are implemented.

[0030] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a side view traffic scene provided in an embodiment of the present application; Figure 2 A flow chart of a device calibration method provided in an embodiment of the present application; Figure 3 A schematic diagram of the processing process of the calibration method of the device provided in the embodiment of the present application; Figure 4 A schematic diagram of a calibration device for a device provided in an embodiment of the present application; Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0033] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0034] Existing automatic device calibration methods (such as camera automatic calibration methods) often require a large amount of data to train and optimize deep learning models, and the data calibration work is also relatively cumbersome, which makes the automatic calibration of equipment not universal.

[0035] Therefore, the present application proposes a device calibration method, which determines multiple license plate image position information and multiple vehicle straight line trajectories based on multiple image information perceived by the device, and then calculates the first vanishing point, the second vanishing point and the third vanishing point based on the multiple license plate image position information and the multiple straight line trajectories, and then calibrates the device by combining the first vanishing point, the second vanishing point and the third vanishing point. In this way, only the image information perceived by the device is needed to determine the vanishing point, and then the device can be automatically calibrated by combining the vanishing point, without the need to use a large amount of data to train and optimize the deep learning model, thereby avoiding tedious data calibration work, and reducing the training and optimization time of the deep learning model, making the automatic calibration of the device more universal.

[0036] The above device may be a camera, so that the camera can be automatically calibrated based on the multiple image information sensed by the camera.

[0037] The above vanishing points (such as the first vanishing point, the second vanishing point, and the third vanishing point) represent intersection points where two parallel straight lines intersect after perspective transformation, which can be understood as intersection points of two parallel lines that are parallel to each other in the real world but intersect in the image.

[0038] Furthermore, the first vanishing point, the second vanishing point, and the third vanishing point correspond to three axes in the world coordinate system.

[0039] For example, the first vanishing point corresponds to the X-axis in the world coordinate system. That is, on the image plane, the intersection of two straight lines parallel to the X-axis in the world coordinate system after perspective transformation reflects the perspective information of the object in the scene in the X-axis direction. In other words, the first vanishing point corresponds to the axis corresponding to the parallel direction of the two straight lines corresponding to the first vanishing point in the world coordinate system. The first vanishing point can be expressed as: .

[0040] The second vanishing point corresponds to the Z axis in the world coordinate system. That is, on the image plane, the intersection of two straight lines parallel to the Z axis in the world coordinate system after perspective transformation reflects the perspective information of the object in the scene in the Z axis direction. In other words, the second vanishing point corresponds to the axis corresponding to the parallel direction of the two straight lines corresponding to the second vanishing point in the world coordinate system. The second vanishing point can be expressed as: .

[0041] The third vanishing point corresponds to the Y axis in the world coordinate system. That is, on the image plane, the intersection of two straight lines parallel to the Y axis in the world coordinate system after perspective transformation reflects the perspective information of the object in the scene in the Y axis direction. In other words, the third vanishing point corresponds to the axis corresponding to the parallel direction of the two straight lines corresponding to the third vanishing point in the world coordinate system. The third vanishing point can be expressed as: .

[0042] Since a large number of vehicles (i.e., vehicle targets) in traffic scenes have their own license plates, the license plates are traffic scene-specific information with standard size information. Figure 1 In the typical side view traffic scene shown, the vehicle has its own license plate. Therefore, the automatic calibration of the device can be achieved through the relevant information of the license plate.

[0043] The top, bottom, left, and right sides of the license plate form two sets of parallel lines that are parallel to each other, and these two sets of parallel lines are orthogonal and parallel to the X and Z axes of the world coordinate system. Therefore, based on the parking space image position information of the license plate, the first vanishing point corresponding to the X axis in the world coordinate system and the second vanishing point corresponding to the Y axis in the world coordinate system can be calculated.

[0044] In addition, in traffic scenes, there are a large number of vehicles running in a near-straight line. It can be considered that the movement direction of the vehicle is parallel to the Y-axis of the world coordinate system. Therefore, the third vanishing point of the Y-axis in the corresponding world coordinate system can be calculated based on the straight-line trajectory of the vehicle.

[0045] Reference Figure 2 The figure is a flow chart of a calibration method for a device provided in an embodiment of the present application, the method comprising: S201, obtaining multiple image information perceived by the device.

[0046] The image information may be image information of a traffic scene. That is, based on the traffic scene, the image information perceived by the device is obtained. The image information may be image information within a time period T (T is a positive number), so that the license plate image position information within the time period T is subsequently collected based on the image information.

[0047] S202, based on the multiple image information, determine the multiple license plate image position information, and determine the linear tracks of the multiple vehicles.

[0048] Optionally, the specific process of determining the multiple license plate image information based on the multiple image information may be as follows: First, the target license plate detection model is obtained, and then each image information is processed by the target license plate detection model to obtain one or more license plate image position information in each image information, so that the determined license plate image position information is more accurate, which helps to improve the accuracy of device calibration based on license plate image position information. Therefore, the target license plate detection model can accurately detect the license plate information (i.e., license plate image position information) in the traffic scene in real time, that is, the license plate information (i.e., license plate image position information) in the image information perceived by the device can be accurately detected in real time.

[0049] In an embodiment of the present application, the above-mentioned target license plate detection model can be a trained license plate detection model, such as a region-based convolutional neural network model (English: Region-based Convolutional Neural Networks, abbreviated as R-CNN), a single shot multi-box detector model (English: Single Shot MultiBox Detector, abbreviated as SSD), etc., thereby avoiding the training of the license plate detection model, further reducing the calibration time of the device automatic calibration, so that the efficiency of the device automatic calibration is improved, thereby further improving the versatility of the device automatic calibration.

[0050] The above-mentioned multiple license plate image position information can be license plate image position information of different vehicles at different times, license plate image position information of the same vehicle at different times, and license plate image position information of different vehicles at the same time. In addition, since the multiple image information obtained is image information within the time period T, the collection of license plate information at different times and locations within the time period T is realized.

[0051] The license plate image position information includes at least the first image coordinate point and the second image coordinate point of the license plate, and may also include the license plate number, the vehicle identification code (English: Identity document, abbreviated as ID), time and other information.

[0052] The above first image coordinate point can be expressed as: ; The above second image coordinate point can be expressed as: . Where V represents the image coordinate system.

[0053] The first image coordinate point can be the upper left image coordinate point of the license plate, that is, the intersection of the left line and the upper line of the license plate; the second image coordinate point can be the lower right image coordinate point of the license plate, that is, the intersection of the right line and the lower line of the license plate. Thus, the third image coordinate point and the fourth image coordinate point of the corresponding license plate can be determined based on the first image coordinate point and the second image coordinate point. In this case, the third image coordinate point is the upper right image coordinate point of the license plate, that is, the intersection of the right line and the upper line of the license plate. The fourth image coordinate point is the lower left image coordinate point of the license plate, that is, the intersection of the left line and the lower line of the license plate.

[0054] Alternatively, the first image coordinate point may be the lower left image coordinate point of the license plate, that is, the intersection of the left line and the lower line of the license plate. The second image coordinate point may be the upper right image coordinate point of the license plate, that is, the intersection of the right line and the upper line of the license plate. Thus, according to the first image coordinate point and the second image coordinate point, the third image coordinate point and the fourth image coordinate point of the corresponding license plate may be determined. In this case, the third image coordinate point is the lower right image coordinate point of the license plate, that is, the intersection of the right line and the lower line of the license plate. The fourth image coordinate point is the upper left image coordinate point of the license plate, that is, the intersection of the left line and the upper line of the license plate.

[0055] Assume that the position information of the first image coordinate point corresponding to the world coordinate system is , where W represents the world coordinate system, and the position information of the second image coordinate point corresponding to the world coordinate system is , then D represents a known standard size information of a license plate, and D can be the difference between the length and width of the license plate. Therefore, the conversion relationship between the image coordinate point of the license plate and the coordinate point in the world coordinate system can be determined by the known D and the first image coordinate point and the second image coordinate point, so as to determine the calibration matrix of the device.

[0056] Optionally, in an embodiment of the present application, after obtaining the position information of multiple license plate images, the Zhang Zhengyou calibration method can also be combined to realize automatic calibration of the device.

[0057] Specifically, the license plate information of known size (i.e., the standard license plate size information) is used as a calibration plate unique to the traffic scene, and combined with the position information of multiple license plate images at different positions, the Zhang Zhengyou calibration method is adopted to realize automatic calibration of the equipment, thereby making full use of the environmental information of the traffic scene, realizing automatic calibration of the equipment, and improving the flexibility of equipment calibration.

[0058] Optionally, the specific process of determining the straight-line trajectories of multiple vehicles based on multiple image information may be as follows: First, based on a plurality of image information, vehicle trajectories of a plurality of vehicles are determined.

[0059] In a possible implementation, the license plate position information is used as a specific embodiment of the vehicle trajectory information. Then the vehicle trajectory can be determined based on the license plate image position information. That is, after obtaining multiple license plate image position information based on multiple image information, the vehicle trajectory of the corresponding vehicle is determined based on the license plate image position information of the same license plate at different times.

[0060] In another possible implementation, the above vehicle trajectory may be determined based on the location information of the vehicle, and the determination process may be as follows: First, the target vehicle detection model is obtained, and then the target vehicle detection model is used to process multiple image information respectively to obtain one or more vehicle information in each image information, so that the determined vehicle information is more accurate. Therefore, the target vehicle detection model can accurately detect vehicle information in traffic scenes in real time, that is, accurately detect vehicle information in image information perceived by the device in real time.

[0061] Based on the position information of the vehicle in the vehicle information, the vehicle track of the corresponding vehicle is determined. Specifically, based on the position information of the same vehicle at different times, the vehicle track of the corresponding vehicle is determined.

[0062] In the embodiment of the present application, the above vehicle information may include not only the vehicle location information, but also the vehicle ID, time and other information, so as to determine the location information of the same vehicle at different times according to the ID. The above vehicle location information may be represented by the coordinate points of the detected vehicle identification frame.

[0063] The target vehicle detection model can be a trained vehicle detection model, such as R-CNN, a vehicle detection model based on an attention mechanism, etc., thereby avoiding the training of the vehicle detection model, further reducing the calibration time of the device automatic calibration, and improving the efficiency of the device automatic calibration, thereby further improving the versatility of the device automatic calibration.

[0064] Furthermore, after the vehicle trajectories of the plurality of vehicles are determined, a straight line trajectory fitting is performed for the vehicle trajectories to obtain a straight line trajectory corresponding to each vehicle, so that the determined straight line trajectory is more accurate.

[0065] In an embodiment of the present application, a random sampling consensus algorithm (RANSAC) may be used to perform straight-line trajectory fitting on the vehicle trajectory to obtain a straight-line trajectory of the corresponding vehicle, but this is not limited to this, and a specific straight-line trajectory fitting method may be selected according to a specific application scenario.

[0066] Optionally, after determining the vehicle trajectories of multiple vehicles, a vehicle trajectory set can be constructed based on the multiple vehicle trajectories, so as to better save the vehicle trajectories of multiple vehicles and facilitate the subsequent better fitting of the straight line trajectory of the corresponding vehicle based on the vehicle trajectory. Then, the above-mentioned straight line trajectory fitting is performed on the vehicle trajectory to obtain the straight line trajectory corresponding to each vehicle, which can be: a straight line trajectory fitting is performed on the multiple vehicle trajectories in the vehicle trajectory set to obtain the vehicle trajectory corresponding to each vehicle.

[0067] When the vehicle trajectory is determined based on the license plate image position information, the constructed vehicle trajectory set includes vehicle trajectories consisting of the license plate image position information.

[0068] When the vehicle trajectory is determined based on the vehicle position information, the constructed vehicle trajectory set includes vehicle trajectories composed of the vehicle position information. The vehicle trajectory set can be expressed as: V ={Pt t}. Among them, P v represents the vehicle trajectory set; t represents time, Pt t Represents the position information of the vehicle at the tth moment, that is, the coordinate point at the tth moment.

[0069] S203, calculating a first vanishing point, a second vanishing point, and a third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories.

[0070] After obtaining the plurality of license plate image position information and the plurality of straight line tracks, a first vanishing point, a second vanishing point, and a third vanishing point are calculated based on the plurality of license plate image position information and the plurality of straight line tracks.

[0071] Specifically, first, based on the first image coordinate point and the second image coordinate point in the position information of each license plate image, the first intersection point of the upper edge line and the lower edge line of the corresponding license plate, and the second intersection point of the left edge line and the right edge line are calculated; the first intersection point is used as the first vanishing point to be determined, and the second intersection point is used as the second vanishing point to be determined, so that the determined first vanishing point to be determined and the second vanishing point to be determined are more accurate.

[0072] The specific calculation process of calculating the first intersection point of the upper side line and the lower side line of the corresponding license plate and the second intersection point of the left side line and the right side line based on the first image coordinate point and the second image coordinate point in the license plate image position information can be shown as follows: Taking the first image coordinate point as the upper left image coordinate point of the license plate, that is, the intersection of the left line and the upper line of the license plate; the second image coordinate point as the lower right image coordinate point of the license plate, that is, the intersection of the right line and the lower line of the license plate as an example, first determine the third image coordinate point and the fourth image coordinate point of the license plate based on the first image coordinate point and the second image coordinate point in the license plate image position information.

[0073] Then, based on the above-mentioned first image coordinate points and the fourth image coordinate points, the image straight line of the left line of the license plate is fitted; based on the above-mentioned second image coordinate points and the third image coordinate points, the image straight line of the right line of the license plate is fitted; based on the above-mentioned first image coordinate points and the third image coordinate points, the image straight line of the upper line of the license plate is fitted; based on the above-mentioned second image coordinate points and the fourth image coordinate points, the image straight line of the lower line of the license plate is fitted.

[0074] Then, based on the image straight line of the upper edge line and the image straight line of the lower edge line of the license plate, the first intersection point of the upper edge line and the lower edge line of the license plate is calculated; based on the image straight line of the left edge line and the image straight line of the right edge line of the license plate, the second intersection point of the left edge line and the right edge line of the license plate is calculated.

[0075] In order to better save and use multiple license plate image position information, a license plate information set including multiple license plate image position information can be constructed based on the obtained multiple license plate image position information. Then, based on the multiple license plate image position information in the license plate information set, the first intersection point and the second intersection point are calculated.

[0076] In the embodiment of the present application, the above license plate information set may include license plate image position information of the same vehicle at different times, license plate image position information of different vehicles at the same time, and license plate position information of different vehicles at different times. Moreover, each license plate image position information in the license plate information set includes a first image coordinate point and a second image coordinate point.

[0077] The above license plate information set can be as follows:

[0078] Among them, P represents the license plate information set; L p Indicates the license plate location information.

[0079] Thus, through the above-mentioned license plate information set, the license plate information at different positions at different times is collected, that is, the position information of multiple license plate images is collected, so as to better preserve the position information of multiple license plate images, and when calculating the vanishing point based on the license plate image position information, the required license plate image position information can be obtained more quickly and accurately.

[0080] Furthermore, after obtaining the position information of multiple license plate images, combined with Zhang Zhengyou's calibration method, the automatic calibration of the device can be achieved: the license plate information of known size (i.e., the standard license plate size information) is used as a calibration plate unique to the traffic scene, and then combined with the license plate information set, Zhang Zhengyou's calibration method is used to achieve automatic calibration of the device.

[0081] Next, based on the multiple vehicle trajectories, the third intersection point of the straight line trajectories between two vehicles is calculated, that is, the third intersection point between every two straight line trajectories is calculated, and the third intersection point is used as the third vanishing point to be determined, so that the calculated third vanishing point to be determined is more accurate.

[0082] The third intersection point between the above two straight line trajectories can be calculated by using the method of finding the intersection point of two straight lines.

[0083] In order to better save and use the multiple straight line tracks, a vehicle straight line track set including the multiple straight line tracks may be constructed based on the obtained multiple straight line tracks, and then the third intersection point may be calculated based on the multiple straight line tracks in the vehicle straight line track set.

[0084] The above vehicle straight line trajectory set can be shown as follows:

[0085] Among them, L V represents the set of vehicle straight-line trajectories, l V Represents a straight line trajectory.

[0086] Therefore, the vehicle straight-line trajectory set is used to better save the vehicle straight-line trajectory, and when calculating the third intersection point, the required straight-line trajectory can be retrieved more quickly and accurately.

[0087] In an embodiment of the present application, the above-mentioned calculation process of calculating the first intersection and the second intersection and the calculation process of calculating the third intersection can be executed simultaneously; the first intersection and the second intersection can be calculated first, and then the third intersection can be calculated; the third intersection can also be calculated first, and then the first intersection and the second intersection can be calculated.

[0088] Further, after calculating the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined, the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined are processed respectively by a clustering algorithm to remove abnormal vanishing points among the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined, so that the most accurate first vanishing point to be determined can be determined from the plurality of first vanishing points to be determined, the most accurate second vanishing point to be determined can be determined from the plurality of second vanishing points to be determined, and the most accurate third vanishing point to be determined can be determined from the plurality of third vanishing points to be determined.

[0089] Then, the most accurate first vanishing point to be determined is used as the first vanishing point, the most accurate second vanishing point to be determined is used as the second vanishing point, and the most accurate third vanishing point to be determined is used as the third vanishing point, so that the obtained first vanishing point, second vanishing point, and third vanishing point are more accurate.

[0090] In order to better preserve and use the multiple first vanishing points to be determined, the multiple second vanishing points to be determined, and the multiple third vanishing points to be determined, a first vanishing point set to be determined including the multiple first vanishing points to be determined can be constructed based on the multiple first vanishing points to be determined, a second vanishing point set to be determined including the multiple second vanishing points to be determined can be constructed based on the multiple second vanishing points to be determined, and a third vanishing point set to be determined including the multiple third vanishing points to be determined can be constructed based on the multiple third vanishing points to be determined. The first vanishing point set to be determined can be expressed as: ; The second vanishing point set to be determined can be expressed as: ; The third set of vanishing points to be determined can be expressed as: .

[0091] Then, the clustering algorithm is used to process the first vanishing point set to be determined, the second vanishing point set to be determined, and the third vanishing point set to be determined, so as to eliminate the influence of abnormal interference points, and obtain the most accurate first vanishing point to be determined in the first vanishing point set to be determined, the most accurate second vanishing point to be determined in the second vanishing point set to be determined, and the most accurate third vanishing point to be determined in the third vanishing point set to be determined. The most accurate first vanishing point in the first vanishing point set to be determined is used as the first vanishing point, the most accurate second vanishing point in the second vanishing point set to be determined is used as the second vanishing point, and the most accurate third vanishing point in the third vanishing point set to be determined is used as the third vanishing point.

[0092] Through the above manner, by adopting the first vanishing point set to be determined, the second vanishing point set to be determined, and the third vanishing point set to be determined, a plurality of first vanishing points to be determined, a plurality of second vanishing points to be determined, and a plurality of third vanishing points to be determined are better preserved, and the clustering algorithm can better perform clustering processing on the plurality of first vanishing points to be determined, the plurality of second vanishing points to be determined, and the plurality of third vanishing points to be determined, thereby further increasing the accuracy of the determined first vanishing points, the second vanishing points, and the third vanishing points.

[0093] S204, automatically calibrating the device based on the first vanishing point, the second vanishing point, and the third vanishing point.

[0094] Since the vanishing point calibration algorithm used to achieve automatic calibration of the device requires three vanishing points in the scene (such as a traffic scene) and a known priori size, and the three vanishing points correspond to the three axes in the world coordinate system, the internal and external parameters of the device can be accurately obtained, that is, automatic calibration of the device can be achieved.

[0095] Therefore, after obtaining the first vanishing point, the second vanishing point, and the third vanishing point, which correspond to the three axes in the world coordinate system, the standard size information of the license plate (ie, the aforementioned D) is first obtained.

[0096] Then, in combination with the standard size information of the license plate, and the first vanishing point, the second vanishing point and the third vanishing point determined based on the first image coordinate point and the second image coordinate point of the license plate, a vanishing point calibration algorithm is adopted to solve the device calibration matrix, thereby calibrating the device and obtaining the device calibration matrix.

[0097] In summary, the device calibration method proposed in this application determines three vanishing points (i.e., the first vanishing point, the second vanishing point, and the third vanishing point) based on the image information perceived by the device (such as a camera), and then combines the three vanishing points with the known standard license plate size information to achieve online automatic calibration of the device (such as a camera).

[0098] In this way, the environmental information in the traffic scene is fully utilized. There is no need for prior configuration information, no high requirements for scene information and carriers, no need for human intervention, and no need for iterative training of deep learning methods. Automatic calibration of equipment (such as cameras) can be achieved, making automatic calibration of equipment (such as cameras) simpler, faster, and more versatile, and improving the accuracy of the calibration results (i.e. calibration accuracy), with excellent usability and feasibility.

[0099] The technical solution of this application is further explained below in conjunction with a specific application process.

[0100] like Figure 3The figure shows a schematic diagram of the processing process of the device calibration method. Taking a camera as an example, first, in the data acquisition module, based on the traffic scene, multiple image information perceived by the camera is obtained. The multiple image information is transmitted to the detection module so that the detection module can detect the image information.

[0101] In the detection module, a target vehicle detection model and a target license plate detection model are obtained. The target vehicle detection model and the target license plate detection model process the image information respectively, and detect the vehicle information and the license plate image position information in the traffic scene in real time. The license plate image position information is transmitted to the first set construction module, so that the first set construction module constructs the license plate information set based on the license plate image position information. And the vehicle information is transmitted to the second set construction module, so that the second set construction module constructs the vehicle straight track set based on the vehicle information.

[0102] In the first set construction module, the license plate image position information transmitted by the detection module is received, and then a license plate information set is constructed based on the license plate image position information, and the license plate information set is transmitted to the vanishing point calculation module, so that the vanishing point calculation module calculates the vanishing point based on the license plate information set.

[0103] In the second set construction module, the vehicle information transmitted by the detection module is received. Then, based on the position information of the vehicle in the vehicle information, a vehicle trajectory set is constructed. Then, RANSAC is used to fit the position information of the same vehicle in the vehicle trajectory set with a straight line trajectory, and a vehicle straight line trajectory set is constructed based on the fitted straight line trajectory. The vehicle straight line trajectory set is then transmitted to the vanishing point calculation module, so that the vanishing point calculation module can calculate the vanishing point based on the vehicle straight line trajectory set.

[0104] In the vanishing point calculation module, based on the first image coordinate point and the second image coordinate point corresponding to each license plate in the license plate information set, the first intersection point of the upper sideline and the lower sideline and the second intersection point of the left sideline and the right sideline of the license plate are calculated, and the first intersection point is used as the first vanishing point to be determined, and the second intersection point is used as the second vanishing point to be determined. Then, based on the multiple first vanishing points to be determined, a first vanishing point set to be determined is constructed, and based on the multiple second vanishing points to be determined, a second vanishing point set to be determined is constructed.

[0105] In addition, in the vehicle straight track set, the third intersection point of the straight track between two vehicles is calculated, and the third intersection point is used as the third vanishing point to be determined. Then, based on the multiple third vanishing points to be determined, the third vanishing point set to be determined is constructed. Then, the first vanishing point set to be determined, the second vanishing point set to be determined, and the third vanishing point set to be determined are processed by a clustering algorithm to obtain the most accurate first vanishing point to be determined (i.e., the first vanishing point), the most accurate second vanishing point to be determined (i.e., the second vanishing point), and the most accurate third vanishing point to be determined (i.e., the third vanishing point). Then, the first vanishing point, the second vanishing point, and the third vanishing point are transmitted to the calibration module so that the calibration module can calibrate the camera in combination with the first vanishing point, the second vanishing point, and the third vanishing point.

[0106] In the calibration module, the standard size information of the license plate is obtained, and then the camera is automatically calibrated by using the vanishing point calibration algorithm in combination with the standard size information of the license plate and the received first vanishing point, second vanishing point, and third vanishing point.

[0107] Based on the same inventive concept, the present application also provides a device calibration device, such as Figure 4 The figure shows a schematic diagram of a calibration device for a device provided by the present application, the device comprising: An acquisition module 401 is used to acquire multiple image information perceived by the device; A determination module 402, for determining a plurality of license plate image position information and a plurality of straight track trajectories of vehicles based on the plurality of image information; A calculation module 403, configured to calculate a first vanishing point, a second vanishing point, and a third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories; The processing module 404 is used to automatically calibrate the device based on the first vanishing point, the second vanishing point, and the third vanishing point.

[0108] In a possible implementation, the determination module 402 is specifically used to determine a target license plate detection model; and to process the plurality of image information respectively through the target license plate detection model to obtain corresponding plurality of license plate image position information.

[0109] In a possible implementation, the determination module 402 is further configured to determine a vehicle trajectory of a corresponding vehicle based on the license plate image position information of the same vehicle at different times; and perform straight line trajectory fitting on the vehicle trajectory to obtain the straight line trajectory corresponding to the corresponding vehicle.

[0110] In a possible implementation, the calculation module 403 is specifically used to construct a license plate information set including a plurality of license plate image position information; and to construct a vehicle straight line trajectory set including a plurality of straight line trajectories; and to calculate the first vanishing point, the second vanishing point, and the third vanishing point based on the license plate information set and the vehicle straight line trajectory set.

[0111] In a possible implementation, the license plate image position information includes a first image coordinate point and a second image coordinate point, and the calculation module 403 is further used to calculate the first intersection point of the upper edge line and the lower edge line of each license plate, and the second intersection point of the left edge line and the right edge line based on the first image coordinate point and the second image coordinate point corresponding to each license plate; and for the straight line trajectory of each vehicle, calculate the third intersection point between every two straight line trajectories; based on multiple first intersection points, multiple second intersection points, and multiple third intersection points, determine the first vanishing point, the second vanishing point, and the third vanishing point.

[0112] In a possible implementation, the calculation module 403 is further configured to process the plurality of first intersection points, the plurality of second intersection points, and the plurality of third intersection points respectively by using a clustering algorithm to obtain the first vanishing point, the second vanishing point, and the third vanishing point.

[0113] In a possible implementation, the processing module 404 is specifically used to obtain standard license plate size information; wherein the standard license plate size information is the difference between the length value and the width value of the license plate; the first vanishing point, the second vanishing point, the third vanishing point and the standard license plate size information are processed by a vanishing point calibration algorithm to determine a calibration matrix of the device.

[0114] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The electronic device can realize the function of the calibration device of the aforementioned device. Figure 5 , the electronic equipment includes: At least one processor 501, and a memory 502 connected to the at least one processor 501. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 In the example, the processor 501 and the memory 502 are connected via a bus 500. The bus 500 is Figure 5 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 500 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 501 can also be called a controller, and there is no limitation on the name.

[0115] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the calibration method of the device discussed above by executing the instructions stored in the memory 502. The processor 501 can implement Figure 4 The functions of each module in the device shown.

[0116] Among them, the processor 501 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 502 and calling data stored in the memory 502, the various functions of the device and processing data, the device can be monitored as a whole.

[0117] In one possible design, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0118] The processor 501 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the calibration method of the device disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0119] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (English: Random Access Memory, abbreviated as RAM), static random access memory (English: Static Random Access Memory, abbreviated as SRAM), programmable read-only memory (English: Programmable Read Only Memory, abbreviated as PROM), read-only memory (English: Read Only Memory, abbreviated as ROM), electrically erasable programmable read-only memory (English: Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), magnetic memory, disk, optical disk, etc. The memory 502 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, which is used to store program instructions and / or data.

[0120] By programming the processor 501, the code corresponding to the calibration method of the device described in the above embodiment can be fixed into the chip, so that the chip can execute the calibration method when running. Figure 2 The steps of the device calibration method of the embodiment shown are as follows: How to design and program the processor 501 is a technology well known to those skilled in the art and will not be described in detail here.

[0121] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the calibration method of the device discussed above.

[0122] In some possible implementations, various aspects of the device calibration method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a device, the program code is used to enable the control device to execute the steps of the device calibration method according to various exemplary embodiments of the present application described above in this specification.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for calibrating a device, characterized in that: include: Acquire multiple image information perceived by the device; Based on the plurality of image information, determining the position information of the plurality of license plate images, and determining the linear trajectories of the plurality of vehicles; Calculating a first vanishing point, a second vanishing point, and a third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories; The device is automatically calibrated in combination with the first vanishing point, the second vanishing point, and the third vanishing point.

2. The method according to claim 1, characterized in that The determining of a plurality of license plate image position information based on the plurality of image information comprises: Determine the target license plate detection model; The target license plate detection model is used to process the plurality of image information respectively to obtain the corresponding plurality of license plate image position information.

3. The method according to claim 1, characterized in that The determining of the straight-line trajectories of the plurality of vehicles comprises: Determining a vehicle trajectory of a corresponding vehicle based on the license plate image position information of the same vehicle at different times; A straight line trajectory is fitted for the vehicle trajectory to obtain the straight line trajectory corresponding to the corresponding vehicle.

4. The method according to claim 1, characterized in that The calculating of the first vanishing point, the second vanishing point and the third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories comprises: Constructing a license plate information set including a plurality of license plate image position information; and constructing a vehicle straight line trajectory set including a plurality of straight line trajectories; The first vanishing point, the second vanishing point, and the third vanishing point are calculated based on the license plate information set and the vehicle straight track set.

5. The method according to claim 1 or 4, characterized in that The license plate image position information includes a first image coordinate point and a second image coordinate point, and the first vanishing point, the second vanishing point and the third vanishing point are calculated based on the plurality of license plate image position information and the plurality of straight line trajectories, including: Based on the first image coordinate point and the second image coordinate point corresponding to each license plate, calculate a first intersection point between an upper sideline and a lower sideline, and a second intersection point between a left sideline and a right sideline of each license plate; and For each of the straight line trajectories of each vehicle, calculating a third intersection point between every two of the straight line trajectories; The first vanishing point, the second vanishing point, and the third vanishing point are determined based on the plurality of the first intersection points, the plurality of the second intersection points, and the plurality of the third intersection points.

6. The method according to claim 1, characterized in that The determining the first vanishing point, the second vanishing point, and the third vanishing point based on the plurality of the first intersection points, the plurality of the second intersection points, and the plurality of the third intersection points comprises: The first intersection points, the second intersection points, and the third intersection points are processed respectively by a clustering algorithm to obtain the first vanishing point, the second vanishing point, and the third vanishing point.

7. The method according to claim 1, characterized in that The automatically calibrating the device by combining the first vanishing point, the second vanishing point, and the third vanishing point includes: Obtaining standard size information of a license plate; wherein the standard size information of a license plate is the difference between a length value and a width value of the license plate; The first vanishing point, the second vanishing point, the third vanishing point and the standard size information of the license plate are processed by a vanishing point calibration algorithm to determine a calibration matrix of the device.

8. A device for calibrating equipment, characterized in that: include: An acquisition module, used to acquire multiple image information perceived by the device; A determination module, used to determine the position information of a plurality of license plate images and the linear tracks of a plurality of vehicles based on the plurality of image information; A calculation module, used for calculating a first vanishing point, a second vanishing point and a third vanishing point based on the plurality of license plate image position information and the plurality of straight line trajectories; A processing module is used to automatically calibrate the device based on the first vanishing point, the second vanishing point, and the third vanishing point.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images