Image acquisition device calibration method and device, electronic device and storage medium

Abstract

This disclosure relates to a calibration method and apparatus for an image acquisition device, an electronic device, and a storage medium. The method includes: performing vehicle detection on video frames acquired by the image acquisition device to determine a target vehicle image; performing wheel feature point detection on the target vehicle image to determine wheel feature points included in the target vehicle image; and calibrating the image acquisition device based on the wheel feature points included in the target vehicle image to determine the device parameters of the image acquisition device.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a calibration method and device for an image acquisition device, an electronic device, and a storage medium. Background Art

[0002] Many applications in the field of video surveillance require calibration of image acquisition devices. After calibrating the device parameters of the image acquisition device, the mapping relationship between the image coordinate system corresponding to the image acquisition device and the world coordinate system can be obtained, which can then be used to determine the physical parameters such as the real size, displacement and speed of the imaging target. The calibration of image acquisition devices can be applied to traffic application scenarios, such as determining the length, speed and distance of vehicles on the road. The calibration of image acquisition devices in related technologies mostly relies on scene reference objects, such as preset calibration plates or on-site reference objects, resulting in low efficiency in calibrating image acquisition devices. Summary of the invention

[0003] The present disclosure proposes a calibration method and device for an image acquisition device, an electronic device, and a technical solution for a storage medium.

[0004] According to one aspect of the present disclosure, a calibration method for an image acquisition device is provided, comprising: determining a target vehicle image by performing vehicle detection on a video frame acquired by the image acquisition device; determining wheel feature points included in the target vehicle image by performing wheel feature point detection on the target vehicle image; and calibrating the image acquisition device according to the wheel feature points included in the target vehicle image to determine device parameters of the image acquisition device.

[0005] In a possible implementation, performing vehicle detection on a video frame acquired by an image acquisition device to determine a target vehicle image includes: performing vehicle detection on the video frame to determine whether a vehicle image exists in the video frame; if it is determined that the vehicle image exists in the video frame, determining whether the vehicle type corresponding to the vehicle image is the target vehicle type; if it is determined that the vehicle type corresponding to the vehicle image is the target vehicle type, determining the vehicle image as the target vehicle image.

[0006] In a possible implementation, the wheel feature points included in the target vehicle image are determined by performing wheel feature point detection on the target vehicle image, including: downsampling feature extraction on the target vehicle image to obtain at least two levels of first feature maps; upsampling feature extraction on the target first feature map to obtain at least two levels of second feature maps, wherein the target first feature map is a feature map with the smallest size among the at least two levels of first feature maps; fusing the at least two levels of first feature maps and the at least two levels of second feature maps to obtain a target feature map; and performing wheel feature point prediction on the target feature map to determine the first coordinates of the wheel feature points in the image coordinate system, wherein the image coordinate system is the coordinate system corresponding to the image acquisition device.

[0007] In a possible implementation, the image acquisition device is calibrated according to the wheel feature points included in the target vehicle image to determine the device parameters of the image acquisition device, including: determining the first coordinates of the wheel feature points in an image coordinate system, wherein the image coordinate system is a coordinate system corresponding to the image acquisition device; determining the second coordinates of the wheel feature points in a world coordinate system; and the image acquisition device is calibrated according to the first coordinates of the wheel feature points in the image coordinate system and the second coordinates in the world coordinate system to determine the device parameters.

[0008] In a possible implementation, determining the second coordinate of the wheel feature point in the world coordinate system includes: determining the second coordinate of the wheel feature point in the world coordinate system according to a preset vehicle length and vehicle width of the target vehicle type in the world coordinate system.

[0009] In a possible implementation, the image acquisition device is calibrated according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system to determine the device parameters, including: constructing a calibration constraint function according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system; and determining the device parameters by iteratively optimizing the calibration constraint function.

[0010] In a possible implementation, the device parameters include: homography matrix parameters corresponding to the image acquisition device, and device external parameters corresponding to the target vehicle image; constructing a calibration constraint function based on the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system, including: constructing a constraint equation corresponding to the wheel feature point based on the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system, the homography matrix parameters, and the device external parameters; constructing the calibration constraint function based on the constraint equation corresponding to the wheel feature point.

[0011] In a possible implementation, the device parameters are determined by iteratively optimizing the calibration constraint function, including: performing the t+1th external parameter optimization on the calibration constraint function according to the tth homography matrix parameter to obtain the t+1th device external parameter, wherein t is an integer greater than or equal to 0, and the tth homography matrix parameter is obtained after the tth internal parameter optimization of the calibration constraint function; performing the t+1th internal parameter optimization on the calibration constraint function according to the t+1th device external parameter to obtain the t+1th homography matrix parameter; when a preset iteration condition is met, determining the device parameters according to the t+1th homography matrix parameter and the t+1th device external parameter.

[0012] In a possible implementation, the calibration constraint function is optimized for the t+1th time external parameters according to the tth homography matrix parameters to obtain the t+1th device external parameters, including: according to the tth homography matrix parameters, using a singular value decomposition algorithm, the calibration constraint function is optimized for the t+1th time external parameters to obtain the t+1th device external parameters.

[0013] In a possible implementation, the calibration constraint function is optimized for the t+1th time internally based on the t+1th device external parameter to obtain the t+1th homography matrix parameter, including: according to the t+1th device external parameter, using the direct linear transformation DLT algorithm, the calibration constraint function is optimized for the t+1th time internally based on the t+1th device external parameter to obtain the t+1th homography matrix parameter.

[0014] In one possible implementation, the preset iteration condition includes: t+1 reaches a preset iteration number threshold; or, the difference between the t+1th homography matrix parameter and the tth homography matrix parameter is less than a first threshold, and the difference between the t+1th device external parameter and the tth device external parameter is less than a second threshold, wherein the tth device external parameter is obtained after the tth external parameter optimization of the calibration constraint function.

[0015] In a possible implementation manner, the method further includes: when t=0, initializing the 0th homography matrix parameters, initializing the 0th homography matrix parameters to a unit matrix.

[0016] In a possible implementation, the method further includes: acquiring an image of the test object through the image acquisition device to obtain a test image, wherein the size of the test object in the world coordinate system is a preset reference size; determining the image size of the test object in the test image, and converting the image size according to the device parameters to obtain a measured size of the test object in the world coordinate system; determining a percentage error between the reference size and the measured size; and determining the calibration accuracy of the device parameters according to the percentage error.

[0017] In a possible implementation, the method further includes: acquiring an image of the test object through the image acquisition device to obtain a test image, wherein the coordinates of a test target point included in the test object in a world coordinate system are preset reference coordinates; determining the image coordinates of the test target point in the test image, and performing coordinate transformation on the image coordinates according to the device parameters to obtain the measured coordinates of the test target point in the world coordinate system; determining a reprojection mean square error between the reference coordinates and the measured coordinates; and determining the calibration accuracy of the device parameters according to the reprojection mean square error.

[0018] According to one aspect of the present disclosure, there is provided a calibration device for an image acquisition device, comprising: a first determination module, used to determine a target vehicle image by performing vehicle detection on a video frame acquired by the image acquisition device; a second determination module, used to determine wheel feature points included in the target vehicle image by performing wheel feature point detection on the target vehicle image; and a calibration module, used to calibrate the image acquisition device according to the wheel feature points included in the target vehicle image, and determine device parameters of the image acquisition device.

[0019] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0020] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented.

[0021] In the disclosed embodiment, vehicle detection is performed on the video frames acquired by the image acquisition device to determine the target vehicle image, and wheel feature points included in the target vehicle image are determined by performing wheel feature point detection on the target vehicle image. Since the wheel feature points are located in the same plane, there is no need to mark the feature points located in the same plane in the target vehicle image through manual interaction. The image acquisition device can be calibrated quickly based on the wheel feature points, and the device parameters of the image acquisition device can be accurately determined, thereby improving the calibration efficiency of the image acquisition device.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0024] Figure 1 A flow chart showing a calibration method for an image acquisition device according to an embodiment of the present disclosure;

[0025] Figure 2 A schematic diagram showing vehicle type detection according to an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram showing wheel feature point detection according to an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram showing calibration and evaluation of an image acquisition device according to an embodiment of the present disclosure is shown;

[0028] Figure 5 A block diagram showing a calibration device for an image acquisition device according to an embodiment of the present disclosure is shown;

[0029] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0030] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0032] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0033] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0034] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0035] Figure 1 A flow chart of a calibration method for an image acquisition device according to an embodiment of the present disclosure is shown. The calibration method for the image acquisition device can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The calibration method for the image acquisition device can be implemented by a processor calling computer-readable instructions stored in a memory. Alternatively, the calibration method for the image acquisition device can be executed by a server. Figure 1 As shown, the calibration method of the image acquisition device may include:

[0036] In step S11, a target vehicle image is determined by performing vehicle detection on the video frames acquired by the image acquisition device.

[0037] The image acquisition device here can be an electronic device that can realize the image acquisition function, for example, it can be a smart camera, a pan-tilt camera, a surveillance camera, or other image acquisition devices that are fixed in position and require parameter calibration. The present disclosure does not make specific limitations on this.

[0038] The video frame acquired by the image acquisition device to be calibrated is obtained, and the vehicle detection is performed on the video frame, and the target vehicle image is obtained according to the detection result. The video frame here can be a video frame acquired by the image acquisition device, or a video frame sequence composed of multiple video frames, which is not specifically limited in the present disclosure.

[0039] The target vehicle image can be obtained after performing vehicle detection on a video frame or a sequence of video frames. The target vehicle image is an image obtained after the image acquisition device shoots the target vehicle. The target vehicle image obtained after vehicle detection can be one or more, and the present disclosure does not specifically limit this. The vehicle detection process will be described in detail later in conjunction with possible implementations of the present disclosure, and will not be repeated here.

[0040] In step S12, wheel feature points included in the target vehicle image are determined by performing wheel feature point detection on the target vehicle image.

[0041] Since the target vehicle image is an image obtained by an image acquisition device shooting the target vehicle, the wheel feature points included in the target vehicle image can be determined by detecting the wheel feature points of the target vehicle image, that is, the wheel feature points corresponding to the target vehicle in the target vehicle image can be determined. The feature extraction process will be described in detail later in conjunction with possible implementations of the present disclosure, and will not be described here in detail.

[0042] In step S13, the image acquisition device is calibrated according to the wheel feature points included in the target vehicle image to determine the device parameters of the image acquisition device.

[0043] When the target vehicle is traveling on a flat road, the wheel feature points corresponding to the target vehicle in the target vehicle image are located in the same lane plane. Therefore, based on the wheel feature points included in the target vehicle image, the mapping relationship between the two-dimensional lane plane and the two-dimensional image plane of the image acquisition device can be directly determined, without the need to mark the feature points located in the same plane in the target vehicle image through manual interaction, so that the image acquisition device can be calibrated quickly and the device parameters of the image acquisition device can be accurately determined. The calibration process of the image acquisition device will be described in detail later in conjunction with the possible implementation methods of the present disclosure, which will not be repeated here.

[0044] In the disclosed embodiments, vehicle detection is performed on the video frames acquired by the image acquisition device to determine the target vehicle image, and wheel feature points included in the target vehicle image are determined by performing wheel feature point detection on the target vehicle image. Since the wheel feature points are located in the same plane, the image acquisition device is calibrated according to the wheel feature points, so the device parameters of the image acquisition device can be determined quickly and accurately, thereby improving the calibration efficiency of the image acquisition device.

[0045] The calibration method of the image acquisition device of the embodiment of the present disclosure includes two steps: vehicle structure analysis and iterative optimization. The processes of vehicle structure analysis and iterative optimization are described in detail below.

[0046] After obtaining the video frames captured by the image acquisition device, a deep convolutional neural network can be used to perform vehicle structured analysis on the video frames to obtain the target vehicle image and the wheel feature points included in the target vehicle image, and the wheel feature points are structured and stored to facilitate the query and acquisition of the wheel feature points in the subsequent iterative optimization process.

[0047] In one possible implementation, a target vehicle image is determined by performing vehicle detection on a video frame acquired by an image acquisition device, including: performing vehicle detection on the video frame to determine whether a vehicle image exists in the video frame; if it is determined that a vehicle image exists in the video frame, determining whether the vehicle type corresponding to the vehicle image is the target vehicle type; if it is determined that the vehicle type corresponding to the vehicle image is the target vehicle type, determining the vehicle image as the target vehicle image.

[0048] By performing vehicle detection and vehicle type detection on video frames, the target vehicle image can be quickly determined from the video frames, thereby improving the efficiency of subsequent calibration of the image acquisition device.

[0049] In one example, a vehicle detection neural network can be used to detect a vehicle on a video frame to determine whether a vehicle image exists in the video frame. For example, based on Faster RCNN, a vehicle detection is performed on a video frame. Specifically, a deep convolutional layer is first used to extract features from the video frame, and the extracted features are input into the region proposal layer. The region proposal layer performs region prediction to obtain multiple first candidate target regions, where the first candidate target region is an area where a vehicle may exist in the video frame. Multiple first candidate target regions are pooled (ROIPooling) to obtain multiple second candidate target regions of uniform size (the size may be pre-set). According to the multiple second candidate target regions, category classification and coordinate regression are performed to obtain the confidence of each second candidate target region, and the confidence is the probability of detecting a vehicle in the second candidate target region. According to the confidence of each second candidate target region, the second candidate target regions with an intersection-over-union ratio greater than a threshold are merged through a maximum suppression algorithm to obtain the final vehicle detection result. In the case where a vehicle image exists in the video frame, the vehicle detection result is the detection frame corresponding to the vehicle in the video frame and the position of the detection frame.

[0050] In the embodiments of the present disclosure, in addition to the Faster RCNN neural network mentioned above, the vehicle detection neural network may also use other deep neural networks that can be used for vehicle detection. The specific form and specific structure of the vehicle detection neural network can be determined according to actual conditions, and the present disclosure does not make specific limitations on this.

[0051] In the embodiments of the present disclosure, the vehicle detection neural network used for vehicle detection can be obtained after training with a preset number (for example, millions of frames) of labeled video frames. The training process can refer to the neural network training process of related technologies (for example, using online difficult example mining / gradient equalization and other strategies for network training), and the present disclosure does not make specific limitations on this.

[0052] When it is determined that there is a vehicle image in the video frame, the video frame is cropped according to the detection frame in the vehicle detection result and the position of the detection frame, and the vehicle image is cropped out of the video frame to facilitate subsequent vehicle type detection.

[0053] In one example, vehicle type detection may be performed on a vehicle image based on a vehicle type detection neural network to determine whether the vehicle type corresponding to the vehicle image is a target vehicle type. Figure 2 FIG. 2 is a schematic diagram showing a vehicle type detection according to an embodiment of the present disclosure. Figure 2 As shown, the vehicle image is input into the vehicle type detection neural network, and after extracting features using resnet18 as the backbone network, two layers of full-connection layers are used to predict the vehicle type to obtain the vehicle type detection result. The vehicle type detection result includes the probability that the vehicle image is each preset vehicle type, and the preset vehicle type corresponding to the maximum probability value is determined as the vehicle type corresponding to the vehicle image. The preset vehicle types include: cars, vans, small trucks, large trucks, SUVs, buses, etc., and the present disclosure does not make specific limitations on this.

[0054] In the embodiments of the present disclosure, the specific form and structure of the vehicle type detection neural network can be determined according to actual conditions, and the present disclosure does not make any specific limitations on this.

[0055] In the embodiments of the present disclosure, the vehicle type detection neural network used for vehicle type detection can be obtained after training with a preset number (for example, millions) of labeled vehicle images. The training process can refer to the neural network training process of the related technology (for example, network training by optimizing cross entropy loss through stochastic gradient descent), and the present disclosure does not make specific limitations on this.

[0056] In a possible implementation, the target vehicle type is a vehicle type whose vehicle length and vehicle width in the world coordinate system are known, and the target vehicle type is one of the preset vehicle types, for example, the target vehicle type is a car with a vehicle length of 3 meters and a vehicle width of 1.56 meters. The specific type of the target vehicle type can be determined according to actual conditions, and the present disclosure does not specifically limit this.

[0057] After performing vehicle type detection on a vehicle image based on a vehicle type detection neural network and determining the vehicle type corresponding to the vehicle image, it is determined whether the vehicle type corresponding to the vehicle image is a target vehicle type, and if it is determined that the vehicle type corresponding to the vehicle image is the target vehicle type, the vehicle image is determined as a target vehicle image.

[0058] The target vehicle image is an image that can be used for subsequent calibration of the image acquisition device. When it is determined that the target vehicle image exists in the video frame, wheel feature point detection is performed on the target vehicle image to determine the wheel feature points included in the target vehicle image.

[0059] In a possible implementation, wheel feature points included in the target vehicle image are determined by performing wheel feature point detection on the target vehicle image, including: downsampling feature extraction on the target vehicle image to obtain at least two levels of first feature maps; upsampling feature extraction on the target first feature map to obtain at least two levels of second feature maps, wherein the target first feature map is a feature map with the smallest size in the at least two levels of first feature maps; fusing the at least two levels of first feature maps and the at least two levels of second feature maps to obtain a target feature map; and predicting wheel feature points on the target feature map to determine first coordinates of the wheel feature points in an image coordinate system, wherein the image coordinate system is a coordinate system corresponding to an image acquisition device.

[0060] By performing feature extraction and feature fusion on the target vehicle image to obtain a target feature map, wheel feature point prediction can be performed on the target feature map to quickly determine the wheel feature points included in the target image for subsequent calibration of the image acquisition device, as well as the first coordinates of the wheel feature points in the image coordinate system, thereby improving the efficiency of subsequent calibration of the image acquisition device.

[0061] In one example, wheel feature point detection may be performed on the target vehicle image based on a wheel feature point detection neural network to determine wheel feature points included in the target vehicle image.

[0062] Figure 3 FIG. 2 is a schematic diagram showing the wheel feature point detection according to an embodiment of the present disclosure. Figure 3As shown in the figure, resnet is used as the backbone network for feature extraction in the wheel feature point detection network. The target vehicle image is input into the wheel feature point detection network, and the backbone network performs down-sampling feature extraction on the target vehicle image to obtain five-level first feature maps of different sizes. The feature pyramid sub-network (FPN) in the wheel feature point detection network is used to up-sample the target first feature map with the smallest size in the five-level first feature map to extract features, and obtain four-level second feature maps of different sizes. Then, the first feature map and the second feature map of the same size are fused to obtain four-level target feature maps of different sizes.

[0063] For example, the size of the target vehicle image is 1024×640. After the backbone network performs downsampling feature extraction on the target vehicle image, the sizes of the five-level first feature maps are 512×320, 256×160, 128×80, 64×40 and 32×20. Using the FPN network, the first feature map of the target with the smallest size (32×20) among the five-level first feature maps is upsampled and feature extracted, and the sizes of the four-level second feature maps are 64×40, 128×80, 256×160 and 512×320. The first feature map of 64×40 and the second feature map of 64×40 are feature fused to obtain a target feature map of 64×40; the first feature map of 128×80 and the second feature map of 128×80 are feature fused to obtain a target feature map of 128×80; the first feature map of 256×160 and the second feature map of 256×160 are feature fused to obtain a target feature map of 256×160; the first feature map of 512×320 and the second feature map of 512×320 are feature fused to obtain a target feature map of 512×320.

[0064] The wheel feature points are predicted for the four levels of target feature degrees of different sizes output by FPN respectively, and the wheel feature point detection results corresponding to each level of target feature map are obtained. Then, the wheel feature point detection results corresponding to each level of target feature map are fused to obtain the first coordinates of the wheel feature points in the image coordinate system.

[0065] In one possible implementation, the same target vehicle image includes at least two of the following wheel feature points: a left front wheel feature point, a left rear wheel feature point, a right front wheel feature point, and a right rear wheel feature point; and the prediction confidence corresponding to the wheel feature point is greater than a confidence threshold.

[0066] Still with the above Figure 3 For example, Figure 3As shown, the wheel feature point detection result includes: a first positioning heat map, a second positioning heat map, a third positioning heat map and a fourth positioning heat map. Among them, the first positioning heat map includes the confidence (confidence is probability) that each pixel point in the target vehicle image is a left front wheel feature point, the second positioning heat map includes the confidence that each pixel point in the target vehicle image is a left rear wheel feature point, the third positioning heat map includes the confidence that each pixel point in the target vehicle image is a right front wheel feature point, and the fourth positioning heat map includes the confidence that each pixel point in the target vehicle image is a right rear wheel feature point. In order to further optimize the accuracy of wheel feature point detection, the wheel feature point detection result also includes: a fifth positioning heat map, and the fifth positioning heat map includes the confidence that each pixel point in the target vehicle image is the background. Using positioning heat map regression as the optimization target, that is, using the sum of the confidences corresponding to the pixels at the same position in the above 5 positioning heat maps as 1 for constraint, the accuracy of each wheel feature point detection is improved.

[0067] In the embodiments of the present disclosure, the specific form and structure of the wheel feature point detection neural network can be determined according to actual conditions, and the present disclosure does not make any specific limitations on this.

[0068] In the embodiments of the present disclosure, the wheel feature point detection neural network used for wheel feature point detection can be obtained after training with a preset number (for example, hundreds of thousands) of vehicle images labeled with wheel feature points. The training process can refer to the neural network training process of related technologies (for example, network training with positioning heat map regression as the optimization goal), and the present disclosure does not make specific limitations on this.

[0069] According to the first to fourth positioning heat maps included in the wheel feature point detection results, the wheel feature points included in the target vehicle image and the first coordinates of the wheel feature points in the image coordinate system can be determined, and then the image acquisition device can be calibrated according to the first coordinates of the wheel feature points in the image coordinate system.

[0070] By presetting the confidence threshold, the wheel feature points with a confidence greater than the confidence threshold can be used to calibrate the image acquisition device, thereby improving the accuracy of calibrating the image acquisition device. The specific value of the confidence threshold can be determined according to actual conditions. For example, the confidence threshold is 0.7, and this disclosure does not make specific limitations on this.

[0071] For example, the confidence threshold is 0.7. For the pixel point (x, y) in the target vehicle image, the confidence that the pixel point (x, y) is the feature point of the left front wheel in the first positioning heat map is 0.8, the confidence that the pixel point (x, y) is the feature point of the left rear wheel in the second positioning heat map is 0.1, the confidence that the pixel point (x, y) is the feature point of the right front wheel in the third positioning heat map is 0.05, and the confidence that the pixel point (x, y) is the feature point of the right rear wheel in the fourth positioning heat map is 0.05. Therefore, it can be determined that the pixel point (x, y) in the target vehicle image corresponds to the feature point of the left front wheel, and the corresponding prediction confidence is 0.8 (greater than the confidence threshold 0.7). Then, according to the position of the pixel point (x, y) in the target vehicle image, the first coordinate of the feature point of the left front wheel in the image coordinate system can be determined.

[0072] In a possible implementation, the image acquisition device is calibrated according to the wheel feature points included in the target vehicle image, and the device parameters of the image acquisition device are determined, including: determining the first coordinates of the wheel feature points in the image coordinate system, wherein the image coordinate system is the coordinate system corresponding to the image acquisition device; determining the second coordinates of the wheel feature points in the world coordinate system; calibrating the image acquisition device according to the first coordinates of the wheel feature points in the image coordinate system and the second coordinates in the world coordinate system to determine the device parameters.

[0073] By determining the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system, the mapping relationship between the image coordinate system and the world coordinate system can be determined, thereby effectively realizing the calibration of the image acquisition device.

[0074] In a possible implementation, determining the second coordinate of the wheel feature point in the world coordinate system includes: determining the second coordinate of the wheel feature point in the world coordinate system according to a preset vehicle length and vehicle width of the target vehicle type in the world coordinate system.

[0075] Since the vehicle type corresponding to the target vehicle image is the target vehicle type, and the vehicle length and vehicle width of the target vehicle type in the world coordinate system are pre-set, the second coordinate of the wheel feature point in the world coordinate system can be quickly determined according to the preset vehicle length and vehicle width of the target vehicle type in the world coordinate system.

[0076] In a possible implementation, the second coordinate of the wheel feature point in the world coordinate system is determined according to the vehicle length and vehicle width of the preset target vehicle type in the world coordinate system, including: when the wheel feature point is a left front wheel feature point, the second coordinate of the left front wheel feature point in the world coordinate system is determined to be (0, 0); when the wheel feature point is a left rear wheel feature point, the second coordinate of the left rear wheel feature point in the world coordinate system is determined to be (0, vehicle length); when the wheel feature point is a right rear wheel feature point, the second coordinate of the right rear wheel feature point in the world coordinate system is determined to be (vehicle length, vehicle width); when the wheel feature point is a right front wheel feature point, the second coordinate of the right front wheel feature point in the world coordinate system is determined to be (vehicle width, 0).

[0077] For example, the preset target vehicle type is a car with a vehicle length of 3 meters and a vehicle width of 1.56 meters in the world coordinate system. The coordinates of the wheel feature points can be modeled according to the vehicle length and vehicle width, and the second coordinate of the left front wheel feature point in the world coordinate system is (0, 0), the second coordinate of the left rear wheel feature point in the world coordinate system is (0, 3), the second coordinate of the right front wheel feature point in the world coordinate system is (1.56, 0), and the second coordinate of the right rear wheel feature point in the world coordinate system is (1.56, 3).

[0078] In a possible implementation, the image acquisition device is calibrated according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system to determine the device parameters, including: constructing a calibration constraint function according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system; and determining the device parameters by iteratively optimizing the calibration constraint function.

[0079] After determining the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system, the image acquisition device can be calibrated by constructing a calibration constraint function and iteratively optimizing the calibration constraint function.

[0080] In one example, the first coordinate of point P in the image coordinate system is (u, v, 1) T , the second coordinate in the world coordinate system is (x,y,1) T In the case of , according to the distortion-free pinhole imaging principle of the image acquisition device, the imaging model of the image acquisition device is shown in the following formula (1):

[0081]

[0082] in, is the normalization factor, and H is the device parameter corresponding to the image acquisition device with 8 degrees of freedom.

[0083] According to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system, a calibration constraint function for calibrating the image acquisition device can be constructed using the imaging model of the image acquisition device shown in the above formula (1).

[0084] In a possible implementation, the device parameters include: homography matrix parameters corresponding to the image acquisition device, and device external parameters corresponding to the target vehicle image; constructing a calibration constraint function according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system, including: constructing a constraint equation corresponding to the wheel feature point based on the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system, the homography matrix parameters, and the device external parameters; constructing a calibration constraint function based on the constraint equation corresponding to the wheel feature point.

[0085] In the vehicle structuring stage, the first coordinates of the wheel feature points included in the target vehicle image in the image coordinate system can be obtained by performing vehicle structuring analysis on the video frames acquired by the image acquisition device. For example, the first coordinates of the jth wheel feature point included in the i-th target vehicle image in the image coordinate system are According to the preset vehicle length and vehicle width of the target vehicle type in the world coordinate system, the second coordinate of the jth wheel feature point included in the i-th target vehicle image in the world coordinate system can be determined as: According to the imaging model of the image acquisition device shown in the above formula (1), the constraint equation corresponding to the j-th wheel feature point included in the i-th target vehicle image can be constructed as shown in the following formula (2):

[0086]

[0087] in, is the normalization factor corresponding to the jth wheel feature point included in the i-th target vehicle image, H i It is a device parameter with 8 degrees of freedom corresponding to the image acquisition device determined based on the i-th target image.

[0088] Since the wheel feature points included in different target vehicle images are all on the same plane, we can use H in the above formula (2) i Perform the following conversion:

[0089] H i =HT i (3)

[0090] Among them, H is the homography matrix parameter corresponding to the image acquisition device, Ti is the external parameter of the device corresponding to the i-th target vehicle image. H=K[r1 r2 t], is the internal parameter of the image acquisition device, f x 、f y is the focal length of the image acquisition device, c x 、c y is the center point of the image acquisition device, and [r1 r2 t] are the preset device external parameters. R i is the rotation matrix corresponding to the i-th target vehicle image, t i is the translation matrix corresponding to the i-th target vehicle image.

[0091] Based on the reprojection mean square error model and the constraint equations shown in the above formulas (2) and (3), the calibration constraint function shown in the following formula (4) can be constructed:

[0092]

[0093] Wherein, n is the number of target vehicle images, and m is the number of wheel feature points whose confidence is greater than the confidence threshold included in the i-th target vehicle image.

[0094] According to the wheel feature points included in the N target vehicle images obtained in the vehicle structure analysis stage, N groups of constraint equations shown in the above formula (2) can be constructed. The homography matrix parameters of the image acquisition device corresponding to different groups of constraint equations are the same, but the device external parameters of the image acquisition device corresponding to different groups of constraint equations are independent of each other. According to the N groups of constraint equations, the calibration constraint function shown in the above formula (4) can be constructed, and then the calibration constraint function is iteratively optimized to determine the homography matrix parameters and device external parameters corresponding to the image acquisition device.

[0095] In practical applications, the degree of freedom of the homography matrix parameters corresponding to the image acquisition device is 8, and the degree of freedom of the external parameters of the device is 3. Therefore, in the embodiment of the present disclosure, the specific value of N can be determined based on the degrees of freedom of the homography matrix parameters and the degrees of freedom of the external parameters of the device.

[0096] For example, one wheel feature point can construct two constraint equations. When constructing the calibration constraint function, there are the following constraints on the number of constraint equations: Since the degree of freedom of the device external parameters is 3, when solving the device external parameters based on the i-th target vehicle image, the i-th target vehicle image needs to include at least two wheel feature points whose prediction confidence is greater than the confidence threshold, so as to construct at least 3 constraint equations to solve the device external parameters. In addition, since the degree of freedom of the homography matrix parameters is 8, at least 8 constraint equations are required when solving the homography matrix parameters.

[0097] Based on the above constraints, when the prediction confidences of the four wheel feature points included in a target vehicle image are all greater than the confidence threshold, that is, m=4, eight constraint equations can be constructed based only on the four wheel feature points included in a target vehicle image to determine the calibration constraint function. By iteratively optimizing the calibration constraint function, the homography matrix parameters and device external parameters can be obtained.

[0098] However, in actual situations, due to the limitation of the acquisition angle of the image acquisition device, the wheel feature points accurately predicted in a target vehicle image, that is, the wheel feature points with a prediction confidence greater than the confidence threshold, may be 2 or 3, that is, m=2 or 3. Therefore, it is necessary to determine the specific value of the number n required for the target vehicle images based on the above-mentioned constraints and the number of wheel feature points with a prediction confidence greater than the confidence threshold included in each target vehicle image. For example, when the number of wheel feature points with a prediction confidence greater than the confidence threshold included in each target vehicle image is 2, at least 8 target vehicle images are required to construct the calibration constraint function. The more target vehicle images used for calibrating the image acquisition device, the more accurate the calibration process. Therefore, more target vehicle images can be appropriately selected to construct the calibration constraint function. For example, the number of target vehicle images is n=30, and the present disclosure does not make specific limitations on this.

[0099] In a possible implementation, the device parameters are determined by iteratively optimizing the calibration constraint function, including: performing the t+1th external parameter optimization on the calibration constraint function according to the tth homography matrix parameter to obtain the t+1th device external parameter, wherein t is an integer greater than or equal to 0, and the tth homography matrix parameter is obtained after the tth internal parameter optimization of the calibration constraint function; performing the t+1th internal parameter optimization on the calibration constraint function according to the t+1th device external parameter to obtain the t+1th homography matrix parameter; when the preset iteration condition is met, the device parameters are determined according to the t+1th homography matrix parameter and the t+1th device external parameter.

[0100] According to the calibration constraint function shown in the above formula (4), the following iterative optimization process is performed.

[0101] Step a: According to the tth homography matrix parameters, the calibration constraint function is optimized for the t+1th external parameters to obtain the t+1th device external parameters.

[0102] In one possible implementation, the calibration constraint function is optimized for the t+1th time external parameters according to the tth homography matrix parameter to obtain the t+1th device external parameters, including: according to the tth homography matrix parameter, using a singular value decomposition algorithm, the calibration constraint function is optimized for the t+1th time external parameters to obtain the t+1th device external parameters.

[0103] For the calibration constraint function shown in the above formula (4), when the calibration constraint function is optimized for the t+1th time, the homography matrix parameter H in the calibration constraint function is fixed to the tth homography matrix parameter obtained after the tth time of internal parameter optimization of the calibration constraint function. The t+1th device external parameter T obtained after the t+1th time of external parameter optimization is solved based on the i-th target vehicle image by using the principle of two-dimensional point cloud matching and the singular value decomposition algorithm. i .

[0104] Assumptions It is the third coordinate after the first coordinate of the j-th wheel feature point included in the i-th target vehicle image in the image coordinate system is converted to the world coordinate system. is the second coordinate of the jth wheel feature point included in the i-th target vehicle image in the world coordinate system. Then the calibration constraint function shown in the above formula (4) can be converted into the following formula (5):

[0105]

[0106] make p' j =p j -u p , q' j =q j -u q , expand the above formula (5) to obtain the following formula (6):

[0107]

[0108] in,

[0109] Therefore, the above formula (6) can be simplified to formula (7):

[0110]

[0111] The singular value decomposition algorithm is used to solve the above formula (7) to determine the t+1th device external parameter T obtained after the t+1th external parameter optimization. i , including the rotation matrix R shown in the following formula (8) i and the translation matrix t i :

[0112]

[0113] t i =u q -R i u p

[0114] in, (Singular Value Decomposition), Q * satisfy Q * The determinant of |Q * |=±1.

[0115] In a possible implementation, when t=0, the 0th homography matrix parameters are initialized to be a unit matrix.

[0116] In the initialization stage, the 0th homography matrix parameters are initialized to the unit matrix, and then the first external parameter optimization of the above step a is performed, that is, according to the 0th homography matrix parameters, the calibration constraint function is optimized for the first time to obtain the first device external parameters.

[0117] Step b: performing a t+1th internal parameter optimization on the calibration constraint function according to the t+1th device external parameter to obtain the t+1th homography matrix parameter.

[0118] In one possible implementation, the calibration constraint function is optimized for the t+1th time internally according to the t+1th device external parameter to obtain the t+1th homography matrix parameter, including: according to the t+1th device external parameter, using a direct linear transformation (DLT, Direct Linear Transform) algorithm, the calibration constraint function is optimized for the t+1th time internally to obtain the t+1th homography matrix parameter.

[0119] For the calibration constraint function shown in the above formula (4), when the calibration constraint function is optimized for the t+1th time, the device external parameter T in the calibration constraint function is i , the t+1th device extrinsic parameter obtained after the t+1th extrinsic parameter optimization of the calibration constraint function is fixed to remain unchanged, and the t+1th homography matrix parameter H obtained after the t+1th intrinsic parameter optimization is solved by the DLT algorithm.

[0120] Assumptions Then the calibration constraint function shown in the above formula (4) can be converted into the following formula (9):

[0121]

[0122] By using the DLT algorithm to solve formula (9), we can obtain the t+1th homography matrix parameter H after the t+1th internal parameter optimization.

[0123] The above steps a and b are performed alternately. When the preset iteration condition is met, the device parameters of the image acquisition device are determined based on the above formula (3) according to the t+1th homography matrix parameters and the t+1th device external parameters.

[0124] In one possible implementation, the preset iteration conditions include: t+1 reaches a preset iteration number threshold; or, the difference between the t+1th homography matrix parameter and the tth homography matrix parameter is less than a first threshold, and the difference between the t+1th device external parameter and the tth device external parameter is less than a second threshold, wherein the tth device external parameter is obtained after the tth external parameter optimization of the calibration constraint function.

[0125] The specific value of the preset iteration number threshold can be determined according to actual conditions. For example, the preset iteration number threshold is 5000 times.

[0126] The difference between the t+1th homography matrix parameter and the tth homography matrix parameter is less than the first threshold, and the difference between the t+1th device external parameter and the tth device external parameter is less than the second threshold, which indicates that the calibration constraint function has reached convergence during the iterative optimization process. The specific values ​​of the first threshold and the second threshold can be determined according to actual conditions. The values ​​of the first threshold and the second threshold can be the same or different. For example, the first threshold and the second threshold are both 10 -5 , this disclosure does not make any specific limitation on this.

[0127] In addition to the above conditions, the preset iteration conditions may also include other conditions according to actual conditions, and the present disclosure does not make any specific limitations on this.

[0128] In the embodiment of the present disclosure, the target vehicle image is determined by performing vehicle detection on the video frame acquired by the image acquisition device, and the wheel feature points included in the target vehicle image are determined by performing feature point detection on the target vehicle image. Since the wheel feature points are located in the same plane, there is no need to mark the feature points located in the same plane in the target vehicle image through manual interaction. The image acquisition device can be calibrated quickly based on the wheel feature points, and the device parameters of the image acquisition device can be accurately determined, thereby improving the calibration efficiency of the image acquisition device. Compared with the method of self-calibration of the image acquisition device using the vanishing points on the lane in the related art, the calibration method of the image acquisition device in the embodiment of the present disclosure is not affected by curves and one-way streets.

[0129] In a possible implementation, the calibration method of the image acquisition device also includes: acquiring an image of the test object through the image acquisition device to obtain a test image, wherein the size of the test object in the world coordinate system is a preset reference size; determining the image size of the test object in the test image, and converting the image size according to the device parameters to obtain the measured size of the test object in the world coordinate system; determining the percentage error between the reference size and the measured size; and determining the calibration accuracy of the device parameters according to the percentage error.

[0130] The calibration method of the image acquisition device of the embodiment of the present disclosure may be evaluated by using the ranging percentage error. Figure 4 The following is a schematic diagram of the calibration and evaluation of an image acquisition device according to an embodiment of the present disclosure. A target rectangle is marked on the target lane, and the target rectangle is determined as the test object. The actual length of the target rectangle is recorded as v, that is, the reference size of the test object in the world coordinate system is v. The image acquisition device calibrated by the calibration method of the image acquisition device according to the embodiment of the present disclosure is used to shoot the target rectangle, and the result is Figure 4 The test image shown in the figure is used to determine the image size of the target rectangle in the test image, and the image size is converted based on the device parameters obtained after calibration to obtain the measured length u of the target rectangle in the world coordinate system, that is, the measured size of the test object in the world coordinate system is u, and then the percentage error between the reference size v and the measured size u can be determined. The calibration accuracy of the device parameters can then be determined based on the percentage error. For example, the larger the percentage error, the lower the calibration accuracy of the device parameters; and the smaller the percentage error, the higher the calibration accuracy of the device parameters.

[0131] After evaluation, it can be known that after calibration using the calibration method of the image acquisition device according to the embodiment of the present disclosure, the percentage error of the image acquisition device in ranging is only 8.75%. Therefore, after calibration using the calibration method of the image acquisition device according to the embodiment of the present disclosure, the calibration accuracy of the device parameters of the image acquisition device is relatively high, which can effectively improve the ranging accuracy when the image acquisition device is subsequently used for ranging.

[0132] In one possible implementation, the calibration method of the image acquisition device also includes: acquiring an image of the test object through the image acquisition device to obtain a test image, wherein the coordinates of a test target point included in the test object in a world coordinate system are preset reference coordinates; determining the image coordinates of the test target point in the test image, and performing coordinate transformation on the image coordinates according to device parameters to obtain the measured coordinates of the test target point in the world coordinate system; determining a reprojection mean square error between the reference coordinates and the measured coordinates; and determining the calibration accuracy of the device parameters according to the reprojection mean square error.

[0133] The reprojection mean square error can be used to evaluate the calibration method of the image acquisition device of the embodiment of the present disclosure. Figure 4For example, a target rectangle is marked on the target lane, the target rectangle is determined as the test object, and the actual coordinates q of the target vertex (i.e., the test target point) in the target rectangle in the world coordinate system are recorded, i.e., the reference coordinates of the test target point in the test object in the world coordinate system are q. The target rectangle is photographed by the image acquisition device calibrated by the calibration method of the image acquisition device of the embodiment of the present disclosure, and the result is Figure 4 The test image shown in the figure is used to determine the image coordinates of the test coordinate point in the test image, and the image coordinates are transformed based on the device parameters obtained after calibration to obtain the measured coordinates p of the target vertex in the target rectangle in the world coordinate system, that is, the measured coordinates of the test target point in the world coordinate system are p, and then the reprojection mean square error (pq) between the reference coordinate q and the measured coordinate p can be determined. 2 , and then the calibration accuracy of the device parameters can be determined according to the reprojection mean square error. For example, the larger the reprojection error, the lower the calibration accuracy of the device parameters; the smaller the reprojection mean square error, the higher the calibration accuracy of the device parameters.

[0134] After evaluation, it can be known that after calibration using the calibration method of the image acquisition device according to the embodiment of the present invention, the mean square error of the reprojection of the image acquisition device is only 0.1732. Therefore, after calibration using the calibration method of the image acquisition device according to the embodiment of the present invention, the calibration accuracy of the device parameters of the image acquisition device is relatively high, which can effectively improve the reprojection accuracy of the image acquisition device.

[0135] The calibration method of the image acquisition device of the embodiment of the present disclosure can be effectively applied to the field of traffic monitoring. After the image acquisition device is calibrated according to the calibration method of the image acquisition device of the embodiment of the present disclosure and the device parameters are obtained, the device parameters can be used to determine the length, speed and direction of the vehicle based on the vehicle image obtained by the image acquisition device on the lane within the coverage range (i.e., the shooting range), and then it can be determined whether the lane has traffic events such as vehicle speeding, road slowing, and road congestion, so that the road traffic conditions can be timely and effectively understood. In addition, by decomposing the device parameters, the physical parameters of the image acquisition device such as the focal length, pitch angle, and installation height relative to the ground can be obtained, and the effective coverage range of the image acquisition device can be further obtained.

[0136] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0137] In addition, the present disclosure also provides a calibration device, an electronic device, a computer-readable storage medium, and a program for an image acquisition device, all of which can be used to implement a calibration method for any image acquisition device provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0138] Figure 5 FIG. 1 is a block diagram of a calibration device for an image acquisition device according to an embodiment of the present disclosure. Figure 5 As shown, the device 50 includes:

[0139] A first determination module 51 is used to determine a target vehicle image by performing vehicle detection on a video frame acquired by an image acquisition device;

[0140] A second determination module 52 is used to determine the wheel feature points included in the target vehicle image by performing wheel feature point detection on the target vehicle image;

[0141] The calibration module 53 is used to calibrate the image acquisition device according to the wheel feature points included in the target vehicle image, and determine the device parameters of the image acquisition device.

[0142] In a possible implementation, the first determining module 51 is specifically configured to:

[0143] Perform vehicle detection on the video frame to determine whether there is a vehicle image in the video frame;

[0144] When it is determined that there is a vehicle image in the video frame, determining whether the vehicle type corresponding to the vehicle image is the target vehicle type;

[0145] When it is determined that the vehicle type corresponding to the vehicle image is the target vehicle type, the vehicle image is determined as the target vehicle image.

[0146] In a possible implementation, the second determining module 52 is specifically configured to:

[0147] Downsampling the target vehicle image for feature extraction to obtain at least two levels of first feature maps;

[0148] Performing upsampling feature extraction on the target first feature map to obtain at least two levels of second feature maps, wherein the target first feature map is the feature map with the smallest size among the at least two levels of first feature maps;

[0149] Fusing at least two levels of first feature maps and at least two levels of second feature maps to obtain a target feature map;

[0150] The wheel feature points are predicted on the target feature map to determine the first coordinates of the wheel feature points in the image coordinate system, wherein the image coordinate system is a coordinate system corresponding to the image acquisition device.

[0151] In a possible implementation, the calibration module 53 includes:

[0152] A first determination submodule is used to determine a first coordinate of the wheel feature point in an image coordinate system, wherein the image coordinate system is a coordinate system corresponding to the image acquisition device;

[0153] A second determination submodule, used to determine a second coordinate of the wheel feature point in the world coordinate system;

[0154] The calibration submodule is used to calibrate the image acquisition device and determine the device parameters according to the first coordinates of the wheel feature points in the image coordinate system and the second coordinates in the world coordinate system.

[0155] In a possible implementation, the second determining submodule is specifically configured to:

[0156] According to the preset vehicle length and vehicle width of the target vehicle type in the world coordinate system, the second coordinate of the wheel feature point in the world coordinate system is determined.

[0157] In a possible implementation, the calibration submodule includes:

[0158] A calibration constraint function construction unit, used to construct a calibration constraint function according to a first coordinate of the wheel feature point in the image coordinate system and a second coordinate in the world coordinate system;

[0159] The iterative optimization unit is used to determine the device parameters by iteratively optimizing the calibration constraint function.

[0160] In a possible implementation, the device parameters include: homography matrix parameters corresponding to the image acquisition device, and device external parameters corresponding to the target vehicle image;

[0161] The calibration constraint function building unit is specifically used for:

[0162] Constructing a constraint equation corresponding to the wheel feature point based on the first coordinate of the wheel feature point in the image coordinate system, the second coordinate in the world coordinate system, homography matrix parameters, and device external parameters;

[0163] Based on the constraint equations corresponding to the wheel feature points, a calibration constraint function is constructed.

[0164] In a possible implementation, the iterative optimization unit includes:

[0165] The extrinsic parameter optimization subunit is used to perform the t+1th extrinsic parameter optimization on the calibration constraint function according to the tth homography matrix parameter to obtain the t+1th device extrinsic parameter, wherein t is an integer greater than or equal to 0, and the tth homography matrix parameter is obtained after the tth internal parameter optimization of the calibration constraint function;

[0166] The internal parameter optimization subunit performs the t+1th internal parameter optimization on the calibration constraint function according to the t+1th device external parameter to obtain the t+1th homography matrix parameter;

[0167] The subunit is determined, and when a preset iteration condition is met, the device parameters are determined according to the t+1th homography matrix parameters and the t+1th device external parameters.

[0168] In a possible implementation, the external parameter optimization subunit is specifically used to:

[0169] According to the tth homography matrix parameters, the singular value decomposition algorithm is used to perform the t+1th external parameter optimization on the calibration constraint function to obtain the t+1th device external parameters.

[0170] In a possible implementation, the internal parameter optimization subunit is specifically used to:

[0171] According to the t+1th device external parameters, the direct linear transformation DLT algorithm is used to perform the t+1th internal parameter optimization on the calibration constraint function to obtain the t+1th homography matrix parameters.

[0172] In a possible implementation, the preset iteration condition includes: t+1 reaches a preset iteration number threshold; or,

[0173] A difference between a t+1th homography matrix parameter and a tth homography matrix parameter is less than a first threshold, and a difference between a t+1th device extrinsic parameter and a tth device extrinsic parameter is less than a second threshold, wherein the tth device extrinsic parameter is obtained after the tth external parameter optimization of the calibration constraint function.

[0174] In a possible implementation, the apparatus 50 further includes:

[0175] The initialization module is used to initialize the 0th homography matrix parameters when t=0, and initialize the 0th homography matrix parameters to the unit matrix.

[0176] In a possible implementation, the apparatus 50 further includes:

[0177] A third determination module is used to acquire an image of the test object through an image acquisition device to obtain a test image, wherein the size of the test object in the world coordinate system is a preset reference size;

[0178] A fourth determination module is used to determine the image size of the test object in the test image, and to convert the image size according to the device parameters to obtain the measured size of the test object in the world coordinate system;

[0179] a fifth determination module, for determining a percentage error between the reference dimension and the measured dimension;

[0180] The sixth determination module is used to determine the calibration accuracy of the device parameters according to the percentage error.

[0181] In a possible implementation, the apparatus 50 further includes:

[0182] A seventh determination module is used to acquire an image of the test object through an image acquisition device to obtain a test image, wherein the coordinates of the test target point included in the test object in the world coordinate system are preset reference coordinates;

[0183] An eighth determination module is used to determine the image coordinates of the test target point in the test image, and to perform coordinate conversion on the image coordinates according to the device parameters to obtain the measurement coordinates of the test target point in the world coordinate system;

[0184] A ninth determination module, for determining a reprojection mean square error between the reference coordinates and the measured coordinates;

[0185] The tenth determination module is used to determine the calibration accuracy of the device parameters according to the reprojection mean square error.

[0186] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0187] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0188] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0189] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code. When the computer-readable code is executed on a device, a processor in the device executes instructions for implementing the calibration method of the image acquisition device provided in any of the above embodiments.

[0190] The embodiments of the present disclosure also provide another computer program product for storing computer-readable instructions, which, when executed, enable a computer to perform the operations of the calibration method for an image acquisition device provided in any of the above embodiments.

[0191] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0192] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 6 As shown, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0193] Reference Figure 6 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0194] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0195] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0196] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0197] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0198] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0199] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0200] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0201] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0202] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0203] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0204] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 7 As shown, the electronic device 1900 may be provided as a server. Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0205] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003).TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0206] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0207] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0208] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0209] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0210] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0211] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0212] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0213] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0214] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0215] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0216] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A calibration method for an image acquisition device, characterized in that: include: By performing vehicle detection on the video frames acquired by the image acquisition device, a target vehicle image is determined; Determining wheel feature points included in the target vehicle image by performing wheel feature point detection on the target vehicle image; Calibrate the image acquisition device according to the wheel feature points included in the target vehicle image to determine device parameters of the image acquisition device; The step of calibrating the image acquisition device according to the wheel feature points included in the target vehicle image to determine the device parameters of the image acquisition device includes: Determine a first coordinate of the wheel feature point in an image coordinate system, wherein the image coordinate system is a coordinate system corresponding to the image acquisition device; Determine the second coordinate of the wheel feature point in the world coordinate system according to the vehicle length and vehicle width of the preset target vehicle type in the world coordinate system; The image acquisition device is calibrated according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system to determine the device parameters.

2. The method according to claim 1, characterized in that The method of performing vehicle detection on the video frames acquired by the image acquisition device to determine the target vehicle image includes: Performing vehicle detection on the video frame to determine whether there is a vehicle image in the video frame; In the case where it is determined that the vehicle image exists in the video frame, determining whether the vehicle type corresponding to the vehicle image is the target vehicle type; In the case where it is determined that the vehicle type corresponding to the vehicle image is the target vehicle type, the vehicle image is determined as the target vehicle image.

3. The method according to claim 1 or 2, characterized in that: The step of determining the wheel feature points included in the target vehicle image by performing wheel feature point detection on the target vehicle image comprises: Performing downsampling feature extraction on the target vehicle image to obtain at least two levels of first feature maps; Performing upsampling feature extraction on the target first feature map to obtain at least two levels of second feature maps, wherein the target first feature map is a feature map with the smallest size among the at least two levels of first feature maps; fusing the at least two levels of first feature maps and the at least two levels of second feature maps to obtain a target feature map; The wheel feature points are predicted on the target feature map to determine the first coordinates of the wheel feature points in the image coordinate system, wherein the image coordinate system is the coordinate system corresponding to the image acquisition device.

4. The method according to claim 1, characterized in that The step of calibrating the image acquisition device according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system to determine the device parameters includes: Constructing a calibration constraint function according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system; The device parameters are determined by iteratively optimizing the calibration constraint function.

5. The method according to claim 4, characterized in that The device parameters include: homography matrix parameters corresponding to the image acquisition device, and device external parameters corresponding to the target vehicle image; The step of constructing a calibration constraint function according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate of the wheel feature point in the world coordinate system comprises: Constructing a constraint equation corresponding to the wheel feature point based on the first coordinate of the wheel feature point in the image coordinate system, the second coordinate in the world coordinate system, the homography matrix parameter, and the device external parameter; The calibration constraint function is constructed based on the constraint equation corresponding to the wheel feature point.

6. The method according to claim 5, characterized in that The determining of the device parameters by iteratively optimizing the calibration constraint function includes: According to the tth homography matrix parameter, the calibration constraint function is optimized for the t+1th time to obtain the t+1th device external parameter, wherein t is an integer greater than or equal to 0, and the tth homography matrix parameter is obtained after the calibration constraint function is optimized for the tth time; According to the t+1th device external parameter, the calibration constraint function is optimized for the t+1th time to obtain the t+1th homography matrix parameter; When a preset iteration condition is met, the device parameters are determined according to the t+1th homography matrix parameters and the t+1th device external parameters.

7. The method according to claim 6, characterized in that The step of performing the t+1th external parameter optimization on the calibration constraint function according to the tth homography matrix parameter to obtain the t+1th device external parameter includes: According to the tth homography matrix parameters, a singular value decomposition algorithm is used to perform a t+1th external parameter optimization on the calibration constraint function to obtain the t+1th device external parameters.

8. The method according to claim 6, characterized in that The step of performing the t+1th internal parameter optimization on the calibration constraint function according to the t+1th device external parameter to obtain the t+1th homography matrix parameter includes: According to the t+1th device external parameter, the direct linear transformation DLT algorithm is used to perform the t+1th internal parameter optimization on the calibration constraint function to obtain the t+1th homography matrix parameter.

9. The method according to claim 6, characterized in that The preset iteration condition includes: t+1 reaches a preset iteration number threshold; or, The difference between the t+1th homography matrix parameter and the tth homography matrix parameter is less than a first threshold, and the difference between the t+1th device external parameter and the tth device external parameter is less than a second threshold, wherein the tth device external parameter is obtained after the tth external parameter optimization of the calibration constraint function.

10. The method according to claim 6, characterized in that The method further comprises: When t=0, the 0th homography matrix parameters are initialized to the unit matrix.

11. The method according to claim 1 or 2, characterized in that: The method further comprises: Capturing an image of the test object by the image acquisition device to obtain a test image, wherein the size of the test object in the world coordinate system is a preset reference size; Determine the image size of the test object in the test image, and perform size conversion on the image size according to the device parameters to obtain the measured size of the test object in the world coordinate system; determining a percentage error between the reference dimension and the measured dimension; The calibration accuracy of the device parameter is determined based on the percentage error.

12. The method according to claim 1 or 2, characterized in that: The method further comprises: The image acquisition device is used to acquire an image of the test object to obtain a test image, wherein the coordinates of the test target points included in the test object in the world coordinate system are preset reference coordinates; Determine the image coordinates of the test target point in the test image, and perform coordinate transformation on the image coordinates according to the device parameters to obtain the measurement coordinates of the test target point in the world coordinate system; determining a reprojection mean square error between the reference coordinates and the measured coordinates; The calibration accuracy of the device parameters is determined according to the reprojection mean square error.

13. A calibration device for an image acquisition device, characterized in that: include: A first determination module is used to determine a target vehicle image by performing vehicle detection on a video frame acquired by an image acquisition device; A second determination module is used to determine the wheel feature points included in the target vehicle image by performing wheel feature point detection on the target vehicle image; A calibration module, used to calibrate the image acquisition device according to the wheel feature points included in the target vehicle image, and determine the device parameters of the image acquisition device; Wherein, the calibration module comprises: A first determination submodule, used to determine a first coordinate of the wheel feature point in an image coordinate system, wherein the image coordinate system is a coordinate system corresponding to the image acquisition device; A second determination submodule, used for determining a second coordinate of the wheel feature point in the world coordinate system according to a preset vehicle length and a vehicle width of the target vehicle type in the world coordinate system; The calibration submodule is used to calibrate the image acquisition device according to the first coordinate of the wheel feature point in the image coordinate system and the second coordinate in the world coordinate system to determine the device parameters.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images