Image acquisition device calibration method, device and computer device

Abstract

The application provides an image acquisition device calibration method and device and a computer device. After determining an image acquisition device at any fixed position in a specified scene, a specified scene image acquired by the image acquisition device is directly obtained. Target object detection is performed on the specified scene image to obtain three-dimensional attributes of multiple target objects at different positions in the specified scene. Three-dimensional coordinates of a group of feature points corresponding to the target objects on the same plane are obtained, and two-dimensional coordinates of the group of feature points mapped into the specified scene image are obtained. Thus, the three-dimensional coordinates and the two-dimensional coordinates of the multiple target objects are processed according to a coordinate system conversion relationship of the image acquisition device to directly obtain calibration parameters of the image acquisition device.

Description

Technical Field

[0001] This application mainly relates to the field of artificial intelligence technology, and more specifically to a calibration method, device and computer equipment for an image acquisition device. Background Technology

[0002] With the development of artificial intelligence technology in fields such as traffic monitoring, image processing, and machine vision, cameras are being used in increasingly wider scenarios. Before using image acquisition devices such as cameras, it is usually necessary to calibrate the camera's intrinsic and extrinsic parameters so that the images acquired by the camera can be processed based on these parameters to meet the needs of practical applications.

[0003] The aforementioned camera calibration typically employs the Zhang Zhengyou calibration method, which involves using the camera to capture images of a calibration board from different angles, obtaining the pixel coordinates of objects in the images using the calibration board, and then calculating the camera's intrinsic and extrinsic parameters. However, this calibration board-based camera calibration method has significant limitations and is unsuitable for camera calibration in complex traffic conditions such as intersections or highways, or even in scenarios where calibration board calibration is not available. Summary of the Invention

[0004] To address the above problems, this application provides the following technical solution:

[0005] This application proposes a calibration method for an image acquisition device, the method comprising:

[0006] Obtain images of a specified scene captured by an image acquisition device; the image acquisition device is deployed at any fixed location within the specified scene;

[0007] Target object detection is performed on the specified scene image to obtain the three-dimensional attributes of multiple target objects; the multiple target objects are located at different positions in the specified scene image.

[0008] Based on the three-dimensional attributes of each target object, obtain the three-dimensional coordinates of a set of feature points corresponding to the target object; the set of feature points includes multiple feature points located on the same plane;

[0009] Obtain the two-dimensional coordinates of the set of feature points mapped to the specified scene image;

[0010] Based on the coordinate system transformation relationship of the image acquisition device, the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the multiple target objects are processed to obtain the calibration parameters of the image acquisition device; the coordinate system transformation relationship can characterize the coordinate transformation relationship between the world coordinate system and the pixel coordinate system of any feature point in the specified scene image.

[0011] Optionally, the three-dimensional attribute includes at least the predicted shape value of the corresponding target object, and the method further includes:

[0012] If the calibration parameters do not meet the calibration parameter requirements, the calibration parameters are adjusted based on the predicted shape value of each target object and the two-dimensional coordinates of the set of feature points to obtain the target calibration parameters of the image acquisition device.

[0013] Optionally, adjusting the calibration parameters based on the predicted shape value of each target object and the two-dimensional coordinates of the set of feature points to obtain the target calibration parameters of the image acquisition device includes:

[0014] Based on the calibration parameters, the shape projection values ​​of the two-dimensional coordinates of the set of feature points onto the corresponding target object are obtained;

[0015] The first projection error between the predicted shape value and the corresponding projected shape value of the target object is obtained;

[0016] By minimizing the first projection error, the calibration parameters are adjusted to obtain the target calibration parameters of the image acquisition device.

[0017] Optionally, the predicted shape value of the target object includes the predicted length, predicted width, and predicted height of the spatial region where the target object is located; the set of feature points includes multiple vertices of any surface of the spatial region where the target object is located.

[0018] The step of obtaining the shape projection value of the two-dimensional coordinates of the set of feature points onto the corresponding target object based on the calibration parameters includes:

[0019] Based on the two-dimensional coordinates of the adjacent vertices of each target object, the pixel distance between the corresponding adjacent vertices is obtained;

[0020] Based on the calibration parameters, the shape projection value of the pixel distance on the corresponding target object space is obtained;

[0021] The shape projection values ​​corresponding to different adjacent vertices of the target object include: the length projection value and width projection value of the plane corresponding to the spatial region where the target object is located, or the length projection value and height projection value, or the width projection value and height projection value.

[0022] Optionally, obtaining the first reprojection error between the predicted shape value and the corresponding projected shape value of the target object includes:

[0023] Obtain the error value between the predicted shape value of the same category and the projected shape value of the corresponding type for each target object;

[0024] By using the error value corresponding to the shape prediction value of each target object, the second projection error of the target object is obtained.

[0025] The first projection error is obtained by averaging the second projection errors of each of the plurality of target objects.

[0026] Optionally, the process of processing the three-dimensional coordinates and two-dimensional coordinates of a set of feature points for each of the plurality of target objects according to the coordinate system transformation relationship of the image acquisition device to obtain the calibration parameters of the image acquisition device includes:

[0027] Based on the coordinate system transformation relationship of the image acquisition device, the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects are calculated to obtain the homography matrix of the corresponding target object;

[0028] Based on the homography matrix of each of the plurality of target objects, the initial internal parameter matrix of the image acquisition device is obtained;

[0029] Based on the initial internal parameter matrix and the homography matrix of each of the plurality of target objects, the initial external parameter matrix of the image acquisition device is obtained;

[0030] The calibration parameters of the image acquisition device are obtained using the initial internal parameter matrix and the initial external parameter matrix.

[0031] Optionally, obtaining the specified scene image acquired by the image acquisition device includes:

[0032] To obtain images of a specified scene at different time frames captured by the image acquisition device; or,

[0033] A first video file acquired by the image acquisition device for a specified scene is determined; the video duration of the first video file is greater than a preset duration.

[0034] Extract images of a specified scene from different time frames of the first video file.

[0035] Optionally, the process of detecting target objects in the specified scene image further includes:

[0036] Obtain the two-dimensional attributes of each of the multiple objects contained in the specified scene image;

[0037] Based on the two-dimensional attributes of each of the plurality of objects, determine the target object among the plurality of objects that meets the calibration conditions;

[0038] The two-dimensional attributes include the two-dimensional bounding box of the corresponding object. Determining the target object that satisfies the calibration conditions among the multiple objects based on their respective two-dimensional attributes includes at least one of the following implementation methods:

[0039] Obtain the number of pixels occupied by the two-dimensional bounding box of the object, and determine that the number of pixels reaches a preset number threshold; the preset number threshold is determined based on the total number of pixels contained in the specified scene image;

[0040] It is determined that the distance between the two-dimensional bounding box of the object and the boundary of the specified scene image is greater than a preset distance threshold;

[0041] It is determined that there are no overlapping regions between the two-dimensional bounding boxes of different objects.

[0042] This application also proposes an image acquisition device calibration device, the device comprising:

[0043] A specified scene image acquisition module is used to acquire specified scene images acquired by an image acquisition device; the image acquisition device is deployed at any fixed position in the specified scene;

[0044] A 3D attribute acquisition module is used to perform target object detection on the specified scene image and obtain the 3D attributes of multiple target objects; the multiple target objects are located at different positions in the specified scene.

[0045] The three-dimensional coordinate acquisition module is used to obtain the three-dimensional coordinates of a set of feature points corresponding to each target object based on the three-dimensional attributes of each target object; the set of feature points includes multiple feature points located on the same plane;

[0046] A two-dimensional coordinate acquisition module is used to obtain the two-dimensional coordinates of the set of feature points mapped to the specified scene image;

[0047] The calibration parameter acquisition module is used to process the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects according to the coordinate system transformation relationship for the image acquisition device, so as to obtain the calibration parameters of the image acquisition device; the coordinate system transformation relationship can characterize the coordinate transformation relationship between the world coordinate system and the pixel coordinate system of any feature point in the specified scene image.

[0048] This application also proposes a computer device, the computer device comprising:

[0049] Communication module;

[0050] The memory is used to store the program that implements the image acquisition device calibration method described above;

[0051] A processor is used to load and execute the program stored in the memory to implement the image acquisition device calibration method described above.

[0052] Therefore, this application provides a method, device, and computer for calibrating an image acquisition device. After determining the image acquisition device at any fixed position in a specified scene, the method directly obtains the specified scene image acquired by the image acquisition device. By performing target object detection on the specified scene image, the method obtains the three-dimensional attributes of multiple target objects at different positions in the specified scene. Based on this, the method obtains the three-dimensional coordinates of a set of feature points on the same plane of the corresponding target objects, and obtains the two-dimensional coordinates of this set of feature points mapped to the specified scene image. Then, based on the coordinate system transformation relationship for the image acquisition device, the method processes the three-dimensional and two-dimensional coordinates of the multiple target objects to directly obtain the calibration parameters of the image acquisition device. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a schematic flowchart of an optional embodiment of the image acquisition device calibration method proposed in this application;

[0055] Figure 2 A traffic scene diagram applicable to the image acquisition device calibration method proposed in this application;

[0056] Figure 3 This is a schematic diagram illustrating the coordinate system transformation relationship of the image acquisition device in the calibration method for the image acquisition device proposed in this application.

[0057] Figure 4 This is a schematic flowchart of an optional embodiment two of the image acquisition device calibration method proposed in this application;

[0058] Figure 5 This is a schematic flowchart of an optional embodiment three of the image acquisition device calibration method proposed in this application;

[0059] Figure 6 A schematic diagram of a three-dimensional key point template for any target object in the calibration of the image acquisition device proposed in this application;

[0060] Figure 7 This is a schematic diagram of an optional embodiment of the image acquisition device calibration equipment proposed in this application;

[0061] Figure 8 This is a schematic diagram of another optional embodiment of the image acquisition device calibration equipment proposed in this application;

[0062] Figure 9 This is a schematic diagram of the hardware structure of an optional embodiment of a computer device suitable for the image acquisition device calibration method proposed in this application. Detailed Implementation

[0063] Regarding the content described in the background technology section, in the process of calibrating the internal and external parameters of image acquisition devices such as cameras, it is desirable to no longer rely on calibration boards and to be applicable to the calibration of internal and external parameters of image acquisition devices in more application scenarios, such as image acquisition devices set up at intersections or highways with complex traffic conditions, as well as other scenarios where calibration boards are not available, to ensure the safety of the calibration process.

[0064] Based on this, this application proposes to use artificial intelligence technologies such as computer vision to perform object detection on images of a specified scene acquired by an image acquisition device for a specified scene, and obtain the three-dimensional coordinates (i.e., world coordinates) of a set of feature points (such as the vertices of the three-dimensional detection box of the detected vehicle in space) of multiple target objects (such as vehicles in a traffic scene) in the specified scene. Then, the image of the specified scene where the target object is located is analyzed to determine the two-dimensional coordinates (i.e., pixel coordinates) of the corresponding set of feature points. After that, the three-dimensional coordinates and two-dimensional coordinates of the set of feature points of each of the multiple target objects can be calculated according to the coordinate system transformation relationship of the image acquisition device to obtain the intrinsic and extrinsic parameters of the image acquisition device, i.e., calibration parameters.

[0065] In order to improve the accuracy of the obtained calibration parameters, this application can obtain the three-dimensional coordinates and two-dimensional coordinates of a set of feature points of a large number of target objects, perform iterative calculations according to the method described above, and optimize the obtained calibration parameters. The implementation process will not be described in detail here.

[0066] Based on the above analysis, the image acquisition device calibration method proposed in this application keeps the installation position fixed throughout the entire calibration process after the image acquisition device is installed at any location in the specified scene. No manual intervention is required, and there is no need to use a calibration board to determine the pixel coordinates of each feature point of the same object in multiple frames of images. This avoids the potential dangers of using a calibration board on dangerous road sections, reduces restrictions, and expands the applicable scenarios.

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] Reference Figure 1 This is a flowchart illustrating an optional embodiment of the image acquisition device calibration method proposed in this application. This method can be applied to computer equipment, such as servers, or terminal devices with certain data processing capabilities, such as… Figure 1 As shown, the method may include:

[0069] Step S11: Obtain the image of the specified scene captured by the image acquisition device; the image acquisition device is deployed at any fixed position in the specified scene.

[0070] In practical applications, the designated scenario in this application can be a roadside traffic scenario or other calibration scenario that requires the configuration of image acquisition devices to meet the needs of monitoring, data collection, and other scenarios. Furthermore, the image acquisition device in this designated scenario can be deployed at any fixed location within it. Throughout the entire calibration process, and even during the use of the image acquisition device in the designated scenario, the installation / placement location of the image acquisition device remains unchanged. This application does not limit the type of designated scenario or the deployment method of the image acquisition device; it can be determined as appropriate. It should be noted that the embodiments in this application only use a roadside traffic scenario as an example for scheme description. The implementation process of the image acquisition device calibration method in other scenarios is similar, and this application will not provide detailed examples for each.

[0071] Based on the above analysis, in order to calibrate the internal and external parameters of the same model of image acquisition device in a specified scene, the image acquisition device can be triggered to enter the working state and acquire images of the currently located specified scene, obtaining at least one frame of the specified scene image. Typically, to ensure the accuracy and reliability of the calibration parameters, the image acquisition device can acquire multiple frames of the specified scene image. This application does not describe in detail the image acquisition process of the image acquisition device in the specified scene. Afterwards, the specified scene image can be sent to a computer device via wireless or wired communication. This application does not restrict the transmission method of the specified scene image and can be determined as appropriate.

[0072] Step S12: Target object detection is performed on the specified scene image to obtain the three-dimensional attributes of multiple target objects; the multiple target objects are located at different positions in the specified scene image.

[0073] In this embodiment of the application, a three-dimensional target detection algorithm can be used to detect target objects in a specified scene image, detect each target object contained in the specified scene image, and predict the three-dimensional attributes of each target object, such as the three-dimensional center, predicted shape (such as the length, height and width of a vehicle), and the direction of movement of the target object.

[0074] Optionally, this application may employ a monocular 3D detection algorithm to detect target objects in a specified scene image and predict the 3D attributes of each target object. In such cases... Figure 2 In the traffic scenario shown, a monocular 3D detection algorithm can be used to predict the 3D center of a vehicle, its shape (length, height, and width), and its orientation. Figure 2 As shown, the three-dimensional bounding box of each vehicle in the image is identified by a monocular 3D detection algorithm. The three-dimensional attributes of the corresponding vehicle are represented by parameters such as the center of the three-dimensional bounding box and the length of each boundary.

[0075] It should be noted that other target object detection algorithms can also be used in this application to predict the three-dimensional attributes of target objects. This application does not limit the implementation method of predicting the three-dimensional attributes of target objects. In addition, in order to obtain the calibration parameters of the image acquisition device, it is necessary to obtain the relevant parameters of at least three target objects in the specified scene. Therefore, after performing target object detection on the acquired specified scene images and determining that a frame of the specified scene image contains fewer than three target objects, target object detection can be performed on other frames of the specified scene images to ensure that the three-dimensional attributes of at least three target objects are obtained.

[0076] In practical applications, to ensure the calibration accuracy of the image acquisition device, the three-dimensional attributes of more target objects can be obtained from multiple frames of specified scene images, such as obtaining the three-dimensional attributes of more than 20 target objects. This application does not limit the number of target objects. Optionally, after obtaining the specified scene images, preprocessing can be performed based on the target object detection results to determine the target objects that meet the calibration conditions, and then predicting the three-dimensional attributes of these target objects. This application does not describe in detail the method for selecting target objects that meet the calibration conditions in the specified scene images, but can refer to, but is not limited to, the implementation method described in the corresponding embodiments below.

[0077] Step S13: Based on the three-dimensional attributes of each target object, obtain the three-dimensional coordinates of a set of feature points corresponding to the target object; a set of feature points includes multiple feature points on the same plane;

[0078] In the methods for calibrating the intrinsic and extrinsic parameters of an image acquisition device, the coordinate system transformation relationship of the image acquisition device model is usually utilized, such as... Figure 3 As shown, the corresponding world coordinates (X, X) of the target object in the world coordinate system. w ,Yw Z w First, the coordinates can be transformed using the RT coordinate transformation matrix into image acquisition device coordinates (X) in the image acquisition device coordinate system (such as the camera coordinate system). c ,Y c Z c Then, using the perspective projection matrix F, the coordinates are converted to image coordinates (x, y) in the image coordinate system. After a second transformation matrix K, they are converted to pixel coordinates (u, v) in the pixel coordinate system. Here, R = (r1, r2, r3) is the rotation matrix of the image acquisition device, T is the position coordinate of the image acquisition device relative to the world coordinate system, which can represent a three-dimensional translation vector, RT can represent the extrinsic parameter matrix of the image acquisition device, and M = K * F can represent the intrinsic parameter matrix of the image acquisition device.

[0079] in,

[0080] In the above matrix, 1 / dx and 1 / dy both represent definite integral operations, which are infinite quantities. γ can represent the optical axis offset of the image acquisition device, and can also be zero as needed. f can represent the focal length of the image acquisition device, that is, the difference between the coordinate system of the image acquisition device and the coordinate system of the image on the Z-axis. u0 and v0 can represent the initial pixel coordinates, and this application does not impose any restrictions on their values.

[0081] Based on the above analysis, the world coordinates and pixel coordinates of the same feature point of multiple target objects can be obtained. Then, through the aforementioned coordinate transformation relationship, the intrinsic and extrinsic parameter matrices of the image acquisition device can be calculated. To obtain the world coordinates of the feature points of the target object, this application can construct a world coordinate system for the target object based on its three-dimensional attributes, thereby obtaining the three-dimensional coordinates of a set of feature points of the target object in that world coordinate system.

[0082] For example, such as Figure 2 The traffic scenario shown uses one of the bottom bounding box endpoints of the vehicle (i.e., the target object) as the origin, and extends the length, width, and height of the vehicle as coordinate axes to establish the world coordinate system of the vehicle. Based on this world coordinate system, the three-dimensional coordinates of each vertex of the vehicle's bounding box can be obtained through the predicted three-dimensional attributes of the vehicle. The three-dimensional coordinates of each endpoint of the bottom bounding box can also be obtained. The implementation process is not detailed in this application, and this application does not limit the method for determining the above set of feature points, which can be determined as appropriate.

[0083] Step S14: Obtain a set of feature points of each target object mapped to two-dimensional coordinates in the specified scene image;

[0084] Following the above analysis, after determining a set of feature points for each target object detected from a specified scene image, such as the vertices of any face of the target object's 3D bounding box, the pixel coordinates of each feature point in the specified scene image can be directly determined. For example, a two-dimensional pixel coordinate system can be constructed using the specified scene image to determine the pixel coordinates of a set of feature points for each target object. Specifically, the two-dimensional coordinates of a set of feature points for the target object can be determined based on the origin of the world coordinate system constructed for the target object and its two coordinate axes on the image, but it is not limited to this pixel coordinate recognition method.

[0085] Optionally, this application can obtain the three-dimensional and two-dimensional coordinates of a set of feature points of multiple target objects according to the method described above, and then associate and store the three-dimensional and two-dimensional coordinates of the same feature point in the specified scene image so that the three-dimensional and two-dimensional coordinates of each feature point of each target object can be queried according to the association relationship in the future.

[0086] Step S15: Based on the coordinate system transformation relationship for the image acquisition device, process the three-dimensional coordinates and two-dimensional coordinates of a set of feature points for each of the multiple target objects to obtain the calibration parameters of the image acquisition device.

[0087] As analyzed above, this coordinate system transformation relationship can characterize the coordinate transformation relationship between any feature point in a specified scene image and the world coordinate system and the pixel coordinate system. After obtaining the three-dimensional and two-dimensional coordinates of a set of feature points of the corresponding target object based on the above world coordinate system, assuming that the bottom fourth vertex of the three-dimensional bounding box of the vehicle is obtained as a set of feature points, and the road surface evidence for the vehicle's travel makes the Z-axis world coordinate of each vehicle's feature points 0, multiple target objects are modeled. Based on the above coordinate system transformation relationship, the mapping relationship between the three-dimensional coordinates and two-dimensional coordinates of the same feature point of the target object can be obtained:

[0088]

[0089] Here, H can be called the homography matrix, which can be a 3×3 matrix and can be represented as:

[0090]

[0091] Based on the mapping relationship between the three-dimensional and two-dimensional coordinates of the target object, the following system of equations can be obtained through matrix operations:

[0092]

[0093] Based on this, this application can use a homogeneous coordinate system and introduce constraint h into the above equations. 33=1, which can be used to jointly solve the three-dimensional and two-dimensional coordinates of a set of feature points of each target object. For example, by using singular value decomposition, the other 8 unknowns contained in the homography matrix corresponding to the target object can be solved to obtain the homography matrix of the target object.

[0094] Subsequently, based on the coordinate system transformation relationship described above and the relationship between each transformation matrix, and according to the homography matrices of the multiple target objects, the internal parameter matrix and external parameter matrix of the image acquisition device are calculated and determined as the calibration parameters of the image acquisition device. The calculation process is not detailed in this application.

[0095] In summary, during the calibration of an image acquisition device at a fixed location in a specified scene, target object detection can be directly performed on the image of the specified scene acquired by the image acquisition device. This identifies multiple target objects located at different positions within the specified scene, predicts the 3D attributes of each target object, and then obtains the 3D coordinates of a set of feature points of the target object, i.e., coordinates in the world coordinate system. Additionally, the specified scene image can be directly analyzed to obtain the 2D coordinates of a set of feature points of the target object, i.e., pixel coordinates in the pixel coordinate system of the image. Thus, based on the coordinate system transformation relationship of the image acquisition device—that is, the coordinate transformation relationship between the world coordinate system and the pixel coordinate system for the same feature point—the 3D and 2D coordinates of a set of feature points for each of the multiple target objects can be calculated to obtain the calibration parameters of the image acquisition device. This calibration process does not require manual intervention to continuously adjust the image acquisition angle of the image acquisition device, nor does it require a calibration board, thus improving calibration efficiency and accuracy and expanding the applicable scenarios for image acquisition device calibration.

[0096] Reference Figure 4 This is a flowchart illustrating an optional embodiment of the image acquisition device calibration method proposed in this application. This embodiment describes an optional refined implementation of the image acquisition device calibration method proposed above. In practical applications, to improve calibration efficiency and reliability, this application can filter objects in a specified scene image to determine objects that meet the calibration conditions as target objects. The implementation process includes, but is not limited to, the filtering method described in this embodiment. Figure 4 As shown, the method may include:

[0097] Step S41: Obtain images of a specified scene at different time frames captured by the image acquisition device; the image acquisition device is deployed at any fixed location in the specified scene.

[0098] In practical applications, for specific scenarios such as traffic scenes, the number and location of objects in a single frame of the specified scene image are not stable and may not meet the required number of target objects for calibration. Therefore, this application can obtain specified scene images at different time frames acquired by the same image acquisition device, and then determine the target objects required for calibration by analyzing all objects contained in the specified scene image of each time frame. It should be noted that the acquisition parameters of the image acquisition device fixedly installed in the specified scene remain unchanged during the image acquisition process, and are usually fixed after installation and debugging.

[0099] In some other embodiments proposed in this application, based on the above analysis, this embodiment can also select specified scene images from different time frames in a longer surveillance video. For example, first, a first video file acquired by the image acquisition device for the specified scene is determined, and the video length of the first video file is longer than a preset length, such as 30 minutes, 1 hour, or 2 hours. This application does not limit the value of the preset length and can determine it according to the specified scene type. Then, specified scene images from different time frames can be extracted from the first video file, and the target object can be determined according to the method described below. The method for acquiring specified scene images from different time frames in this application includes, but is not limited to, the method described above.

[0100] Step S42: Perform object detection on the specified scene image of each time frame to obtain the two-dimensional attributes of each object contained in the specified scene image;

[0101] Step S43: Based on the two-dimensional attributes of each object, determine the target objects that meet the calibration conditions in the specified scene images of different time frames;

[0102] For each time frame of a specified scene image captured by the image acquisition device, an algorithm such as a convolutional neural network can be used to detect objects present in that specified scene image. For example, detecting vehicles on a traffic road from a traffic scene image, which is the region of interest in that specified scene image. The two-dimensional attributes of the object can include, for example, Figure 2 The two-dimensional bounding box shown can specifically refer to at least one of the following: the number of pixels contained in the two-dimensional bounding box, its content, and its positional proportion in the specified scene image. The two-dimensional attributes of the object to be detected can be determined based on preset calibration conditions.

[0103] Optionally, after detecting each object and its two-dimensional attributes from the specified scene image of each time frame, further filtering can be performed according to preset calibration conditions to determine multiple target objects that meet the calibration conditions from the detected objects. These multiple target objects typically come from the specified scene images of multiple time frames, and each specified scene image of a time frame does not necessarily contain one or more target objects, depending on the situation. The target objects that meet the calibration conditions can include, but are not limited to, at least one of the following implementation methods:

[0104] The number of pixels occupied by the two-dimensional bounding box of the detected object is obtained. If this number of pixels reaches a preset threshold, the corresponding object can be considered a target object that meets the calibration conditions. This preset threshold can be determined based on the total number of pixels contained in the specified scene image, such as 900 or 1600, etc. This application does not limit the value of this preset threshold. It is understood that this application can use this method to identify objects with a small number of pixels (i.e., small display size) in the detected specified scene image, thus avoiding the use of such objects for image acquisition device calibration, which would increase the calibration result error due to larger detection error.

[0105] In other words, objects with smaller display sizes in a specified scene image will have relatively large errors in the detection results of their three-dimensional and two-dimensional coordinates. If such objects are continued to be used to calculate the calibration parameters of the image acquisition device, the accuracy and reliability of the calibration parameters will be reduced. Therefore, this application can use the above method to filter out target objects with larger display sizes in a specified scene image to achieve the subsequent image processing device calibration steps.

[0106] Furthermore, if the predicted calibration information of an object in a specified scene image is not fully displayed, the accuracy of the calibration result will be reduced if the calibration information is used to calculate the calibration parameters of the image acquisition device. Therefore, this application can determine the object whose distance between the two-dimensional bounding box and the boundary of the specified scene image is greater than a preset distance threshold from the multiple detected objects. In other words, it determines the object located in the middle or slightly below the middle of the specified scene image as the target object and filters out the object located at the edge of the specified scene image.

[0107] Based on the above analysis, this application can also identify target objects whose two-dimensional bounding boxes do not overlap, that is, filter out multiple objects that overlap in a specified scene image, and avoid the adverse effects of detection error of such objects on the calibration results.

[0108] In summary, following the method described above, after detecting multiple objects from specified scene images at different time frames, such as vehicles driving on the road in a traffic scene, objects that are too small in display size, located at the edge of the image, and / or overlap with other objects can be filtered out, and the remaining objects can be identified as target objects. In other words, this application can select to display target objects with larger size and not overlap with other objects in the middle or slightly below the middle of the specified scene image. Of course, as needed, objects that meet but are not limited to some (one or more) of the calibration conditions described above can also be selected as target objects to ensure the reliability and accuracy of the calibration results achieved accordingly. The implementation process will not be described in detail in this application.

[0109] Step S44: Based on the target object detection results, obtain the three-dimensional attributes of each target object; the three-dimensional attributes include at least the predicted shape value of the corresponding target object;

[0110] Following the method described above, after identifying the target object in the specified scene image that meets the calibration conditions, a monocular 3D detection algorithm or other 3D detection algorithms can be used to predict the 3D attributes of the target object in the specified scene image. For example, the 3D bounding box of the target object in the specified scene image can be determined, and then the side length, center and / or shape of the 3D bounding box can be identified to obtain the predicted 3D attributes of the corresponding target object. However, this method of 3D attribute prediction is not limited and no restrictions are imposed.

[0111] Step S45: Based on the three-dimensional attributes of each target object, obtain the three-dimensional coordinates of a set of feature points corresponding to the target object; a set of feature points includes multiple feature points on the same plane;

[0112] Step S46: Obtain a set of feature points of each target object mapped to two-dimensional coordinates in the specified scene image;

[0113] As analyzed above, for each target object detected in a specified scene image, this application can determine a feature point of the target object (such as any feature point in the aforementioned set of feature points) as the origin, and extend the connecting line from that feature point to other feature points in the same set of feature points as coordinate axes, thus constructing the world coordinate system of the target object itself. Based on the detected 3D bounding box of the target object, the 3D coordinates of a set of feature points of the target object in its world coordinate system can be obtained. Figure 2 The world coordinates (X, Y, Z) of the four vertices of the bottom bounding box of the target vehicle in the traffic scene shown are in its own world coordinate system.

[0114] Furthermore, after determining a set of feature points for each target object using the above method, the two-dimensional coordinates of the feature points mapped to the specified scene image can be determined, that is, the pixel coordinates (u,v) of the set of feature points in the pixel coordinate system of the specified scene image. This application does not limit the method for obtaining the three-dimensional and two-dimensional coordinates of each target object, including but not limited to the implementation method described above.

[0115] It should be noted that for each target object in the same and / or different specified scene images, their respective world coordinate systems can be constructed in the same way. That is, feature points at the same location of the target object are selected as the origin, and the extensions of the lines connecting these feature points to other feature points are used as coordinate axes to form the world coordinate system of the target object. Of course, different methods can be used to construct the corresponding world coordinate systems for different target objects. This application does not limit the implementation method of constructing the world coordinate system for each target object, and it can be made consistent with the two-dimensional coordinates in the two-dimensional coordinate system on the image plane. The embodiments of this application use the same method of constructing the world coordinate system as an example for illustration.

[0116] Step S47: Based on the coordinate system transformation relationship for the image acquisition device, calculate the three-dimensional coordinates and two-dimensional coordinates of a set of feature points for each of the multiple target objects to obtain the homography matrix of the corresponding target object;

[0117] Based on the description of the corresponding part of the above embodiment, assuming that the four bottom vertices of each target object are selected as a group of feature points, and that the Z-axis coordinate of each target object in its own world coordinate system is 0, after determining the coordinate system transformation relationship described above, the three-dimensional coordinates and two-dimensional coordinates of the same feature point of each target object are substituted into the above mapping formula. After solving multiple equations together, the homography matrix H of the target object is obtained. The calculation process will not be described in detail in this embodiment.

[0118] Step S48: Obtain the initial internal parameter matrix of the image acquisition device based on the homography matrix of each of the multiple target objects;

[0119] Based on the above description of the transformation process between different coordinate systems, the homography matrix H of each target object has the following transformation relationship:

[0120] H=[h1 h2 h3]=λK·[r1 r2 t]

[0121] Here, r1 and r2 are both rotation matrix vectors, i.e., mutually orthogonal unit vectors. These two vectors have the same norm, which is equal to 1. Therefore, the following relationship exists between these two vectors:

[0122]

[0123] ||r1||=||r2||=1

[0124] Substituting the transformation relationship of H above into the above formula, we can obtain:

[0125]

[0126]

[0127] Assume K -T K -1 =B, then we can obtain the constraints used to calculate the internal parameter matrix of the image acquisition device, that is, K -T K -1 Substituting B into the above formula, we can obtain:

[0128]

[0129] Based on the description of the quadratic transformation matrix K above, this matrix K is an upper triangular matrix, and K and B are related by K... - T K -1 =B, where matrix B is a symmetric matrix. Based on the above analysis, matrix B contains six unknowns about the parameters of the image acquisition device. The homography matrices H of at least three target objects can be substituted into the matrix, and the resulting multiple equations can be solved jointly to obtain the quadratic transformation matrix K of the image acquisition device. Based on the above transformation relationship, the initial internal parameter matrix K·F of the image acquisition device can be obtained. The calculation process is not detailed in this application.

[0130] It should be noted that, in order to ensure the accuracy and reliability of the calibration results, more than 20 target objects can usually be identified according to the method described above. Then, multiple sets of three-dimensional and two-dimensional coordinates of these target objects with mapping relationships are substituted into the mapping formula that represents the coordinate system transformation relationship to obtain the initial internal parameter matrix of the image acquisition device.

[0131] Step S49: Obtain the initial external parameter matrix of the image acquisition device based on the initial internal parameter matrix and the homography matrices of each of the multiple target objects;

[0132] Step S410: Obtain the calibration parameters of the image acquisition device using the initial internal parameter matrix and the initial external parameter matrix;

[0133] For the image acquisition device to be calibrated in this application, after fixing it at any position in the specified scene and configuring the parameters, the internal parameters of the image acquisition device will remain unchanged. In this way, after obtaining the homography matrix H of each target object and the quadratic transformation matrix K in the internal parameter matrix according to the method described above, the initial external parameter matrix of the image acquisition device can be obtained by combining the transformation relationship between H, K and RT.

[0134] Based on the transformation relationships between H, K, and RT mentioned above, we can derive the expressions for the rotation matrix vectors r1 and r2, as well as the elements of the homography matrix:

[0135]

[0136]

[0137] r3 = r1 × r2

[0138]

[0139] Substituting the quadratic transformation matrix K obtained above, and the homography matrices H of each of the multiple target objects into the above formula, the external parameter matrix RT of the image acquisition device is solved. Then, the initial internal parameter matrix and the initial external parameter matrix of the image acquisition device obtained from the solution can be used to construct the calibration parameters of the image acquisition device.

[0140] Therefore, it can be seen that in the process of obtaining calibration parameters, there is no need to adjust the shooting parameters of the image acquisition device. After fixing it to the specified scene, multiple time frames of the specified scene images are acquired. Using the target object detection method, multiple target objects that meet the calibration conditions are identified. A world coordinate system is constructed using the feature points of each target object to obtain the three-dimensional coordinates of each feature point, and the two-dimensional coordinates of each feature point mapped to the corresponding specified scene image are obtained. Substituting the three-dimensional coordinates and two-dimensional coordinates with the mapping relationship into the coordinate transformation relationship for the image acquisition device, the calibration parameters of the image acquisition device can be directly solved without the need for auxiliary tools such as calibration boards, reducing scene restrictions and making the image acquisition device calibration method proposed in this application applicable to more scenarios.

[0141] Step S411: Determine that the calibration parameter does not meet the calibration parameter requirements, and adjust the calibration parameter based on the predicted shape value of each target object and the two-dimensional coordinates of a set of feature points to obtain the target calibration parameter of the image acquisition device.

[0142] In practical applications, the calibration parameters of the image acquisition device obtained by directly solving the method described above often have relatively large errors. In order to ensure that the obtained calibration parameters are accurate and reliable, the calibration parameters of the image acquisition device can be checked to see if they meet the calibration parameter requirements. The calibration parameter requirements may include that the attribute loss value of the target object obtained based on the calibration parameters (such as reprojection error) is less than a threshold (which can be a small value; this application does not limit its value and it can be determined according to the situation). This application does not limit the content of the calibration parameter requirements and can determine them according to the actual situation.

[0143] Based on this, if the solved calibration parameters meet the above calibration parameter requirements, that is, the attribute loss value (such as reprojection error) of the target object obtained based on the calibration parameters is less than the threshold, the calibration parameters can be determined as the target calibration parameters of the image acquisition device; conversely, if the calibration parameters do not meet the calibration parameter requirements, that is, the attribute loss value of the target object obtained based on the calibration parameters is greater than or equal to the threshold, it is necessary to optimize the internal parameter matrix and / or external parameter matrix of the calibration parameters according to the attribute loss value characterized by the calibration parameter requirements, so as to reduce the attribute loss value. In other words, this application can optimize the above calibration parameters based on the calibration parameters obtained this time by minimizing the reprojection error of the target object's attribute value, so as to meet the application requirements of the image acquisition device in a specified scenario.

[0144] Reference Figure 5 This is a flowchart illustrating an optional embodiment three of the image acquisition device calibration method proposed in this application. This embodiment describes an optional refined implementation of the above-mentioned image acquisition device calibration method, such as describing the optimization process of the initially solved calibration parameters, including but not limited to the implementation methods described in the embodiments below. Regarding the process of obtaining the initial calibration parameters, please refer to the description of the corresponding part of the above embodiments; this embodiment will not elaborate further. Figure 5 As shown, the method may include:

[0145] Step S51: Determine the shape prediction value of the spatial region where each target object is located; the shape prediction value includes the predicted length, predicted width, and predicted height;

[0146] Based on the description of the corresponding parts of the above embodiments, a monocular 3D detection algorithm can be used to analyze a specified scene image and predict the 3D attributes of the target object. This can include at least the shape prediction value of the target object. For example, target object detection can be performed on images collected in a traffic scene to predict the center, length, width, height, and direction of each vehicle. The implementation process will not be described in detail in this application.

[0147] Step S52: Based on the two-dimensional coordinates of the adjacent vertices of each target object, obtain the pixel distance between the corresponding adjacent vertices;

[0148] Step S53: Based on the initial calibration parameters of the image acquisition device, obtain the shape projection value of the pixel distance in the corresponding target object space;

[0149] In practical applications of this application, the initial calibration parameters of the image acquisition device refer to the calibration parameters obtained by direct solution as described above. Based on these initial calibration parameters, the shape projection values ​​of a set of feature points projected onto the corresponding target object can be obtained. Optionally, as described in steps S52 and S53 above, the set of feature points for each target object can be multiple vertices of any surface in the spatial region where the target object is located, such as the four vertices of the bottom bounding box of a vehicle, sequentially denoted as m0, m1, m2, and m3. In the specified scene image to which the target object belongs, the difference between the two-dimensional coordinates of two adjacent vertices can be calculated to obtain the pixel distance between these two vertices, i.e., the length of the bounding box where these two vertices are located. This may be the length, width, or height of the target object, which can be determined based on the position of these two adjacent vertices in the target object.

[0150] Subsequently, this application can obtain the two-dimensional coordinates of each vertex projected onto the world coordinate system of the corresponding target object, and the three-dimensional coordinates of each vertex projected onto the corresponding world coordinate system, which are denoted as the three-dimensional projected coordinates. Then, the difference operation is performed on the three-dimensional projected coordinates of the two adjacent vertices to obtain the shape projection value of the corresponding target pair, which is of the same type as the projection length, projection width, or projection height. The implementation process can refer to the transformation method described in the formula above. The transformation matrix used in this transformation process is the initial calibration parameter obtained in this solution. The implementation process is not described in detail.

[0151] Step S54: Obtain the error value between the predicted value of the same category shape and the projected value of the corresponding type shape for each target object;

[0152] Following the above analysis, this application can calculate the difference between the shape projection value of the target object obtained by reprojection and the shape prediction value obtained directly from the image to obtain the error value of the target object's shape of that type. This application can determine the shape projection value based on the shape category for which the error value needs to be calculated. Therefore, the shape projection values ​​corresponding to different adjacent vertices of the target object can include, but are not limited to: the length projection value and width projection value of the plane corresponding to the spatial region where the target object is located, or the length projection value and height projection value, or the width projection value and height projection value, that is, the bounding box projection value on any surface of the target object. Of course, it can also include the bounding box projection values ​​on multiple surfaces, that is, obtain the length projection value, width projection value, and height projection value, etc.

[0153] Step S55: Using the error value corresponding to the shape prediction value of each target object, obtain the second projection error of the corresponding target object;

[0154] Step S56: Calculate the average of the second projection errors of each of the multiple target objects to obtain the first projection error;

[0155] In some embodiments, the following reprojection error calculation method is used:

[0156]

[0157] Where n represents the number of target objects identified; l i This can represent the predicted length of the i-th target object obtained by performing target object detection on a specified scene image, w. i This can represent the predicted width of the i-th target object. This can represent the shape projection value of the pixel distance between two adjacent vertices projected onto the world coordinate system based on the calibration parameters of the i-th target object obtained above. This application can use, but is not limited to, the maximum likelihood estimation function to calculate this value; the implementation process is not detailed here. This application can use the reprojection error calculation formula described above to obtain the first reprojection error between the predicted shape value of the target object and the corresponding shape projection value, i.e., the average of the second reprojection errors of multiple target objects, but is not limited to the calculation method described in this application.

[0158] Optionally, the reprojection error corresponding to the predicted length and predicted width of the target object can be obtained according to the reprojection error calculation method described above; alternatively, the reprojection error corresponding to the predicted length and predicted height of the target object can be obtained, or the reprojection error corresponding to the predicted width and predicted height of the target object can be obtained. This can be determined based on the position of a set of feature points of the selected target object on the target object. The implementation process is not detailed in this application.

[0159] Step S57: By minimizing the first projection error, the calibration parameters are adjusted to obtain the target calibration parameters of the image acquisition device.

[0160] Following the above analysis, this application can use methods such as Newton's iteration method, LM algorithm, or other loss functions to iteratively solve for the approximate optimal solution with the minimum first projection error, and determine the corresponding calibration parameters as the target calibration parameters of the image acquisition device. This application will not describe the solution process in detail.

[0161] Optionally, during the above iterative solution process, the calibration parameters of the image acquisition device can be adjusted according to the changing trend of the first projection error. Then, based on the adjusted calibration parameters, the first projection error of the predicted values ​​of different shapes of the target object can be recalculated, and the changing trend of the first projection error can be updated. The calibration parameters of the image acquisition device can be adjusted accordingly until the latest obtained first projection error is less than the error threshold (which is a small value, and this application does not limit the size of the error threshold), or the change value of the first projection error is less than the change threshold (i.e., the first projection error gradually decreases and tends to stabilize), etc., and the iterative solution is stopped. The calibration parameters of the image acquisition device obtained at the end are determined as the target calibration parameters.

[0162] In summary, the method proposed in this application for optimizing the calibration parameters of an image acquisition device through reprojection error calculation involves directly reprojecting the two-dimensional coordinates of a set of feature points of a target object in a specified scene image to the world coordinate system of the target object. Following the coordinate system transformation relationship described above, and based on the calibration parameters obtained, the corresponding three-dimensional projection coordinates are calculated. The projection values ​​of the corresponding shape type, such as projection length, projection width, or projection height, are then calculated through difference calculations. Subsequently, the calibration parameters can be adjusted to minimize the error between the shape projection value and the shape prediction value directly predicted from the specified scene image. For example, the difference between the projection length and the predicted length of the same target object should be minimized, as should the difference between the projection width and the predicted width of the target object. After multiple iterative optimizations, the target calibration parameters of the image acquisition device are obtained.

[0163] As can be seen, compared with the optimization method of projecting the predicted three-dimensional coordinates of the target object onto the two-dimensional coordinates of the specified scene image and then comparing them with the two-dimensional coordinates of the corresponding feature points mapped on the specified scene image, and adjusting the initial calibration parameters based on the difference between the two, the reprojection error solution method proposed in this application only requires one projection, which improves the accuracy and reliability of the calibration parameters.

[0164] In some further embodiments of this application, in order to improve the accuracy of the calibration of the intrinsic and extrinsic parameters of the image acquisition device, this application can also pre-obtain three-dimensional key point templates for different types of target objects, such as... Figure 6 As shown, in the process of detecting target objects in a specified scene image, object attributes such as brand, model, and shape of the target object can be obtained. Based on the object attributes, a 3D key point template matching the object attributes is selected from a variety of pre-stored 3D key point templates for each object. Then, the 3D key point template can be projected onto the corresponding target object in the specified scene image to obtain the 2D coordinates of each 3D key point mapped onto the specified scene image. The implementation process is not detailed in this application.

[0165] Subsequently, in the process of solving the reprojection error, the two-dimensional coordinates of a set of feature points (or other three-dimensional key points, etc.) of each target object can be determined according to the method described above. The pixel distance between two corresponding feature points is obtained by difference calculation. Based on the calibration parameters that have been solved or adjusted above, the reprojection error of the length, width and / or height of the target object in space is calculated. By minimizing the reprojection error, the calibration parameters of the image acquisition device are adjusted until the target calibration parameters are obtained.

[0166] Optionally, in the above process of solving the reprojection error, the three-dimensional coordinates of a set of feature points of the target object (such as the four vertices on a plane, or the eight vertices of the three-dimensional bounding box of the target object) can be determined based on the selected three-dimensional key point template of the target object (this is more accurate than the three-dimensional coordinates predicted above), and their two-dimensional coordinates (i.e., the real pixel coordinates) mapped onto the specified scene image. Then, the first reprojection error is obtained according to the first reprojection error calculation method described above. By minimizing the first reprojection error, the calibration parameters of the image acquisition device are adjusted until the target calibration parameters are obtained. The implementation process can be referred to the description of the corresponding part of the above embodiment, which will not be described in detail here.

[0167] Reference Figure 7 This is a schematic diagram of an optional embodiment of the image acquisition device calibration equipment proposed in this application, as shown below. Figure 7 As shown, the calibration device for the image acquisition device may include:

[0168] The specified scene image acquisition module 71 is used to acquire a specified scene image acquired by the image acquisition device; the image acquisition device is deployed at any fixed position in the specified scene;

[0169] The 3D attribute acquisition module 72 is used to perform target object detection on the specified scene image and obtain the 3D attributes of multiple target objects; the multiple target objects are located at different positions in the specified scene.

[0170] The three-dimensional coordinate acquisition module 73 is used to obtain the three-dimensional coordinates of a set of feature points corresponding to each target object based on the three-dimensional attributes of each target object; the set of feature points includes multiple feature points located on the same plane;

[0171] The two-dimensional coordinate acquisition module 74 is used to obtain the two-dimensional coordinates of the set of feature points mapped to the specified scene image;

[0172] The calibration parameter acquisition module 75 is used to process the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects according to the coordinate system transformation relationship for the image acquisition device, so as to obtain the calibration parameters of the image acquisition device; the coordinate system transformation relationship can characterize the coordinate transformation relationship between the world coordinate system and the pixel coordinate system of any feature point in the specified scene image.

[0173] In some other embodiments, the three-dimensional attributes of the target object may at least include the predicted shape value of the corresponding target object. Based on this, such as Figure 8 As shown, the above-mentioned image acquisition device calibration equipment may further include:

[0174] The calibration parameter adjustment module 76 is used to determine that the calibration parameters do not meet the calibration parameter requirements, and adjust the calibration parameters according to the shape prediction value of each target object and the two-dimensional coordinates of the set of feature points to obtain the target calibration parameters of the image acquisition device.

[0175] Optional, such as Figure 8 As shown, the calibration parameter adjustment module 76 may include:

[0176] The shape projection value acquisition unit 761 is used to obtain the shape projection value of the two-dimensional coordinates of the set of feature points projected onto the corresponding target object according to the calibration parameters.

[0177] The first projection error acquisition unit 762 is used to obtain the first projection error between the predicted shape value and the corresponding projected shape value of the target object.

[0178] The target calibration parameter acquisition unit 763 is used to adjust the calibration parameters by minimizing the first projection error to obtain the target calibration parameters of the image acquisition device.

[0179] In one possible implementation, the predicted shape value of the target object includes the predicted length, predicted width, and predicted height of the spatial region where the target object is located; the set of feature points includes multiple vertices of any surface of the spatial region where the target object is located.

[0180] Based on this, the shape projection value obtaining unit 761 mentioned above may include:

[0181] The pixel distance acquisition unit is used to obtain the pixel distance between adjacent vertices based on the two-dimensional coordinates of the adjacent vertices of each target object;

[0182] The first obtaining unit is used to obtain the shape projection value of the pixel distance in the corresponding target object space based on the calibration parameters;

[0183] The shape projection values ​​corresponding to different adjacent vertices of the target object include: the length projection value and width projection value of the plane corresponding to the spatial region where the target object is located, or the length projection value and height projection value, or the width projection value and height projection value.

[0184] In some other embodiments, the first projection error obtaining unit 762 described above may include:

[0185] An error value acquisition unit is used to obtain the error value between the shape prediction value of the same category and the shape projection value of the corresponding type for each target object;

[0186] The second projection error obtaining unit is used to obtain the second projection error of the target object by using the error value corresponding to the shape prediction value of each target object;

[0187] The mean calculation unit is used to perform mean calculation on the second projection error of each of the plurality of target objects to obtain the first projection error.

[0188] Based on the descriptions of the embodiments above, the calibration parameter acquisition module 75 may include:

[0189] The homography matrix acquisition unit is used to perform calculations on the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects according to the coordinate system transformation relationship for the image acquisition device, so as to obtain the homography matrix of the corresponding target object;

[0190] An initial intrinsic parameter matrix obtaining unit is used to obtain the initial intrinsic parameter matrix of the image acquisition device based on the homography matrix of each of the plurality of target objects;

[0191] An initial extrinsic parameter matrix obtaining unit is used to obtain the initial extrinsic parameter matrix of the image acquisition device based on the initial intrinsic parameter matrix and the homography matrix of each of the plurality of target objects;

[0192] The calibration parameter acquisition unit is used to obtain the calibration parameters of the image acquisition device using the initial internal parameter matrix and the initial external parameter matrix.

[0193] Based on the image acquisition device calibration device described in the above embodiments, the specified scene image acquisition module 71 may include:

[0194] The second acquisition unit is used to acquire specified scene images at different time frames captured by the image acquisition device; or,

[0195] The first video file determination unit is used to determine a first video file acquired by the image acquisition device for a specified scene; the video duration of the first video file is greater than a preset duration.

[0196] The extraction unit is used to extract specified scene images from different time frames of the first video file.

[0197] In some other embodiments, the image acquisition device calibration device described above may further include:

[0198] The two-dimensional attribute acquisition module is used to obtain the two-dimensional attributes of each of the multiple objects contained in the specified scene image during the target object detection process of the specified scene image;

[0199] The target object determination module is used to determine the target object that meets the calibration conditions among the multiple objects based on the two-dimensional attributes of each of the multiple objects.

[0200] Optionally, if the two-dimensional attribute includes the two-dimensional bounding box of the corresponding object, the target object determination module may include at least one of the following determination units:

[0201] The first determining unit is used to obtain the number of pixels occupied by the two-dimensional bounding box of the object, and determine that the number of pixels reaches a preset number threshold; the preset number threshold is determined based on the total number of pixels contained in the specified scene image;

[0202] The second determining unit is used to determine that the distance between the two-dimensional bounding box of the object and the boundary of the specified scene image is greater than a preset distance threshold.

[0203] The third determining unit is used to determine that there is no overlapping area between the two-dimensional bounding boxes of different objects.

[0204] It should be noted that the various modules and units in the above-mentioned device embodiments can all be stored as program modules in the memory of a computer device. The processor of the computer device executes the program modules stored in the memory to achieve the corresponding functions. The functions achieved by each program module and its combination, as well as the technical effects achieved, can be referred to the description of the corresponding part of the above-mentioned method embodiments, which will not be repeated in this embodiment.

[0205] This application also provides a computer-readable storage medium on which at least one set of computer instructions can be stored. This set of computer instructions can be called and loaded by a processor to implement the image acquisition device calibration method described in the above embodiments. The implementation process can be referred to the description of the corresponding part of the above method embodiments.

[0206] Reference Figure 9This is a schematic diagram of the hardware structure of a computer device for an optional embodiment of the image acquisition device calibration method proposed in this application. The computer device can be a server, such as a single physical server, a service cluster composed of multiple physical servers, or a cloud server that supports cloud computing services. In some embodiments, the computer device can also be a terminal device with certain data processing capabilities, such as a laptop, desktop computer, intelligent transportation equipment, robot, etc. This application does not limit the product type of the computer device and can be used as appropriate.

[0207] Based on the above description of the calibration method for image acquisition devices, such as Figure 9 As shown, the computer device that performs the image acquisition device calibration method may include: a communication module 91, a memory 92, and a processor 93, wherein:

[0208] The number of each of the communication module 91, memory 92, and processor 93 can be at least one, and they can be connected to each other via a bus, but are not limited to this. Figure 9 The connection method shown, and its connection method with other components of the computer device, are not described in detail in this application. The bus may include an address bus, a data bus, a control bus, etc. Figure 9 Not all of them are shown; they can be determined based on actual needs, and this application will not elaborate on them.

[0209] The communication module 91 may include a communication module capable of data interaction using a wireless communication network, such as a WIFI module, a 5G / 6G (fifth-generation mobile communication network / sixth-generation mobile communication network) module, a GPRS module, etc., to realize communication between the computer device and the image acquisition device or other terminal devices or servers, and to receive images of a specified scene acquired by the image acquisition device. It should be understood that the communication module 91 may also include a communication interface for data interaction between internal components of the computer device, such as a USB interface, a serial / parallel port, etc. This application does not limit the specific content included in the communication module 91.

[0210] The memory 92 can be used to store a program that implements the image acquisition device calibration method proposed in this application; the processor 93 can be used to load and execute the program stored in the memory 92 to implement the image acquisition device calibration method proposed in this application. The implementation process will not be described in detail in this embodiment.

[0211] In practical applications of this application, memory 92 can be non-volatile memory, such as hard disk drive (HDD) or solid-state drive (SSD), or it can be volatile memory, such as random-access memory (RAM). As analyzed above, memory 92 can be any available storage medium accessible to processor 93 and / or computer device. It should be understood that memory 92 in this application can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data. This application does not limit the type of memory 92 or its physical connection relationship with the other devices mentioned above; it can be determined as appropriate.

[0212] Optionally, the processor 93 may include one or more processors to execute the image acquisition device calibration method proposed in this application. The processor may include, but is not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a digital signal processing device (DSPD), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device (PLD), a central processing unit (CPU), processing circuits, or other suitable hardware, firmware, and / or combinations of hardware and software. This application does not limit the composition and structure of the processor 93 of the computer device.

[0213] It should be understood that, Figure 9 The structure of the computer device shown does not constitute a limitation on the computer device in the embodiments of this application. In practical applications, the computer device may include more than Figure 9 In addition to the components shown, or combinations of certain components, if the computer device is a terminal device, the computer device may also include a display screen, microphone, speaker, camera, antenna, power management module, various sensors, etc., which will not be listed here.

[0214] Finally, it should be noted that, regarding the above embodiments, relational terms such as "first" and "second" are merely used to distinguish one operation, unit, or module from another, and do not necessarily require or imply any such actual relationship or order between these units, operations, or modules. Furthermore, they should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0215] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0216] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, or system that includes said element.

[0217] Furthermore, the various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus and computer devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0218] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A calibration method for an image acquisition device, the method comprising: To obtain images of a specified scene captured by an image acquisition device; The image acquisition device is deployed at any fixed location within the specified scene; Target object detection is performed on the specified scene image to obtain the three-dimensional attributes of multiple target objects; the multiple target objects are located at different positions in the specified scene image. Based on the three-dimensional attributes of each target object, obtain the three-dimensional coordinates of a set of feature points corresponding to the target object; the set of feature points includes multiple feature points located on the same plane; Obtain the two-dimensional coordinates of the set of feature points mapped to the specified scene image; Based on the coordinate system transformation relationship of the image acquisition device, the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects are processed to obtain the calibration parameters of the image acquisition device; the coordinate system transformation relationship can characterize the coordinate transformation relationship between the world coordinate system and the pixel coordinate system of any feature point in the specified scene image. Wherein, the three-dimensional attributes include at least the shape prediction value of the corresponding target object, and the shape prediction value of the target object includes the predicted length, predicted width, and predicted height of the spatial region where the target object is located. The method further includes: Based on the two-dimensional coordinates of the adjacent vertices of each target object, the pixel distance between the corresponding adjacent vertices is obtained; Based on the calibration parameters, the shape projection value of the pixel distance on the corresponding target object space is obtained; If the calibration parameters are determined not to meet the calibration parameter requirements, the calibration parameters are adjusted based on the shape prediction value and the corresponding shape projection value of each target object to obtain the target calibration parameters of the image acquisition device.

2. The method according to claim 1, wherein adjusting the calibration parameters to obtain the target calibration parameters of the image acquisition device comprises: The first projection error between the predicted shape value and the corresponding projected shape value of the target object is obtained; By minimizing the first projection error, the calibration parameters are adjusted to obtain the target calibration parameters of the image acquisition device.

3. The method according to claim 2, wherein the set of feature points includes multiple vertices of any surface of the spatial region where the target object is located; in, The shape projection values ​​corresponding to different adjacent vertices of the target object include: the length projection value and width projection value of the plane corresponding to the spatial region where the target object is located, or the length projection value and height projection value, or the width projection value and height projection value.

4. The method according to claim 2, wherein obtaining the first reprojection error between the predicted shape value and the corresponding projected shape value of the target object includes: Obtain the error value between the predicted shape value of the same category and the projected shape value of the corresponding type for each target object; By using the error value corresponding to the shape prediction value of each target object, the second projection error of the target object is obtained. The first projection error is obtained by averaging the second projection errors of each of the plurality of target objects.

5. The method according to any one of claims 1-4, wherein processing the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects based on the coordinate system transformation relationship of the image acquisition device to obtain the calibration parameters of the image acquisition device includes: Based on the coordinate system transformation relationship of the image acquisition device, the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects are calculated to obtain the homography matrix of the corresponding target object; Based on the homography matrix of each of the plurality of target objects, the initial internal parameter matrix of the image acquisition device is obtained; Based on the initial internal parameter matrix and the homography matrix of each of the plurality of target objects, the initial external parameter matrix of the image acquisition device is obtained; The calibration parameters of the image acquisition device are obtained using the initial internal parameter matrix and the initial external parameter matrix.

6. The method according to any one of claims 1-4, wherein obtaining the specified scene image acquired by the image acquisition device comprises: To obtain images of a specified scene at different time frames captured by the image acquisition device; or, Determine the first video file captured by the image acquisition device for the specified scene; The duration of the first video file is longer than the preset duration; Extract images of a specified scene from different time frames of the first video file.

7. The method according to any one of claims 1-4, further comprising, during the target object detection process of the specified scene image: Obtain the two-dimensional attributes of each of the multiple objects contained in the specified scene image; Based on the two-dimensional attributes of each of the plurality of objects, determine the target object among the plurality of objects that meets the calibration conditions; The two-dimensional attributes include the two-dimensional bounding box of the corresponding object. Determining the target object that satisfies the calibration conditions among the multiple objects based on their respective two-dimensional attributes includes at least one of the following implementation methods: Obtain the number of pixels occupied by the two-dimensional bounding box of the object, and determine that the number of pixels reaches a preset number threshold; the preset number threshold is determined based on the total number of pixels contained in the specified scene image; It is determined that the distance between the two-dimensional bounding box of the object and the boundary of the specified scene image is greater than a preset distance threshold; It is determined that there are no overlapping regions between the two-dimensional bounding boxes of different objects.

8. An image acquisition device calibration device, the device comprising: The specified scene image acquisition module is used to acquire specified scene images captured by the image acquisition device; The image acquisition device is deployed at any fixed location within the specified scene; A 3D attribute acquisition module is used to perform target object detection on the specified scene image and obtain the 3D attributes of multiple target objects; the multiple target objects are located at different positions in the specified scene. The three-dimensional coordinate acquisition module is used to obtain the three-dimensional coordinates of a set of feature points corresponding to each target object based on the three-dimensional attributes of each target object; the set of feature points includes multiple feature points located on the same plane; A two-dimensional coordinate acquisition module is used to obtain the two-dimensional coordinates of the set of feature points mapped to the specified scene image; The calibration parameter acquisition module is used to process the three-dimensional coordinates and the two-dimensional coordinates of a set of feature points of each of the plurality of target objects according to the coordinate system transformation relationship for the image acquisition device, so as to obtain the calibration parameters of the image acquisition device; the coordinate system transformation relationship can characterize the coordinate transformation relationship between the world coordinate system and the pixel coordinate system of any feature point in the specified scene image; The three-dimensional attributes include at least the predicted shape value of the corresponding target object. The predicted shape value of the target object includes the predicted length, predicted width, and predicted height of the spatial region where the target object is located. The calibration parameter adjustment module is used to obtain the pixel distance between the adjacent vertices based on the two-dimensional coordinates of the adjacent vertices of each target object; obtain the shape projection value of the pixel distance on the corresponding target object space based on the calibration parameters; determine that the calibration parameters do not meet the calibration parameter requirements, and adjust the calibration parameters based on the predicted shape value and the corresponding shape projection value of each target object to obtain the target calibration parameters of the image acquisition device.

9. A computer device, the computer device comprising: Communication module; A memory for storing a program that implements the image acquisition device calibration method as described in any one of claims 1-7; A processor is configured to load and execute the program stored in the memory to implement the image acquisition device calibration method as described in any one of claims 1-7.

Images