A camera extrinsic parameter calibration method, device and equipment

Abstract

The application provides a camera extrinsic parameter calibration method, device and equipment. The method comprises the following steps: acquiring image pixel coordinates of a real-time dynamic measurement (RTK) device in video image data, wherein the video image data is a calibration process video record acquired by a to-be-calibrated camera; acquiring positioning data output by the RTK device, wherein the positioning data comprises at least one latitude and longitude information of the RTK device; establishing a corresponding relationship between the image pixel coordinates and the latitude and longitude information, wherein the image pixel coordinates and the latitude and longitude information correspond to each other; and determining a camera extrinsic parameter matrix according to the corresponding image pixel coordinates and latitude and longitude information. According to the scheme of the application, the calibration process can be completed without the cooperation of multiple persons, the calibration process is simple, and the labor cost is relatively small.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication technology, and in particular to a camera extrinsic parameter calibration method, device and equipment. BACKGROUND

[0002] Camera extrinsic parameter calibration refers to obtaining a transformation matrix from a world coordinate system to a camera coordinate system of a camera after installation of the camera. The transformation matrix is also called camera pose or camera extrinsic parameter. Camera extrinsic parameter is a necessary condition for connecting the position of an object in the real world with the pixel position of the object in an image, and is widely used in fields that need to locate the position of an object in an image, such as autonomous driving, security, intelligent traffic monitoring and control, etc.

[0003] In the prior art, when calibrating the extrinsic parameters of a camera that has been installed on a rod, 4-20 calibration points are generally preset in the image captured by the camera, and 2-3 people work together, one or two people carry a real-time dynamic measurement (RTK) device to the scene to obtain the latitude and longitude coordinates at the preset calibration points. One person is responsible for obtaining real-time video streams of the camera using tools in front of a computer and maintaining communication with the on-site personnel to supervise whether the on-site personnel accurately arrive at the preset calibration points. In this scheme, multiple people are needed to complete the calibration process, and the calibration process is cumbersome and labor-intensive. SUMMARY

[0004] The present application provides a camera extrinsic parameter calibration method, device and equipment, which solves the problem of high labor cost when calibrating the extrinsic parameters of a camera that has been installed on a rod.

[0005] In a first aspect, an embodiment of the present application provides a camera extrinsic parameter calibration method, comprising:

[0006] Obtaining image pixel coordinates of a real-time dynamic measurement (RTK) device in video image data, wherein the video image data is a calibration process video recording obtained by a camera to be calibrated;

[0007] Obtaining positioning data output by the RTK device, wherein the positioning data includes at least one latitude and longitude information of the RTK device;

[0008] Establishing a corresponding relationship between the image pixel coordinates and the latitude and longitude information, wherein the image pixel coordinates and the latitude and longitude information correspond to each other one by one;

[0009] Determining a camera extrinsic parameter matrix according to the image pixel coordinates and the latitude and longitude information corresponding to each other.

[0010] Optionally, the step of obtaining image pixel coordinates of a real-time dynamic measurement (RTK) device in video image data comprises:

[0011] perform target detection on each frame of video image in the video image data;

[0012] obtain a stay duration of the RTK device when the RTK device is detected;

[0013] obtain image pixel coordinates of the RTK device in the video image frame according to the stay duration.

[0014] Optionally, the obtaining of the image pixel coordinates of the RTK device in the video image data according to the stay duration comprises:

[0015] obtain a use posture of the RTK device when the stay duration of the RTK device in a first pixel region is greater than a first threshold value; wherein the first pixel region is any pixel region in a video image frame displaying the RTK device;

[0016] obtain the image pixel coordinates of the RTK device according to a coordinate range of the first pixel region when the use posture is an upright posture.

[0017] Optionally, the obtaining of the stay duration of the RTK device comprises:

[0018] obtain M continuous video image frames including the RTK device from N video image frames in a first target duration; N≥M>0, N and M are integers;

[0019] obtain a center point coordinate of the RTK device in the M continuous video image frames; wherein each video image corresponds to a center point coordinate;

[0020] determine whether the RTK device moves in a second target duration according to M center point coordinates; wherein the second target duration is a duration corresponding to the M continuous video image frames;

[0021] if not, the second target duration is taken as the stay duration.

[0022] Optionally, the determining of whether the RTK device moves in the second target duration according to M center point coordinates comprises:

[0023] calculate a horizontal variance according to horizontal coordinates of M center point coordinates;

[0024] calculate a vertical variance according to vertical coordinates of M center point coordinates;

[0025] determine that the RTK device does not move in the second target duration when the horizontal variance is less than a second threshold value and the vertical variance is less than a third threshold value.

[0026] Optionally, after the image pixel coordinates of the RTK device are obtained according to the coordinate range of the first pixel region, the method further comprises:

[0027] determining a Euclidean distance between the first pixel coordinate and the second pixel coordinate; wherein the first pixel coordinate is an image pixel coordinate of the RTK device in an i-th frame of video image; the second pixel coordinate is an image pixel coordinate of the RTK device in an i-1-th frame of video image; i is a positive integer;

[0028] in a case where the Euclidean distance is greater than or equal to a fourth threshold value, saving the first pixel coordinate.

[0029] Optionally, after the coordinate corresponding to the first pixel region is saved, the method further comprises:

[0030] in a case where the staying duration is greater than a fifth threshold value, obtaining a use posture of the RTK device; wherein the fifth threshold value is greater than the first threshold value;

[0031] in a case where the use posture is a horizontal posture, deleting the saved first pixel coordinate.

[0032] In a second aspect, an embodiment of the present application provides a camera extrinsic parameter calibration device, comprising a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the camera extrinsic parameter calibration method according to the first aspect.

[0033] In a third aspect, an embodiment of the present application provides a camera extrinsic parameter calibration apparatus, comprising:

[0034] a first obtaining module configured to obtain image pixel coordinates of a real-time kinematic (RTK) device in video image data, the video image data being video recording data of a calibration process obtained by a camera to be calibrated;

[0035] a second obtaining module configured to obtain positioning data output by the RTK device, the positioning data comprising at least one latitude and longitude information of the RTK device;

[0036] a third obtaining module configured to establish a corresponding relationship between the image pixel coordinates and the latitude and longitude information, wherein the image pixel coordinates and the latitude and longitude information correspond to each other in one-to-one manner;

[0037] a first determining module configured to determine a camera extrinsic parameter matrix according to the image pixel coordinates and the latitude and longitude information corresponding to each other.

[0038] In a third aspect, embodiments of the present application provide a computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the steps of the camera extrinsic parameter calibration method according to the first aspect.

[0039] The above technical solution of the present application has the following advantages:

[0040] In the above solution, the video recording of the calibration process only needs one staff to carry the RTK device to complete, after the on-site calibration is completed, the latitude and longitude coordinates output by the RTK device are exported, and the corresponding relationship between the image pixel coordinates of the RTK device in the video image data and the latitude and longitude coordinates is established, so that the camera extrinsic parameter matrix can be obtained. In this way, a single person can complete the calibration task, and the calibration process does not need to set the preset point, and the suitable point can be selected on site for calibration, and the image pixel coordinates are obtained without human error, so that the camera extrinsic parameter can be automatically generated, and the labor cost is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 One of the flowcharts of the camera extrinsic parameter calibration method according to the embodiments of the present application;

[0042] Figure 2 The second flowchart of the camera extrinsic parameter calibration method according to the embodiments of the present application;

[0043] Figure 3 The structural block diagram of the camera extrinsic parameter calibration device according to the embodiments of the present application;

[0044] Figure 4 The hardware structure schematic diagram of the camera extrinsic parameter calibration device according to the embodiments of the present application. DETAILED DESCRIPTION

[0045] In order to make the technical problems, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the accompanying drawings. In the following description, specific details such as specific configurations and components are provided only to help a comprehensive understanding of the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, in order to be clear and concise, the description of known functions and structures is omitted.

[0046] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0047] In various embodiments of the present application, it should be understood that the size of the serial number of the following processes does not mean the order of execution, and the execution order of the processes should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0048] In addition, the terms "system" and "network" are often used interchangeably herein.

[0049] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that the determination of B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0051] Specifically, the embodiments of the present application provide a camera extrinsic parameter calibration method, device and equipment, which solves the problem of high labor cost in calibrating the extrinsic parameters of a camera mounted on a rod in the prior art.

[0052] First embodiment

[0053] As shown in Figure 1 The embodiments of the present application provide a camera extrinsic parameter calibration method, which specifically includes the following steps:

[0054] Step 101: Obtain the image pixel coordinates of a real-time dynamic measurement RTK device in video image data, wherein the video image data is a calibration process video recording obtained by a to-be-calibrated camera.

[0055] In specific implementation, the recording process of the video image data is as follows: a staff member obtains the latitude and longitude coordinates of a plurality of calibration points by an RTK device at a calibration site, and video collection of the calibration site is performed by a to-be-calibrated camera, so as to ensure that the entire process of on-site calibration of the staff member is recorded by video. The size of the video picture is generally consistent with the production environment or the maximum picture size of the camera. Here, the calibration points can not be pre-set, and suitable points can be selected on site for calibration, thereby reducing the workload.

[0056] Further, the video image data recorded by the camera is exported, and the structured information of the target (i.e., the RTK device) is output by the target and tracking detection algorithm, including: the lateral offset x and the longitudinal offset y of the image pixel position of the target, the width w and the height h of the target, and the ID of the target. Currently, only one valid category is detected, i.e., the RTK device and the long rod connected below the device. That is, the RTK device and the long rod connected below the device are taken as a whole detection target to obtain a target detection box, which is the position of the RTK device and the long rod connected below the device in the video image frame.

[0057] Step 102: Obtain the positioning data output by the RTK device, the positioning data including at least one latitude and longitude information of the RTK device.

[0058] It can be understood that the RKT device outputs the latitude and longitude coordinates of each selected calibration point position. One calibration point position corresponds to one latitude and longitude coordinate.

[0059] Step 103: Establish a corresponding relationship between the image pixel coordinates and the latitude and longitude information: wherein the image pixel coordinates and the latitude and longitude information correspond one-to-one.

[0060] In specific implementation, as shown in Figure 2 , the exported video image data (video image file) and the latitude and longitude information can be loaded into an automatic calibration program. The automatic calibration program buffers the video image frames and the latitude and longitude information to queues according to the time sequence of the calibration process, respectively. Here, when data is obtained based on the video image frame queue and the latitude and longitude queue, the first-in first-out principle is used, i.e., data elements are successively entered into the queue from the tail and successively out of the queue from the head, so that the data stored first is taken out first. Each time a video image frame is obtained from the head of the video image frame queue, the image pixel coordinates of the RTK device in the dotting state are obtained when the RTK device in the dotting state is detected in the video image frame, and the latitude and longitude information is correspondingly obtained from the head of the latitude and longitude information queue. The image pixel coordinates corresponding to each RTK device in the dotting state are combined with the head data of the latitude and longitude information queue in turn, so that a one-to-one corresponding relationship between the image pixel coordinates and the latitude and longitude information can be established.

[0061] When the latitude and longitude data queue is empty, it is considered that all calibration points have been paired with image pixel coordinates.

[0062] Optionally, the RTK device in the vertical posture and staying for a duration longer than a preset threshold can be confirmed as the RTK device in the dotting state. The vertical posture means that the long rod connected below the RTK device is perpendicular to the ground, and the RTK device is installed on the long rod.

[0063] Step 104: determining the camera extrinsic matrix according to the corresponding image pixel coordinates and the latitude and longitude information.

[0064] It should be noted that the camera extrinsic is a necessary condition for connecting the position of the object in the real world and the pixel position of the object in the image. Through the camera extrinsic, the conversion between the position coordinates of the object in the real world and the pixel position coordinates of the object in the image can be completed.

[0065] Since the image pixel coordinates are the position coordinates of the RKT device in the video image frame, and the latitude and longitude information is the position coordinates of the RKT device in the real world, the latitude and longitude information output by the RKT device corresponds to the position coordinates of the RKT device in the pixel coordinate system. Through N sets of corresponding image pixel coordinates and latitude and longitude information, the camera extrinsic matrix can be generated.

[0066] In specific implementation, all latitude and longitude coordinates are first converted to the Universal Transverse Mercator (UTM), so that the earth ellipsoid coordinates can be converted to the plane coordinate system. Then, the perspective transformation method is used to generate the perspective transformation matrix of the image pixel coordinates to the UTM, that is, the camera extrinsic matrix.

[0067] In formal use, the input image pixel coordinate point is calculated through the camera extrinsic matrix to obtain the coordinate on the UTM, and then converted to the latitude and longitude output. In this way, the image pixel coordinates can be converted to the latitude and longitude coordinates through the camera extrinsic matrix.

[0068] In the above embodiment, the video recording of the recording calibration process can be completed by only one worker carrying the RTK device. After the on-site calibration is completed, the latitude and longitude coordinates output by the RTK device are exported, and the corresponding relationship between the image pixel coordinates and the latitude and longitude coordinates of the RTK device in the video image data is established to obtain the camera extrinsic matrix. In this way, a single person can complete the calibration task, and the calibration process does not need to set a preset point. The suitable point can be selected on site for calibration, and the image pixel coordinates are obtained without human-induced error. The process is automated, and the labor cost is reduced.

[0069] In an embodiment of the present application, the image pixel coordinates of the real-time dynamic measurement RTK device in the video image data are obtained, including:

[0070] Target detection is performed on each frame of video image in the video image data.

[0071] When the RTK device is detected, the staying time of the RTK device is obtained.

[0072] According to the staying time, the image pixel coordinates of the RTK device in the video image frame are obtained.

[0073] Optionally, when the stay duration is greater than or equal to the preset threshold, it is determined that the RTK device is in the dotting state, otherwise it may be in the process of being carried or installed by the staff, such as the RTK device may be in the process of moving from one calibration point to the next calibration point.

[0074] In a specific implementation, when the target (RTK device) is detected, the pixel position area of the RTK device and the connected long rod in the video image frame can be located through the target detection frame, and the center point of the lower edge of the target detection frame can be used as the image pixel coordinates of the RTK device.

[0075] In addition, the target detection for each frame of video image in the video image data can include: using a deep learning network to detect the RTK device and the long rod connected to the device in the video image. In terms of network structure design, a lightweight network structure is used as a whole. The network is composed of a backbone, an FPN layer and a detection head. The backbone network is responsible for down-sampling the input image data to obtain high-dimensional features. The FPN layer fuses features at multiple feature map sizes, uses high-resolution low-level features and high-level semantic information to improve target detection accuracy and recall rate. The detection head is a network designed for target detection tasks. Specifically, a two-branch network design can be used, one branch outputs the center point coordinates (x, y) of the target detection frame and the distance from the center point to the two corners of the target (i.e. the distance from the center point to the long side and the width of the target detection frame). The other branch outputs the target category. After the input image is processed by the neural network and post-processed, the final output is the target detection frame and the category. This application only detects the RTK device, so there is only one category.

[0076] Further, the Hungarian algorithm can be used to maximize the matching between all detected targets in the current video image and the target queue in the tracker, using the shape and motion of the target detection frame as the measurement unit. The formula for measuring the distance between two target frames is defined as follows:

[0077]

[0078] Where T represents the tracking target information, D represents the detected target information, the target information includes the horizontal coordinate x and the vertical coordinate y of the target detection frame (i.e. the upper left corner of the target detection frame is located at the horizontal coordinate x and the vertical coordinate y in the pixel coordinate system), the width w and the height h of the target detection frame, a is the weight parameter of the shape measurement, β is the weight parameter of the motion measurement, and exp is the exponential function with the natural constant e as the base.

[0079] The RTK device in the video image can be detected through target tracking. The target detection box contains the RTK device and the long rod. After one detection and tracking is performed, the output includes the number of detected targets (the target refers to the RTK device), the offsets x and y of the upper left corner of the target detection box in the pixel coordinate system, the size w and h of the target detection box in the pixel coordinate system, the target ID, and the confidence of target tracking.

[0080] In the above embodiment, the RTK device in the video image frame can be obtained through the target detection algorithm, and whether the image pixel coordinates of the RKT device are obtained is determined according to the stay duration of the RTK device. The image pixel coordinates of the RKT device in movement are avoided to be obtained. Since the RKT device in movement does not perform dotting, the latitude and longitude information will not be output. Therefore, through the embodiment, it can be ensured that the image pixel coordinates obtained by the RTK device are all output with latitude and longitude information, the image pixel points and the latitude and longitude information are accurately corresponded, and false detection is avoided.

[0081] In a specific embodiment, when the RTK device is detected, the stay duration of the RTK device is obtained, including:

[0082] When the RTK device is detected, the number of detected RKT devices is determined. When the number is 1, the stay duration of the RTK device is obtained.

[0083] In a specific implementation, when the target (RTK device) is detected, the number of targets is determined. When the number of targets is equal to 0, it is considered that the current image does not have the RTK device that needs to be searched, the current sub-process is ended, and a new image is obtained from the image queue for target detection and tracking. When the number of targets is greater than 1, it is considered that there may be false detection and the like, and a window containing the current image frame will be popped up to request manual selection of the correct target detection box. After the selection is completed, the program will delete the remaining target detection boxes according to the manual selection result (for example, the target detection box selected manually can be retained by using the target ID). When the number of targets is equal to 1, it is considered that the RTK device that needs to be searched is located, and the subsequent steps are performed.

[0084] In a specific embodiment of the present application, the image pixel coordinates of the RTK device in the video image data are obtained according to the stay duration, including:

[0085] When the stay duration of the RTK device in the first pixel region is greater than a first threshold, the use posture of the RTK device is obtained. The first pixel region is any pixel region in the video image frame that displays the RTK device.

[0086] In the case where the use posture is a vertical posture, the image pixel coordinates of the RTK device are obtained according to the coordinate range of the first pixel region.

[0087] In a specific implementation, when the staying duration of the target detection frame at the same position is greater than a first threshold (for example, 6 seconds), the use posture estimation of the RTK device is started, for example, the video image frame data in the last 1 second can be used to estimate whether the RTK device and the long rod are in an upright posture or a horizontal posture. When the use posture neither satisfies the upright posture nor the horizontal posture, the process is ended. In the case where the staying duration of the RTK device in the first pixel region is greater than the first threshold and the use posture is the upright posture, the image pixel coordinates of the RTK device are obtained according to the coordinate range of the first pixel region.

[0088] The first pixel region can refer to the pixel coordinate range corresponding to the target detection frame. The lower edge center point coordinates of the target detection frame can be used as the image pixel coordinates of the RTK device.

[0089] Specifically, the upright posture can be understood as a "dotting" signal. The determination method of the use posture can be: according to the shape particularity of the target detection frame, for example, the width and height of the target detection frame have a large ratio, when the height is greater than or equal to 1.5 times (reference value, which can be customized) of the width, it is considered that the target is in the upright posture. When the width is greater than or equal to 1.5 times (reference value, which can be customized) of the height, it is considered that the target is in the horizontal posture.

[0090] In the above embodiment, since the RTK device in the dotting state is in the upright state, whether to obtain the image pixel coordinates of the RKT device is determined through the staying duration and the use posture of the RTK device, which can avoid obtaining the image pixel coordinates of the RTK device in the dotting state, avoid false detection, and realize accurate correspondence between the image pixel points and the latitude and longitude information.

[0091] In a specific embodiment, the obtaining of the staying duration of the RTK device includes:

[0092] From N frames of video images in the first target duration, M continuous video image frames including the RTK device are obtained; N≥M>0, N and M are integers;

[0093] The center point coordinates of the RTK device in the M continuous video image frames are obtained; wherein each frame of video image corresponds to a center point coordinate;

[0094] According to the M center point coordinates, whether the RTK device moves in a second target duration is determined; wherein the second target duration is the duration corresponding to the M continuous video image frames;

[0095] If not, the second target duration is used as the staying duration.

[0096] In a specific implementation, the position of the RKT device and the connected long rod in the video image frame can be located by the target detection box. In an actual application scenario, even if the RTK device is stationary, the corresponding target detection box will still have jitter. In rare cases, the coordinates of the target detection box in each video image frame are completely consistent. The length of time that the target (RTK device) stays at the same position can be calculated. The structured information of the target can be used to establish an archive of the position information and survival time of each target in the second target time period, with the number being in frames. Further, the center point coordinates of the target detection box can be used to determine whether the RTK device moves in the second target time period.

[0097] Alternatively, if the target does not move, the average of the position coordinates of the target detection box in the continuous multiple video image frames can be used as the image pixel coordinates of the RTK device, or the position coordinates of an optional target detection box can be used as the image pixel coordinates of the RTK device.

[0098] In a specific embodiment, the determination of whether the RTK device moves in the second target time period based on the M center point coordinates includes:

[0099] The horizontal variance is calculated based on the horizontal coordinates of the M center point coordinates.

[0100] The vertical variance is calculated based on the vertical coordinates of the M center point coordinates.

[0101] When the horizontal variance is less than a second threshold value and the vertical variance is less than a third threshold value, it is determined that the RTK device does not move in the second target time period.

[0102] In a specific implementation, the indicators for calculating the length of time that the RTK device stays are mainly time and displacement. Regarding the time statistics, the frame rate of the video is obtained from the video Meta information transmitted by the application layer. Thus, it can be known that how many frames of images are composed in one second of the video. The frame rate of 25 fps can be used by default. Regarding the displacement statistics, the image frames in the first target time period are used to obtain the center position coordinates of the target detection box in each video image by video data analysis. The data is extracted in units of seconds (25 frames). It is assumed that 12 groups of center point coordinates (x, y) are extracted. Next, the horizontal and vertical distances between the center points of the first group and the second group are calculated based on the Euclidean distance formula. Similarly, the distances between the center points of the first group and the third group,..., and the center points of the first group and the twelfth group are calculated. Finally, the horizontal and vertical variances are calculated using the distance values obtained from the 12 groups of data.

[0103] In this embodiment, the lateral displacement variance and the longitudinal displacement variance of the M center point coordinates within the second target time length (such as 6 seconds, which can be defined by the application layer) are calculated. If the lateral variance is less than the second threshold value and the longitudinal variance is less than the third threshold value, it is considered that the RTK device is not moving. If it is greater than or equal to the preset threshold value, it is considered that the RTK device is in a moving state.

[0104] In an embodiment of the present application, after obtaining the image pixel coordinates of the RTK device according to the coordinate range of the first pixel region, the method further comprises:

[0105] determining the Euclidean distance between the first pixel coordinates and the second pixel coordinates; wherein the first pixel coordinates are the image pixel coordinates of the RTK device in the i-th frame of video image; the second pixel coordinates are the image pixel coordinates of the RTK device in the i-1-th frame of video image; i is a positive integer;

[0106] in the case where the Euclidean distance is greater than or equal to the fourth threshold value, saving the first pixel coordinates.

[0107] In specific implementation, when the stay time length of the target (referring to the RTK device) satisfies the first threshold value and the use posture satisfies the vertical posture, the Euclidean distance between the image pixel coordinates of the target currently obtained and the image pixel coordinates saved last time is calculated. If the Euclidean distance is greater than or equal to the fourth threshold value, it is considered that it is not the same pixel coordinate point, and the data storage step is executed. If the Euclidean distance is less than the fourth threshold value, it is considered that it is the same pixel coordinate point, and repeated storage is avoided, and the sub-process is ended and the next frame of image is detected. In this way, repeated storage of the same image pixel coordinates can be avoided, and the situation that the image pixel coordinates and the latitude and longitude coordinates cannot be accurately corresponded can be avoided.

[0108] It should be noted that, in the calibration site, the calibration point position is selected by the staff on site, so it is possible that the selected point position is not suitable and needs to be replaced with another point position to re-mark. At this time, in order to avoid obtaining image pixel points that the user does not want, the present application provides an embodiment to solve this problem.

[0109] Specifically, in an embodiment of the present application, after saving the coordinates corresponding to the first pixel region, the method further comprises:

[0110] when the stay time length is greater than the fifth threshold value, obtaining the use posture of the RTK device; wherein the fifth threshold value is greater than the first threshold value;

[0111] in the case where the use posture is a horizontal posture, deleting the saved first pixel coordinates.

[0112] In this embodiment, the horizontal posture is used to delete the recorded last image pixel coordinates. Since the image pixel coordinates of the RTK device are obtained after the stay duration meets the first threshold value, if the field staff does not want to use the calibration point position, the RTK device can be made to stay at the calibration point position for more than the fifth threshold value after the stay duration meets the first threshold value, and the RTK device and the long rod are placed horizontally (the length direction is parallel to the ground level). After recording the process through the camera, when the stay duration of the RTK device is greater than the fifth threshold value and the use posture is the horizontal posture based on the video image data, the operation of deleting the last saved image pixel coordinates is triggered, and the latitude and longitude information is marked back to the head of the latitude and longitude information queue.

[0113] A specific example of the camera extrinsic calibration method is introduced below. Mainly includes:

[0114] Load the video file of the calibration process into the video image frame queue, detect the target (RTK device) through the target detection tracking algorithm, judge the number of detected RTK devices, when the number is equal to 0, reacquire a new image from the image queue for target detection tracking; when the number is greater than 1, request manual selection of the correct target detection box; when the number is equal to 1, it is considered that the RTK device to be found is located, and the stay duration of the RTK device is calculated.

[0115] When the stay duration meets 6 seconds (the first threshold value), the posture of the RTK device is estimated, if it is a vertical posture, the distance between the current acquired image pixel coordinates and the last saved image pixel coordinates is calculated, if the distance is greater than or equal to the distance threshold value (the fourth threshold value), the current acquired image pixel coordinates are stored; if the distance is less than the distance threshold value (the fourth threshold value), the next image frame is continuously detected.

[0116] When the stay duration does not meet 6 seconds (the first threshold value), the next image frame is continuously detected.

[0117] When the stay duration meets 9 seconds (the fifth threshold value), the posture of the RTK device is estimated, if it is a horizontal posture, the last saved image pixel coordinates are deleted.

[0118] When the latitude and longitude information queue is empty, it is considered that all calibration points have been paired with image pixel coordinate points.

[0119] According to the multiple sets of paired latitude and longitude information and image pixel coordinates, the camera extrinsic matrix is generated.

[0120] In the above embodiment, by acquiring the position of the RTK device in the camera imaging screen, when the RTK device stays at a certain place for a duration greater than or equal to a first threshold and the use posture is kept in an upright posture, the image pixel coordinates of the RTK device are automatically recorded, and the latitude and longitude information output by the RTK device is automatically associated; when the RTK device stays at a certain place for a duration greater than a fifth threshold and the use posture is kept in a horizontal posture, the last recorded image pixel coordinates are deleted.

[0121] Second embodiment

[0122] As Figure 3 shown, a camera extrinsic parameter calibration device 300 of an embodiment of the present application comprises:

[0123] A first acquisition module 301 is configured to acquire image pixel coordinates of a real-time dynamic measurement RTK device in video image data, wherein the video image data is a calibration process video recording acquired by a camera to be calibrated.

[0124] A second acquisition module 302 is configured to acquire positioning data output by the RTK device, wherein the positioning data comprises at least one latitude and longitude information of the RTK device.

[0125] A third acquisition module 303 is configured to establish a corresponding relationship between the image pixel coordinates and the latitude and longitude information, wherein the image pixel coordinates and the latitude and longitude information correspond to each other in one-to-one correspondence.

[0126] A first determination module 304 is configured to determine a camera extrinsic parameter matrix according to the image pixel coordinates and the latitude and longitude information corresponding to each other.

[0127] Optionally, the first acquisition module 301 comprises:

[0128] A target detection sub-module is configured to perform target detection on each frame of video image in the video image data.

[0129] A first acquisition sub-module is configured to acquire a stay duration of the RTK device when the RTK device is detected.

[0130] A second acquisition sub-module is configured to acquire image pixel coordinates of the RTK device in the video image frame according to the stay duration.

[0131] Optionally, the second acquisition sub-module comprises:

[0132] A first acquisition unit is configured to acquire a use posture of the RTK device when the RTK device stays in a first pixel area for a duration greater than a first threshold; wherein the first pixel area is any pixel area in a video image frame displaying the RTK device.

[0133] The second acquisition unit is configured to acquire image pixel coordinates of the RTK device according to the coordinate range of the first pixel region when the use posture is the vertical posture.

[0134] Optionally, the first acquisition sub-module comprises:

[0135] The third acquisition unit is configured to acquire M continuous video image frames comprising the RTK device from N video image frames within a first target time length; N≥M>0, and N and M are integers.

[0136] The fourth acquisition unit is configured to acquire a center point coordinate of the RTK device in the M continuous video image frames; wherein each video image corresponds to one center point coordinate.

[0137] The fifth acquisition unit is configured to determine whether the RTK device moves within a second target time length according to M center point coordinates; wherein the second target time length is a time length corresponding to the M continuous video image frames.

[0138] The sixth acquisition unit is configured to take the second target time length as the stay time length if the RTK device does not move.

[0139] Optionally, the fifth acquisition unit is specifically configured to:

[0140] calculate a horizontal variance according to abscissas of the M center point coordinates;

[0141] calculate a vertical variance according to ordinates of the M center point coordinates;

[0142] determine that the RTK device does not move within the second target time length when the horizontal variance is less than a second threshold value and the vertical variance is less than a third threshold value.

[0143] Optionally, the device 300 further comprises:

[0144] The second determination module is configured to determine an Euclidean distance between a first pixel coordinate and a second pixel coordinate; wherein the first pixel coordinate is an image pixel coordinate of the RTK device in an i-th video image, the second pixel coordinate is an image pixel coordinate of the RTK device in an (i-1)-th video image, and i is a positive integer.

[0145] The storage module is configured to save the first pixel coordinate when the Euclidean distance is greater than or equal to a fourth threshold value.

[0146] Optionally, the device 300 further comprises:

[0147] The fourth acquisition module is used to acquire the usage posture of the RTK device when the dwell time is greater than a fifth threshold; wherein the fifth threshold is greater than the first threshold;

[0148] The deletion module is used to delete the saved first pixel coordinates when the usage posture is horizontal.

[0149] The camera extrinsic parameter calibration device 300 is a device corresponding to the method in the first embodiment described above. All implementation means in the above method embodiment are applicable to the embodiment of the camera extrinsic parameter calibration device and can achieve the same technical effect.

[0150] Third Embodiment

[0151] To better achieve the above objectives, such as Figure 4 As shown, the fourth embodiment of the present invention also provides a camera extrinsic parameter calibration device, comprising:

[0152] The processor 400; and the memory 420 connected to the processor 400 via a bus interface, the memory 420 being used to store programs and data used by the processor 400 during operation, and the processor 400 calling and executing the programs and data stored in the memory 420.

[0153] The transceiver 410 is connected to a bus interface and is used to receive and send data under the control of the processor 400; the processor 400 is used to read the program in the memory 420 to implement the following steps:

[0154] The image pixel coordinates of the real-time dynamic measurement RTK device are obtained from the video image data, which is a video recording of the calibration process obtained by the camera to be calibrated;

[0155] Acquire the positioning data output by the RTK device, wherein the positioning data includes at least one latitude and longitude information of the RTK device;

[0156] Establish a correspondence between the image pixel coordinates and the latitude and longitude information: wherein the image pixel coordinates and the latitude and longitude information correspond one-to-one;

[0157] The camera extrinsic matrix is ​​determined based on the corresponding image pixel coordinates and latitude and longitude information.

[0158] Among them, Figure 4In particular embodiments, the bus architecture can include any number of interconnecting buses and bridges, and the various circuitry representative of the processor 400 and the memory 420 that can be linked together by the bus architecture. The bus architecture can also link various other circuitry, such as peripheral devices, voltage regulators, and power management circuitry, all of which are well known in the art, and therefore, will not be further described herein. The bus interface provides an interface to the transceiver 410. The transceiver 410 can be a plurality of elements including a transmitter and a receiver, providing a means for communicating with various other apparatus over a transmission medium. The user interface 430 can also be an interface to external or internal devices, depending on the end terminal, including, but not limited to, a keypad, a display, a speaker, a microphone, a joystick, etc. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 can store data used by the processor 400 in executing its operations.

[0159] Optionally, the processor 400 is specifically configured to read the program in the memory 420 to implement the following steps:

[0160] target detection is performed on each frame of video image in the video image data;

[0161] When the RTK device is detected, a stay duration of the RTK device is obtained;

[0162] According to the stay duration, an image pixel coordinate of the RTK device in the video image frame is obtained.

[0163] Optionally, the processor 400 is specifically configured to read the program in the memory 420 to implement the following steps:

[0164] When the stay duration of the RTK device in the first pixel region is greater than a first threshold, a use posture of the RTK device is obtained; wherein the first pixel region is any pixel region in the video image frame displaying the RTK device;

[0165] When the use posture is an upright posture, according to the coordinate range of the first pixel region, an image pixel coordinate of the RTK device is obtained.

[0166] Optionally, the processor 400 is specifically configured to read the program in the memory 420 to implement the following steps:

[0167] From N frames of video image within a first target duration, M continuous video image frames including the RTK device are obtained; N > M > 0, N and M belong to integers;

[0168] A center point coordinate of the RTK device in the M continuous video image frames is obtained; wherein each frame of video image corresponds to a center point coordinate;

[0169] determine whether the RTK device moves within a second target time length according to the M center point coordinates; wherein the second target time length is a time length corresponding to the M continuous video image frames;

[0170] If not, the second target time length is taken as the stay time length.

[0171] Optionally, the processor 400 is specifically configured to read a program in the memory 420 to implement the following steps:

[0172] calculate a horizontal variance according to horizontal coordinates of the M center point coordinates;

[0173] calculate a vertical variance according to vertical coordinates of the M center point coordinates;

[0174] determine that the RTK device does not move within the second target time length when the horizontal variance is less than a second threshold value and the vertical variance is less than a third threshold value.

[0175] Optionally, the processor 400 is further configured to read a program in the memory 420 to implement the following steps:

[0176] determine an Euclidean distance between a first pixel coordinate and a second pixel coordinate; wherein the first pixel coordinate is an image pixel coordinate of the RTK device in an i-th video image frame; the second pixel coordinate is an image pixel coordinate of the RTK device in an i-1-th video image frame; i is a positive integer;

[0177] save the first pixel coordinate when the Euclidean distance is greater than or equal to a fourth threshold value.

[0178] Optionally, the processor 400 is further configured to read a program in the memory 420 to implement the following steps:

[0179] obtain a use posture of the RTK device when the stay time length is greater than a fifth threshold value; wherein the fifth threshold value is greater than the first threshold value;

[0180] delete the saved first pixel coordinate when the use posture is a horizontal posture.

[0181] The camera external parameter calibration device provided by the application can be calibrated by one person, and the calibration process does not need to set a preset point, but can select a suitable point for calibration on site, and the image pixel coordinates are obtained without human error, the process is automated, and the labor cost is reduced.

[0182] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by instructing the relevant hardware by a computer program including instructions for executing part or all of the steps of the above-mentioned method, and the computer program can be stored in a readable storage medium, which can be any form of storage medium.

[0183] In addition, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the method in the first embodiment described above. The same technical effects can be achieved, and details are not repeated here to avoid repetition.

[0184] In addition, it should be noted that in the device and method of the present application, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the present application. Moreover, the steps of performing the above series of processes can be naturally executed in time sequence according to the order of description, but it is not necessary to be executed in time sequence. Some steps can be executed in parallel or independently of each other. It can be understood by those skilled in the art that all or any steps or components of the method and device of the present application can be implemented in hardware, firmware, software or a combination thereof in any computing device (including processors, storage media, etc.) or network of computing devices, which can be achieved by those skilled in the art using their basic programming skills after reading the description of the present application.

[0185] Therefore, the object of the present application can also be achieved by running a program or a set of programs on any computing device. The computing device can be a commonly known general-purpose device. Therefore, the object of the present application can also be achieved only by providing a program product containing program code for implementing the method or device. That is, such a program product also constitutes the present application, and a storage medium storing such a program product also constitutes the present application. Obviously, the storage medium can be any commonly known storage medium or any storage medium developed in the future. It should be noted that in the device and method of the present application, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the present application. Moreover, the steps of performing the above series of processes can be naturally executed in time sequence according to the order of description, but it is not necessary to be executed in time sequence. Some steps can be executed in parallel or independently of each other.

[0186] The above is the preferred embodiment of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for calibrating camera extrinsic parameters, characterized in that, include: The image pixel coordinates of the real-time dynamic measurement RTK device are obtained from the video image data, which is a video recording of the calibration process obtained by the camera to be calibrated; Acquire the positioning data output by the RTK device, wherein the positioning data includes at least one latitude and longitude information of the RTK device; Establish a correspondence between the image pixel coordinates and the latitude and longitude information: wherein the image pixel coordinates and the latitude and longitude information correspond one-to-one; The camera extrinsic parameter matrix is ​​determined based on the corresponding image pixel coordinates and latitude / longitude information; The step of acquiring the image pixel coordinates of the real-time dynamic measurement RTK device in the video image data includes: Target detection is performed on each frame of the video image data; When the RTK device is detected, the dwell time of the RTK device is obtained; When the dwell time of the RTK device in the first pixel region exceeds a first threshold, the usage posture of the RTK device is obtained; wherein, the first pixel region is any pixel region in the video image frame that displays the RTK device; When the usage posture is vertical, the image pixel coordinates of the RTK device are obtained according to the coordinate range of the first pixel region; The step of obtaining the dwell time of the RTK device includes: From N video frames within the first target duration, obtain M consecutive video image frames including the RTK device; N≥M>0, where N and M are integers; Obtain the center point coordinates of the RTK device in the M consecutive video image frames; wherein, each video image frame corresponds to one center point coordinate; Based on the coordinates of the M center points, it is determined whether the RTK device moves within the second target duration; wherein, the second target duration is the duration corresponding to the M consecutive video image frames; If no movement is made, the second target duration will be used as the dwell time.

2. The camera extrinsic parameter calibration method according to claim 1, characterized in that, The step of determining whether the RTK device has moved within the second target time period based on the coordinates of the M center points includes: Calculate the lateral variance based on the x-coordinates of the M center points. Calculate the longitudinal variance based on the ordinates of the M center points; When the lateral variance is less than a second threshold and the longitudinal variance is less than a third threshold, it is determined that the RTK device has not moved within the second target duration.

3. The camera extrinsic parameter calibration method according to claim 1, characterized in that, After obtaining the image pixel coordinates of the RTK device based on the coordinate range of the first pixel region, the method further includes: Determine the Euclidean distance between the first pixel coordinates and the second pixel coordinates; wherein, the first pixel coordinates are the image pixel coordinates of the RTK device in the i-th frame of the video image; the second pixel coordinates are the image pixel coordinates of the RTK device in the (i-1)-th frame of the video image; i is a positive integer; If the Euclidean distance is greater than or equal to the fourth threshold, the coordinates of the first pixel are saved.

4. The camera extrinsic parameter calibration method according to claim 3, characterized in that, After saving the coordinates corresponding to the first pixel region, the method further includes: When the dwell time exceeds a fifth threshold, the usage posture of the RTK device is obtained; wherein the fifth threshold is greater than the first threshold; When the usage posture is horizontal, delete the saved first pixel coordinates.

5. A calibration device for camera extrinsic parameters, comprising: A transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the camera extrinsic parameter calibration method as described in any one of claims 1 to 4.

6. A calibration device for camera extrinsic parameters, characterized in that, include: The first acquisition module is used to acquire the image pixel coordinates of the real-time dynamic measurement RTK device in the video image data, wherein the video image data is a video recording of the calibration process acquired by the camera to be calibrated; The second acquisition module is used to acquire the positioning data output by the RTK device, the positioning data including at least one latitude and longitude information of the RTK device; The third acquisition module is used to establish the correspondence between the image pixel coordinates and the latitude and longitude information: wherein the image pixel coordinates and the latitude and longitude information correspond one-to-one; The first determining module is used to determine the camera extrinsic parameter matrix based on the corresponding image pixel coordinates and the latitude and longitude information; The first acquisition module includes: The target detection submodule performs target detection on each frame of the video image data; The first acquisition submodule is used to acquire the dwell time of the RTK device when the RTK device is detected; The second acquisition submodule is used to acquire the usage posture of the RTK device when the dwell time of the RTK device in the first pixel area is greater than a first threshold; wherein, the first pixel area is any pixel area in the video image frame that displays the RTK device; when the usage posture is a vertical posture, the image pixel coordinates of the RTK device are acquired according to the coordinate range of the first pixel area. The first acquisition submodule includes: The third acquisition unit is used to acquire M consecutive video image frames, including the RTK device, from N video images within the first target duration; N≥M>0, where N and M are integers; The fourth acquisition unit is used to acquire the center point coordinates of the RTK device in the M consecutive video image frames; wherein each video image frame corresponds to one center point coordinate. The fifth acquisition unit is used to determine whether the RTK device has moved within a second target duration based on the M center point coordinates; wherein the second target duration is the duration corresponding to the M consecutive video image frames; The sixth acquisition unit is used to take the second target duration as the dwell time if no movement is made.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the camera extrinsic parameter calibration method as described in any one of claims 1 to 4.

Images