Camera calibration method, device and system, image acquisition method and device

By obtaining the image and three-dimensional coordinate data of the surveillance camera during the movement of the positioning device and performing automatic calibration, it solves the problem that traditional surveillance cameras are difficult to achieve two-dimensional to three-dimensional conversion, improves positioning accuracy and efficiency, and is suitable for urban-level digital twin systems. .

CN118470136BActive Publication Date: 2024-11-26ALIBABA CLOUD COMPUTING CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410910446.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-26
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

Traditional surveillance cameras are difficult to restore two-dimensional image information to three-dimensional space in real time, resulting in reduced accuracy of target recognition, detection and tracking, and it is difficult to achieve accurate positioning in urban market scenarios.

Method used

By obtaining the target image and three-dimensional coordinate data collected by the camera to be calibrated during the movement of the positioning device, image recognition and time stamp synchronization are performed, and the camera is automatically calibrated to improve positioning accuracy and achieve accurate conversion between two-dimensional and three-dimensional information.

Benefits of technology

It improves the efficiency and accuracy of camera calibration, reduces costs, expands the scope of application, solves the calibration problem in scenes without obvious texture characteristics, and realizes accurate conversion between image two-dimensional coordinates and spatial three-dimensional coordinates.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470136B_ABST
    Figure CN118470136B_ABST
Patent Text Reader

Abstract

The embodiments of the present specification provide a camera calibration method, device and system, and an image acquisition method and device, wherein the camera calibration method comprises: obtaining a target image acquired by a positioning device during the movement of the positioning device, as well as target three-dimensional coordinates acquired by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; performing image recognition on the target image to obtain target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; calibrating the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data, and automatically calibrating the camera to be calibrated through the first time data and the second time data, thereby improving the efficiency and accuracy of camera calibration; based on the calibration result of the camera to be calibrated, accurate conversion between two-dimensional information and three-dimensional information can be achieved, thereby improving positioning accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular to a camera calibration method, device and system, and an image acquisition method and device. Background Art

[0002] The application of large-scale cloud computing and large-model AI technology has provided conditions for city-level digital twin systems. Surveillance cameras, as a common video acquisition sensor in urban scenes, are widely used in transportation, municipal administration and other fields. They are currently the most widely used sensors in cities.

[0003] However, the two-dimensional images of traditional surveillance cameras lack relevance to real three-dimensional scenes. They usually achieve semantic understanding of two-dimensional images, such as target recognition, detection, and tracking, but it is difficult to restore the two-dimensional information obtained from the image to three-dimensional space in real time. This makes it difficult for the surveillance system to accurately locate the target, reducing the accuracy of target recognition, detection, and tracking. Summary of the invention

[0004] In view of this, an embodiment of the present specification provides a camera calibration method. One or more embodiments of the present specification also relate to a camera calibration device, a camera calibration system, an image acquisition method, an image acquisition device, a computing device, a computer-readable storage medium, and a computer program product to solve the problem of difficulty in accurately locating a target.

[0005] According to a first aspect of an embodiment of this specification, a camera calibration method is provided, including:

[0006] Acquire a target image captured by the positioning device during the movement of the positioning device, a target three-dimensional coordinates collected by the positioning device during the movement, and first time data corresponding to the target three-dimensional coordinates;

[0007] Performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates;

[0008] The camera to be calibrated is calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0009] According to a second aspect of an embodiment of this specification, a camera calibration method is provided, including:

[0010] Obtaining, in the target city scene, a target image captured by the monitoring camera during the movement of the positioning device, as well as the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates;

[0011] Performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates;

[0012] The monitoring camera is calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0013] According to a third aspect of the embodiments of this specification, there is provided an image acquisition method, including:

[0014] Acquire an image to be processed collected by the target camera for the target object during the movement of the target object, and a target calibration parameter of the target camera, wherein the target calibration parameter is obtained based on the calibration of the target camera, and calibrate the target camera, including: acquiring a target image collected by the positioning device during the movement of the positioning device, and a target three-dimensional coordinate collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinate; perform image recognition on the target image to obtain a target two-dimensional coordinate of the positioning device and second time data corresponding to the target two-dimensional coordinate; calibrate the target camera according to the target three-dimensional coordinate, the target two-dimensional coordinate, the first time data and the second time data;

[0015] Perform image recognition on the image to be processed to obtain the two-dimensional position information of the target object;

[0016] The three-dimensional position information of the target object is generated according to the two-dimensional position information and the target calibration parameters.

[0017] According to a fourth aspect of an embodiment of this specification, a camera calibration system is provided, the camera calibration system comprising a server, a camera to be calibrated, and a positioning device;

[0018] The camera to be calibrated is configured to collect target images of the positioning device during movement;

[0019] A positioning device configured to collect the three-dimensional coordinates of the target during the movement and the first time data corresponding to the three-dimensional coordinates of the target;

[0020] The server is configured to obtain the target image collected by the positioning device during the movement of the positioning device, the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates; perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates; calibrate the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0021] According to a fifth aspect of the embodiments of this specification, a camera calibration device is provided, including:

[0022] A first acquisition module is configured to acquire a target image captured by the positioning device during the movement of the positioning device, a target three-dimensional coordinates collected by the positioning device during the movement, and first time data corresponding to the target three-dimensional coordinates;

[0023] A first recognition module is configured to perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates;

[0024] The first calibration module is configured to calibrate the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0025] According to a sixth aspect of the embodiments of this specification, a camera calibration device is provided, including:

[0026] The second acquisition module is configured to acquire, in the target city scene, a target image captured by the monitoring camera during the movement of the positioning device, as well as the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates;

[0027] A second recognition module is configured to perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates;

[0028] The second calibration module is configured to calibrate the monitoring camera according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0029] According to a seventh aspect of the embodiments of this specification, there is provided an image acquisition device, including:

[0030] The third acquisition module is configured to acquire an image to be processed collected by the target camera for the target object during the movement of the target object, and a target calibration parameter of the target camera, wherein the target calibration parameter is obtained based on the calibration of the target camera, and the calibration of the target camera includes: acquiring a target image collected by the positioning device during the movement of the positioning device, and a target three-dimensional coordinate collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinate; performing image recognition on the target image to obtain a target two-dimensional coordinate of the positioning device and second time data corresponding to the target two-dimensional coordinate; and calibrating the target camera according to the target three-dimensional coordinate, the target two-dimensional coordinate, the first time data and the second time data;

[0031] A third recognition module is configured to perform image recognition on the image to be processed to obtain two-dimensional position information of the target object;

[0032] The generating module is configured to generate three-dimensional position information of the target object according to the two-dimensional position information and the target calibration parameters.

[0033] According to an eighth aspect of the embodiments of this specification, a computing device is provided, including:

[0034] Memory and processor;

[0035] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned camera calibration method or image acquisition method are implemented.

[0036] According to a ninth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned camera calibration method or image acquisition method are implemented.

[0037] According to a tenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned camera calibration method or image acquisition method when executed by a processor.

[0038] An embodiment of the present specification realizes obtaining a target image captured by a positioning device during the movement of the positioning device, as well as target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; performing image recognition on the target image to obtain target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; and calibrating the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0039] In this way, by acquiring the target image captured by the camera to be calibrated, based on image recognition, obtaining the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates, and acquiring the target three-dimensional coordinates collected by the positioning device and the corresponding first time data, it is possible to realize automatic calibration of the camera to be calibrated, and improve the efficiency and accuracy of calibration of the camera to be calibrated; further, based on the calibration result of the camera to be calibrated, it is possible to convert the two-dimensional coordinates of the image captured by the camera to be calibrated into spatial three-dimensional coordinates, and it is also possible to convert the spatial three-dimensional coordinates into the image two-dimensional coordinates in the image captured by the camera to be calibrated, thereby realizing accurate conversion between two-dimensional information and three-dimensional information, and improving positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is an architecture diagram of a camera calibration system provided by an embodiment of this specification;

[0041] Figure 2is a flow chart of a camera calibration method provided by an embodiment of this specification;

[0042] Figure 3 is a flow chart of another camera calibration method provided by an embodiment of this specification;

[0043] Figure 4 is a flow chart of an image acquisition method provided by an embodiment of this specification;

[0044] Figure 5 is a process flow chart of a camera calibration method provided by an embodiment of this specification;

[0045] Figure 6 is a structural schematic diagram of a camera calibration device provided by an embodiment of this specification;

[0046] Figure 7 is a schematic diagram of the structure of another camera calibration device provided by an embodiment of this specification;

[0047] Figure 8 is a structural schematic diagram of an image acquisition device provided by an embodiment of this specification;

[0048] Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0049] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0050] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0051] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0053] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0054] When the big model is used in practice, the pre-trained model can be fine-tuned based on a small number of samples and applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0055] First, the terms involved in one or more embodiments of this specification are explained.

[0056] Camera calibration: refers to the process of establishing a geometric model of camera imaging and solving the parameters of camera intrinsic parameters and camera extrinsic parameters.

[0057] Camera intrinsic parameters: refers to the parameters that describe the inherent properties of the camera itself. These parameters define the projection process of the camera from three-dimensional space to two-dimensional images and do not depend on the position and direction of the camera in the environment. Camera intrinsic parameters can mainly include camera focal length, principal point position, shear coefficient and distortion parameters, among which distortion parameters include radial distortion parameters and tangential distortion parameters.

[0058] Camera extrinsics: refers to the parameters that describe the position and posture of the camera relative to the world coordinate system (or scene coordinate system). These parameters define the direction and position of the camera in three-dimensional space and determine the relative transformation between the camera coordinate system and the world coordinate system. Camera extrinsics can mainly include rotation matrices and translation vectors.

[0059] High-precision positioning: refers to technologies that can achieve centimeter-level spatial positioning, including outdoor RTK (Real-Time Kinematic) satellite positioning, indoor UWB (Ultra-Wideband) positioning, etc.

[0060] Timestamp soft synchronization: A method of synchronizing time between different systems or hardware to ensure data consistency in communication systems.

[0061] NTP (Network Time Protocol): is a standard time synchronization protocol applied to computer networks. It can enable different devices to maintain the same time on the Internet Protocol network, achieve high-precision time correction, and ensure the accuracy and consistency of time.

[0062] At present, the calibration of surveillance camera parameters can be based on accurately reconstructed 3D models and high-precision maps. The matching point pairs between the image and the 3D model can be obtained through manual annotation or algorithms, and then the internal and external parameters of the camera model can be solved using optimization algorithms. However, in urban scenes, the time and cost required to build 3D models and high-precision maps are high. When the scene changes or there are no obvious texture features, it is difficult to effectively calculate the camera parameters, and it is difficult to reasonably place calibration objects within the field of view of the surveillance camera. Therefore, it is difficult to achieve camera parameter calibration through conventional camera calibration algorithms, and it is difficult to accurately associate 2D information with 3D information in urban scenes.

[0063] Based on this, an embodiment of the present specification provides a camera calibration method, which obtains a target image captured by a positioning device during the movement of the positioning device, as well as the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates; performs image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates; and calibrates the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0064] In this way, by acquiring the target image captured by the camera to be calibrated, based on image recognition, obtaining the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates, and acquiring the target three-dimensional coordinates collected by the positioning device and the corresponding first time data, it is possible to realize automatic calibration of the camera to be calibrated, and improve the efficiency and accuracy of calibration of the camera to be calibrated; further, based on the calibration result of the camera to be calibrated, it is possible to convert the two-dimensional coordinates of the image captured by the camera to be calibrated into spatial three-dimensional coordinates, and it is also possible to convert the spatial three-dimensional coordinates into the image two-dimensional coordinates in the image captured by the camera to be calibrated, thereby realizing accurate conversion between two-dimensional information and three-dimensional information, and improving positioning accuracy.

[0065] In this specification, a camera calibration method is provided. This specification also relates to a camera calibration device, a camera calibration system, an image acquisition method, an image acquisition device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0066] See also Figure 1 , Figure 1 The camera calibration system 100 includes a server 102 , a camera to be calibrated 104 , and a positioning device 106 , wherein the server 102 includes a time synchronization unit 1022 , a detection unit 1024 , and a calibration unit 1026 .

[0067] The camera 104 to be calibrated is configured to collect target images of the positioning device 106 during movement.

[0068] Positioning device 106: configured to collect the three-dimensional coordinates of the target during the movement and the first time data corresponding to the three-dimensional coordinates of the target.

[0069] The server 102 is configured to obtain the target image captured by the camera 104 to be calibrated during the movement of the positioning device 106, as well as the target three-dimensional coordinates collected by the positioning device 106 during the movement and the first time data corresponding to the target three-dimensional coordinates; perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device 106 and the second time data corresponding to the target two-dimensional coordinates; calibrate the camera 104 to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0070] Furthermore, the server 102 includes a time synchronization unit 1022, a detection unit 1024, and a calibration unit 1026. The time synchronization unit 1022 is configured to synchronize the time of the camera to be calibrated 104 and the positioning device 106, so as to achieve soft synchronization of the timestamps of the camera to be calibrated 104 and the positioning device 106. The detection unit 1024 is configured to identify the positioning device in the target image, determine the target two-dimensional coordinates of the positioning device in the target image; obtain the second time data when the camera to be calibrated captures the target image, and associate the target two-dimensional coordinates with the second time data.

[0071] Optionally, the positioning device 106 is mounted on a mobile object; further, the detection unit 1024 is configured to perform mobile object recognition on the target image to obtain position information of the mobile object in the target image, wherein the mobile object carries the positioning device and moves within the field of view of the camera to be calibrated; based on the position information, determine the target two-dimensional coordinates of the positioning device in the target image.

[0072] The calibration unit 1026 is configured to obtain the target three-dimensional coordinates and the first time data corresponding to the target three-dimensional coordinates, as well as the target two-dimensional coordinates and the second time data corresponding to the target two-dimensional coordinates; and calibrate the camera 104 to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0073] In actual applications, the server 102 is a server for calibrating the camera to be calibrated. It should be noted that the server 102 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server (cloud-side device) for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0074] By applying the embodiments of the present specification, the time synchronization unit 1022 performs synchronous timing on the camera 104 to be calibrated and the positioning device 106, so as to achieve soft synchronization of the timestamps of the camera 104 to be calibrated and the positioning device 106, thereby improving the accuracy of the correlation between the second time data corresponding to the two-dimensional coordinates and the first time data corresponding to the three-dimensional coordinates, thereby improving the camera calibration accuracy; the detection unit 1024 recognizes the target image, obtains the target two-dimensional coordinates and the first time data of the positioning device 106 in the target image, and enables the calibration unit 1026 to complete the automatic calibration of the camera to be calibrated according to the target two-dimensional coordinates and the second time data as well as the target three-dimensional coordinates and the first time data, without changing the existing layout conditions, without adding additional calibration equipment, and without building a high-precision map and a three-dimensional model, thereby improving the efficiency and accuracy of camera calibration, thereby achieving accurate conversion between image two-dimensional coordinates and spatial three-dimensional coordinates, and improving positioning efficiency.

[0075] See also Figure 2 , Figure 2 A flowchart of a camera calibration method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0076] Step 202: obtaining a target image captured by the positioning device during the movement of the positioning device, the target three-dimensional coordinates collected by the positioning device during the movement, and the first time data corresponding to the target three-dimensional coordinates.

[0077] In practical applications, the target image captured by the positioning device during the movement of the positioning device, the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates can be obtained.

[0078] Specifically, the camera to be calibrated can be understood as a camera that needs to calibrate internal and external parameters, which can capture two-dimensional images or record videos, such as a surveillance camera. The positioning device can be understood as a device that can perform real-time high-precision positioning of the three-dimensional space, such as RTK equipment, laser radar, satellite signal receiver, etc. The target image can be understood as the image collected by the camera to be calibrated during the movement of the positioning device.

[0079] Optionally, the camera to be calibrated can shoot at a preset frame rate (e.g., 30 frames per second) to obtain multiple images while the positioning device is moving. The camera to be calibrated can integrate a timestamp saving function to record the timestamp corresponding to each target image while shooting each target image. The camera to be calibrated can save timestamps accurate to milliseconds.

[0080] Specifically, the target three-dimensional coordinates can be understood as the spatial three-dimensional coordinate information collected by the positioning device during movement, that is, the actual position information of the positioning device in the real scene. The target three-dimensional coordinates can include various types of coordinates in the spatial coordinate system, such as longitude, latitude and altitude in the geographic coordinate system, or three-dimensional coordinates in other spatial coordinate systems, etc., which can be determined according to the needs of actual applications. The first time data can be understood as the corresponding timestamp saved by the positioning device while collecting the target three-dimensional coordinates. The positioning device can also save timestamps accurate to milliseconds.

[0081] In actual applications, due to considerations of deployment costs and feasibility, the camera to be calibrated and the positioning device are often two independent hardware devices, or deployed in two independent hardware systems. Therefore, it is difficult for the time systems of the two devices to maintain high consistency. If there is a large time deviation between the timestamps recorded by the two devices, the error of camera calibration will be large, making it difficult to achieve accurate positioning.

[0082] Based on this, according to an optional embodiment of the present specification, before obtaining the target image captured by the positioning device during the movement of the positioning device, the target three-dimensional coordinates collected by the positioning device during the movement, and the first time data corresponding to the target three-dimensional coordinates, the method may also include:

[0083] The time data of the image collected by the camera to be calibrated and the time data of the three-dimensional coordinates collected by the positioning device are calibrated.

[0084] Optionally, calibrating the time data of the camera to be calibrated to acquire images and the positioning device to acquire three-dimensional coordinates may include: synchronizing the time of the camera to be calibrated and the positioning device through a time synchronization unit. Further, the time synchronization unit may be an NTP time server or other time service that can achieve soft synchronization of timestamps between different devices.

[0085] For example, through the NTP timing server, the time of the camera to be calibrated and the time of the positioning device can be aligned at the second level.

[0086] By applying this embodiment, by calibrating the time data of the image captured by the camera to be calibrated and the three-dimensional coordinates captured by the positioning device, it is possible to ensure that the time deviation between the timestamp of the target image captured by the camera to be calibrated and the timestamp of the three-dimensional coordinates captured by the positioning device is as small as possible, for example, the time is kept consistent at the second level, thereby improving the parameter accuracy in the camera automatic calibration process and improving the accuracy of the calibration parameters.

[0087] Step 204: Perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates.

[0088] According to an optional embodiment of the present specification, during the movement of the camera to be calibrated in the positioning device, based on the target image captured by the positioning device, image recognition can be performed on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates.

[0089] Optionally, image recognition of the target image can be achieved through an AI detection algorithm or an image recognition model.

[0090] In practical applications, during the time period when the camera to be calibrated is shooting, the positioning device may not be in the camera field of view of the camera to be calibrated, or the positioning device may be blocked by other objects in the camera field of view. Therefore, according to another optional embodiment of this specification, when obtaining the target image captured by the positioning device during the movement of the camera to be calibrated, the target image can be filtered through a filtering algorithm to filter noise, retain the target image that can identify the positioning device from the image, and perform image recognition on the filtered target image.

[0091] Specifically, the target two-dimensional coordinates may be understood as the two-dimensional position coordinates of the positioning device in the target image, and may specifically be pixel point position coordinates.

[0092] Exemplarily, assuming that the pixel resolution of the target image is 1920*1080, the pixel point position coordinates of the positioning device may be (1000, 500).

[0093] According to an optional embodiment of the present specification, performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates may include:

[0094] Identify the positioning device in the target image and determine the target two-dimensional coordinates of the positioning device in the target image;

[0095] The second time data when the camera to be calibrated captures the target image is obtained, and the target two-dimensional coordinates and the second time data are associated.

[0096] Further, according to an optional embodiment of the present specification, identifying the positioning device in the target image and determining the target two-dimensional coordinates of the positioning device in the target image may include:

[0097] The positioning device in the target image is identified through the AI ​​detection algorithm, and the target two-dimensional coordinates of the positioning device in the target image are obtained through target detection.

[0098] According to another optional embodiment of the present specification, identifying the positioning device in the target image and determining the target two-dimensional coordinates of the positioning device in the target image may include:

[0099] The positioning device in the target image is identified through the image recognition model to obtain the target two-dimensional coordinates of the positioning device in the target image, wherein the image recognition model is a pre-trained model.

[0100] In practical applications, when the camera to be calibrated captures the target image, it will save the timestamp corresponding to each target image, which can be understood as the second time data corresponding to the target image. When the positioning device in the target image is identified and the target two-dimensional coordinates are obtained, the second time data corresponding to the target image can be associated with the target two-dimensional coordinates of the positioning device in the target image.

[0101] By applying this embodiment, the target two-dimensional coordinates of the positioning device in the target image are determined by identifying the positioning device in the target image, and the target two-dimensional coordinates of the positioning device in the image can be obtained. By associating the target two-dimensional coordinates and the second time data, the second time data corresponding to each target two-dimensional coordinate can be obtained, thereby facilitating the subsequent automatic calibration of the camera to be calibrated according to the target two-dimensional coordinates and the second time data, thereby improving the accuracy of the calibration parameters.

[0102] According to another optional embodiment of the present specification, identifying the positioning device in the target image and determining the target two-dimensional coordinates of the positioning device in the target image may also include:

[0103] Perform moving object recognition on the target image to obtain the position information of the moving object in the target image, wherein the moving object is equipped with a positioning device and moves within the field of view of the camera to be calibrated;

[0104] According to the position information, the target two-dimensional coordinates of the positioning device in the target image are determined.

[0105] Specifically, the mobile object can carry the positioning device and move in the real scene, that is, it can move within the field of view of the camera to be calibrated. The mobile object can be a movable platform such as a vehicle or a flying device, or it can be a relevant staff member. The position information of the mobile object can include the three-dimensional coordinates of the mobile object, and can also include the area of ​​interest of the mobile object in the target image.

[0106] In actual applications, the positioning device is mounted on a moving object, which is often larger and easier to identify. Therefore, through the AI ​​detection algorithm, the position information of the moving object in the target image can be identified first, and then the target two-dimensional coordinates of the positioning device in the target image can be identified based on the position information.

[0107] It should be noted that before the camera to be calibrated captures the target image, the physical features of the positioning device and the moving object may be enhanced, thereby improving the recognition accuracy of the positioning device after being photographed by the camera to be calibrated.

[0108] Optionally, the feature enhancement of the positioning device can be achieved by placing the positioning device at a more conspicuous place of the moving object and / or painting the positioning device with a special conspicuous color, such as placing it on the roof of a vehicle and painting it red.

[0109] Optionally, the feature enhancement of the moving object can be achieved by adding marker information, such as adding a large internship marker on the body of the vehicle, or adding a red strip marker on the hood.

[0110] By applying this embodiment, the target image is subjected to moving object recognition, and the target two-dimensional coordinates of the positioning device in the target image are determined according to the position information of the moving object, so that the detection accuracy of the target detection can be improved, thereby improving the recognition accuracy of the target two-dimensional coordinates.

[0111] Step 206: Calibrate the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data, and the second time data.

[0112] In practical applications, based on obtaining the target three-dimensional coordinates collected by the positioning device during movement and the first time data corresponding to the target three-dimensional coordinates, and obtaining the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates, the camera to be calibrated can be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0113] According to an optional embodiment of the present specification, calibrating the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data, and the second time data may include:

[0114] Determine the target three-dimensional coordinates and the target two-dimensional coordinates that have a projection relationship according to the first time data and the second time data;

[0115] Based on the target three-dimensional coordinates and the target two-dimensional coordinates that have a projection relationship, and the projection relationship, calibration parameter information of the camera to be calibrated is generated.

[0116] Specifically, the target three-dimensional coordinates and target two-dimensional coordinates that have a projection relationship can be understood as the target three-dimensional coordinates and target two-dimensional coordinates at the same time. The calibration parameter information may include the camera intrinsic parameters and camera extrinsic parameters of the camera to be calibrated.

[0117] In practical applications, based on the target three-dimensional coordinates and target two-dimensional coordinates with a projection relationship, as well as the projection relationship, the calibration parameter information of the camera to be calibrated is generated, so that the camera to be calibrated can be automatically calibrated.

[0118] In the actual implementation process, based on the pinhole imaging principle, the projection relationship between the target three-dimensional coordinates and the target two-dimensional coordinates at the same time can be seen in the following formula (1).

[0119] Formula (1)

[0120] Specifically, λ is a constant, (u, v) is the two-dimensional pixel coordinate, that is, the two-dimensional coordinate of the target, K and R represent the camera internal parameters, t represents the camera external parameters, P is the three-dimensional coordinate of the target, X i It is a point in the camera coordinate system, that is, the three-dimensional coordinates of the target. It is the three-dimensional coordinate obtained by projecting the two-dimensional pixel coordinate (u, v).

[0121] Furthermore, for the camera to be calibrated with distortion coefficients, radial distortion and tangential distortion need to be added through the following equations (2) and (3). K and R can be understood as the camera intrinsic parameters of the camera to be calibrated, and t can be understood as the camera extrinsic parameters of the camera to be calibrated. u and v are points in the image coordinate system, that is, the two-dimensional coordinates of the target.

[0122] Formula (2)

[0123] Formula (3)

[0124] Specifically, u and v are the two-dimensional coordinates of the target after radial distortion and tangential distortion, k 1 , k 2 , k 3 are the radial distortion coefficients of the camera to be calibrated, p 1 、p 2 are the tangential distortion coefficients of the camera to be calibrated. and represents the pixel coordinates after distortion correction. r is calculated according to the following formula (4).

[0125] Formula (4)

[0126] Optionally, multiple sets of target 3D coordinates and target 2D coordinates with projection relationship can be determined according to the first time data and the second time data. Further, calibration parameter information of the camera to be calibrated can be generated based on the multiple sets of target 3D coordinates and target 2D coordinates with projection relationship.

[0127] Optionally, generating calibration parameter information of the camera to be calibrated based on multiple groups of target three-dimensional coordinates and target two-dimensional coordinates that have a projection relationship may include: substituting each group of target three-dimensional coordinates and target two-dimensional coordinates that have a projection relationship into the above formula (1) to obtain multiple equations, and solving the multiple equations simultaneously to generate the calibration parameter information of the camera to be calibrated.

[0128] Exemplarily, the calibration parameter information of the camera to be calibrated can be calculated based on at least 5 groups of target 3D coordinates and target 2D coordinates that have a projection relationship and the above formula (1).

[0129] According to this embodiment, by determining the target three-dimensional coordinates and the target two-dimensional coordinates that have a projection relationship; based on the target three-dimensional coordinates and the target two-dimensional coordinates that have a projection relationship, and the projection relationship, the calibration parameter information of the camera to be calibrated is generated, and the calibration parameter information of the camera can be calculated according to multiple groups of target two-dimensional coordinates and target three-dimensional coordinates that have an associated relationship through an automatic calibration algorithm, thereby realizing automatic calibration of the camera to be calibrated and improving the efficiency and accuracy of camera calibration.

[0130] According to an optional embodiment of the present specification, determining the target three-dimensional coordinates and the target two-dimensional coordinates having a projection relationship according to the first time data and the second time data may include:

[0131] Determine, from the first time data and the second time data, target first time data and target second time data whose time deviation is less than a preset time threshold;

[0132] The three-dimensional coordinates of the target corresponding to the first time data of the target and the two-dimensional coordinates of the target corresponding to the second time data of the target are obtained.

[0133] In practical applications, the target first time data and target second time data whose time deviation is less than a preset time threshold can be determined from the first time data and the second time data, thereby obtaining the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data.

[0134] It is easy to understand that since the camera to be calibrated and the positioning device are usually located in two independent hardware systems, the time system of the camera to be calibrated and the time system of the positioning device are often time-aligned based on the timestamp soft synchronization technology. That is, when the camera to be calibrated and the positioning device collect images and three-dimensional coordinates at the same time, there is often a small time deviation between the time data carried by the image and the time data carried by the three-dimensional coordinates.

[0135] Based on this, at the same moment, the target second time data corresponding to the target two-dimensional coordinates obtained by the camera to be calibrated and the target first time data corresponding to the target three-dimensional coordinates obtained by the positioning device can often be aligned at the second level, while there is a certain time deviation in more precise time measurement units (such as milliseconds).

[0136] Exemplarily, the time deviation may be 50 milliseconds.

[0137] Specifically, the preset time threshold may be determined according to requirements in actual applications. For example, if the time of the camera to be calibrated and the positioning device are aligned to the second level, the preset time threshold may be 100 milliseconds.

[0138] In practical applications, due to the projection relationship between the target three-dimensional coordinates and the target two-dimensional coordinates at the same time based on the pinhole imaging principle, it can be determined that the target first time data and the target second time data with a time deviation less than a preset time threshold belong to the same time; there is a projection relationship between the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data.

[0139] By applying this embodiment, by determining target first time data and target second time data whose time deviation is less than a preset time threshold from the first time data and the second time data, target three-dimensional coordinates and target two-dimensional coordinates that have a projection relationship can be determined based on the target first time data and the target second time data, thereby enabling automatic calibration of the camera to be calibrated based on the target three-dimensional coordinates and target two-dimensional coordinates that have a projection relationship.

[0140] According to an optional embodiment of the present specification, generating calibration parameter information of a camera to be calibrated based on target three-dimensional coordinates and target two-dimensional coordinates having a projection relationship and the projection relationship may include:

[0141] constructing a coordinate pair based on the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data;

[0142] Based on the coordinate alignment, the projection relationship between the target two-dimensional coordinates and the target three-dimensional coordinates, the calibration parameter information of the camera to be calibrated is generated.

[0143] Specifically, the coordinate pair is formed by pairing the target three-dimensional coordinates and the target two-dimensional coordinates. The target three-dimensional coordinates and the target two-dimensional coordinates are paired according to the first time data and the second time data.

[0144] In practical applications, the positioning device can collect the three-dimensional coordinates of the target during movement and the first time data corresponding to the three-dimensional coordinates of the target based on a preset frame rate. The camera to be calibrated can also shoot the positioning device based on a preset frame rate and record the image timestamp corresponding to each frame of the target image. The preset frame rate of the positioning device and the preset frame rate of the camera to be calibrated can be the same or different.

[0145] For example, the positioning device can collect three-dimensional coordinate data every 100 milliseconds and record the corresponding first time data. The camera to be calibrated can capture 30 frames of target images in one second and record the second time data corresponding to the target images.

[0146] Therefore, by identifying the target image captured by the camera to be calibrated, multiple target two-dimensional coordinates corresponding to the movement of the positioning device and the second time data corresponding to each target two-dimensional coordinate can be obtained; and based on the positioning device, multiple target three-dimensional coordinates corresponding to the movement of the positioning device and the first time data corresponding to each target three-dimensional coordinate can be obtained.

[0147] Optionally, the target three-dimensional coordinates and the target two-dimensional coordinates may be paired according to the first time data and the second time data to obtain multiple groups of coordinate pairs.

[0148] Optionally, the camera to be calibrated is calibrated according to the coordinate pairs, specifically, the camera to be calibrated is calibrated according to no less than a preset number of coordinate pairs. The preset number can be determined according to the needs of actual applications, and can usually be 5 or 10, that is, the camera to be calibrated is calibrated according to at least 5 groups, or at least 10 groups of different coordinate pairs.

[0149] By applying this embodiment, by constructing a coordinate pair, the target three-dimensional coordinates and the target two-dimensional coordinates that are associated with each other can be paired to form a coordinate pair, so that the camera to be calibrated can be calibrated according to the coordinate pair, thereby improving the efficiency and accuracy of camera calibration.

[0150] Optionally, constructing a coordinate pair according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data, and the second time data may include:

[0151] Determine the target first time data and the target second time data whose time deviation is less than a preset time threshold;

[0152] A coordinate pair is constructed based on the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data.

[0153] In practical applications, the target first time data and the target second time data whose time deviation is less than a preset time threshold can be determined, and the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data are paired to obtain a set of coordinate pairs. Similarly, multiple sets of coordinate pairs can be obtained through pairing.

[0154] Specifically, the preset time threshold may be determined according to requirements in actual applications. For example, if the time of the camera to be calibrated and the positioning device are aligned to the second level, the preset time threshold may be 100 milliseconds.

[0155] By applying this embodiment, a coordinate pair is constructed based on the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data. This ensures that the camera is calibrated according to the two-dimensional coordinates and three-dimensional coordinates with corresponding relationships, thereby improving the accuracy of the camera calibration process and thus improving the accuracy of the calibration parameters.

[0156] It is worth mentioning that since the camera to be calibrated and the positioning device implement soft synchronization of timestamps, that is, time alignment is achieved at the second level, but there is still a fixed time deviation at the millisecond level. Therefore, it is difficult to ensure that the three-dimensional position information and the two-dimensional image information are completely aligned only through soft synchronization of timestamps.

[0157] Based on this, according to an optional embodiment of the present specification, before constructing the coordinate pair based on the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data, the following may also be included:

[0158] Determine the target time deviation between the three-dimensional coordinates acquired by the positioning device and the image acquired by the camera to be calibrated;

[0159] Generate target third time data according to the target time deviation and the target second time data;

[0160] A reference three-dimensional coordinate corresponding to the target second time data is generated according to the target third time data, the target first time data, and the target three-dimensional coordinate corresponding to the target first time data.

[0161] Specifically, the target time deviation can be understood as the time deviation between the time system of the camera to be calibrated and the time system of the positioning device, so the target time deviation can be a fixed value. The target time deviation can also be called a fixed time deviation.

[0162] In practical applications, in order to optimize the target time deviation in the process of calculating the calibration parameter information and improve the accuracy of the calibration results, the target time deviation can be added to the target second time data, so as to achieve the alignment of the target second time data with the target first time data. Specifically, the target third time data can be understood as the time data corresponding to each three-dimensional coordinate collected by the positioning device after the target time deviation is optimized. The reference three-dimensional coordinates can be understood as the three-dimensional coordinates that the positioning device should actually collect after optimizing the target time deviation, that is, the three-dimensional coordinates corresponding to the target third time data.

[0163] Exemplarily, assuming that the positioning device collects three-dimensional coordinates every 100 milliseconds, and the first time data corresponding to the three-dimensional coordinates of the first target is 0 milliseconds, then the first time data corresponding to the three-dimensional coordinates of the second target is 100 milliseconds, and the first time data corresponding to the three-dimensional coordinates of the third target is 200 milliseconds..., assuming that the camera to be calibrated also captures the target image every 100 milliseconds, and the second time data corresponding to the two-dimensional coordinates of the first target is 50 milliseconds, then the second time data corresponding to the two-dimensional coordinates of the second target is 150 milliseconds, and the second time data corresponding to the two-dimensional coordinates of the third target is 250 milliseconds..., then there is a fixed time deviation of 50 milliseconds between the positioning device and the camera to be calibrated.

[0164] In practical applications, in order to avoid the influence of the fixed time deviation on the calibration accuracy, the target third time data can be generated according to the target time deviation and the target second time data. The generation formula of the target third time data is shown in the following formula (5).

[0165] Formula (5)

[0166] Among them, t is the target third time data, t c is the first time data, t δ is a fixed time deviation.

[0167] Furthermore, based on the target third time data, the target first time data, and the target three-dimensional coordinates corresponding to the target first time data, a reference three-dimensional coordinate corresponding to the target second time data can be generated. For a specific generation formula, see the following formula (6).

[0168] Formula (6)

[0169] in, The image coordinates taken by the camera The corresponding three-dimensional coordinates are also the reference three-dimensional coordinates. It is The three-dimensional coordinates of the target, yes The corresponding first time data. It is The three-dimensional coordinates of the target, yes The corresponding first time data. It is obtained by adding the first time data to the fixed time deviation.

[0170] Accordingly, constructing a coordinate pair based on the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data may include:

[0171] A coordinate pair is constructed based on the target two-dimensional coordinates corresponding to the target second time data and the reference three-dimensional coordinates corresponding to the target second time data.

[0172] By applying this embodiment, by determining a fixed time deviation, the third time data corresponding to the positioning device after optimizing the fixed time deviation is calculated based on the fixed time deviation, and the reference three-dimensional coordinates corresponding to the third time data are obtained. The reference three-dimensional coordinates can be determined as the three-dimensional coordinates corresponding to the image coordinates, thereby constructing a coordinate pair for camera calibration, realizing timestamp synchronization between the positioning device and the camera to be calibrated, and improving the accuracy of camera calibration.

[0173] According to an optional embodiment of the present specification, after calibrating the camera to be calibrated according to the coordinate pair, the method may further include:

[0174] Acquire reference two-dimensional coordinates and calibration parameter information of the camera to be calibrated, wherein the reference two-dimensional coordinates are obtained by projecting the target three-dimensional coordinates corresponding to the second time data onto a two-dimensional coordinate system, and the calibration parameter information is obtained by calibrating the camera to be calibrated;

[0175] Generate projection error information according to the target two-dimensional coordinates and the reference two-dimensional coordinates;

[0176] The calibration parameter information is adjusted according to the projection error information to obtain the adjusted calibration parameter information.

[0177] Specifically, the reference two-dimensional coordinates are obtained by projecting the target three-dimensional coordinates corresponding to the second time data to a two-dimensional coordinate system. The calibration parameter information is obtained based on calibrating the camera to be calibrated, and may include camera intrinsic parameters and camera extrinsic parameters. The projection error information can be understood as the coordinate distance difference between the target two-dimensional coordinates obtained by identifying the target image and the reference two-dimensional coordinates obtained by projecting the three-dimensional coordinates at the same time to a two-dimensional coordinate system.

[0178] In practical applications, when the camera to be calibrated is calibrated through the pinhole imaging principle and multiple sets of coordinate pairs to obtain the camera intrinsic parameters and camera extrinsic parameters, the camera intrinsic parameters and camera extrinsic parameters can be further optimized according to the projection error information, and a more accurate fixed time deviation between the camera to be calibrated and the positioning device can be obtained.

[0179] Optionally, the projection error information is generated according to the target two-dimensional coordinates and the reference two-dimensional coordinates, which can be achieved by the following formula (7).

[0180] Formula (7)

[0181] Specifically, K and R can be understood as camera internal parameters, t can be understood as camera external parameters, and k 1 ,k 2 ,k 3It can be understood as the radial distortion parameter of the camera, p 1 ,p 2 It can be understood as the tangential distortion parameter of the camera, t δ It can be understood as the optimized fixed time deviation. It can be understood as the target two-dimensional coordinates. It can be understood as a reference to two-dimensional coordinates.

[0182] By applying this embodiment, by performing image recognition on the target image, the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates are obtained, and the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates are obtained. The target three-dimensional coordinates and the target two-dimensional coordinates with a correlation relationship can be determined according to the first time data and the second time data. There is no need to build a high-precision three-dimensional model or a plane map, which can greatly reduce the camera calibration cost; there is no need to mark the two-dimensional-three-dimensional matching points of the image and the three-dimensional model, which can greatly improve the camera calibration efficiency; it can also solve the camera calibration in scenes without obvious texture features, greatly improving the scope of application; in addition, there is no need to modify the monitoring equipment and its network, and the time deviation can be used to characterize the data differences between multiple devices, and the time deviation can be optimized to achieve multi-device collaborative calibration, which has the characteristics of multiple usage scenarios, a wide range, and strong anti-interference ability.

[0183] See also Figure 3 , Figure 3 A flowchart of another camera calibration method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0184] Step 302: Acquire a target image captured by a surveillance camera during the movement of the positioning device in a target city scene, as well as target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates.

[0185] Step 304: Perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates.

[0186] Step 306: Calibrate the surveillance camera according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data, and the second time data.

[0187] It should be noted that the specific implementation method of step 302 to step 306 is the same as the specific implementation method of the above-mentioned step 202 to step 206. The detailed implementation method of step 302 to step 306 can refer to the above-mentioned step 202 to step 206, and this specification will not repeat it here.

[0188] By applying this embodiment, by acquiring the target image captured by the surveillance camera, based on image recognition, the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates are obtained, as well as the target three-dimensional coordinates collected by the positioning device and the corresponding first time data are obtained. The three-dimensional coordinates and the two-dimensional coordinates with a projection relationship can be determined according to the first time data and the second time data, so that the calibration parameter information of the surveillance camera can be calculated based on the projection relationship, and the automatic calibration of the surveillance camera in the target city scene can be realized; in this way, according to the calibration parameter information of the camera, the conversion relationship between the image two-dimensional coordinates and the spatial three-dimensional coordinates can be established, and the precise conversion between two-dimensional information and three-dimensional information can be realized, thereby improving the positioning accuracy and better meeting the needs of the city-level digital twin system.

[0189] See also Figure 4 , Figure 4 A flow chart of an image acquisition method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0190] Step 402: obtaining an image to be processed captured by the target camera for the target object during the movement of the target object, and target calibration parameters of the target camera, wherein the target calibration parameters are obtained based on calibration of the target camera, and calibrating the target camera includes: obtaining a target image captured by the positioning device during the movement of the positioning device, and target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; performing image recognition on the target image to obtain target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; and calibrating the target camera according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0191] Specifically, the target camera can be understood as a surveillance camera in a real scene. The target object can be understood as a person or object (such as a vehicle, flying equipment, etc.) moving in the display scene. The image to be processed can be understood as a two-dimensional image taken by the target camera of the target object during movement. The target calibration parameters can be understood as the camera intrinsic parameters and camera extrinsic parameters of the target camera.

[0192] Step 404: Perform image recognition on the image to be processed to obtain the two-dimensional position information of the target object.

[0193] Specifically, the two-dimensional position information can be understood as the two-dimensional pixel coordinates of the target object in the image to be processed, or the position of the region of interest of the target object in the image to be processed.

[0194] Step 406: Generate three-dimensional position information of the target object according to the two-dimensional position information and the target calibration parameters.

[0195] Specifically, the three-dimensional position information can be understood as the three-dimensional coordinate information of the target object in the real scene, which can be longitude and latitude information, etc.

[0196] By applying this embodiment, the target camera collects the processed images of the target object during its movement and the target calibration parameters of the target camera, the two-dimensional position information of the target object in the processed images can be converted into the three-dimensional position information of the target object in the real scene, thereby restoring the two-dimensional information to three-dimensional space, and improving the positioning accuracy.

[0197] In practical applications, based on the target calibration parameters of the target camera, in addition to converting the two-dimensional image information captured by the target camera into three-dimensional spatial information, it is also possible to convert the three-dimensional spatial information of the object into two-dimensional image information.

[0198] For example, when a vehicle is traveling on a road, relatively accurate latitude and longitude information of the vehicle can be obtained through high-precision positioning equipment. Through the target calibration parameters of each monitoring camera on the road, it can be determined which specific camera on the road can capture the vehicle at a specific time.

[0199] Therefore, by applying the camera calibration method provided in this manual to realize automatic calibration of camera intrinsic and extrinsic parameters, it is possible not only to restore the two-dimensional position information in the image to three-dimensional space, but also to restore the three-dimensional position information to the two-dimensional image, thereby realizing accurate conversion between three-dimensional coordinates and two-dimensional coordinates.

[0200] The following combination Figure 5 , taking the application of the camera calibration method provided in this specification in an urban scene as an example, the camera calibration method is further described. Figure 5 A processing flow chart of a camera calibration method provided in an embodiment of the present specification is shown, which specifically includes the following steps.

[0201] Step 502: Synchronize the time of the surveillance camera and the high-precision positioning device through the time synchronization unit.

[0202] Step 504: Obtain the target image captured by the monitoring camera during the movement of the high-precision positioning device, as well as the target three-dimensional coordinates captured by the high-precision positioning device during the movement and the first timestamp corresponding to the target three-dimensional coordinates.

[0203] Step 506: Automatically detect and identify the high-precision positioning device in each frame of the target image through the AI ​​algorithm, calculate and obtain the target two-dimensional coordinates of the high-precision positioning device, and associate the second timestamp corresponding to the target image with the target two-dimensional coordinates.

[0204] Step 508: Call the automatic calibration algorithm through the target three-dimensional coordinates, the target two-dimensional coordinates, the first timestamp and the second timestamp to calculate the camera intrinsic parameters and camera extrinsic parameters of the monitoring camera.

[0205] By applying this embodiment, by performing image recognition on the target image, the target two-dimensional coordinates of the high-precision positioning device in the target image and the second timestamp corresponding to the target two-dimensional coordinates are obtained, and the target three-dimensional coordinates collected by the high-precision positioning device during the movement and the first timestamp corresponding to the target three-dimensional coordinates are obtained. The target three-dimensional coordinates and the target two-dimensional coordinates with an associated relationship can be determined according to the first timestamp and the second timestamp. There is no need to build a high-precision three-dimensional model or a plane map, which can greatly reduce the camera calibration cost; there is no need to mark the two-dimensional-three-dimensional matching points of the image and the three-dimensional model, which can greatly improve the camera calibration efficiency; it can also solve the camera calibration in scenes without obvious texture features, greatly expand the application scope, and improve the camera calibration efficiency and accuracy.

[0206] Corresponding to the above method embodiment, this specification also provides a camera calibration device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a camera calibration device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0207] The first acquisition module 602 is configured to acquire the target image captured by the positioning device during the movement of the positioning device, the target three-dimensional coordinates collected by the positioning device during the movement, and the first time data corresponding to the target three-dimensional coordinates.

[0208] The first recognition module 604 is configured to perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates.

[0209] The first calibration module 606 is configured to calibrate the camera to be calibrated according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0210] Optionally, the camera calibration device further includes a calibration module configured to:

[0211] The time data of the image collected by the camera to be calibrated and the time data of the three-dimensional coordinates collected by the positioning device are calibrated.

[0212] Optionally, the first identification module 604 is further configured to:

[0213] Identify the positioning device in the target image and determine the target two-dimensional coordinates of the positioning device in the target image;

[0214] The second time data when the camera to be calibrated captures the target image is obtained, and the target two-dimensional coordinates and the second time data are associated.

[0215] Optionally, the first identification module 604 is further configured to:

[0216] Perform moving object recognition on the target image to obtain the position information of the moving object in the target image, wherein the moving object is equipped with a positioning device and moves within the field of view of the camera to be calibrated;

[0217] According to the position information, the target two-dimensional coordinates of the positioning device in the target image are determined.

[0218] Optionally, the first calibration module 606 is further configured to:

[0219] Determine the target three-dimensional coordinates and the target two-dimensional coordinates that have a projection relationship according to the first time data and the second time data;

[0220] Based on the target three-dimensional coordinates and the target two-dimensional coordinates that have a projection relationship, and the projection relationship, calibration parameter information of the camera to be calibrated is generated.

[0221] Optionally, the first calibration module 606 is further configured to:

[0222] Determine, from the first time data and the second time data, target first time data and target second time data whose time deviation is less than a preset time threshold;

[0223] The three-dimensional coordinates of the target corresponding to the first time data of the target and the two-dimensional coordinates of the target corresponding to the second time data of the target are obtained.

[0224] Optionally, the first calibration module 606 is further configured to:

[0225] constructing a coordinate pair based on the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data;

[0226] Based on the coordinate alignment, the projection relationship between the target two-dimensional coordinates and the target three-dimensional coordinates, the calibration parameter information of the camera to be calibrated is generated.

[0227] Optionally, the first calibration module 606 is further configured to:

[0228] Determine the target time deviation between the three-dimensional coordinates acquired by the positioning device and the image acquired by the camera to be calibrated;

[0229] Generate target third time data according to the target time deviation and the target second time data;

[0230] Generate reference three-dimensional coordinates corresponding to the target second time data according to the target third time data, the target first time data, and the target three-dimensional coordinates corresponding to the target first time data; the first calibration module 606 is further configured to:

[0231] A coordinate pair is constructed based on the target two-dimensional coordinates corresponding to the target second time data and the reference three-dimensional coordinates corresponding to the target second time data.

[0232] Optionally, the first calibration module 606 is further configured to:

[0233] Acquire reference two-dimensional coordinates and calibration parameter information of the camera to be calibrated, wherein the reference two-dimensional coordinates are obtained by projecting the target three-dimensional coordinates corresponding to the second time data onto a two-dimensional coordinate system, and the calibration parameter information is obtained by calibrating the camera to be calibrated;

[0234] Generate projection error information according to the target two-dimensional coordinates and the reference two-dimensional coordinates;

[0235] The calibration parameter information is adjusted according to the projection error information to obtain the adjusted calibration parameter information.

[0236] By applying this embodiment, by acquiring the target image captured by the camera to be calibrated, based on image recognition, the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates are obtained, and the target three-dimensional coordinates collected by the positioning device and the corresponding first time data are obtained. The three-dimensional coordinates and the two-dimensional coordinates with a projection relationship can be determined according to the first time data and the second time data, so that the calibration parameter information of the camera to be calibrated can be calculated based on the projection relationship, and the automatic calibration of the camera to be calibrated can be realized; in this way, according to the calibration parameter information of the camera, the conversion relationship between the image two-dimensional coordinates and the space three-dimensional coordinates can be established, and the precise conversion between the two-dimensional information and the three-dimensional information can be realized, thereby improving the positioning accuracy.

[0237] The above is a schematic scheme of a camera calibration device of this embodiment. It should be noted that the technical scheme of the camera calibration device and the technical scheme of the camera calibration method described above are of the same concept, and the details not described in detail in the technical scheme of the camera calibration device can be referred to the description of the technical scheme of the camera calibration method described above.

[0238] Corresponding to the above method embodiment, this specification also provides another camera calibration device embodiment, Figure 7 FIG. 2 shows a schematic diagram of the structure of another camera calibration device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0239] The second acquisition module 702 is configured to acquire, in the target city scene, a target image captured by the monitoring camera during the movement of the positioning device, as well as target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates.

[0240] The second recognition module 704 is configured to perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates.

[0241] The second calibration module 706 is configured to calibrate the surveillance camera according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0242] By applying this embodiment, by acquiring the target image captured by the surveillance camera, based on image recognition, the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates are obtained, as well as the target three-dimensional coordinates collected by the positioning device and the corresponding first time data are obtained. The three-dimensional coordinates and the two-dimensional coordinates with a projection relationship can be determined according to the first time data and the second time data, so that the calibration parameter information of the surveillance camera can be calculated based on the projection relationship, and the automatic calibration of the surveillance camera in the target city scene can be realized; in this way, according to the calibration parameter information of the camera, the conversion relationship between the image two-dimensional coordinates and the spatial three-dimensional coordinates can be established, and the precise conversion between two-dimensional information and three-dimensional information can be realized, thereby improving the positioning accuracy and better meeting the needs of the city-level digital twin system.

[0243] The above is a schematic scheme of another camera calibration device of this embodiment. It should be noted that the technical scheme of the camera calibration device and the technical scheme of the camera calibration method described above belong to the same concept, and the details not described in detail in the technical scheme of the camera calibration device can be referred to the description of the technical scheme of the camera calibration method described above.

[0244] Corresponding to the above method embodiment, this specification also provides an image acquisition device embodiment, Figure 8 FIG. 1 is a schematic diagram showing the structure of an image acquisition device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0245] The third acquisition module 802 is configured to acquire the to-be-processed image captured by the target camera for the target object during the movement of the target object, and the target calibration parameters of the target camera, wherein the target calibration parameters are acquired based on the calibration of the target camera, and the calibration of the target camera includes: acquiring the target image captured by the positioning device during the movement of the positioning device, and the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates; performing image recognition on the target image to acquire the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates; and calibrating the target camera according to the target three-dimensional coordinates, the target two-dimensional coordinates, the first time data and the second time data.

[0246] The third recognition module 804 is configured to perform image recognition on the image to be processed to obtain the two-dimensional position information of the target object.

[0247] The generating module 806 is configured to generate the three-dimensional position information of the target object according to the two-dimensional position information and the target calibration parameters.

[0248] By applying this embodiment, the target camera collects the processed images of the target object during its movement and the target calibration parameters of the target camera, the two-dimensional position information of the target object in the processed images can be converted into the three-dimensional position information of the target object in the real scene, thereby restoring the two-dimensional information to three-dimensional space, and improving the positioning accuracy.

[0249] The above is a schematic scheme of an image acquisition device of this embodiment. It should be noted that the technical scheme of the image acquisition device and the technical scheme of the above-mentioned image acquisition method belong to the same concept, and the details not described in detail in the technical scheme of the image acquisition device can be referred to the description of the technical scheme of the above-mentioned image acquisition method.

[0250] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0251] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0252] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0253] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0254] The processor 920 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned camera calibration method or image acquisition method.

[0255] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the camera calibration method or image acquisition method described above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the camera calibration method or image acquisition method described above.

[0256] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned camera calibration method or image acquisition method.

[0257] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the camera calibration method or image acquisition method described above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the camera calibration method or image acquisition method described above.

[0258] An embodiment of the present specification further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned camera calibration method or image acquisition method when executed by a processor.

[0259] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the camera calibration method or image acquisition method described above belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the camera calibration method or image acquisition method described above.

[0260] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0261] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0262] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0263] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0264] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A camera calibration method, comprising: Acquire a target image captured by the positioning device during the movement of the positioning device, a target three-dimensional coordinates captured by the positioning device during the movement, and first time data corresponding to the target three-dimensional coordinates; Performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; Determine, from the first time data and the second time data, target first time data and target second time data whose time deviation is less than a preset time threshold; Acquire the target three-dimensional coordinates corresponding to the target first time data, and the target two-dimensional coordinates corresponding to the target second time data; Determine a target time deviation between the three-dimensional coordinates acquired by the positioning device and the image acquired by the camera to be calibrated; Generate target third time data according to the target time deviation and the target first time data, wherein the target third time data is between the target first time data and the next first time data of the target first time data; Generate reference three-dimensional coordinates corresponding to the target second time data according to the target third time data, the difference between the target first time data and the next first time data, the target three-dimensional coordinates corresponding to the target first time data, and the target three-dimensional coordinates corresponding to the next first time data; Based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, calibration parameter information of the camera to be calibrated is generated, wherein the projection relationship represents the reference three-dimensional coordinates and the target two-dimensional coordinates, and is associated with the camera intrinsic parameters and camera extrinsic parameters of the camera to be calibrated, and the calibration parameter information includes the camera intrinsic parameters and the camera extrinsic parameters.

2. The method according to claim 1, before obtaining the target image captured by the positioning device during the movement of the positioning device, the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates, further comprises: Calibrate the time data of the image collected by the camera to be calibrated and the three-dimensional coordinates collected by the positioning device.

3. The method according to claim 1, wherein the step of performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates comprises: Identify the positioning device in the target image and determine the target two-dimensional coordinates of the positioning device in the target image; The second time data when the camera to be calibrated captures the target image is obtained, and the target two-dimensional coordinates and the second time data are associated.

4. The method according to claim 3, wherein the step of identifying the positioning device in the target image and determining the target two-dimensional coordinates of the positioning device in the target image comprises: Performing mobile object recognition on the target image to obtain position information of the mobile object in the target image, wherein the mobile object carries the positioning device and moves within the field of view of the camera to be calibrated; The target two-dimensional coordinates of the positioning device in the target image are determined according to the position information.

5. The method according to claim 1, wherein generating calibration parameter information of the camera to be calibrated based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data comprises: constructing a coordinate pair based on the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data; Based on the coordinate pairing, the projection relationship between the target two-dimensional coordinates and the reference three-dimensional coordinates, calibration parameter information of the camera to be calibrated is generated.

6. The method according to claim 5, after generating the calibration parameter information of the camera to be calibrated based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, further comprising: Acquire reference two-dimensional coordinates and calibration parameter information of the camera to be calibrated, wherein the reference two-dimensional coordinates are obtained by projecting the reference three-dimensional coordinates corresponding to the second time data onto a two-dimensional coordinate system, and the calibration parameter information is obtained by calibrating the camera to be calibrated; Generate projection error information according to the target two-dimensional coordinates and the reference two-dimensional coordinates; The calibration parameter information is adjusted according to the projection error information to obtain adjusted calibration parameter information.

7. A camera calibration method, comprising: Acquire, in a target city scene, a target image captured by a monitoring camera during the movement of the positioning device, as well as target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; Performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; Determine, from the first time data and the second time data, target first time data and target second time data whose time deviation is less than a preset time threshold; Acquire the target three-dimensional coordinates corresponding to the target first time data, and the target two-dimensional coordinates corresponding to the target second time data; Determine a target time deviation between the three-dimensional coordinates acquired by the positioning device and the image acquired by the monitoring camera; Generate target third time data according to the target time deviation and the target first time data, wherein the target third time data is between the target first time data and the next first time data of the target first time data; Generate reference three-dimensional coordinates corresponding to the target second time data according to the target third time data, the difference between the target first time data and the next first time data, the target three-dimensional coordinates corresponding to the target first time data, and the target three-dimensional coordinates corresponding to the next first time data; Based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, the calibration parameter information of the monitoring camera is generated, wherein the projection relationship represents the reference three-dimensional coordinates and the target two-dimensional coordinates, and is associated with the camera intrinsic parameters and camera extrinsic parameters of the monitoring camera, and the calibration parameter information includes the camera intrinsic parameters and the camera extrinsic parameters.

8. An image acquisition method, comprising: Acquire an image to be processed, which is captured by a target camera during the movement of the target object, and target calibration parameters of the target camera, wherein the target calibration parameters are obtained based on calibration of the target camera, and the calibration of the target camera comprises: acquiring a target image, which is captured by the target camera during the movement of the positioning device, and target three-dimensional coordinates captured by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; performing image recognition on the target image to obtain target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; determining target first time data and target second time data whose time deviation is less than a preset time threshold from the first time data and the second time data; acquiring target three-dimensional coordinates corresponding to the target first time data, and target two-dimensional coordinates corresponding to the target second time data; determining the positioning The target time deviation between the three-dimensional coordinates collected by the device and the image collected by the target camera; generating target third time data according to the target time deviation and the target first time data, wherein the target third time data is between the target first time data and the next first time data of the target first time data; generating reference three-dimensional coordinates corresponding to the target second time data according to the difference between the target third time data, the target first time data and the next first time data, and the target three-dimensional coordinates corresponding to the target first time data and the target three-dimensional coordinates corresponding to the next first time data; generating target calibration parameters of the target camera based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, wherein the projection relationship characterizes the relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates, and is associated through the camera intrinsic parameters and camera extrinsic parameters of the target camera; Performing image recognition on the image to be processed to obtain two-dimensional coordinate data of the target object; The three-dimensional coordinate data of the target object is generated according to the two-dimensional coordinate data and the target calibration parameters.

9. A camera calibration system, comprising a server, a camera to be calibrated and a positioning device; The camera to be calibrated is configured to capture a target image of the positioning device during movement; The positioning device is configured to collect the three-dimensional coordinates of the target during movement and the first time data corresponding to the three-dimensional coordinates of the target; The server is configured to obtain a target image captured by the camera to be calibrated during the movement of the positioning device, as well as target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; Performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; Determine, from the first time data and the second time data, target first time data and target second time data whose time deviation is less than a preset time threshold; Acquire the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data; determine the target time deviation between the three-dimensional coordinates collected by the positioning device and the image collected by the camera to be calibrated; Generate target third time data according to the target time deviation and the target first time data, wherein the target third time data is located between the target first time data and the next first time data of the target first time data; generate reference three-dimensional coordinates corresponding to the target second time data according to the difference between the target third time data, the target first time data and the next first time data, and the target three-dimensional coordinates corresponding to the target first time data and the target three-dimensional coordinates corresponding to the next first time data; generate calibration parameter information of the camera to be calibrated based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, wherein the projection relationship represents the reference three-dimensional coordinates and the target two-dimensional coordinates, and is associated through the camera intrinsic parameters and camera extrinsic parameters of the camera to be calibrated, and the calibration parameter information includes the camera intrinsic parameters and the camera extrinsic parameters.

10. The system according to claim 9, wherein the server is further configured to identify the positioning device in the target image and determine the target two-dimensional coordinates of the positioning device in the target image; The second time data when the camera to be calibrated captures the target image is obtained, and the target two-dimensional coordinates and the second time data are associated.

11. The system according to claim 10, wherein the positioning device is mounted on a mobile object; The server is further configured to perform moving object recognition on the target image to obtain position information of the moving object in the target image, wherein: The mobile object carries the positioning device and moves within the field of view of the camera to be calibrated; The target two-dimensional coordinates of the positioning device in the target image are determined according to the position information.

12. A camera calibration device, comprising: A first acquisition module is configured to acquire a target image captured by the positioning device during the movement of the positioning device, as well as target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; A first recognition module is configured to perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; A first calibration module is configured to determine, from the first time data and the second time data, target first time data and target second time data whose time deviation is less than a preset time threshold; Acquire the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data; determine the target time deviation between the three-dimensional coordinates collected by the positioning device and the image collected by the camera to be calibrated; Generate target third time data according to the target time deviation and the target first time data, wherein the target third time data is located between the target first time data and the next first time data of the target first time data; generate reference three-dimensional coordinates corresponding to the target second time data according to the difference between the target third time data, the target first time data and the next first time data, and the target three-dimensional coordinates corresponding to the target first time data and the target three-dimensional coordinates corresponding to the next first time data; generate calibration parameter information of the camera to be calibrated based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, wherein the projection relationship represents the reference three-dimensional coordinates and the target two-dimensional coordinates, and is associated through the camera intrinsic parameters and camera extrinsic parameters of the camera to be calibrated, and the calibration parameter information includes the camera intrinsic parameters and the camera extrinsic parameters.

13. A camera calibration device, comprising: The second acquisition module is configured to acquire, in the target city scene, a target image captured by the monitoring camera during the movement of the positioning device, as well as target three-dimensional coordinates collected by the positioning device during the movement and first time data corresponding to the target three-dimensional coordinates; A second recognition module is configured to perform image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and second time data corresponding to the target two-dimensional coordinates; A second calibration module is configured to determine, from the first time data and the second time data, target first time data and target second time data whose time deviation is less than a preset time threshold; Obtaining the target three-dimensional coordinates corresponding to the target first time data and the target two-dimensional coordinates corresponding to the target second time data; determining the target time deviation between the three-dimensional coordinates collected by the positioning device and the image collected by the monitoring camera; Generate target third time data according to the target time deviation and the target first time data, wherein the target third time data is located between the target first time data and the next first time data of the target first time data; generate reference three-dimensional coordinates corresponding to the target second time data according to the difference between the target third time data, the target first time data and the next first time data, and the target three-dimensional coordinates corresponding to the target first time data and the target three-dimensional coordinates corresponding to the next first time data; generate calibration parameter information of the monitoring camera based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, wherein the projection relationship represents the reference three-dimensional coordinates and the target two-dimensional coordinates, and is associated through the camera intrinsic parameters and camera extrinsic parameters of the monitoring camera, and the calibration parameter information includes the camera intrinsic parameters and the camera extrinsic parameters.

14. An image acquisition device, comprising: The third acquisition module is configured to acquire the to-be-processed image captured by the target camera for the target object during the movement of the target object, and the target calibration parameters of the target camera, wherein the target calibration parameters are obtained based on the calibration of the target camera, and the calibration of the target camera includes: acquiring the target image captured by the target camera for the positioning device during the movement of the positioning device, and the target three-dimensional coordinates collected by the positioning device during the movement and the first time data corresponding to the target three-dimensional coordinates; performing image recognition on the target image to obtain the target two-dimensional coordinates of the positioning device and the second time data corresponding to the target two-dimensional coordinates; determining the target first time data and the target second time data whose time deviation is less than a preset time threshold from the first time data and the second time data; acquiring the target three-dimensional coordinates corresponding to the target first time data, and the target two-dimensional coordinates corresponding to the target second time data; determining Determine the target time deviation between the three-dimensional coordinates collected by the positioning device and the image collected by the target camera; generate target third time data according to the target time deviation and the target first time data, wherein the target third time data is between the target first time data and the next first time data of the target first time data; generate reference three-dimensional coordinates corresponding to the target second time data according to the difference between the target third time data, the target first time data and the next first time data, and the target three-dimensional coordinates corresponding to the target first time data and the target three-dimensional coordinates corresponding to the next first time data; generate target calibration parameters of the target camera based on the projection relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates corresponding to the target second time data, wherein the projection relationship represents the relationship between the reference three-dimensional coordinates and the target two-dimensional coordinates, which are associated through the camera intrinsic parameters and camera extrinsic parameters of the target camera; A third recognition module is configured to perform image recognition on the image to be processed to obtain two-dimensional coordinate data of the target object; The generating module is configured to generate the three-dimensional coordinate data of the target object according to the two-dimensional coordinate data and the target calibration parameters.

15. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

16. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

17. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Citation Information

Cited By

  • Method and system for establishing coordinates of target region

    TWI941508B