Method and apparatus for determining a monitoring camera's field of view

Abstract

The embodiment of the application discloses a method and device for determining the visual field of a monitoring camera. The method comprises: acquiring a first image collected by the monitoring camera at a current pose; acquiring a second image matched with the first image from a pre-established pre-calibration data set, wherein the pre-calibration data set comprises preset position images collected by the monitoring camera at multiple poses and camera parameters corresponding to the preset position images; determining matching feature points of the first image and the second image, determining camera parameters corresponding to the first image by using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image; and determining the visual field of the monitoring camera at the current pose by using the camera parameters corresponding to the first image and a camera imaging model. The application can realize dynamic calculation of the visual field of the monitoring camera, and accurately determine the visual field of the monitoring camera even if the pose of the monitoring camera changes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, in particular to a method and device for determining a visual field of a monitoring camera. BACKGROUND

[0002] With the rapid development of cities and social technology, security monitoring technology has become an important technology for maintaining social security. Visual field calculation is a key technology for ensuring monitoring systems. Through visual field calculation, the visual range of each monitoring camera can be obtained, and the visual blind area of the monitoring system can be accurately reflected. Based on the visual field and visual blind area of the monitoring camera, the camera distribution can be better planned, and the utilization rate of the equipment can be improved while ensuring the coverage of the monitoring system.

[0003] There are many types of monitoring cameras, and some monitoring cameras with large visual range and flexible control have gradually developed, such as ball cameras (i.e., ball-type cameras). The installation ratio of ball cameras is high in actual deployment and application. Taking a ball camera as an example, during the operation of the ball camera, human control or natural factors (wind blowing, vibration, mechanical deviation of preset position, etc.) can cause the pose (i.e., position and attitude) of the ball camera to change, which brings great difficulty to the visual field calculation of the ball camera and limits the further application of the ball camera. Therefore, how to accurately determine the visual field of the monitoring camera such as the ball camera becomes a challenging and urgent problem to be solved. SUMMARY

[0004] Therefore, the present application provides a method and device for determining the visual field of a monitoring camera to improve the accuracy of determining the visual field of the monitoring camera.

[0005] The present application provides the following solutions:

[0006] In a first aspect, a method for determining the visual field of a monitoring camera is provided, and the method comprises:

[0007] obtaining a first image collected by the monitoring camera at a current pose;

[0008] obtaining a second image matched with the first image from a pre-established pre-calibration data set, wherein the pre-calibration data set contains preset position images collected by the monitoring camera at multiple poses and camera parameters corresponding to each preset position image;

[0009] determining matching feature points of the first image and the second image, and determining camera parameters corresponding to the first image by using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image;

[0010] determining the visual field of the monitoring camera at the current pose by using the camera parameters corresponding to the first image and a camera imaging model.

[0011] According to one achievable method in an embodiment of this application, the establishment of the pre-calibration data set includes:

[0012] Preset position images of the monitoring camera in multiple poses are acquired in advance, and the camera parameters corresponding to the preset position images are determined respectively.

[0013] According to one achievable method in the embodiments of this application, the camera parameters corresponding to the preset bit image are determined as follows:

[0014] Obtain the feature points calibrated for the preset position image and the corresponding points in three-dimensional space, and obtain the initial camera parameters calibrated for the preset position image;

[0015] The images captured by the monitoring camera during continuous rotation and scaling are obtained, and the captured images are combined with the preset position images to form an image sequence.

[0016] Determine the matching feature points of each image in the image sequence, and determine the map points corresponding to each matching feature point, wherein the map points are points in three-dimensional space;

[0017] The initial camera parameters corresponding to the preset position image are optimized by using the coordinate relationship of the same map point in each image of the image sequence.

[0018] According to one achievable method in an embodiment of this application, determining the map points corresponding to each matching feature point includes:

[0019] For each map point, the following steps are performed: the initial coordinates of the map point are obtained by minimizing the first loss function. The first loss function reflects the difference between the coordinates obtained by transforming the initial coordinates of the map point to the image plane using the initial camera parameters of each image and the coordinates of the corresponding feature point of the map point in each image.

[0020] According to one achievable method in an embodiment of this application, the initial camera parameters corresponding to the preset position image are optimized using the coordinate relationship of the same map point in each image of the image sequence, resulting in camera parameters corresponding to the preset position image including:

[0021] By minimizing the second loss function, the camera parameters corresponding to the preset position image are obtained. The second loss function reflects the difference between the coordinates of each map point transformed to the preset position image plane using the camera parameters corresponding to the preset position image and the corresponding points of the map points on the preset position image.

[0022] According to one achievable method in an embodiment of this application, after obtaining the camera parameters corresponding to the preset image, the method further includes:

[0023] convert a camera extrinsic parameter in the camera parameters corresponding to the preset position image to a geographic coordinate system by using a transformation matrix;

[0024] The transformation matrix is obtained by minimizing a third loss function, and the third loss function reflects a difference between a coordinate of each map point converted to a preset position image plane by using the transformation matrix and a corresponding point of the map point on the preset position image.

[0025] According to an implementable manner in the embodiments of the present application, obtaining the second image matching the first image from a pre-established pre-calibration data set comprises:

[0026] performing feature extraction and first matching processing on feature points of the first image and the preset position image in the pre-calibration data set, respectively;

[0027] If the number of matching feature points between the preset position image and the first image is greater than or equal to a preset threshold, the preset position image is determined as the second image; otherwise, a historical image having determined corresponding camera parameters is obtained from historical images collected by the monitoring camera before the present, and the obtained historical image and the first image are subjected to first matching processing;

[0028] If the number of matching feature points between the obtained historical image and the first image is greater than or equal to a preset threshold, the historical image is determined as the second image; otherwise, the first image, the preset position image in the pre-calibration data set, and the obtained historical image are subjected to second matching processing;

[0029] If the number of matching feature points between an image in the preset position image in the pre-calibration data set and the first image is greater than or equal to a preset threshold, the image is determined as the second image.

[0030] According to an implementable manner in the embodiments of the present application, determining the camera parameters corresponding to the first image by using a coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image comprises:

[0031] determining a homographic transformation matrix between the first image and the second image;

[0032] determining initial camera parameters corresponding to the first image by using the camera parameters corresponding to the second image and the homographic transformation matrix;

[0033] On the basis of the initial camera parameters, camera parameters corresponding to the first image are obtained by minimizing a fourth loss function, wherein the fourth loss function reflects a difference between coordinates of the matching feature points on the second image after being converted to the first image plane by using the camera parameters of the first image and the camera parameters of the second image and coordinates of the corresponding matching feature points on the first image.

[0034] According to an implementable manner in embodiments of the present application, the camera parameters include camera intrinsic parameters and camera extrinsic parameters.

[0035] The initial camera parameters corresponding to the first image are determined by using the camera parameters corresponding to the second image and the homographic transformation matrix, including:

[0036] The camera intrinsic parameters corresponding to the second image are taken as the initial camera intrinsic parameters corresponding to the first image.

[0037] The initial camera extrinsic parameters corresponding to the first image are determined by using the initial camera intrinsic parameters of the first image, the homographic transformation matrix, and the camera extrinsic parameters corresponding to the second image.

[0038] According to an implementable manner in embodiments of the present application, the visual field of the monitoring camera at the current pose is determined by using the camera parameters corresponding to the first image and a camera imaging model, including:

[0039] The current position and attitude angle of the monitoring camera are determined by using the camera parameters corresponding to the first image.

[0040] The frustum of the monitoring camera at the current pose is determined based on a camera imaging model by using the current position, attitude angle, and field of view angle of the monitoring camera, wherein the field of view angle is determined by a focal length of the monitoring camera and an image resolution of the first image.

[0041] The visual field of the camera is determined by using a range covered by the frustum.

[0042] In a second aspect, a method for determining a visual field of a monitoring camera is provided, which is executed by a cloud server and includes:

[0043] A first image collected by a monitoring camera in a monitoring system at a current pose is acquired.

[0044] A second image matched with the first image is acquired from a pre-established pre-calibration data set, wherein the pre-calibration data set includes preset position images respectively collected by the monitoring camera at multiple poses and camera parameters corresponding to the preset position images.

[0045] determining matching feature points of the first image and the second image, determining the camera parameters corresponding to the first image by using coordinate relations of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image;

[0046] determining the visual field of the monitoring camera in the current pose by using the camera parameters corresponding to the first image and a camera imaging model;

[0047] sending information of the visual field to a display device in the monitoring system for visualization.

[0048] In a third aspect, an apparatus for determining a visual field of a monitoring camera is provided, and the apparatus can include:

[0049] an image acquisition unit configured to acquire a first image captured by the monitoring camera in a current pose;

[0050] an image matching unit configured to acquire a second image matched with the first image from a pre-established pre-calibration data set, the pre-calibration data set containing preset position images respectively captured by the monitoring camera in multiple poses and camera parameters corresponding to each preset position image;

[0051] a parameter estimation unit configured to determine matching feature points of the first image and the second image, and determine the camera parameters corresponding to the first image by using coordinate relations of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image;

[0052] a visual field determination unit configured to determine the visual field of the monitoring camera in the current pose by using the camera parameters corresponding to the first image and a camera imaging model.

[0053] According to a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of the first aspect and the second aspect.

[0054] According to a fifth aspect, an electronic device is provided, and the electronic device includes:

[0055] one or more processors; and

[0056] a memory associated with the one or more processors, the memory configured to store program instructions that, when executed by the one or more processors, perform the steps of the method according to any one of the first aspect and the second aspect.

[0057] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0058] 1) The application acquires a first image collected by the monitoring camera in real time in the current pose, acquires a second image matching the first image from the pre-calibration data set containing the preset pose image, and then determines the camera parameters of the first image by using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image, and further determines the visual field of the monitoring camera in the current pose in combination with the camera imaging model. This method can realize dynamic calculation of the visual field of the monitoring camera, and can accurately determine the visual field of the monitoring camera even if the pose of the monitoring camera changes.

[0059] 2) The application matches the first image collected by the monitoring camera with the preset pose image in the pre-calibration data set to dynamically determine the visual field, which does not depend on hardware devices and can be applied to devices of various manufacturers, models and styles, has strong universality and higher accuracy.

[0060] 3) In the calibration process of the preset pose image, the application acquires an image sequence composed of the preset pose image and the images collected by the monitoring camera in the continuous rotation and scaling process, optimizes the initial camera parameters corresponding to the preset pose image by using the coordinate relationship of the same map point in each image in the image sequence, thereby realizing automatic calibration of the preset pose image in a complex environment and improving the calibration accuracy of the preset pose image.

[0061] 4) In the process of matching the first image with the preset pose image to acquire the second image, the application adopts a multi-level matching method, which improves the matching success rate while taking into account the calculation efficiency.

[0062] 5) When determining the camera parameters corresponding to the first image, the application not only estimates the parameters based on the homographic transformation matrix, but also uses the camera parameters determined based on the homographic transformation matrix as the initial camera parameters, and then further optimizes the initial camera parameters by minimizing the fourth loss function (the fourth loss function reflects the difference between the coordinates of the matching feature points on the second image converted to the first image plane by using the camera parameters of the first image and the camera parameters of the second image and the coordinates of the corresponding matching feature points on the first image), to obtain the camera parameters corresponding to the first image, thereby improving the accuracy of the camera parameters corresponding to the first image and further improving the accuracy of the determined visual field.

[0063] Of course, any product implementing the application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only illustrate some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

[0065] Figure 1 A system architecture diagram applicable to the embodiments of the present application;

[0066] Figure 2 A main method flowchart for determining the visual field of the monitoring camera provided by the embodiments of the present application;

[0067] Figure 3 A method flowchart for establishing a pre-calibration data set provided by the embodiments of the present application;

[0068] Figure 4 A method flowchart for determining the camera parameters corresponding to the first image provided by the embodiments of the present application;

[0069] Figure 5 A schematic diagram of the monitoring camera visual cone provided by the embodiments of the present application;

[0070] Figure 6 A schematic block diagram of the apparatus for determining the visual field of the monitoring camera provided by the embodiments of the present application;

[0071] Figure 7 A schematic block diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0073] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0074] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0075] Depending on context, the word "if" can be interpreted to mean "when" or "while" or "in response to determining," or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "when it is determined" or "in response to determining" or "when [the stated condition or event] is detected" or "in response to detecting [the stated condition or event]."

[0076] At present, there are some methods for determining the visual field of a monitoring model such as a ball camera, for example, using a hardware device such as a pan-tilt head to obtain the rotation angle and focal length of the ball camera. A user can use a ball camera SDK (Software Development Kit) for secondary development to calibrate the ball camera, and then determine the value of the visual field. However, the visual field determined by the current calibration scheme based on hardware devices has low accuracy and poor universality.

[0077] In view of this, the present application provides a new visual field determination method. In order to facilitate the understanding of the present application, Figure 1 An exemplary system architecture to which embodiments of the present application can be applied is shown. The system architecture mainly includes a visual field determination apparatus and a monitoring camera.

[0078] The monitoring camera is arranged in the actual environment as needed to collect image data of the surrounding environment. The camera referred to in the embodiments of the present application refers to a visual sensor, which is a relatively broad meaning, and is an instrument for obtaining external environment images using optical elements and imaging devices. It can be a traditional camera, a digital camera, a camera, a video camera, etc. The monitoring camera referred to in the embodiments of the present application can be flexibly rotated and zoomed, for example, a ball camera or the like can be used. The ball camera is a camera that can realize real-time monitoring in all directions, can realize 360-degree panoramic shooting, and can perform pan-tilt control, multi-zoom, etc.

[0079] The visual field determination apparatus is used to obtain the images collected by the monitoring camera, and the visual field of each monitoring camera is determined by using the method provided in the embodiments of the present application. After the visual field of each monitoring camera is determined, it can be provided to a monitoring control system, and the monitoring control system can evaluate the coverage of the monitoring camera, adjust the layout of the monitoring camera using the visual field of each monitoring camera, etc.

[0080] In addition, other devices can also be included in the system, for example, a display device for displaying the images collected by the monitoring camera, a device for tracking a target using the images collected by the monitoring camera, etc. Figure 1 None of these devices are shown in the figure.

[0081] The visual field determination device described above can be arranged on a server, can be arranged in a single server, can be arranged in a server group composed of multiple servers, or can be arranged in a cloud server. The cloud server is also called a cloud computing server or a cloud host, which is a host product in a cloud computing service system, and is used to solve the defects of large management difficulty and weak service scalability in traditional physical hosts and virtual private server (VPS, Virtual Private Server) services. It can also be arranged in a computer terminal with strong computing power.

[0082] It should be understood that Figure 1 The number of visual field determination devices, monitoring control devices and monitoring cameras in the above-mentioned system is only illustrative. According to the needs of implementation, there can be any number of visual field determination devices, monitoring control devices and monitoring cameras.

[0083] Figure 2 The main method flow chart for determining the visual field of the monitoring camera provided by the embodiment of the present application can be executed by the visual field determination device in the system shown in Figure 1 As shown in Figure 2 The method can include the following steps:

[0084] Step 202: Obtain a first image collected by the monitoring camera at a current pose.

[0085] Step 204: Obtain a second image matched with the first image from a pre-established pre-calibration data set, and the pre-calibration data set contains preset position images collected by the monitoring camera at multiple poses and camera parameters corresponding to each preset position image.

[0086] Step 206: Determine the matching feature points of the first image and the second image, and determine the camera parameters corresponding to the first image by using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image.

[0087] Step 208: Determine the visual field of the monitoring camera at the current pose by using the camera parameters corresponding to the first image and the camera imaging model.

[0088] As can be seen from the above flow, the present application pre-calibrates the camera parameters of the preset position images, obtains a first image collected by the monitoring camera at a current pose in real time, obtains a second image matched with the first image from a pre-calibration data set containing the preset position images, and then determines the camera parameters of the first image by using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image, and further determines the visual field of the monitoring camera at the current pose in combination with the camera imaging model. This method can realize dynamic calculation of the visual field of the monitoring camera, and can accurately determine the visual field of the monitoring camera even if the pose of the monitoring camera changes.

[0089] The steps in the above process will be described in detail below in conjunction with embodiments. It should be noted that the "first", "second", and the like in the present disclosure do not have the limitations of size, order, and quantity, and are only used to distinguish the names. For example, "first image" and "second image" are used to distinguish two images in name to facilitate description. For another example, "first loss function", "second loss function", "third loss function", and "fourth loss function" are used to distinguish four loss functions in name.

[0090] Since the method flow of determining the visual field of the monitoring camera needs to use the pre-calibration data set, in order to facilitate understanding, the establishment process of the pre-calibration data set will be described in detail first.

[0091] The pre-calibration data set is pre-established, for example, the pre-calibration data set can be pre-established in an offline manner for online determination of the visual field of the monitoring camera. The pre-calibration data set includes pre-set position images respectively captured by the monitoring camera at multiple poses and camera parameters corresponding to each pre-set position image.

[0092] The so-called pre-set position image refers to an image captured at a pre-set pose. In the embodiments of the present application, the monitoring camera can be controlled to capture images at multiple poses as pre-set position images, and the corresponding pose information of each pre-set position image can be embodied by, for example, view angle information. Preferably, images under different poses, different seasons, weather conditions, and light conditions can be captured, and some of them (for example, 1-2 images under different poses, different seasons, weather conditions, and light conditions) can be selected as pre-set position images.

[0093] The pre-set position images captured need to meet a preset overlap degree, so as to ensure that the pre-set position images captured within all possible view angle ranges cover all points in the environment that can be captured by the monitoring camera.

[0094] For each pre-set position image, the feature points on the image can be manually labeled with the corresponding three-dimensional feature points in the three-dimensional model, and the camera parameters of each pre-set position image can be calibrated. The feature points can be pixel points on the image, or corner points on the image, etc. The so-called corner point is an extreme point, which is usually a point with particularly prominent properties in the image, and is usually a very important feature in the image, which can be, for example, an edge intersection point, a point with a local maximum gradient value in the image, etc.

[0095] The camera parameters involved in the embodiments of the present application mainly include camera intrinsic parameters and camera extrinsic parameters, and can also include distortion coefficients and other parameters. The camera intrinsic parameters can include camera focal length, principal point position, skew coefficient, etc. The principal point is the center of the camera image. The camera extrinsic parameters can include the position and attitude of the camera in the geographic coordinate system. The distortion coefficient is a parameter describing the distortion that occurs in the camera imaging process. Due to the deviation of lens manufacturing precision and assembly process, distortion occurs in the camera imaging process, which can be generally divided into radial distortion and tangential distortion. Radial distortion is caused by lens shape, and tangential distortion is caused by the non-parallelism of the lens itself and the imaging plane. After the distortion coefficient is calibrated, it can be used to restore the distortion correction image.

[0096] As one of the implementable modes, the process of calibrating the camera parameters of each preset position image can be calculated from the preset position image.

[0097] However, in order to further improve the accuracy of the camera parameters corresponding to the preset position image, a more preferred embodiment provided by the present application adopts a nonlinear optimization mode, and the camera parameters calculated from the preset position image are used as initial camera parameters. The initial camera parameters are used as the basis for nonlinear optimization, and the camera parameters corresponding to the preset position image are obtained and stored in the pre-calibration data set. The process can be as shown in Figure 3

[0098] Step 302: Obtain the feature points calibrated for the preset position image and the three-dimensional feature points corresponding to the feature points in the three-dimensional model, and obtain the initial camera parameters calibrated for the preset position image.

[0099] In this step, the feature points calibrated by the human for the preset position image and the corresponding points in the three-dimensional space can be obtained, that is, the 2D-3D point pairs are calibrated. The initial camera parameters can be calculated from the preset position image.

[0100] Suppose the focal lengths of the monitoring camera in two directions are f x and f y , the skew coefficient is s, the principal point position coordinates are (c x , c y ), and the image row height and column width are rows and cols, respectively, then the initial camera intrinsic parameter matrix of the monitoring camera is represented as K:

[0101]

[0102] Step 304: Obtain the images collected by the monitoring camera in the continuous rotation and scaling process, and form an image sequence with the collected images and the preset position image.

[0103] ​This step involves acquiring supplementary images from the monitoring camera to supplement more constraint information, thereby helping to optimize the camera parameters of the preset position images. Similarly, the preset position images acquired by the monitoring camera during continuous rotation and scaling need to meet a preset overlap to ensure that the preset position images acquired within all possible viewing angles cover all points in the environment that the monitoring camera can capture, and have different degrees of rotation and scaling.

[0104] The preset image and the supplementary acquired images together form an image sequence {I1,I2,…,I...} N}, where N is the total number of images in the image sequence.

[0105] Step 306: Determine the matching feature points of each image in the image sequence, and determine the map points corresponding to each matching feature point.

[0106] In this embodiment, a feature extraction model can be used to extract feature points from each image in an image sequence, and then a feature matching model can be used to perform feature matching on each image in the image sequence to determine the matching relationship of feature points in each image. For example, for each pair of images, the correspondence of the same feature point matched in the two images can be determined.

[0107] Feature extraction and feature matching can both utilize pre-trained feature extraction and matching models. The feature extraction model can be a deep learning-based visual feature extractor such as Superpoint, which detects feature points (or keypoints) in an input image and calculates a descriptor for each feature point, capturing the image's structural, textural, and semantic information. The feature matching model can be a graph neural network-based matcher such as Superglue, which matches feature points in two images and outputs the matching confidence score for each pair of feature points. In addition to Superpoint and Superglue, other deep neural network models can also be used to implement the feature extraction and matching models in this embodiment.

[0108] The training process of the feature extraction model and the feature matching model is not limited by the embodiments of the present application, and any existing training method can be used. For example, after the Superpoint feature extraction model and the Superglue feature matching model are pre-trained using the COCO (Common Objects in COntext) dataset, the Superpoint feature extraction model and the Superglue feature matching model are fine-tuned using the urban monitoring image dataset. The COCO dataset is a large-scale image dataset with more than 330K images, containing 1.5 million targets, 80 target categories, 91 material categories, 5 sentences of image description for each image, and 250,000 feature point annotated pedestrians, and is widely used in image detection, semantic segmentation, image title generation and other scenarios.

[0109] Further, in order to reduce false matches, the matching relationship of the obtained feature points in each image can be filtered in combination with existing algorithms such as RANSAC (Random Sample Consensus) algorithm, homographic transformation model, etc. Among them, the RANSAC algorithm can use the MAGSAC++ (Marginalizing Sample Consensus) algorithm.

[0110] Under the projective transformation, the corresponding points on the two image planes satisfy the homographic transformation. Therefore, the homographic transformation matrix between each image in the image sequence can be obtained. The homographic transformation matrix represents the coordinate transformation matrix between the corresponding points on the two image planes, for example, the homographic transformation matrix H ij represents the coordinate transformation matrix between the corresponding points on the image I i and the image I j , i≠j.

[0111] The rotation matrix R ij between the images in the image sequence can be obtained.

[0112] R ij = K -1 H ij K (2)

[0113] In order to increase more constraint relationships in the optimization process of the camera parameters, the position of the map point is introduced in the embodiments of the present application. The so-called map point refers to a feature point existing in the actual three-dimensional space. The feature point existing in the actual three-dimensional space can usually be reflected in the map, so it is called map point. The matching relationship of the feature points in each image can be combined to obtain a track of the feature points, and one track corresponds to one map point.

[0114] In determining the map point corresponding to each matched feature point, actually, the initial value of the map point is needed to be calculated by using the initial value of the camera parameter. Specifically, the following process can be performed for each map point: by minimizing the first loss function, the initial coordinates of the map point are obtained, the first loss function reflects the difference between the coordinates obtained by converting the initial coordinates of the map point to the image plane by using the initial camera parameters of each image and the coordinates of the corresponding feature points of the map point on each image; then the initial coordinates of each map point are normalized and / or distorted.

[0115] Suppose the initial coordinates of the map point are represented as X k , k∈[1, M], M represents the number of map points involved in the image sequence. The first loss function F(X k ) described above can be represented as:

[0116]

[0117] Wherein, n is the number of images in the image sequence containing the corresponding feature points of the map point, that is, the number of images in the image sequence capturing the map point. x ik represents the corresponding feature point coordinates of the map point X k on the image I i , that is, the corresponding feature point coordinates of the map point X k on the image I i . K i R i X k is the coordinate obtained by converting X i to the image plane by using the initial camera intrinsic matrix K i corresponding to the image I i and the rotation matrix R i from the image I k to the reference coordinate system. The reference coordinate system can be the plane coordinate system corresponding to a reference image selected from the image sequence, so R i is the rotation matrix of the image I i relative to the reference image.

[0118] By minimizing F(X k ), the initial coordinates of the map point can be obtained by gradually adjusting and optimizing X k . The above process is performed for each map point to obtain the initial coordinates of each map point.

[0119] In addition to the above-mentioned method of determining the initial coordinates of the map point, other methods can also be used, and the present application only describes the preferred method as an example.

[0120] Step 308: optimizing the initial camera parameters corresponding to the preset position image by using the coordinate relationship of the same map point in each image in the image sequence, to obtain the camera parameters corresponding to the preset position image.

[0121] In this step, the camera parameters corresponding to the preset position image can be obtained by minimizing a second loss function, which reflects the difference between the coordinates of each map point converted to the preset position image plane by using the camera parameters corresponding to the preset position image and the corresponding points of the map points on the preset position image.

[0122] Assuming that the optical centers of the monitoring cameras such as spherical cameras coincide, the map points have no real depth information, but are a ray emitted by the optical center of the camera, therefore, the initial coordinates of the map points can be normalized. After normalization, X k represents a normalized direction vector in three-dimensional space.

[0123] Assuming that X k is captured by the image I i , then the corresponding feature point of X k on the image I i is denoted as x ik , and the coordinates of x ik after normalization to the corresponding plane coordinate system of the reference image are X' ik = [x', y', 1] T , which are obtained by the following formula:

[0124] X' ik = R i X k (4)

[0125] Since the images collected by the monitoring cameras usually have distortion, the coordinates obtained after distortion processing of X' ik are X” ik = [x”, y”, 1], which are obtained by the following formula:

[0126] x” = x'(1 + k1 r 2 +k2 r 4 )+2p1 x'y'+p2(r 2 +2x' 2 ) (5)

[0127] y” = y'(1 + k1 r 2 +k2 r 4 )+2p2 x'y'+p1(r 2 +2y' 2 ) (6)

[0128] where k1 and k2 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients, r2 =x' 2 +y' 2 .

[0129] For example, the second loss function is expressed as F(K) i ,R i ,k1,k2,p1,p2,X k The following formula can be used:

[0130]

[0131] Among them, K i X” ik Indicates the use of image I i The camera intrinsic parameter matrix for X” ik Convert to image I i The coordinates obtained on the plane. By minimizing F(K) i ,R i ,k1,k2,p1,p2,X k ), gradually adjust and optimize K i ,R i ,k1,k2,p1,p2,X k Image I can then be obtained i The corresponding camera parameters and map point coordinates. From these, the camera parameters for the preset image can be obtained.

[0132] In addition to the above-mentioned method for determining the camera parameters of the preset image, other methods can also be used. This application only describes the preferred method described above as an example.

[0133] In the calibration process of the preset position image, this application obtains an image sequence composed of images acquired by the monitoring camera during continuous rotation and scaling, and the preset position image. By using this method of supplementary image acquisition, the coordinate relationship of the same map point in each image in the image sequence is used to optimize the initial camera parameters corresponding to the preset position image, thereby realizing automatic calibration of the preset position image in complex environments and improving the calibration accuracy of the preset position image.

[0134] Due to the camera extrinsic parameters (mainly R) obtained in the above process i All of these are camera extrinsic parameters in the reference coordinate system. In practical applications, the camera extrinsic parameters are usually transformed to the geographic coordinate system. In this embodiment, a transformation matrix can be used to transform the camera extrinsic parameters in the camera parameters corresponding to the preset position image to the geographic coordinate system. The transformation matrix is ​​obtained by minimizing a third loss function, which reflects the difference between the coordinates of each map point transformed to the preset position image plane using the transformation matrix and the corresponding points of the map points on the preset position image.

[0135] Suppose the transformation matrix of the geographic coordinate system to the reference coordinate system is represented as image I i The external parameter corresponding to the reference coordinate system is t i image I i The corresponding translation vector, | represents the R i and t i Splicing, the calculation of the transformation matrix can be optimized by the following third loss function Get:

[0136]

[0137] Where n is the number of preset bit images, and m is the number of artificial marking feature point and three-dimensional feature point pairs in the preset bit image. Indicates that the K i , and Convert the map point coordinates X k to the coordinates on the preset bit image I i Gradually adjust the optimization Can get the final

[0138] Then use the optimized Can get the camera external parameter of the preset bit image under the geographic coordinate system For example, the following formula can be used:

[0139]

[0140] After obtaining the camera parameters such as camera intrinsic parameters and camera extrinsic parameters corresponding to the preset bit image, the pre-calibration data set is established.

[0141] The following embodiments will be described in detail. Figure 2 The step 204 in the flow shown in the figure, that is, "obtaining the second image matched with the first image from the pre-established pre-calibration data set, and the pre-calibration data set includes the preset bit image collected by the monitoring camera in multiple poses respectively and the camera parameters corresponding to each preset bit image" will be described in detail.

[0142] After obtaining the first image collected by the monitoring camera in the current pose, first, the first image is matched in the pre-calibration data set to determine the preset bit image matched with the first image as the second image.

[0143] As one of the implementable ways, feature extraction and matching of feature points can be performed on the first image and the preset bit image in the pre-calibration data set respectively, and the preset bit image with the number of matched feature points exceeding the preset threshold is determined as the second image, where the preset threshold can be an empirical value or an experimental value.

[0144] The feature extraction can employ a pre-trained feature extraction model, and the feature matching can employ a pre-trained feature matching model. The feature extraction model can employ a Superpoint feature extraction model, and the feature matching model can employ a Superglue feature matching model.

[0145] As another more preferred embodiment, the present application can employ a multi-stage matching strategy, i.e., a matching including the following three stages:

[0146] The first stage: feature extraction and first matching processing of feature points are respectively performed on the first image and the preset bit image in the pre-calibration data set; if the number of matching feature points between the preset bit image and the first image is greater than or equal to a preset threshold, the preset bit image is determined as the second image, otherwise, the next stage of matching is performed.

[0147] The feature extraction can employ a pre-trained feature extraction model, and the feature extraction model can employ a Superpoint feature extraction model. The first matching processing can be performed by a processing method such as KNN (K-Nearest Neighbor) matching, which has a small amount of calculation and high efficiency. Through the first matching processing, the matching feature points that are mutually matched on the first image and the preset bit image can be determined. Assuming that the preset bit image I i has matching feature points with the first image, and the number of matching feature points is greater than or equal to a preset threshold, the preset bit image I i is determined as the second image. If the number of matching feature points between all preset bit images and the first image is less than the preset threshold, the first stage of matching fails, and the next stage of matching is entered.

[0148] The second stage: a history image in which the corresponding camera parameter has been determined is acquired from the history images collected by the monitoring camera before the first image, and the acquired history image is subjected to the first matching processing with the first image; if the number of matching feature points between the history image and the first image is greater than or equal to a preset threshold, the acquired history image is determined as the second image, otherwise, the next stage of matching is performed.

[0149] If the manner in the embodiments of the present application is performed to determine the corresponding camera parameters for the images collected by the monitoring camera as the first image, or the manner in the embodiments of the present application is periodically performed to determine the corresponding camera parameters for the images collected by the monitoring camera as the first image, or the manner in the embodiments of the present application is triggered based on a specific event to determine the corresponding camera parameters for the images collected by the monitoring camera as the first image, there are historical images that have determined the corresponding camera parameters, and the camera parameters of the current first image can be determined by using the historical images. Therefore, the historical image in which the first image is successfully matched with the preset position image can be subjected to the first matching processing, and if the historical image is matched, that is, the number of matching feature points of the historical image and the first image is greater than or equal to the preset threshold, the historical image is determined as the second image.

[0150] It should be noted that the historical image described above can be the last image for which the corresponding camera parameters are determined by using the manner in the embodiments of the present application. Because the last image is usually relatively close in time to the current first image, the pose, focal length, etc. will not differ greatly, and it is easier to match.

[0151] The third stage: the first image and the preset position image in the pre-calibration data set and the historical image obtained in the second stage are subjected to the second matching processing, and if there is an image in the preset position image and the historical image and the number of matching feature points of the image and the first image is greater than or equal to the preset threshold, the image is determined as the second image. Otherwise, the pose or focal length of the monitoring camera can be adjusted, and the above steps are re-executed. Figure 2 The flowchart is shown.

[0152] The second matching processing used in the third stage can adopt a deep matching manner, for example, a Superglue feature matching model is used to match the feature points. Because the calculation amount of deep matching is large, the matching in the third stage is performed only when the matching in the previous two stages fails.

[0153] In this step, the first image and the preset position image in the pre-calibration data set can be subjected to the second matching processing first, and if the matching fails, the first image and the historical image are subjected to the second matching processing.

[0154] The multi-stage matching manner described above improves the matching success rate while taking into account the calculation efficiency.

[0155] The above step 206, that is, “determining the matching feature points of the first image and the second image, determining the camera parameters of the first image by using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters of the second image” will be described in detail in combination with embodiments.

[0156] After the matching second image is determined, the matching feature points of the first image and the second image are obtained. Then, the following steps can be performedFigure 4 The process shown in the figure determines the camera parameters corresponding to the first image, such as Figure 4 As shown, it may specifically include the following steps:

[0157] Step 402: Determine the homography transformation matrix between the first image and the second image.

[0158] Assume the first image is I c The second image is I i After a successful match, the homography transformation matrix H can be obtained directly through the linear transformation between the matched feature points. ci .

[0159] Step 404: Determine the initial camera parameters corresponding to the first image using the camera parameters and homography transformation matrix corresponding to the second image.

[0160] In this step, the camera intrinsic parameter K corresponding to the second image can be... i The initial camera intrinsic parameter K corresponding to the first image c , is represented as:

[0161] K c =K i (10)

[0162] Using the initial camera intrinsics K of the first image c Homography transformation matrix H ci And the rotation matrix R in the camera extrinsic parameters corresponding to the second image. i Determine the rotation matrix R in the initial camera extrinsic parameters corresponding to the first image. c .For example:

[0163]

[0164] Step 406: Based on the initial camera parameters, the camera parameters corresponding to the first image are obtained by minimizing the fourth loss function. The fourth loss function reflects the difference between the coordinates of the matching feature points on the second image after being transformed to the plane of the first image using the camera parameters of the first image and the camera parameters of the second image, and the coordinates of the corresponding matching feature points on the first image.

[0165] If the matching feature points on the first image and the second image are respectively represented as and Where k∈[1,m], and m is the number of matching feature point pairs. As one possible implementation, the fourth loss function F(K) c ,R c This can be represented as follows:

[0166]

[0167] is the coordinate obtained by transforming K c , R c and R i to the first image plane. By minimizing F(K c , R c ), K c and R c are gradually adjusted and optimized.

[0168] R c is the rotation matrix of the first image in the reference coordinate system, and the camera extrinsic parameter corresponding to the first image in the reference coordinate system is t c is the translation vector corresponding to the first image, and | indicates that R c and t c are spliced. Then, the camera extrinsic parameter of the first image can be converted to the geographic coordinate system by using the following formula:

[0169]

[0170] wherein, is the camera extrinsic parameter corresponding to the first image in the geographic coordinate system.

[0171] It can be seen that, in the determination of the camera parameter corresponding to the first image, the application not only estimates the parameter based on the homographic transformation matrix, but also takes the camera parameter determined based on the homographic transformation matrix as the initial camera parameter, and then further optimizes the initial camera parameter in the manner of minimizing the fourth loss function (the fourth loss function reflects the difference between the coordinates of the matching feature points on the second image converted to the first image plane by using the camera parameter of the first image and the camera parameter of the second image, and the coordinates of the corresponding matching feature points on the first image), to obtain the camera parameter corresponding to the first image, thereby improving the accuracy of the camera parameter corresponding to the first image, and further improving the accuracy of the determined visual domain.

[0172] In addition to the manner shown in the above Figure 4 , other manners can also be used, and the application is described herein only by taking the above preferred manner as an example.

[0173] The above step 208, i.e., “determining the visual domain of the monitoring camera at the current pose by using the camera parameter corresponding to the first image and the camera imaging model”, will be described in detail below in combination with embodiments.

[0174] ​In this step, the current position and attitude angle of the monitoring camera can be determined first using the camera parameters corresponding to the first image. After determining the camera's intrinsic and extrinsic parameters, the current position and attitude angle of the monitoring camera can be determined using a geometric algorithm. This part is based on existing algorithms and will not be elaborated here.

[0175] Then, using the current position, attitude angle, and field of view of the monitoring camera, and based on the mapping relationship provided by the camera imaging model, the viewing cone of the monitoring camera in its current pose can be determined. The field of view is determined by the focal length of the monitoring camera and the image resolution of the first image. Finally, the visible field of view of the camera is determined using the area covered by the viewing cone.

[0176] like Figure 5 As shown, the area that the camera's field of view can cover in three-dimensional space is the camera's visible field of view, and the area that it cannot cover is the camera's blind zone.

[0177] When the position or focal length of the monitoring camera changes, it can be used as follows: Figure 2 The method described re-determines the field of view of the monitoring camera and sends the camera pose information, frustum information, and field of view information to a display device for visualization. For example, the camera pose information can be displayed digitally at the corner of the monitoring screen. Another example is displaying the frustum information and field of view information of the monitoring camera on the monitoring screen using a 3D model.

[0178] This application uses the first image captured by the monitoring camera to match the preset position image in the pre-calibrated data set to dynamically determine the field of view. This method is not dependent on hardware devices and can be applied to devices of various manufacturers, models and styles. It has strong versatility and higher accuracy.

[0179] After determining the field of view of the monitoring camera using the above method, it can be applied to various subsequent applications, such as the following:

[0180] First application: System coverage assessment

[0181] The field of view (LAD) of a monitoring camera is an important theoretical indicator for system coverage evaluation, reflecting the coverage range and accuracy of the monitoring system. By overlaying the LAD area and proportion of the monitoring cameras in the system, the rationality of the camera installation and layout can be reflected, helping users determine whether the system layout can effectively cover the target area. The BAD area and proportion of the monitoring cameras reflect the system's coverage, helping users to promptly identify system deficiencies and take timely measures to supplement and improve the system.

[0182] Second application: Monitoring camera layout

[0183] A series of supporting resources are needed for installing monitoring cameras, including control devices, networks and analysis computing devices, and how to effectively use these monitoring cameras to achieve full coverage of the space becomes an important problem. The traditional monitoring camera layout scheme is mostly set by manual experience, and the installation and testing process is time-consuming and easily affected by the experience of the installer, thereby affecting the effect. If the visual field calculation result of the monitoring camera provided in the embodiments of the present application is used, the monitoring camera layout problem is converted into an optimization problem, and effective full coverage of the area can be achieved at minimum cost, the layout cost of the monitoring camera is saved, and the environmental perception effect is improved. Through the present application, the visual field calibration between multiple monitoring cameras can be realized, and the connection between the visual fields is realized, forming a continuous visual coverage area, effectively avoiding the problem of visual blind area of the monitoring system.

[0184] The third application: dynamic target tracking

[0185] Visual field calculation has a great effect on dynamic target tracking. By calculating the camera parameters and visual field of the monitoring camera, the viewing angle of the monitoring camera can be adjusted according to the different motion trajectories of the target, thereby improving the accuracy and stability of the moving target tracking. In addition, the potential observation camera of the target can be predicted by combining the visual range of the monitoring camera and the motion trajectory of the target, and the cross-camera tracking function can be realized.

[0186] The above describes certain embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0187] According to another aspect, embodiments of a device for determining a visual field of a monitoring camera are provided. Figure 6 A schematic block diagram of a device for determining a visual field of a monitoring camera according to one embodiment is shown, which corresponds Figure 1 The visual field determination device in the architecture shown. As Figure 6 shown, the device 600 includes an image acquisition unit 601, an image matching unit 602, a parameter estimation unit 603, and a visual field determination unit 604, and can also include a pre-calibration unit 605. The main functions of each component unit are as follows:

[0188] The image acquisition unit 601 is configured to acquire a first image collected by the monitoring camera at a current pose.

[0189] The image matching unit 602 is configured to acquire a second image matching the first image from a pre-established pre-calibration data set, the pre-calibration data set containing preset position images respectively captured by the monitoring camera at multiple poses and camera parameters corresponding to the preset position images.

[0190] The parameter estimation unit 603 is configured to determine matching feature points of the first image and the second image, determine the camera parameters corresponding to the first image by using coordinate relationships of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image.

[0191] The visual field determination unit 604 is configured to determine the visual field of the monitoring camera at the current pose by using the camera parameters corresponding to the first image and a camera imaging model.

[0192] Further, the pre-calibration unit 605 in the device is configured to pre-acquire preset position images respectively set by the monitoring camera at multiple poses and respectively determine camera parameters corresponding to the preset position images.

[0193] As one of the implementable manners, the pre-calibration unit 605 can be specifically configured to, when respectively determining the camera parameters corresponding to the preset position images:

[0194] acquire feature points calibrated for the preset position images and three-dimensional feature points corresponding to the feature points in a three-dimensional model, and acquire initial camera parameters calibrated for the preset position images;

[0195] acquire images captured by the monitoring camera in a continuous rotation and scaling process, and form an image sequence by using the captured images and the preset position images;

[0196] determine matching feature points of each image in the image sequence, and determine map points corresponding to each matching feature point;

[0197] optimize the initial camera parameters corresponding to the preset position images by using coordinate relationships of the same map point in each image in the image sequence, to obtain the camera parameters corresponding to the preset position images.

[0198] As one of the implementable manners, the pre-calibration unit 605 can be specifically configured to, when determining the map points corresponding to each matching feature point:

[0199] respectively perform, for each map point: by minimizing a first loss function, obtain initial coordinates of the map point, the first loss function reflecting differences between coordinates obtained by converting the initial coordinates of the map point to planes of each image by using initial camera parameters of each image and coordinates of feature points corresponding to the map point on each image.

[0200] As one of the implementable manners, the pre-calibration unit 605 can be specifically configured to, when optimizing the initial camera parameters corresponding to the preset position image by using the coordinate relationship of the same map point in each image in the image sequence, to obtain the camera parameters corresponding to the preset position image:

[0201] By minimizing the second loss function, the camera parameters corresponding to the preset position image are obtained, and the second loss function reflects the difference between the coordinates of each map point converted to the preset position image plane by using the camera parameters corresponding to the preset position image and the corresponding points of the map point on the preset position image.

[0202] As one of the implementable manners, the pre-calibration unit 605 can be further configured to, after obtaining the camera parameters corresponding to the preset position image:

[0203] Convert the camera extrinsic parameters in the camera parameters corresponding to the preset position image to the geographic coordinate system by using the transformation matrix;

[0204] Wherein the transformation matrix is obtained by minimizing the third loss function, and the third loss function reflects the difference between the coordinates of each map point converted to the preset position image plane by using the transformation matrix and the corresponding points of the map point on the preset position image.

[0205] As one of the implementable manners, the image matching unit 602 can be specifically configured to:

[0206] Respectively perform feature extraction and first matching processing of feature points on the first image and the preset position image in the pre-calibration data set;

[0207] If the number of matching feature points between the preset position image and the first image is greater than or equal to the preset threshold, it is determined that the preset position image is the second image; otherwise, a historical image in which the corresponding camera parameters have been determined is obtained from the historical images collected by the monitoring camera before, and the obtained historical image is subjected to the first matching processing with the first image;

[0208] If the number of matching feature points between the obtained historical image and the first image is greater than or equal to the preset threshold, it is determined that the historical image is the second image; otherwise, the first image and the preset position image in the pre-calibration data set and the obtained historical image are subjected to the second matching processing;

[0209] If the number of matching feature points between the image in the pre-calibration data set and the obtained historical image and the first image is greater than or equal to the preset threshold, it is determined that the image is the second image.

[0210] As one of the implementable manners, the parameter estimation unit 603 can be specifically configured to:

[0211] Determine the homography transformation matrix between the first image and the second image;

[0212] determine initial camera parameters corresponding to the first image by using the camera parameters corresponding to the second image and the homographic transformation matrix;

[0213] obtain camera parameters corresponding to the first image by minimizing a fourth loss function based on the initial camera parameters, wherein the fourth loss function reflects a difference between coordinates of the matched feature points on the second image after being converted to a plane of the first image by using the camera parameters of the first image and the camera parameters of the second image and coordinates of the corresponding matched feature points on the first image.

[0214] wherein the camera parameters include camera intrinsic parameters and camera extrinsic parameters. When determining the initial camera parameters corresponding to the first image by using the camera parameters corresponding to the second image and the homographic transformation matrix, the parameter estimation unit 603 can be specifically configured to:

[0215] use the camera intrinsic parameters corresponding to the second image as the initial camera intrinsic parameters corresponding to the first image;

[0216] determine the initial camera extrinsic parameters corresponding to the first image by using the initial camera intrinsic parameters of the first image, the homographic transformation matrix and the camera extrinsic parameters corresponding to the second image.

[0217] As one of the implementable manners, the viewable area determination unit 604 can be specifically configured to:

[0218] determine a current position and an attitude angle of the monitoring camera by using the camera parameters corresponding to the first image;

[0219] determine a view cone of the monitoring camera at the current position and the attitude angle by using the current position, the attitude angle and a field of view angle of the monitoring camera, wherein the field of view angle is determined by a focal length of the monitoring camera and an image resolution of the first image;

[0220] determine the viewable area of the camera by using a range covered by the view cone.

[0221] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be referred to the part of the method embodiments. The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. they can be located in one place, or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those skilled in the art can understand and implement it without creative labor.

[0222] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0223] In addition, the embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the method in any one of the preceding method embodiments.

[0224] And an electronic device, comprising:

[0225] One or more processors; and

[0226] A memory associated with the one or more processors, the memory is used to store program instructions, the program instructions are executed by the one or more processors to perform the steps of the method in any one of the preceding method embodiments.

[0227] The present application also provides a computer program product, comprising a computer program, which realizes the steps of the method in any one of the preceding method embodiments when executed by a processor.

[0228] Wherein, Figure 7An exemplary architecture of the electronic device can include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 can be communicatively connected through a communication bus 730.

[0229] The processor 710 can be implemented in the form of a general-purpose CPU, a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the present application.

[0230] The memory 720 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, a basic input / output system (BIOS) 722 for controlling the low-level operation of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and a visual field determination apparatus 725, and the like can also be stored. The visual field determination apparatus 725 can be an application program for implementing the above-mentioned steps in the embodiments of the present application. In summary, when the technical solutions provided in the present application are implemented by software or firmware, the related program codes are stored in the memory 720 and executed by the processor 710.

[0231] The input / output interface 713 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, and the like, and the output device can include a display, a speaker, a vibrator, an indicator, and the like.

[0232] The network interface 714 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0233] Bus 730 includes a path for transferring information between the various components of the device (e.g., processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720).

[0234] It should be noted that although the above device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., but in the process of implementation, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also contain only the components necessary to implement the scheme of the present application, and does not have to contain all the components shown in the figure.

[0235] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, disk, optical disk, etc., including a number of instructions for making a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0236] The above provides a detailed description of the technical solutions of the present application. The specific examples are applied to the principles and implementation of the present application. The above description of the embodiments is only to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for determining the field of view of a monitoring camera, characterized in that, The method includes: Acquire the first image captured by the monitoring camera in the current pose; A second image matching the first image is obtained from a pre-established pre-calibration data set, wherein the pre-calibration data set includes preset position images collected by the monitoring camera in multiple poses and camera parameters corresponding to each preset position image; The matching feature points of the first image and the second image are determined. The camera parameters corresponding to the first image are determined by using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image. Using the camera parameters and camera imaging model corresponding to the first image, the field of view of the monitoring camera in the current pose is determined; The establishment of the pre-calibration data set includes: The monitoring camera is pre-acquired in multiple poses, and the feature points calibrated for the preset position images and the corresponding points in three-dimensional space are obtained. The initial camera parameters calibrated for the preset position images are also obtained. The images captured by the monitoring camera during continuous rotation and scaling are obtained, and the captured images are combined with the preset position images to form an image sequence. Determine the matching feature points of each image in the image sequence, and determine the map points corresponding to each matching feature point, wherein the map points are points in three-dimensional space; The initial camera parameters corresponding to the preset position image are optimized by using the coordinate relationship of the same map point in each image of the image sequence, so as to obtain the camera parameters corresponding to the preset position image.

2. The method according to claim 1, characterized in that, The process of determining the map points corresponding to each matching feature point includes: For each map point, the following steps are performed: the initial coordinates of the map point are obtained by minimizing the first loss function. The first loss function reflects the difference between the coordinates obtained by transforming the initial coordinates of the map point to the image plane using the initial camera parameters of each image and the coordinates of the corresponding feature point of the map point in each image.

3. The method according to claim 1, characterized in that, The initial camera parameters corresponding to the preset position image are optimized by using the coordinate relationship of the same map point in each image of the image sequence, resulting in the following camera parameters corresponding to the preset position image: By minimizing the second loss function, the camera parameters corresponding to the preset position image are obtained. The second loss function reflects the difference between the coordinates of each map point transformed to the preset position image plane using the camera parameters corresponding to the preset position image and the corresponding points of the map points on the preset position image.

4. The method according to claim 1, characterized in that, After obtaining the camera parameters corresponding to the preset image, the process also includes: Using a transformation matrix, the camera extrinsic parameters in the camera parameters corresponding to the preset image are transformed to the geographic coordinate system; The transformation matrix is ​​obtained by minimizing a third loss function, which reflects the difference between the coordinates of each map point transformed to the preset position image plane using the transformation matrix and the corresponding points of the map points on the preset position image.

5. The method according to claim 1, characterized in that, Obtaining a second image that matches the first image from a pre-established pre-calibrated dataset includes: Feature extraction of feature points and first matching processing are performed on the first image and the preset image in the pre-calibrated data set, respectively. If the number of matching feature points between a preset image and the first image is greater than or equal to a preset threshold, then the preset image is determined to be the second image; otherwise, a historical image with corresponding camera parameters determined from the historical images previously acquired by the monitoring camera is obtained, and the obtained historical image is matched with the first image for the first matching process. If the number of matching feature points between the acquired historical image and the first image is greater than or equal to a preset threshold, then the historical image is determined to be the second image; otherwise, the first image, the preset bit image in the pre-calibrated data set, and the acquired historical image are subjected to a second matching process. If the number of matching feature points between an image and the first image in the preset bit image in the precalibrated data set and the acquired historical images is greater than or equal to a preset threshold, then the image is determined to be the second image.

6. The method according to claim 1, characterized in that, Using the coordinate relationship of the matching feature points in the first image and the second image, and the camera parameters corresponding to the second image, the camera parameters corresponding to the first image are determined as follows: Determine the homography transformation matrix between the first image and the second image; Using the camera parameters corresponding to the second image and the homography transformation matrix, the initial camera parameters corresponding to the first image are determined; Based on the initial camera parameters, the camera parameters corresponding to the first image are obtained by minimizing the fourth loss function, wherein the fourth loss function reflects the difference between the coordinates of the matching feature points on the second image after transforming them to the first image plane using the camera parameters of the first image and the camera parameters of the second image, and the coordinates of the corresponding matching feature points on the first image.

7. The method according to claim 6, characterized in that, The camera parameters include intrinsic camera parameters and extrinsic camera parameters; Determining the initial camera parameters corresponding to the first image using the camera parameters corresponding to the second image and the homography transformation matrix includes: Use the camera intrinsics corresponding to the second image as the initial camera intrinsics corresponding to the first image; The initial camera extrinsic parameters corresponding to the first image are determined using the initial camera intrinsic parameters of the first image, the homography transformation matrix, and the camera extrinsic parameters corresponding to the second image.

8. The method according to any one of claims 1 to 7, characterized in that, Using the camera parameters and camera imaging model corresponding to the first image, the field of view of the monitoring camera in the current pose is determined as follows: Using the camera parameters corresponding to the first image, the current position and attitude angle of the monitoring camera are determined; Using the current position, attitude angle, and field of view of the monitoring camera, the frustum of the monitoring camera in the current pose is determined based on the camera imaging model, wherein the field of view is determined by the focal length of the monitoring camera and the image resolution of the first image; The field of view of the camera is determined by using the area covered by the view cone.

9. The method according to claim 8, characterized in that, The method is executed by a cloud server.

10. A device for determining the field of view of a monitoring camera, characterized in that, The device includes: The image acquisition unit is configured to acquire the first image captured by the monitoring camera in the current pose. The image matching unit is configured to obtain a second image that matches the first image from a pre-established pre-calibration data set, wherein the pre-calibration data set includes preset position images acquired by the monitoring camera in multiple poses and camera parameters corresponding to each preset position image; The parameter estimation unit is configured to determine matching feature points between the first image and the second image, and to determine the camera parameters corresponding to the first image using the coordinate relationship of the matching feature points on the first image and the second image and the camera parameters corresponding to the second image. The field of view determination unit is configured to determine the field of view of the monitoring camera in the current pose using the camera parameters and camera imaging model corresponding to the first image. The pre-calibration unit is configured to acquire preset position images of the monitoring camera in multiple poses, acquire feature points calibrated for the preset position images and the corresponding points in three-dimensional space, and acquire initial camera parameters calibrated for the preset position images. The images captured by the monitoring camera during continuous rotation and scaling are obtained, and the captured images are combined with the preset position images to form an image sequence. Determine the matching feature points of each image in the image sequence, and determine the map points corresponding to each matching feature point, wherein the map points are points in three-dimensional space; The initial camera parameters corresponding to the preset position image are optimized by using the coordinate relationship of the same map point in each image of the image sequence, so as to obtain the camera parameters corresponding to the preset position image.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

Images