Spatial information and video three-dimensional dynamic fusion method and system and storage medium

Abstract

The invention discloses a spatial information and video three-dimensional dynamic fusion method and system and a storage medium, and the method comprises the steps: setting a preset frame based on the corresponding relation between an initial video frame image and a control point; constructing an image index table according to Z, P and T values of the PTZ camera when a preset frame is collected; according to the P, T and Z values of the PTZ camera when the real-time video frame image is acquired, a preset frame is inquired from the image index table by adopting a grading-blocking method, and a corresponding control point is obtained; a real-time video frame image and a preset frame are dynamically matched by combining an SURF algorithm, an FLANN algorithm and an RANSAC algorithm, pixel coordinates of a control point are obtained, three-dimensional space information is mapped to a two-dimensional video frame image through three-dimensional direct linear transformation, and fusion from the space information to a monitoring video is achieved. According to the method, the preset frame image with the control point is matched, and the three-dimensional space information is mapped into the two-dimensional video frame image by adopting three-dimensional direct linear transformation, so that the fusion of the three-dimensional geographic space information and the video is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of information fusion, and in particular to a method, system and storage medium for dynamic fusion of spatial information and video in three dimensions. Background Art

[0002] With the development of traditional PTZ (Pan-Tilt-Zoom) video surveillance technology and artificial intelligence (AI), by installing PTZ cameras and applying them to natural resource monitoring fields such as farmland protection, mine monitoring, and ecological restoration in the form of video + AI, an all-weather, all-time domain automated monitoring system has been built to achieve automatic or semi-automatic identification of monitoring targets. At the same time, in order to better identify the effective position of the target in many fragmented surveillance videos, a corresponding three-dimensional geographic scene is created in the surveillance video background to provide a geographic location for any point in the video image, laying a data foundation for positioning and identifying the target. At present, the main methods for fusing video with spatial information include photogrammetry, camera imaging models, and homography transformation. The camera imaging model requires a known mathematical model of the camera to establish the correspondence between three-dimensional geographic coordinates and pixel coordinates; the homography transformation is a transformation parameter established by matching the image with the feature points of the two-dimensional plane, which constructs the correspondence between the two planes to achieve the fusion effect.

[0003] Existing methods can map objects in three-dimensional scenes to monitoring images, but since the viewing angle of a PTZ camera can change and can also be zoomed in and out, it is very difficult to obtain the camera's internal and external parameters. Homography transformation can achieve transformations of different viewing angles without knowing the camera's internal and external parameters, but the premise is based on the transformation between two planes, which is not suitable for scenes with undulating terrain. Therefore, it is necessary to develop a scene mapping method for undulating terrain and achieve dynamic fusion when the camera rotates. Summary of the invention

[0004] The purpose of the present invention is to provide a method, system and storage medium for the three-dimensional dynamic fusion of spatial information and video. In view of the characteristics of terrain undulation, PTZ camera viewing angle flexibility and camera scalability, the present invention matches a preset frame with control points as a reference image, adopts a three-dimensional direct linear transformation method to solve the conversion relationship between object coordinates and pixel coordinates, maps the spatial data to the monitoring screen, thereby achieving the fusion of three-dimensional geographic spatial information and video.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for dynamic fusion of spatial information and video in three dimensions, the method comprising the following steps:

[0007] S1, based on the initial video data and spatial data of the monitoring area, respectively obtain an initial video frame image and a number of control points in the initial video frame image, and set a preset frame based on the correspondence between the initial video frame image and the control points;

[0008] S2, constructing an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame;

[0009] S3, obtaining a real-time video frame image of the monitoring area, and according to the P, T, and Z values ​​of the PTZ camera when obtaining the real-time video frame image, using a hierarchical-blocking method to query a preset frame from the image index table to obtain the corresponding control point;

[0010] S4, combines SURF, FLANN and RANSAC algorithms to dynamically match real-time video frame images with preset frames, obtains the pixel coordinates of control points, and maps three-dimensional spatial information to two-dimensional video frame images through three-dimensional direct linear transformation, realizing the fusion of spatial information to surveillance video.

[0011] Furthermore, in S1, the control points are feature points of known spatial coordinates in the initial video frame image; the control points are arranged in advance in the monitoring area, and the arrangement method of the control points is:

[0012] Select multiple fixed locations in the monitoring area and evenly distribute several control points.

[0013] Furthermore, in S1, the step of setting the preset frame is performed according to the hierarchical-blocking principle, the horizontal maximum steering angle and the vertical maximum steering angle between adjacent video frames are determined, and based on the horizontal maximum steering angle and the vertical maximum steering angle, the initial video frame image obtained by continuously and evenly rotating the PTZ camera is used as the preset frame image.

[0014] Furthermore, the specific steps of setting the preset frame are as follows:

[0015] S101, horizontal setting preset frame:

[0016] First, set the PTZ camera pitch angle to 0, shoot a video frame forward, record the left or right edge mark point in the current video frame, turn the PTZ camera horizontally to the left or right, and observe that the left or right edge mark point just disappears. At this time, there is no overlap between the two video frames before and after the rotation, that is, the angle rotated is the horizontal maximum steering angle H. max ; According to the working field of the PTZ camera, select the shooting angle ɑ so that ɑ <H max , capture initial video frame images in sequence;

[0017] S102, vertically set a preset frame:

[0018] Keep the PTZ camera pan level, record the lower or upper edge mark point in the current video frame, and rotate the PTZ camera vertically downward or upward to observe that the lower or upper edge mark just disappears. At this time, there is no overlap between the two video frames before and after the rotation, that is, the angle rotated is the vertical maximum steering angle V max ; According to the working field of the PTZ camera, select the shooting angle β so that β <V max , sequentially capture the initial video frame images;

[0019] S103, the acquired initial video frame image is expressed in the form of a five-tuple: pic=<pcid,z,p,t,pts> , where pcid is the number of the initial video frame image, z, p, t are the magnification, rotation, and pitch angle values ​​of the PTZ camera when shooting the image, and pts is the set of control points in the initial video frame image.<Ic1,Pc1> ,<Ic1,Pc1> ,…,<Icn,Pcn>}, where PCi is a control point, ICi is a point in the initial video frame image corresponding to the control point, i = 1, 2, ..., n, n is a positive integer; IC =<pcid,u,v> It consists of a 3-tuple, where u and v represent the pixel coordinates of the control point.

[0020] Further, the S2 constructs an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame, specifically including:

[0021] Following the principle of grading and blocking, the image index is constructed according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame, that is, Z is sorted in ascending order as the main index key, and then the P and T values ​​corresponding to the Z value are also arranged in ascending order to establish an image index table, where P values ​​are rows and T values ​​are columns, and the content in the table is the ID of the initial video frame image in the preset frame corresponding to the corresponding P / T value.

[0022] Further, the S3 obtains a real-time video frame image of the monitoring area, and according to the P, T, and Z values ​​of the PTZ camera when obtaining the real-time video frame image, a hierarchical-block method is used to query a preset frame from the image index table to obtain the corresponding control point, specifically including:

[0023] The classification is obtained by comparing the Z values. On this basis, the image index table is queried according to the P and T values. The specific method is as follows: the P, T, and Z values ​​of the real-time video frame image are recorded as P r , P t , P z ; The P, T, and Z values ​​of the preset frame are P p 、T p , Z p , where P r -ɑ≤P p≤P r +ɑ,T r -β≤T p ≤T r +β, when the preset frame sets that meet the required P value and T value range are retrieved respectively, the intersection of the two is taken to complete the retrieval, and the ID set of the initial video frame image is obtained. The corresponding initial video frame image is found according to the ID to obtain the corresponding control point.

[0024] Furthermore, the S4 combines SURF, FLANN and RANSAC algorithms to dynamically match the real-time video frame image with the preset frame, obtains the pixel coordinates of the control point, maps the three-dimensional spatial information to the two-dimensional video frame image by three-dimensional direct linear transformation, and realizes the fusion of spatial information to the surveillance video, specifically including:

[0025] S401, large-scale scene matching:

[0026] 1) Using a feature matching algorithm, the current video frame image and the initial video frame image set in the preset frame are matched with each other in turn, and the transformation matrix between the current video frame image and each initial video frame image is calculated;

[0027] 2) According to each transformation matrix, the pixel coordinates of the control points in the initial video frame image are transformed into the current video frame image, and based on the control points in the current video frame image and the corresponding pixel coordinates, a three-dimensional direct linear transformation is used to establish a mapping relationship between ground points and pixel points, thereby realizing the fusion of three-dimensional spatial information and the current video frame;

[0028] 3) When the PTZ camera's viewing angle changes, the changed current video frame image is sequentially subjected to the above steps 1) to 2) to achieve dynamic fusion of real-time monitoring video and spatial information;

[0029] S402, small-scale scene matching:

[0030] A virtual video frame method is proposed based on the principle of image feature matching. That is, according to the P, T values ​​and focal length value f of the current video frame, the focal length value is adjusted to the focal length value with the lowest magnification in the image index table, and a video frame is obtained, which is a virtual video frame containing the minimum number of control points, denoted as VF. vir ;

[0031] Adjust the focal length of the PTZ camera to f, and obtain the real-time video frame at this time, which is recorded as VF cur , then VF vir For the reference video frame, perform large-scale scene matching and construct VF vir The mapping relationship with the three-dimensional space information; on this basis, VF is calculated according to formula (1) cur With VF virThe relationship between cur Relationship with three-dimensional space information:

[0032] [u cur v cur 1] T =H[u vir v vir 1] T =HH′[x vir y vir z vir 1] T (1)

[0033] In formula (1), u cur 、v cur 、u vir 、v vir They are real-time video frames VF cur The pixel in the virtual video frame VF vir The corresponding pixel in x vir ,y vir 、z vir VF vir Corresponding to the geographic coordinates in the three-dimensional space information; H is VF vir and VF cur The transformation matrix between image pixels is calculated by the dynamic feature matching algorithm; H' is the mapping matrix between geographic space and image, based on VF vir Calculated using three-dimensional direct linear transformation;

[0034] As the PTZ camera rotates, the real-time video frame changes. It is determined whether the real-time video frame exceeds the range of the virtual video frame, and the virtual video frame is dynamically updated.

[0035] Further, the determining whether the real-time video frame exceeds the range of the virtual video frame and dynamically updating the virtual video frame specifically includes:

[0036] By swapping the parameters in formula (1), as shown in formula (2), it can be used to determine the inclusion relationship between the virtual video frame and the real-time video frame:

[0037] [u vir v vir 1] T =H[u cur v cur 1] T (2)

[0038] Assume real-time video frame VF cur The size is w*h, and the four corners are (0,0), (w,0), (0,h), (w,h). In formula (2), u cur, v cur These are the pixel coordinates of the four corners. H is the transformation matrix from the real-time video frame to the virtual video frame, which is also obtained through the dynamic feature matching algorithm. u vir and v vir These are the pixel coordinates of the same name corresponding to (0,0), (w,0), (0,h), and (w,h) in the virtual video frame. If the calculated result of a corner exceeds the image size limit, it means that the real-time video frame exceeds the range of the virtual video frame and needs to be recalculated to obtain a new virtual video frame.

[0039] The present invention also provides a spatial information and video three-dimensional dynamic fusion system, which is applied to the above-mentioned spatial information and video three-dimensional dynamic fusion method, comprising:

[0040] A preset frame setting module is used to obtain an initial video frame image and a number of control points in the initial video frame image based on the initial video data and spatial data of the monitoring area, and to set a preset frame based on the corresponding relationship between the initial video frame image and the control points;

[0041] An index building module, used to build an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame;

[0042] The image query module is used to obtain the real-time video frame image of the monitoring area, and according to the P, T, Z values ​​of the PTZ camera when obtaining the real-time video frame image, a hierarchical-blocking method is used to query the preset frame from the image index table to obtain the corresponding control point;

[0043] The dynamic matching and mapping module is used to dynamically match the real-time video frame image with the preset frame by combining the SURF, FLANN and RANSAC algorithms, obtain the pixel coordinates of the control points, and map the three-dimensional spatial information to the two-dimensional video frame image by three-dimensional direct linear transformation, so as to realize the fusion of spatial information to the surveillance video.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for dynamic fusion of spatial information and video in three dimensions as described above is implemented.

[0045] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention provides a method and system for dynamic fusion of spatial information and video in three dimensions, and sets preset frames according to the monitoring range and zoom ratio of the PTZ camera and the different terrain conditions within the field of view. These preset frames establish the corresponding relationship between the image and the control point, and establish an image index table according to the P, T, and Z values ​​of the PTZ camera; then, according to the P, T, and Z values ​​of the PTZ camera that shoots the real-time video, the preset frames are queried from the image index table using a hierarchical-blocking method; on this basis, the real-time video frame and the preset frame are dynamically matched to obtain the pixel coordinates of the control point by combining the SURF, FLANN, and RANSAC algorithms, and the three-dimensional spatial information is mapped to the two-dimensional image by a three-dimensional direct linear transformation, thereby realizing the fusion of geographic spatial information to the surveillance video. The present invention does not require camera calibration, avoids the complex calibration steps, and improves the fusion efficiency; the present invention combines the viewing angle and flexible zoom characteristics of the PTZ camera, can consider the spatial information of the undulating terrain, and is suitable for application scenarios based on undulating terrain. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A schematic diagram of the flow of the method for dynamic fusion of spatial information and video 3D provided by the present invention;

[0048] Figure 2 A block diagram of the working principle of the method for dynamic fusion of spatial information and video 3D provided by an embodiment of the present invention;

[0049] Figure 3 A flowchart of preset frame indexes according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of matching preset frames and real-time video frames for a large-scale scene according to an embodiment of the present invention;

[0051] Figure 5 It is a schematic diagram of small scene matching according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The purpose of the present invention is to match the preset frame images with control points, solve the conversion relationship between object coordinates and pixel coordinates by using three-dimensional direct linear transformation, and map the spatial data to the monitoring screen, so as to achieve the fusion of video and three-dimensional geographic space information. Based on this idea, a method for the dynamic fusion of spatial information and video in three dimensions taking into account the undulating terrain is proposed.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] like Figure 1 and Figure 2 As shown, the present invention provides a method for dynamic fusion of spatial information and video three-dimensionally, comprising the following steps:

[0056] S1, based on the initial video data and spatial data of the monitoring area, respectively obtain an initial video frame image and a number of control points in the initial video frame image, and set a preset frame based on the correspondence between the initial video frame image and the control points;

[0057] S2, constructing an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame;

[0058] S3, obtaining a real-time video frame image of the monitoring area, and according to the P, T, and Z values ​​of the PTZ camera when obtaining the real-time video frame image, using a hierarchical-blocking method to query a preset frame from the image index table to obtain the corresponding control point;

[0059] S4, combines SURF, FLANN and RANSAC algorithms to dynamically match real-time video frame images with preset frames, obtains the pixel coordinates of control points, and maps three-dimensional spatial information to two-dimensional video frame images through three-dimensional direct linear transformation, realizing the fusion of spatial information to surveillance video.

[0060] For example, in S1, the control point is a feature point of known spatial coordinates in the initial video frame image; the control point is arranged in advance in the monitoring area, and the arrangement method of the control point is:

[0061] A certain number of control points are arranged in the monitoring area. These control points are distributed as evenly as possible in the entire area. At the same time, the selected positions also need to be relatively fixed to ensure that the position changes little in different time periods. Each control point has a unique point number for subsequent identification and matching, which is expressed as: P c =<cid,X,Y,Z> , where cid is the control point number and X, Y, Z are the geospatial coordinates of the control point.

[0062] For example, in S1, setting the preset frame is to select some initial video frame images in advance, each of which contains one or more control points, and establish a preset frame index table based on the P, T, and Z values ​​of the video frame to facilitate subsequent retrieval and matching. At the same time, in order to ensure the continuity of the preset frames and the correctness of the matching, there should be overlapping parts between the preset frames. The specific setting process follows the grading-blocking principle. The grading is to determine how many levels of zoom ratio to obtain based on the size of the monitored object of the PTZ camera; the blocking is to obtain continuous video frames through horizontal and vertical rotation and shooting after the magnification is determined. In order to ensure that each video frame contains control points and that adjacent video frames have overlapping parts, it is necessary to first calculate the maximum steering angle between adjacent video frames (this angle is divided into the horizontal maximum steering angle H max and vertical maximum steering angle V max ) to avoid excessive rotation angle during shooting, resulting in non-overlapping of adjacent video frames; then, based on the maximum steering angle, continuously and evenly rotate the captured image as the preset video frame. The specific operations are as follows:

[0063] S101, horizontal setting preset frame:

[0064] First, set the PTZ camera pitch angle to 0, shoot a video frame forward, record the left or right edge mark point in the current video frame, turn the PTZ camera horizontally to the left or right, and observe that the left or right edge mark point just disappears. At this time, there is no overlap between the two video frames before and after the rotation, that is, the angle rotated is the horizontal maximum steering angle H. max ; According to the working field of the PTZ camera, select the shooting angle ɑ so that ɑ <H max , take the initial video frame images in sequence; the selection of ɑ <H max In addition to meeting the requirements, it is also necessary to combine the specific working field range of the PTZ camera.

[0065] S102, vertically set a preset frame:

[0066] Keep the PTZ camera pan level, record the lower or upper edge mark point in the current video frame, and rotate the PTZ camera vertically downward or upward to observe that the lower or upper edge mark just disappears. At this time, there is no overlap between the two video frames before and after the rotation, that is, the angle rotated is the vertical maximum steering angle Vmax ; According to the working field of the PTZ camera, select the shooting angle β so that β <V max , sequentially capture the initial video frame images;

[0067] S103, the acquired initial video frame image is expressed in the form of a five-tuple: pic=<pcid,z,p,t,pts> , where pcid is the number of the initial video frame image, z, p, t are the magnification, rotation, and pitch angle values ​​of the PTZ camera when shooting the image, and pts is the set of control points in the initial video frame image.<Ic1,Pc1> ,<Ic1,Pc1> ,…,<Icn,Pcn>}, where PCi is a control point, ICi is a point in the initial video frame image corresponding to the control point, i = 1, 2, ..., n, n is a positive integer; IC =<pcid,u,v> It consists of 3 tuples, where u and v represent the pixel coordinates of the control point. So pts can also be expressed as: pts = {<pcid1,u1,v1,cid1,x1,y1,z1> ,…,<pcidn,un,vn,cidn,xn,yn,zn>}.

[0068] The S2, constructing an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame, specifically includes:

[0069] In order to quickly retrieve the relevant preset frame images during the later dynamic matching, it is necessary to build an index for the preset frame images. Following the principle of grading and blocking, the image index is built according to the Z, P, and T values ​​of the PTZ camera when capturing the preset frame. That is, Z is used as the main index key and sorted in ascending order, and then the P and T values ​​corresponding to the Z value are also sorted in ascending order to build an index table, where P values ​​are rows and T values ​​are columns. The content in the table is the ID of the preset frame image corresponding to the P / T value, such as Figure 3 As shown in (a).

[0070] S3, obtaining a real-time video frame image of the monitoring area, and according to the P, T, Z values ​​of the PTZ camera when obtaining the real-time video frame image, using a hierarchical-blocking method to query a preset frame from the image index table to obtain the corresponding control point, specifically includes:

[0071] When dynamic matching is performed, the preset frame is queried based on the Z, P, and T values ​​of the video frame. First, the classification is obtained by comparing the Z value, and then the P and T values ​​are queried in the index table. The specific method is as follows: record the P, T, and Z values ​​of the real-time frame as P r , P t , P z ; The P, T, and Z values ​​of the preset frame are P p 、T p , Zp First, the P value direction is queried. Assuming that the horizontal rotation range of the camera is 0 to 360°, the P value range is 0 to 360°. In order to ensure that the preset frame around the real-time video frame can be found during retrieval and the control points in the preset frame are transformed to the current video frame, the angle of the retrieval range should not be less than the steering angle ɑ when the preset frame is shot, that is, the retrieval range of the P value is ±ɑ. The P value of the real-time video frame r The value range is: P r -ɑ≤P p ≤P r +ɑ, when the search range value exceeds 360, subtract 360, and when a negative number appears, add 360. The same method is used to select the index range for the T value in the vertical direction, that is, T r -β≤T p ≤T r +β. When the preset frame ID sets that meet the required P value and T value range are retrieved (such as Figure 3 (as shown in b), take the intersection of the two to complete the search and obtain the preset frame image index ID set, such as Figure 3 The intersection of the two dotted rectangles shown in c can be used to find the corresponding preset frame image according to the ID.

[0072] S4 combines SURF, FLANN and RANSAC algorithms to dynamically match the real-time video frame image with the preset frame, obtains the pixel coordinates of the control point, maps the three-dimensional spatial information to the two-dimensional video frame image by three-dimensional direct linear transformation, and realizes the fusion of spatial information to the surveillance video, specifically including:

[0073] The fusion of spatial information and surveillance video is essentially the fusion of spatial information and sequence images, which is a process of dynamic image matching and three-dimensional transformation mapping. The embodiment of the present invention adopts a method combining SURF (Speeded Up Robust Features), FLANN (Fast Library For Approximate Nearest Neighbors) and RANSAC (Random Sample Consensus) algorithms to achieve dynamic matching between sequence images, and maps three-dimensional spatial information to two-dimensional images by three-dimensional direct linear transformation. Facing real application scenarios, the present invention mainly deals with the dynamic matching and mapping of the following two types of scenarios.

[0074] S401, large-scale scene matching:

[0075] The so-called large-scale matching is to ensure that the preset frame set for the current video frame matching contains at least 6 evenly distributed control points when the PTZ camera has a small magnification and a wide field of view, such as Figure 4In this scenario, first compare the magnifications, select the preset frame index table under the magnification, and query the preset frame set related to the current video frame through the retrieval method described in step 4; the matching and mapping process is as follows:

[0076] 1) Using a feature matching algorithm, the current video frame image and the initial video frame image set in the preset frame are matched with each other in turn, and the transformation matrix between the current video frame image and each initial video frame image is calculated;

[0077] 2) According to each transformation matrix, the pixel coordinates of the control points in the initial video frame image are transformed into the current video frame image. According to the control points in the current video frame image and the corresponding pixel coordinates, a three-dimensional direct linear transformation is used to establish a mapping relationship between ground points and pixel points, that is, a mapping relationship between geographic space coordinates and pixel coordinates, thereby realizing the fusion of three-dimensional space information and the current video frame;

[0078] 3) When the PTZ camera's viewing angle changes, the changed current video frame image is sequentially subjected to the above steps 1) to 2) to achieve dynamic fusion of real-time monitoring video and spatial information;

[0079] S402, small-scale scene matching:

[0080] Small-scale scene matching is when the focal length of the surveillance video is increased, due to the smaller field of view, the minimum number of control points required for 3D-2D mapping cannot be met when the camera is rotated (such as Figure 5 In view of this situation, the embodiment of the present invention proposes a method of virtualizing video frames based on the principle of image feature matching, that is, according to the P, T values ​​and focal length value f of the current video frame, the focal length value is adjusted to the focal length value with the lowest magnification in the index table, and a video frame is obtained, which is a virtual video frame containing the minimum number of control points, denoted as VF vir ,like Figure 5 As shown in ②; then according to the current video frame focal length f, the camera focal length value is adjusted to f, and the real-time video frame at this time is obtained (denoted as VF cur ), then VF vir As the reference video frame, perform the matching process of large-scale scenes and construct VF vir The mapping relationship with the three-dimensional scene; on this basis, VF is calculated according to formula (1) cur With VF vir The relationship between cur Relationship with the three-dimensional scene.

[0081] [u cur v cur 1] T =H[u virv vir 1] T =HH′[x vir y vir z vir 1] T (1)

[0082] In formula (1), u cur 、v cur 、u vir 、v vir They are real-time video frames VF cur The pixel in the virtual video frame VF vir The corresponding pixel in x vir ,y vir 、z vir VF vir Corresponding to the geographic coordinates in the three-dimensional space information; H is VF vir and VF cur The transformation matrix between image pixels is calculated by the dynamic feature matching algorithm; H' is the mapping matrix between geographic space and image, based on VF vir Calculated using three-dimensional direct linear transformation;

[0083] As the PTZ camera rotates, the real-time video frame changes. It is determined whether the real-time video frame exceeds the virtual video frame range and the virtual video frame is dynamically updated, including:

[0084] By swapping the parameters in formula (1), as shown in formula (2), it can be used to determine the inclusion relationship between the virtual video frame and the real-time video frame:

[0085] [u vir v vir 1] T =H[u cur v cur 1] T (2)

[0086] Assume real-time video frame VF cur The size is w*h, and the four corners are (0,0), (w,0), (0,h), (w,h). In formula (2), u cur , v cur These are the pixel coordinates of the four corners. H is the transformation matrix from the real-time video frame to the virtual video frame, which is also obtained through the dynamic feature matching algorithm. u vir and v vir These are the pixel coordinates of the same name corresponding to (0,0), (w,0), (0,h), and (w,h) in the virtual video frame. If the calculated result of a corner exceeds the image size limit, it means that the real-time video frame exceeds the range of the virtual video frame and needs to be recalculated to obtain a new virtual video frame.

[0087] The present invention also provides a spatial information and video three-dimensional dynamic fusion system, which is applied to the above-mentioned spatial information and video three-dimensional dynamic fusion method, comprising:

[0088] A preset frame setting module is used to obtain an initial video frame image and a number of control points in the initial video frame image based on the initial video data and spatial data of the monitoring area, and to set a preset frame based on the corresponding relationship between the initial video frame image and the control points;

[0089] An index building module, used to build an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame;

[0090] The image query module is used to obtain the real-time video frame image of the monitoring area, and according to the P, T, Z values ​​of the PTZ camera when obtaining the real-time video frame image, a hierarchical-blocking method is used to query the preset frame from the image index table to obtain the corresponding control point;

[0091] The dynamic matching and mapping module is used to dynamically match the real-time video frame image with the preset frame by combining the SURF, FLANN and RANSAC algorithms, obtain the pixel coordinates of the control points, and map the three-dimensional spatial information to the two-dimensional video frame image by three-dimensional direct linear transformation, so as to realize the fusion of spatial information to the surveillance video.

[0092] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for dynamic fusion of spatial information and video in three dimensions as described above is implemented.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for dynamic fusion of spatial information and video three-dimensional, characterized in that: The following steps are involved: S1, based on the initial video data and spatial data of the monitoring area, respectively obtain an initial video frame image and a number of control points in the initial video frame image, and set a preset frame based on the correspondence between the initial video frame image and the control points; S2, constructing an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame; S3, obtaining a real-time video frame image of the monitoring area, and according to the P, T, and Z values ​​of the PTZ camera when obtaining the real-time video frame image, using a hierarchical-blocking method to query a preset frame from the image index table to obtain the corresponding control point; S4, combines SURF, FLANN and RANSAC algorithms to dynamically match real-time video frame images with preset frames, obtains the pixel coordinates of control points, and maps three-dimensional spatial information to two-dimensional video frame images through three-dimensional direct linear transformation, realizing the fusion of spatial information to surveillance video.

2. The method for dynamic fusion of spatial information and video 3D according to claim 1, characterized in that: In S1, the control points are feature points with known spatial coordinates in the initial video frame image; the control points are arranged in advance in the monitoring area, and the arrangement method of the control points is: Select multiple fixed locations in the monitoring area and evenly distribute several control points.

3. The method for dynamic fusion of spatial information and video 3D according to claim 1, characterized in that: In S1, the step of setting the preset frame is performed according to the hierarchical-blocking principle, and the horizontal maximum steering angle and the vertical maximum steering angle between adjacent video frames are determined. Based on the horizontal maximum steering angle and the vertical maximum steering angle, the initial video frame image obtained by continuously and evenly rotating the PTZ camera is used as the preset frame image.

4. The method for dynamic fusion of spatial information and video 3D according to claim 3, characterized in that: The specific steps of setting the preset frame are as follows: S101, horizontal setting preset frame: First, set the PTZ camera pitch angle to 0, shoot a video frame forward, record the left or right edge mark point in the current video frame, turn the PTZ camera horizontally to the left or right, and observe that the left or right edge mark point just disappears. At this time, there is no overlap between the two video frames before and after the rotation, that is, the angle rotated is the horizontal maximum steering angle H. max ; According to the working field of the PTZ camera, select the shooting angle ɑ so that ɑ <H max , capture initial video frame images in sequence; S102, vertically set the preset frame: Keep the PTZ camera pan level, record the lower or upper edge mark point in the current video frame, and rotate the PTZ camera vertically downward or upward to observe that the lower or upper edge mark just disappears. At this time, there is no overlap between the two video frames before and after the rotation, that is, the angle rotated is the vertical maximum steering angle V max ; According to the working field of the PTZ camera, select the shooting angle β so that β <V max , sequentially capture the initial video frame images; S103, the acquired initial video frame image is expressed in the form of a five-tuple: pic=<pcid,z,p,t,pts> , where pcid is the number of the initial video frame image, z, p, t are the magnification, rotation, and pitch angle values ​​of the PTZ camera when shooting the image, and pts is the set of control points in the initial video frame image.<Ic1,Pc1> ,<Ic1,Pc1> ,…,<Icn,Pcn>}, where PCi is a control point, ICi is a point in the initial video frame image corresponding to the control point, i = 1, 2, ..., n, n is a positive integer; IC =<pcid,u,v> It consists of a 3-tuple, where u and v represent the pixel coordinates of the control point.

5. The method for dynamic fusion of spatial information and video 3D according to claim 1, characterized in that: The S2, constructing an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame, specifically includes: Following the principle of grading and blocking, the image index is constructed according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame, that is, Z is sorted in ascending order as the main index key, and then the P and T values ​​corresponding to the Z value are also arranged in ascending order to establish an image index table, where P values ​​are rows and T values ​​are columns, and the content in the table is the ID of the initial video frame image in the preset frame corresponding to the corresponding P / T value.

6. The method for dynamic fusion of spatial information and video 3D according to claim 4, characterized in that: S3, obtaining a real-time video frame image of the monitoring area, and according to the P, T, Z values ​​of the PTZ camera when obtaining the real-time video frame image, using a hierarchical-blocking method to query a preset frame from the image index table to obtain the corresponding control point, specifically includes: The classification is obtained by comparing the Z values. On this basis, the P and T values ​​are queried in the image index table. The specific method is as follows: the P, T, and Z values ​​corresponding to the real-time video frame image are recorded as P r , P t , P z , the P, T, and Z values ​​corresponding to the preset frame are P p , T p , Z p , where P r -ɑ≤P p ≤P r +ɑ,T r -β≤T p ≤T r +β, when the preset frame sets that meet the required P value and T value range are retrieved respectively, the intersection of the two is taken to complete the retrieval, and the ID set of the initial video frame image is obtained. The corresponding initial video frame image is found according to the ID to obtain the corresponding control point.

7. The method for dynamic fusion of spatial information and video 3D according to claim 6, characterized in that: S4 combines SURF, FLANN and RANSAC algorithms to dynamically match the real-time video frame image with the preset frame, obtains the pixel coordinates of the control point, maps the three-dimensional spatial information to the two-dimensional video frame image by three-dimensional direct linear transformation, and realizes the fusion of spatial information to the surveillance video, specifically including: S401, large-scale scene matching: 1) Using a feature matching algorithm, the current video frame image and the initial video frame image set in the preset frame are matched with each other in turn, and the transformation matrix between the current video frame image and each initial video frame image is calculated; 2) According to each transformation matrix, the pixel coordinates of the control points in the initial video frame image are transformed into the current video frame image, and based on the control points in the current video frame image and the corresponding pixel coordinates, a three-dimensional direct linear transformation is used to establish a mapping relationship between ground points and pixel points, thereby realizing the fusion of three-dimensional spatial information and the current video frame; 3) When the PTZ camera's viewing angle changes, the changed current video frame image is sequentially subjected to the above steps 1) to 2) to achieve dynamic fusion of real-time monitoring video and spatial information; S402, small-scale scene matching: A virtual video frame method is proposed based on the principle of image feature matching. That is, according to the P, T values ​​and focal length value f of the current video frame, the focal length value is adjusted to the focal length value with the lowest magnification in the image index table, and a video frame is obtained, which is a virtual video frame containing the minimum number of control points, denoted as VF. vir ; Adjust the focal length of the PTZ camera to f, and obtain the real-time video frame at this time, which is recorded as VF cur , then VF vir As the reference video frame, perform S401 large-scale scene matching to build VF vir The mapping relationship with the three-dimensional space information; on this basis, VF is calculated according to formula (1) cur With VF vir The relationship between cur Relationship with three-dimensional space information: [u cur v cur 1] T =H[u vir v vir 1] T =HH′[x vir y vir z vir 1] T (1) In formula (1), u cur 、v cur 、u vir 、v vir They are real-time video frames VF cur The pixel in the virtual video frame VF vir The corresponding pixel in x vir ,y vir 、z vir VF vir Corresponding to the geographic coordinates in the three-dimensional space information; H is VF vir and VF cur The transformation matrix between image pixels is calculated by the dynamic feature matching algorithm; H' is the mapping matrix between geographic space and image, based on VF vir Calculated using three-dimensional direct linear transformation; As the PTZ camera rotates, the real-time video frame changes. It is determined whether the real-time video frame exceeds the range of the virtual video frame, and the virtual video frame is dynamically updated.

8. The method for dynamic fusion of spatial information and video 3D according to claim 7, characterized in that: The determining whether the real-time video frame exceeds the range of the virtual video frame and dynamically updating the virtual video frame specifically includes: By swapping the parameters in formula (1), as shown in formula (2), the inclusion relationship between the virtual video frame and the real-time video frame can be determined: [u vir v vir 1] T =H[u cur v cur 1] T (2) Assume real-time video frame VF cur The size is w*h, and the four corners are (0,0), (w,0), (0,h), (w,h). In formula (2), u cur , v cur These are the pixel coordinates of the four corners. H is the transformation matrix from the real-time video frame to the virtual video frame, which is also obtained through the dynamic feature matching algorithm. u vir and v vir These are the pixel coordinates of the same name corresponding to (0,0), (w,0), (0,h), and (w,h) in the virtual video frame. If the calculated result of a corner exceeds the image size limit, it means that the real-time video frame exceeds the range of the virtual video frame and needs to be recalculated to obtain a new virtual video frame.

9. A spatial information and video three-dimensional dynamic fusion system, applied to the spatial information and video three-dimensional dynamic fusion method according to any one of claims 1 to 8, characterized in that: include: A preset frame setting module is used to obtain an initial video frame image and a number of control points in the initial video frame image based on the initial video data and spatial data of the monitoring area, and to set a preset frame based on the corresponding relationship between the initial video frame image and the control points; An index building module, used to build an image index table according to the Z, P, and T values ​​of the PTZ camera when capturing the initial video frame image in the preset frame; The image query module is used to obtain the real-time video frame image of the monitoring area, and according to the P, T, Z values ​​of the PTZ camera when obtaining the real-time video frame image, a hierarchical-blocking method is used to query the preset frame from the image index table to obtain the corresponding control point; The dynamic matching and mapping module is used to dynamically match the real-time video frame image with the preset frame by combining the SURF, FLANN and RANSAC algorithms, obtain the pixel coordinates of the control points, and map the three-dimensional spatial information to the two-dimensional video frame image by three-dimensional direct linear transformation, so as to realize the fusion of spatial information to the surveillance video.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a method for dynamic fusion of spatial information and video in three dimensions is implemented as described in any one of claims 1 to 8.

Images