Video and GIS fusion method and device and electronic equipment

By obtaining the video data and GIS data of the surveillance camera, determining the internal and external parameters, evaluating the matching degree and fusion, the problem of unsatisfactory fusion effect during the movement of the surveillance camera in video GIS technology is solved, and high-quality video and GIS data are achieved Integration improves the authenticity and comprehensiveness of GIS.

CN118941745APending Publication Date: 2024-11-12CHINA TOWER CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410964217.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing video GIS technology has poor integration effect with GIS during monitoring camera movement, and cannot achieve high-quality data matching and fusion.

Method used

By obtaining the video data and GIS data of the surveillance camera, determining the internal and external parameters corresponding to each frame of the image in the video data, calculating the geographical coordinates of the feature points, evaluating the matching degree of the video data and GIS data, and fusing it when available. To improve the fusion effect.

Benefits of technology

It realizes accurate matching and integration of video data and GIS data, improves the integration effect of video and GIS during the movement of the surveillance camera, overcomes the problem of unsatisfactory fusion effect in traditional technology, and makes GIS more realistic and comprehensive.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941745A_ABST
    Figure CN118941745A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a video and GIS (Geographic Information System) fusion method and device and electronic equipment, which are used for improving the fusion effect of video data and GIS data, and the method comprises the following steps: obtaining video data and GIS data shot by a monitoring camera; determining an internal parameter and an external parameter of a monitoring camera corresponding to each frame of image in the video data; determining geographic coordinates of at least one feature point in the video data according to internal parameters and external parameters of a monitoring camera corresponding to a target image in the video data, determining a matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, and determining availability of the GIS data according to the matching degree; and under the condition that the GIS data is available, according to the internal reference and the external reference of the monitoring camera corresponding to each frame of image in the video data, determining the geographic coordinates of each pixel point in each frame of image in the video data, and according to the geographic coordinates of each pixel point in each frame of image in the video data, fusing the video data with the GIS data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, device and electronic device for integrating video and GIS. Background Art

[0002] With the rapid development of smart cities and intelligent transportation systems, the demand for real-time and accurate geographic information is growing. Video GIS technology can combine video data with Geographic Information System (GIS) to provide strong support for urban management and traffic monitoring. Video GIS technology, namely video + digital twin technology, is a multidisciplinary, multi-dimensional, and multi-scale comprehensive technology application that integrates geographic information system, video virtual-reality fusion, digital twin and other technologies.

[0003] The existing video GIS technology belongs to simple data splicing when performing video GIS, and has high requirements on the posture of the monitoring camera. The fusion effect of the video captured by the monitoring camera during movement and GIS is not ideal. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for fusing video and GIS, which are used to solve the problem that the existing video GIS fusion technology has an unsatisfactory fusion effect on the video captured by the monitoring camera during the movement and GIS.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, a method for integrating video and GIS is provided, the method comprising:

[0007] Obtain video data captured by surveillance cameras and geographic information system GIS data;

[0008] Determine the internal parameters and external parameters of the surveillance camera corresponding to each frame image in the video data;

[0009] Determine the geographic coordinates of at least one feature point in the video data according to the internal and external parameters of the surveillance camera corresponding to the target image in the video data, determine the matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, and determine the availability of the GIS data according to the matching degree;

[0010] When the GIS data is available, the geographic coordinates of each pixel in each frame of the video data are determined based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data, and the video data is fused with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

[0011] In a second aspect, a video and GIS fusion device is provided, the device comprising:

[0012] A data acquisition module is used to acquire video data captured by surveillance cameras and geographic information system GIS data;

[0013] A first determination module, used to determine the internal parameters and external parameters of the monitoring camera corresponding to each frame image in the video data;

[0014] A second determination module is used to determine the geographic coordinates of at least one feature point in the video data according to the internal parameters and external parameters of the surveillance camera corresponding to the target image in the video data, determine the matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, and determine the availability of the GIS data according to the matching degree;

[0015] A data fusion module is used to determine the geographic coordinates of each pixel in each frame of the video data based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data when the GIS data is available, and to fuse the video data with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

[0016] In a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, a computer program product is provided. The program product is stored in a storage medium and is executed by at least one processor to implement the method as described in the first aspect.

[0019] A method for fusing video and GIS is proposed in an embodiment of the present application. On the one hand, the internal and external parameters of the monitoring camera corresponding to each frame of the video data can be calibrated, so that the geographic coordinates of each pixel in each frame of the video data can be accurately determined according to the internal and external parameters of the monitoring camera corresponding to each frame of the video data. Then, the video data and the GIS data are fused according to the geographic coordinates of each pixel in each frame of the video data, instead of simply splicing the video data and the GIS data. Therefore, the fusion effect of the video captured during the exercise and the GIS can be improved. On the other hand, before the formal fusion, the matching degree of the video data and the GIS data can be determined according to the geographic coordinates of a small number of feature points in the video data, and the availability of the GIS data can be determined according to the matching degree. When the GIS data is available, the video data and the GIS data are fused again, instead of blindly splicing. Therefore, the fusion effect of the video and GIS can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 It is a flowchart of a method for integrating video and GIS provided in one embodiment of the present application.

[0022] Figure 2 It is a schematic diagram of an implementation flow of step 101 provided in an embodiment of the present application.

[0023] Figure 3A It is a schematic diagram of an implementation flow of step 102 provided in an embodiment of the present application (a schematic diagram of a camera parameter calibration flow diagram).

[0024] Figure 3B It is a schematic diagram of an implementation flow of step 102 provided in another embodiment of the present application (a schematic diagram of a camera parameter calibration flow diagram).

[0025] Figure 4 It is a schematic diagram of an implementation flow of step 103 provided in an embodiment of the present application.

[0026] Figure 5 It is a schematic diagram of an implementation flow of step 104 provided in an embodiment of the present application.

[0027] Figure 6 It is a structural diagram of a video and GIS fusion device provided in one embodiment of the present application.

[0028] Figure 7It is a detailed structural diagram of module 601 provided in an embodiment of the present application.

[0029] Fig. 8A It is a detailed structural diagram of module 602 provided in an embodiment of the present application.

[0030] Figure 8B It is a detailed structural diagram of module 602 provided in another embodiment of the present application.

[0031] Fig. 9 It is a detailed structural diagram of module 603 provided in an embodiment of the present application.

[0032] Fig.10 It is a detailed structural diagram of module 604 provided in an embodiment of the present application.

[0033] Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0035] In order to solve the problem that the existing video GIS fusion technology does not achieve an ideal fusion effect between the video captured by the surveillance camera during movement and the GIS, the embodiment of the present application provides a method and device for fusion of video and GIS. The method can be executed by an electronic device, such as a terminal device or a server, or the method can be executed by software installed in the electronic device.

[0036] It should be noted that the embodiments of the present application provide a method and device for fusing video and GIS, which can fuse video shot by a fixed camera with GIS, and can also fuse video shot by a motion camera with GIS.

[0037] The following first describes a method for integrating video and GIS provided in an embodiment of the present application.

[0038] like Figure 1 As shown, a method for integrating video and GIS provided in an embodiment of the present application may include:

[0039] Step 101, obtaining video data captured by a surveillance camera and geographic information system GIS data.

[0040] The monitoring camera can be any camera that can shoot video, and the monitoring camera can be a fixed monitoring camera or a monitoring camera that moves along a specified route. For example, the monitoring camera can include a fixed gun camera or a non-fixed panoramic (PTZ) camera. For a fixed gun camera, two people can work together, one to operate in the monitoring room and the other to use a total station or other measuring equipment on site to determine its orientation (such as external orientation). The measurement of a camera point takes about 10 minutes. For non-fixed PTZ cameras, such as pan / tilt cameras and ball cameras, the external orientation elements and the panoramic image can be calibrated in advance, and the internal orientation can be determined later by image matching.

[0041] In one implementation, a surveillance camera is directly connected to an electronic device that executes a method for integrating video and GIS provided in an embodiment of the present application, and the electronic device can obtain video data captured by the surveillance camera.

[0042] In another implementation, the surveillance camera may first be connected to a third-party platform (such as a cloud platform) and the captured video data may be uploaded to the third-party platform. An electronic device that executes a method for integrating video and GIS provided in an embodiment of the present application may read the video data captured by the surveillance camera from the third-party platform.

[0043] The GIS data may include maps, vector data and elevation information. The vector data may include at least one of building, road and terrain information.

[0044] As an example, Figure 2 As shown, step 101 may specifically include:

[0045] Step 1011: access the video data collected by the surveillance camera and transmit the video data to a processing center in real time. The processing center may be a data processing center on an electronic device that executes a method for integrating video and GIS proposed in an embodiment of the present application.

[0046] Step 1012, obtaining map data based on the GIS data, extracting and collecting vector data and elevation information of the area corresponding to the video data.

[0047] It can be understood that when GIS data contains elevation information, the geographic information contained in the GIS data can be two-dimensional or three-dimensional. This allows a video and GIS fusion method provided in an embodiment of the present application to achieve the fusion of video and two-dimensional GIS data or three-dimensional GIS data, overcoming the shortcomings of traditional two-dimensional GIS, such as the lack of intuitiveness, complex virtual reality modeling, and low timeliness of remote sensing, making GIS more realistic and comprehensive, and injecting more vitality into GIS.

[0048] Optionally, after step 101 and before step 102, the video captured by the surveillance camera may be preprocessed to improve the quality of the video data, thereby improving the visual effect of the subsequently fused video GIS. The preprocessing may include but is not limited to at least one of format conversion, noise reduction, and color correction.

[0049] Step 102: Determine the internal parameters and external parameters of the surveillance camera corresponding to each frame image in the video data.

[0050] According to the above description, when the surveillance camera is a fixed gun, its internal and external parameters remain unchanged. After the internal and external parameters of the surveillance camera at the initial position are calibrated, the internal and external parameters corresponding to each frame image in the video data taken by the surveillance camera are all the internal and external parameters of the surveillance camera at the initial position.

[0051] For example, Figure 3A As shown, the process of calibrating the intrinsic and extrinsic parameters of the surveillance camera at the initial position includes:

[0052] Step 1021, use the calibration board to capture multiple images containing the calibration target, extract the pixel coordinates of the feature points on the calibration target in the image, and perform intrinsic calibration using camera calibration software (such as OpenCV) based on the actual size and shape of the calibration target and the pixel coordinates of the feature points in the image.

[0053] Among them, the internal parameters may include focal length, principal point position, etc. The calibration of the camera internal parameters is to calibrate the error factors in the imaging processing of the imaging sensor (camera) itself in the camera. Through repeated simulation of mathematical model parameters, the camera imaging error is adjusted, thereby realizing the process of 2D image to 3D simulation. The surveillance camera that has completed the internal parameter calibration can perform 3D scene inversion, that is, depth perception.

[0054] Step 1022: Select a known landmark as a calibration target, and solve the corresponding camera extrinsic parameters according to the positions of the calibration target in the camera coordinate system and the world coordinate system.

[0055] For example, the PnP extrinsic calibration algorithm is used to solve the rotation and translation matrices between the camera coordinate system and the world coordinate system to obtain the camera extrinsic parameters to represent the position of the surveillance camera in the world coordinate system.

[0056] For a surveillance camera moving along a specified route, the intrinsic parameters and extrinsic parameters corresponding to images of different frames in the video data taken by the surveillance camera may be different. The following introduces a method for determining the intrinsic parameters and extrinsic parameters corresponding to each frame of the video data taken by the surveillance camera moving along the specified route.

[0057] In some embodiments, Figure 3BAs shown, in the case where the video data is obtained by the surveillance camera moving along a specified route, the step 102 may specifically include:

[0058] Step 1023: extract frames from the video data to obtain multiple frames of first images.

[0059] Exemplarily, frames may be extracted at equal intervals with m frames as an interval, and at least one frame may be extracted each time, where m is an integer greater than 0.

[0060] Step 1024 , performing structure-from-motion processing on the multiple frames of first images to obtain intrinsic parameters and extrinsic parameters of the surveillance camera corresponding to the multiple frames of first images.

[0061] For example, for the multiple frames of first images, an incremental SFM algorithm, a global SFM algorithm, and a hybrid SFM algorithm may be used to perform SFM processing, so that the intrinsic and extrinsic parameters of the surveillance camera matching each first image can be set. SFM processing belongs to the prior art, so it will not be described in detail in this article.

[0062] Step 1025, according to the frame rate of the video data and the designated route, linear interpolation processing is performed on the internal parameters and external parameters of the monitoring camera corresponding to the multiple frames of first images to obtain the internal parameters and external parameters of the monitoring camera corresponding to the multiple frames of second images, wherein the multiple frames of second images are images in the video data other than the multiple frames of first images.

[0063] Specifically, linear interpolation processing is performed based on the frame rate in the video data and the internal and external parameters of the surveillance cameras that have been assigned to the multiple frames of the first image, so that the internal and external parameters of the surveillance cameras that are assigned to each second image can be set. The internal and external parameters of each group of surveillance cameras can be set in the form of a camera projection matrix to achieve the purpose of automatically calibrating the camera internal and external parameters for each frame of the video data.

[0064] For example, assuming that the number of frames of a video data is 1500 and the frame rate is 25fps, 60 frames of images with equal intervals are extracted from the video data as multiple first images, and the images not extracted in the aforementioned video data are second images. After the 60 frames of first images are processed by the incremental SFM algorithm, the 60 sets of camera intrinsic parameters and extrinsic parameters corresponding to the 60 frames of first images are linearly interpolated at 25fps, so that each frame of the second image is set with the corresponding camera intrinsic parameters and extrinsic parameters. It can be understood that this method of extracting frames for motion recovery structure processing to obtain the intrinsic parameters and extrinsic parameters of the monitoring camera corresponding to multiple frames of the first image, and then obtaining the intrinsic parameters and extrinsic parameters of the monitoring camera corresponding to the remaining frames through linear interpolation processing can avoid the 1440 frames of second images from being put into the motion recovery structure process, reduce the amount of calculation of motion recovery structure processing, help improve calculation efficiency, and at the same time take into account the accuracy of camera parameter configuration for the 1500 frames of images in the video data.

[0065] Of course, if the computing power is sufficient, in other embodiments, when the video data is obtained by the surveillance camera shooting along a specified route, the step 102 may specifically include: performing motion recovery structure processing on each frame image in the video data, and obtaining the internal and external parameters of the surveillance camera corresponding to each frame image in the video data.

[0066] Step 103: Determine the geographic coordinates of at least one feature point in the video data based on the internal and external parameters of the surveillance camera corresponding to the target image in the video data, determine the degree of matching between the video data and the GIS data based on the geographic coordinates of the at least one feature point, and determine the availability of the GIS data based on the matching degree.

[0067] The target image may be any frame image in the video data.

[0068] For example, Figure 4 As shown, step 103 may include:

[0069] Step 1031: select at least one first feature point from the target image in the video data, and determine the pixel coordinates of the at least one feature point.

[0070] The first feature point may be a corner point and / or an edge point of a feature object in the scene captured by the video data.

[0071] Step 1032: Determine at least one second feature point in the GIS data that corresponds to the at least one first feature point.

[0072] Corresponding corner points and / or edge points of the above-mentioned feature objects are found in the GIS data as second feature points.

[0073] Step 1033: Convert the pixel coordinates of the at least one feature point into geographic coordinates according to the intrinsic parameters and extrinsic parameters of the surveillance camera corresponding to the target image.

[0074] After determining the internal and external parameters of the surveillance camera corresponding to the target image, the specific process of converting the pixel coordinates in the target image into geographic coordinates belongs to the prior art and will not be described in detail herein.

[0075] Step 1034, determining the degree of matching between the video data and the GIS data based on the geographical coordinates of the at least one first feature point obtained by conversion and the geographical coordinates of the at least one second feature point in the GIS data.

[0076] This process can be understood as a preliminary registration of the video data and the GIS data, and a process of evaluating the matching degree of the registered video data and the GIS data.

[0077] For example, step 1034 may include:

[0078] Determine a registration deviation evaluation index between the video data and the GIS data according to the geographic coordinates of the at least one first feature point obtained by conversion, the geographic coordinates of the at least one second feature point in the GIS data, and a first formula;

[0079] The matching degree between the video data and the GIS data is determined according to the registration deviation evaluation index and the registration deviation evaluation threshold.

[0080] Among them, the first formula is:

[0081]

[0082] Among them, G i is the registration deviation evaluation index, P i is the geographic coordinates of the first feature point in the target image, Q i is the geographic coordinate of the second feature point corresponding to the first feature point in the GIS data, N is the total number of the first feature points, ||P i -Q i || is the Euclidean distance.

[0083] Optionally, after step 1034, step 103 may further include: using the geographical coordinates of the at least one first feature point after registration as attributes of the video data, and assigning them to the target image and the first feature point.

[0084] Furthermore, the matching degree may be represented by a matching level, and the corresponding relationship between the matching level and the registration deviation evaluation threshold may include:

[0085] Excellent matching level: G i ≤P y At this time, the registration effect of the first feature point and the second feature point in the GIS data is excellent, which means that the position of the first feature point in the geographic coordinate system is close to the position of the second feature point in the GIS data, with a small deviation. The registration effect of the video data and the geographic spatial information is good, and the error between the pixel coordinates and the geographic coordinates is small. The video data can be accurately associated with the geographic information, supporting subsequent geospatial analysis and applications;

[0086] Good match level: P y <G i ≤P c ,At this time, the registration effect of the first feature point and the second feature point in the GIS data is good, the registration deviation is moderate, and there is a deviation that has no effect on practical applications;

[0087] General Matching Level: G i >P c ,At this time, the registration effect between the first feature point and the second feature point in the GIS data is poor, the ,registration deviation is large, and the video data is difficult to correspond to the ,geographic information, which is not suitable for geospatial analysis and ,applications.

[0088] Among them, G i is the registration deviation evaluation index, P y is the registration deviation evaluation threshold corresponding to the excellent registration level, P c The threshold for evaluating the registration deviation corresponding to a good registration grade.

[0089] Of course, other matching level division methods can be used according to actual needs. The above is just an example.

[0090] On this basis, in step 103, determining the availability of the GIS data according to the matching degree may include:

[0091] When the matching level between the video data and the GIS data is higher than or equal to a preset level, determining that the GIS data is available;

[0092] When the matching level between the video data and the GIS data is lower than the preset level, it is determined that the GIS data is unavailable.

[0093] The preset level may be flexibly set, for example, the preset level may be the above-mentioned good matching level.

[0094] Optionally, when the GIS data is unavailable, available GIS data is reselected.

[0095] For example, suppose the first feature point in the video data is the corner point of a building on Street A, and the second feature point that matches it in the GIS data is the corner point of a building on Street B, resulting in the matching level of the two being at a general matching level, that is, the two do not match, and it is necessary to reselect available GIS data.

[0096] Step 104, when the GIS data is available, determine the geographic coordinates of each pixel in each frame of the video data based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data, and fuse the video data with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

[0097] Optionally, before step 104, Figure 1 The method shown may also include: determining the alignment accuracy of the first feature point and the corresponding second feature point based on the converted geographic coordinates of the at least one first feature point, the geographic coordinates of the at least one second feature point in the GIS data, and the total number of the first feature points.

[0098] Accordingly, when the registration accuracy meets the preset conditions, step 104 is executed again; when the registration accuracy does not meet the preset conditions, it means that the matching accuracy between the video data and the GIS data is poor and needs to be adjusted. Specifically, step 102 can be returned to recalibrate the internal and external parameters of the surveillance camera corresponding to each frame image in the video data, and finally find the best matching position between the pixel coordinates in the video data and the geographic coordinates in the GIS data to achieve precise fusion of video and GIS.

[0099] The registration accuracy includes at least one of geographic coordinate accuracy, geographic coordinate projection accuracy and target measurement accuracy.

[0100] Exemplarily, when the registration accuracy includes geographic coordinate accuracy, geographic coordinate projection accuracy, and target measurement accuracy, the preset conditions may include:

[0101] 1) The spatial positioning error of geographic coordinate accuracy is less than 1 meter;

[0102] 2) The reprojection error of geographic coordinate video images is less than 10 pixels;

[0103] 3) The measurement size error of the target measurement accuracy is less than a certain percentage relative to the target itself, such as less than 15%.

[0104] The calculation formula for geographic coordinate accuracy is:

[0105]

[0106] Among them, Pz is the Euclidean distance between the geographic coordinates of the first feature point in the target image and the corresponding second feature point in the GIS data; wherein (x g ,y g ) represents the geographic coordinates of the second feature point in the GIS data, (x v ,y v ) represents the geographic coordinates of the first feature point; or, (x g ,y g ) represents the position of the calibration target g in the actual scene, (x v ,y v ) represents the pixel position of the calibration target g in the video frame v, and the at least one first feature point is located in the calibration target.

[0107] The calculation formula for geographic coordinate projection accuracy is:

[0108]

[0109] Among them, k is the scaling factor between pixel coordinates and geographic coordinates, which is used to project pixel coordinates into GIS data;

[0110] The calculation formula for the target measurement accuracy is:

[0111]

[0112] Among them, s g is the actual size of the calibrated target in geographic space, s v is a measured size of the calibration target in the target image, and the at least one first feature point is located in the calibration target.

[0113] In some embodiments, step 104 may specifically include: assigning geographic coordinates to the video data based on the geographic coordinates of each pixel point in each frame image in the video data to map the pixel coordinates in the video data to the geographic coordinate system; superimposing the video data assigned with geographic coordinates with the GIS data to project the video data into the geographic space.

[0114] In other embodiments, Figure 5 As shown, step 104 may specifically include:

[0115] Step 1041, adding a video layer to the GIS data;

[0116] Step 1042, projecting the pixels contained in each frame image in the video data onto the video layer in time sequence according to the geographic coordinates of each pixel point in each frame image in the video data;

[0117] Step 1043, superimpose the GIS data with the video layer to obtain fused video GIS data.

[0118] Optionally, after step 104, Figure 1 The method shown may also include: displaying the video GIS data, that is, superimposing the GIS data onto the video data for visual display, thereby realizing a visual presentation of video fusion.

[0119] Specifically, displaying the video GIS data may include:

[0120] Visualizing the GIS data in the form of layers to form corresponding geographic information layers;

[0121] Performing pixel-level overlay processing on the geographic information layer and the image frame corresponding to the video data, so that the geographic information and the video content can be visually integrated;

[0122] Add interactive functions to the video after integrating geographic information, allowing users to view detailed information of different geographic areas and adjust the display method and level of geographic information by clicking or dragging.

[0123] The embodiment of the present application provides a method for fusing video and GIS. On the one hand, the internal and external parameters of the monitoring camera corresponding to each frame image in the video data can be calibrated, so that the geographic coordinates of each pixel point in each frame image in the video data can be accurately determined according to the internal and external parameters of the monitoring camera corresponding to each frame image, and then the video data and the GIS data are fused according to the geographic coordinates of each pixel point in each frame image in the video data, instead of simply splicing the video data and the GIS data, so that the fusion effect of the video captured during the exercise and the GIS can be improved; on the other hand, before the formal fusion, the matching degree of the video data and the GIS data can be determined according to the geographic coordinates of a small number of feature points in the video data, and the availability of the GIS data can be determined according to the matching degree. When the GIS data is available, the video data and the GIS data are fused again, instead of blindly splicing, so that the fusion effect of the video and GIS can be further improved.

[0124] In addition, the video and GIS fusion method provided in an embodiment of the present application can also determine the alignment accuracy of the first feature point and the corresponding second feature point based on the geographic coordinates of the at least one first feature point obtained by conversion, the geographic coordinates of the at least one second feature point in the GIS data, and the total number of the first feature points before the formal fusion, and when the alignment accuracy meets the preset conditions, execute step 104 to fuse the video data and the GIS data, which can achieve a better fusion effect of the video data and the GIS data.

[0125] Furthermore, a method for integrating video and GIS provided in an embodiment of the present application can access multiple video streams in real time, flexibly embed intelligent recognition models, obtain the position, posture, coverage of the camera and the spatial position corresponding to each pixel in the video image through technical means, so that the video data is integrated with the scene, especially the three-dimensional scene, and the video access is fully compatible with mainstream video surveillance equipment, and has the functions of monitoring camera calibration, two-dimensional and three-dimensional video and geographic space integration, and integrated visualization presentation. In addition, a method for integrating video and GIS provided in an embodiment of the present application is based on the internal and external parameters of the monitoring camera, and the video data after being assigned with geographic coordinates is superimposed with the GIS data, and the video is enhanced with GIS or GIS enhanced video to achieve the integration of video and two-dimensional or three-dimensional GIS data, which overcomes the shortcomings of traditional video GIS technology, such as the unintuitive two-dimensional GIS, the complex virtual reality modeling, and the low timeliness of remote sensing, making GIS more real and comprehensive, and injecting more vitality into GIS. Further, a method for integrating video and GIS provided in an embodiment of the present application superimposes the video data after being assigned with geographic coordinates with the GIS data, making the traditional map more vivid and fresh, and further enhances the sense of reality of the immersive visual experience by integrating the three-dimensional model and the video real scene.

[0126] The above introduces a method for integrating video and GIS proposed in an embodiment of the present application. Corresponding to the above method for integrating video and GIS, an embodiment of the present application also proposes a device for integrating video and GIS, which will be described below.

[0127] like Figure 6 As shown, a video and GIS fusion device 600 proposed in an embodiment of the present application includes: a data acquisition module 601, a first determination module 602, a second determination module 603 and a data fusion module 604.

[0128] The data acquisition module 601 is used to acquire video data captured by a surveillance camera and geographic information system GIS data.

[0129] The monitoring camera may be any camera capable of shooting video, and the monitoring camera may be a fixed monitoring camera or a monitoring camera moving along a specified route.

[0130] The GIS data may include maps, vector data and elevation information. The vector data may include at least one of building, road and terrain information.

[0131] As an example, Figure 7 As shown, the data acquisition module 601 may specifically include:

[0132] The video data access submodule 6011 is used to access the video data collected by the monitoring camera and transmit the video data to the processing center in real time. The processing center can be a data processing center on an electronic device that executes a video and GIS fusion method proposed in an embodiment of the present application.

[0133] The GIS data collection submodule 6012 is used to obtain map data based on the GIS data, and extract and collect vector data and elevation information of the area corresponding to the video data.

[0134] It can be understood that when GIS data contains elevation information, the geographic information contained in the GIS data can be two-dimensional or three-dimensional. This allows a video and GIS fusion method provided in an embodiment of the present application to achieve the fusion of video and two-dimensional GIS data or three-dimensional GIS data, overcoming the shortcomings of traditional two-dimensional GIS, such as the lack of intuitiveness, complex virtual reality modeling, and low timeliness of remote sensing, making GIS more realistic and comprehensive, and injecting more vitality into GIS.

[0135] Optionally, the device 600 may further include: a data preprocessing module for preprocessing the video captured by the surveillance camera to improve the quality of the video data, thereby improving the visual effect of the subsequently fused video GIS. The preprocessing may include but is not limited to at least one of format conversion, noise reduction, and color correction.

[0136] The first determination module 602 is used to determine the internal parameters and external parameters of the surveillance camera corresponding to each frame image in the video data.

[0137] According to the above description, when the surveillance camera is a fixed gun, its internal and external parameters remain unchanged. After the internal and external parameters of the surveillance camera at the initial position are calibrated, the internal and external parameters corresponding to each frame image in the video data taken by the surveillance camera are all the internal and external parameters of the surveillance camera at the initial position.

[0138] For example, Fig. 8A As shown, the first determination module 602 may include:

[0139] The intrinsic parameter calibration submodule 6021 is used to use the calibration board to capture multiple images containing the calibration target, extract the pixel coordinates of the feature points on the calibration target in the image, and perform intrinsic parameter calibration using camera calibration software (such as OpenCV) based on the actual size and shape of the calibration target and the pixel coordinates of the feature points in the image.

[0140] Among them, the internal parameters may include focal length, principal point position, etc. The calibration of the camera internal parameters is to calibrate the error factors in the imaging processing of the imaging sensor (camera) itself in the camera. Through repeated simulation of mathematical model parameters, the camera imaging error is adjusted, thereby realizing the process of 2D image to 3D simulation. The surveillance camera that has completed the internal parameter calibration can perform 3D scene inversion, that is, depth perception.

[0141] The extrinsic parameter calibration submodule 6022 is used to select a known landmark as a calibration target, and solve the corresponding camera extrinsic parameters according to the positions of the calibration target in the camera coordinate system and the world coordinate system.

[0142] For example, the PnP extrinsic calibration algorithm is used to solve the rotation and translation matrices between the camera coordinate system and the world coordinate system to obtain the camera extrinsic parameters to represent the position of the surveillance camera in the world coordinate system.

[0143] For a surveillance camera moving along a specified route, the intrinsic parameters and extrinsic parameters corresponding to images of different frames in the video data taken by the surveillance camera may be different. The following introduces a method for determining the intrinsic parameters and extrinsic parameters corresponding to each frame of the video data taken by the surveillance camera moving along the specified route.

[0144] In some embodiments, Figure 8B As shown, in the case where the video data is obtained by the surveillance camera moving along a specified route, the first determination module 602 may specifically include:

[0145] The frame extraction submodule 6023 is used to extract frames from the video data to obtain multiple frames of first images.

[0146] Exemplarily, frames may be extracted at equal intervals with m frames as an interval, and at least one frame may be extracted each time, where m is an integer greater than 0.

[0147] The motion structure recovery submodule 6024 is used to perform motion recovery structure processing on the multiple frames of first images to obtain the intrinsic parameters and extrinsic parameters of the monitoring camera corresponding to the multiple frames of first images.

[0148] For example, for the multiple frames of first images, an incremental SFM algorithm, a global SFM algorithm, and a hybrid SFM algorithm may be used to perform SFM processing, so that the intrinsic and extrinsic parameters of the surveillance camera matching each first image can be set. SFM processing belongs to the prior art, so it will not be described in detail in this article.

[0149] The linear interpolation submodule 6025 is used to perform linear interpolation processing on the internal parameters and external parameters of the monitoring camera corresponding to the multiple frames of first images according to the frame rate of the video data and the specified route, so as to obtain the internal parameters and external parameters of the monitoring camera corresponding to the multiple frames of second images, wherein the multiple frames of second images are images in the video data other than the multiple frames of first images.

[0150] Specifically, linear interpolation processing is performed based on the frame rate in the video data and the internal and external parameters of the surveillance cameras that have been assigned to the multiple frames of the first image, so that the internal and external parameters of the surveillance cameras that are assigned to each second image can be set. The internal and external parameters of each group of surveillance cameras can be set in the form of a camera projection matrix to achieve the purpose of automatically calibrating the camera internal and external parameters for each frame of the video data.

[0151] Of course, if the computing power is sufficient, in other embodiments, when the video data is obtained by the surveillance camera shooting along a specified route, the step 102 may specifically include: performing motion recovery structure processing on each frame image in the video data, and obtaining the internal and external parameters of the surveillance camera corresponding to each frame image in the video data.

[0152] The second determination module 603 is used to determine the geographic coordinates of at least one feature point in the video data based on the internal parameters and external parameters of the surveillance camera corresponding to the target image in the video data, determine the matching degree between the video data and the GIS data based on the geographic coordinates of the at least one feature point, and determine the availability of the GIS data based on the matching degree.

[0153] The target image may be any frame image in the video data.

[0154] For example, Fig. 9 As shown, the second determination module 603 may include:

[0155] The first feature point determination submodule 6031 is used to select at least one first feature point from the target image in the video data, and determine the pixel coordinates of the at least one feature point.

[0156] The first feature point may be a corner point and / or an edge point of a feature object in the scene captured by the video data.

[0157] The second feature point determination submodule 6032 is used to determine at least one second feature point in the GIS data corresponding to the at least one first feature point.

[0158] Specifically, the corner points and / or edge points of the above-mentioned feature objects may be found in the GIS data as the second feature points.

[0159] The pixel coordinate conversion submodule 6033 is used to convert the pixel coordinates of the at least one feature point into geographic coordinates according to the intrinsic parameters and extrinsic parameters of the monitoring camera corresponding to the target image.

[0160] After determining the internal and external parameters of the surveillance camera corresponding to the target image, the specific process of converting the pixel coordinates in the target image into geographic coordinates belongs to the prior art and will not be described in detail herein.

[0161] The matching degree determination submodule 6034 is used to determine the matching degree between the video data and the GIS data based on the geographical coordinates of the at least one first feature point obtained by conversion and the geographical coordinates of the at least one second feature point in the GIS data.

[0162] This process can be understood as a preliminary registration of the video data and the GIS data, and a process of evaluating the matching degree of the registered video data and the GIS data.

[0163] Exemplarily, the matching degree determination submodule 6034 may be used to:

[0164] Determine a registration deviation evaluation index between the video data and the GIS data according to the geographic coordinates of the at least one first feature point obtained by conversion, the geographic coordinates of the at least one second feature point in the GIS data, and a first formula;

[0165] The matching degree between the video data and the GIS data is determined according to the registration deviation evaluation index and the registration deviation evaluation threshold.

[0166] Among them, the first formula is:

[0167]

[0168] Among them, G i is the registration deviation evaluation index, P i is the geographic coordinates of the first feature point in the target image, Q iis the geographic coordinate of the second feature point corresponding to the first feature point in the GIS data, N is the total number of the first feature points, ||P i -Q i || is the Euclidean distance.

[0169] Optionally, the second determination module 603 may further include: an attribute adding submodule, configured to use the geographical coordinates of the at least one first feature point after registration as attributes of the video data, and assign them to the target image and the first feature point.

[0170] Furthermore, the matching degree may be represented by a matching level, and the corresponding relationship between the matching level and the registration deviation evaluation threshold may include:

[0171] Excellent matching level: G i ≤P y At this time, the registration effect of the first feature point and the second feature point in the GIS data is excellent, which means that the position of the first feature point in the geographic coordinate system is close to the position of the second feature point in the GIS data, with a small deviation. The registration effect of the video data and the geographic spatial information is good, and the error between the pixel coordinates and the geographic coordinates is small. The video data can be accurately associated with the geographic information, supporting subsequent geospatial analysis and applications;

[0172] Good match level: P y <G i ≤P c ,At this time, the registration effect of the first feature point and the second feature point in the GIS data is good, the registration deviation is moderate, and there is a deviation that has no effect on practical applications;

[0173] General Matching Level: G i >P c ,At this time, the registration effect between the first feature point and the second feature point in the GIS data is poor, the ,registration deviation is large, and the video data is difficult to correspond to the ,geographic information, which is not suitable for geospatial analysis and ,applications.

[0174] Among them, G i is the registration deviation evaluation index, P y is the registration deviation evaluation threshold corresponding to the excellent registration level, P c The threshold for evaluating the registration deviation corresponding to a good registration grade.

[0175] Of course, other matching level division methods can be used according to actual needs. The above is just an example.

[0176] On this basis, the second determining module 603 can be specifically used to:

[0177] When the matching level between the video data and the GIS data is higher than or equal to a preset level, determining that the GIS data is available;

[0178] When the matching level between the video data and the GIS data is lower than the preset level, it is determined that the GIS data is unavailable.

[0179] The preset level may be flexibly set, for example, the preset level may be the above-mentioned good matching level.

[0180] Optionally, when the GIS data is unavailable, available GIS data is reselected.

[0181] For example, suppose the first feature point in the video data is the corner point of a building on Street A, and the second feature point that matches it in the GIS data is the corner point of a building on Street B, resulting in the matching level of the two being at a general matching level, that is, the two do not match, and it is necessary to reselect available GIS data.

[0182] The data fusion module 604 is used to determine the geographic coordinates of each pixel in each frame of the video data based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data when the GIS data is available, and to fuse the video data with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

[0183] Optionally, before the data fusion module 604 fuses the video data with the GIS data, Figure 5 The device 600 shown may also include: a registration accuracy determination module, used to determine the registration accuracy of the first feature point and the corresponding second feature point based on the geographical coordinates of the at least one first feature point obtained by conversion, the geographical coordinates of the at least one second feature point in the GIS data, and the total number of the first feature points.

[0184] Correspondingly, when the registration accuracy meets the preset conditions, the data fusion module 604 is triggered again; when the registration accuracy does not meet the preset conditions, it means that the matching accuracy between the video data and the GIS data is poor and needs to be adjusted. Specifically, the second determination module 602 can be triggered to recalibrate the internal and external parameters of the surveillance camera corresponding to each frame image in the video data, and finally find the best matching position between the pixel coordinates in the video data and the geographic coordinates in the GIS data, so as to achieve precise fusion of video and GIS.

[0185] The registration accuracy includes at least one of geographic coordinate accuracy, geographic coordinate projection accuracy and target measurement accuracy.

[0186] Exemplarily, when the registration accuracy includes geographic coordinate accuracy, geographic coordinate projection accuracy, and target measurement accuracy, the preset conditions may include:

[0187] 1) The spatial positioning error of geographic coordinate accuracy is less than 1 meter;

[0188] 2) The reprojection error of geographic coordinate video images is less than 10 pixels;

[0189] 3) The measurement size error of the target measurement accuracy is less than a certain percentage relative to the target itself, such as less than 15%.

[0190] The calculation formula for geographic coordinate accuracy is:

[0191]

[0192] Among them, F z is the Euclidean distance between the geographic coordinates of the first feature point in the target image and the corresponding second feature point in the GIS data; wherein (x g ,y g ) represents the geographic coordinates of the second feature point in the GIS data, (x v ,y v ) represents the geographic coordinates of the first feature point; or, (x g ,y g ) represents the position of the calibration target g in the actual scene, (x v ,y v ) represents the pixel position of the calibration target g in the video frame v, and the at least one first feature point is located in the calibration target.

[0193] The calculation formula for geographic coordinate projection accuracy is:

[0194]

[0195] Among them, k is the scaling factor between pixel coordinates and geographic coordinates, which is used to project pixel coordinates into GIS data;

[0196] The calculation formula for the target measurement accuracy is:

[0197]

[0198] Among them, s g is the actual size of the calibrated target in geographic space, s v is a measured size of the calibration target in the target image, and the at least one first feature point is located in the calibration target.

[0199] In some embodiments, the data fusion module 604 can be specifically used to: assign geographic coordinates to the video data based on the geographic coordinates of each pixel point in each frame image in the video data, so as to map the pixel coordinates in the video data to the geographic coordinate system; and superimpose the video data assigned with geographic coordinates with the GIS data, so as to project the video data into the geographic space.

[0200] In other embodiments, Fig.10 As shown, the data fusion module 604 may specifically include:

[0201] Layer adding submodule 6041, used for adding a video layer to the GIS data;

[0202] The pixel projection submodule 6042 is used to project the pixels contained in each frame image in the video data onto the video layer in time sequence according to the geographical coordinates of each pixel point in each frame image in the video data;

[0203] The data fusion submodule 6043 is used to superimpose the GIS data with the video layer to obtain fused video GIS data.

[0204] Optional, Figure 6 The device 600 shown may also include: a display module for displaying the video GIS data, that is, superimposing the GIS data onto the video data for visual display, thereby realizing visual presentation of video fusion.

[0205] Specifically, the display module can be used for:

[0206] Visualizing the GIS data in the form of layers to form corresponding geographic information layers;

[0207] Performing pixel-level overlay processing on the geographic information layer and the image frame corresponding to the video data, so that the geographic information and the video content can be visually integrated;

[0208] Add interactive functions to the video after integrating geographic information, allowing users to view detailed information of different geographic areas and adjust the display method and level of geographic information by clicking or dragging.

[0209] The embodiment of the present application proposes a device for fusing video and GIS. On the one hand, the internal and external parameters of the monitoring camera corresponding to each frame of the video data can be calibrated, so that the geographic coordinates of each pixel in each frame of the video data can be accurately determined according to the internal and external parameters of the monitoring camera corresponding to each frame of the video data. Then, the video data and the GIS data are fused according to the geographic coordinates of each pixel in each frame of the video data, instead of simply splicing the video data and the GIS data. Therefore, the fusion effect of the video captured during the exercise and the GIS can be improved. On the other hand, before the formal fusion, the matching degree of the video data and the GIS data can be determined according to the geographic coordinates of a small number of feature points in the video data, and the availability of the GIS data can be determined according to the matching degree. When the GIS data is available, the video data and the GIS data are fused again, instead of blindly splicing. Therefore, the fusion effect of the video and GIS can be further improved.

[0210] In addition, a video and GIS fusion device provided in an embodiment of the present application can determine the alignment accuracy of the first feature point and the corresponding second feature point based on the geographic coordinates of the at least one first feature point obtained by conversion, the geographic coordinates of the at least one second feature point in the GIS data, and the total number of the first feature points before the formal fusion, and fuse the video data and the GIS data when the alignment accuracy meets the preset conditions, which can achieve a better fusion effect of the video data and the GIS data.

[0211] Furthermore, a video and GIS fusion device provided in an embodiment of the present application can access multiple video streams in real time, flexibly embed intelligent recognition models, obtain the position, posture, coverage of the camera and the spatial position corresponding to each pixel in the video image through technical means, so that the video data is integrated with the scene, especially the three-dimensional scene, and the video access is fully compatible with mainstream video surveillance equipment, and has the functions of monitoring camera calibration, two-dimensional and three-dimensional video and geographic space fusion, and fusion visualization presentation. In addition, a video and GIS fusion device provided in an embodiment of the present application is based on the internal and external parameters of the monitoring camera, and the video data after being assigned with geographic coordinates is superimposed with the GIS data, and the video is enhanced with GIS or GIS enhanced video to achieve the fusion of video and two-dimensional or three-dimensional GIS data, which overcomes the shortcomings of the traditional video GIS technology, such as the unintuitive two-dimensional GIS, the complex virtual reality modeling, and the low timeliness of remote sensing, making GIS more real and comprehensive, and injecting more vitality into GIS. Further, a video and GIS fusion device provided in an embodiment of the present application superimposes the video data after being assigned with geographic coordinates with the GIS data, making the traditional map more vivid and fresh, and further enhances the sense of reality of the immersive visual experience by using the fusion of three-dimensional models and video real scenes.

[0212] It should be noted that Figure 6 The video and GIS fusion device 400 can achieve Figure 1 The method of the method embodiment of the present invention can achieve the same technical effect, and the specific reference can be made to the above Figure 1 The introduction of the method for integrating video and GIS in the illustrated embodiment will not be repeated here.

[0213] Fig.11 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.11 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0214] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0215] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0216] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a fusion device of video and GIS at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0217] Obtain video data captured by surveillance cameras and geographic information system GIS data;

[0218] Determine the internal parameters and external parameters of the surveillance camera corresponding to each frame image in the video data;

[0219] Determine the geographic coordinates of at least one feature point in the video data according to the internal and external parameters of the surveillance camera corresponding to the target image in the video data, determine the matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, and determine the availability of the GIS data according to the matching degree;

[0220] When the GIS data is available, the geographic coordinates of each pixel in each frame of the video data are determined based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data, and the video data is fused with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

[0221] The above application Figure 1The method for integrating video and GIS disclosed in the embodiment shown above can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in software form. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in one or more embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in one or more embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0222] The electronic device may also perform Figure 1 The method of integrating video and GIS is not described in detail in this application.

[0223] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations:

[0224] Obtain video data captured by surveillance cameras and geographic information system GIS data;

[0225] Determine the internal parameters and external parameters of the surveillance camera corresponding to each frame image in the video data;

[0226] Determine the geographic coordinates of at least one feature point in the video data according to the internal and external parameters of the surveillance camera corresponding to the target image in the video data, determine the matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, and determine the availability of the GIS data according to the matching degree;

[0227] When the GIS data is available, the geographic coordinates of each pixel in each frame of the video data are determined based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data, and the video data is fused with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

[0228] The embodiment of the present application also proposes a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the video and GIS fusion method provided in the embodiment of the present application.

[0229] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0230] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0231] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0233] It should be noted that the various embodiments in this application are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0234] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0235] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for integrating video and GIS, characterized in that: The method comprises: Obtain video data captured by surveillance cameras and geographic information system GIS data; Determine the internal parameters and external parameters of the surveillance camera corresponding to each frame image in the video data; Determine the geographic coordinates of at least one feature point in the video data according to the internal and external parameters of the surveillance camera corresponding to the target image in the video data, determine the matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, and determine the availability of the GIS data according to the matching degree; When the GIS data is available, the geographic coordinates of each pixel in each frame of the video data are determined based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data, and the video data is fused with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

2. The method according to claim 1, characterized in that The video data is obtained by the monitoring camera moving along a specified route, wherein the determining of the internal parameters and external parameters of the monitoring camera corresponding to each frame image in the video data includes: Extract frames from the video data to obtain multiple frames of first images; Performing structure-from-motion processing on the multiple frames of first images to obtain intrinsic parameters and extrinsic parameters of the surveillance camera corresponding to the multiple frames of first images; According to the frame rate of the video data and the designated route, linear interpolation processing is performed on the intrinsic parameters and extrinsic parameters of the monitoring camera corresponding to the multiple frames of first images to obtain the intrinsic parameters and extrinsic parameters of the monitoring camera corresponding to the multiple frames of second images, wherein the multiple frames of second images are images in the video data other than the multiple frames of first images.

3. The method according to claim 1, characterized in that Determining the geographic coordinates of at least one feature point in the video data according to the internal parameters and external parameters of the surveillance camera corresponding to the target image in the video data, and determining the matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, includes: Selecting at least one first feature point from a target image in the video data, and determining pixel coordinates of the at least one feature point; Determining at least one second feature point in the GIS data corresponding to the at least one first feature point; Converting pixel coordinates of the at least one feature point into geographic coordinates according to the intrinsic parameters and extrinsic parameters of the surveillance camera corresponding to the target image; The matching degree between the video data and the GIS data is determined according to the geographical coordinates of the at least one first feature point obtained by conversion and the geographical coordinates of the at least one second feature point in the GIS data.

4. The method according to claim 3, characterized in that in, The determining the matching degree between the video data and the GIS data according to the geographical coordinates of the at least one first feature point obtained by conversion and the geographical coordinates of the at least one second feature point in the GIS data comprises: Determine a registration deviation evaluation index between the video data and the GIS data according to the geographic coordinates of the at least one first feature point obtained by conversion, the geographic coordinates of the at least one second feature point in the GIS data, and a first formula; Determining the matching degree between the video data and the GIS data according to the registration deviation evaluation index and the registration deviation evaluation threshold; Among them, the first formula is: Among them, G i is the registration deviation evaluation index, P i is the geographic coordinates of the first feature point in the target image, Q i is the geographic coordinate of the second feature point corresponding to the first feature point in the GIS data, N is the total number of the first feature points, ‖P i -Q i ‖ is the Euclidean distance.

5. The method according to claim 4, characterized in that The matching degree is represented by a matching level, and the corresponding relationship between the matching level and the registration deviation evaluation threshold includes: Excellent matching level: G i ≤P y ; Good match level: P y <G i ≤P c ; General Matching Level: G i >P c ; Among them, G i is the registration deviation evaluation index, P y is the registration deviation evaluation threshold corresponding to the excellent registration level, P c The threshold for evaluating the registration deviation corresponding to a good registration grade.

6. The method according to claim 5, characterized in that Determining the availability of the GIS data according to the matching degree includes: When the matching level between the video data and the GIS data is higher than or equal to a preset level, determining that the GIS data is available; When the matching level between the video data and the GIS data is lower than the preset level, it is determined that the GIS data is unavailable.

7. The method according to any one of claims 3 to 6, characterized in that: Before fusing the video data with the GIS data according to the geographic coordinates of each pixel in each frame image in the video data, the method further includes: Determine the registration accuracy of the first feature point and the corresponding second feature point according to the converted geographic coordinates of the at least one first feature point, the geographic coordinates of the at least one second feature point in the GIS data, and the total number of the first feature points; The step of fusing the video data with the GIS data according to the geographic coordinates of each pixel in each frame of the video data includes: When the registration accuracy meets the preset conditions, the video data is fused with the GIS data according to the geographic coordinates of each pixel in each frame image in the video data.

8. The method according to claim 7, characterized in that The registration accuracy includes at least one of geographic coordinate accuracy, geographic coordinate projection accuracy and target measurement accuracy, wherein: The calculation formula of the geographic coordinate accuracy is: Among them, F z is the Euclidean distance between the geographic coordinates of the first feature point in the target image and the corresponding second feature point in the GIS data; wherein (x g ,y g ) represents the geographic coordinates of the second feature point in the GIS data, (x v ,y v ) represents the geographic coordinates of the first feature point; The calculation formula of the geographic coordinate projection accuracy is: Among them, k is the scaling factor between pixel coordinates and geographic coordinates, which is used to project pixel coordinates into GIS data; The calculation formula of the target quantity measurement accuracy is: Among them, s g is the actual size of the calibrated target in geographic space, s v is a measured size of the calibration target in the target image, and the at least one first feature point is located in the calibration target.

9. The method according to any one of claims 1 to 6 and 8, characterized in that: The step of fusing the video data with the GIS data according to the geographic coordinates of each pixel in each frame of the video data includes: Adding a video layer to the GIS data; Projecting the pixels contained in each frame image in the video data onto the video layer in time sequence according to the geographic coordinates of each pixel point in each frame image in the video data; The GIS data is superimposed on the video layer to obtain fused video GIS data.

10. The method according to claim 9, characterized in that The method further comprises: Display the video GIS data.

11. A video and GIS fusion device, characterized in that: The device comprises: A data acquisition module is used to acquire video data captured by surveillance cameras and geographic information system GIS data; A first determination module, used to determine the internal parameters and external parameters of the monitoring camera corresponding to each frame image in the video data; A second determination module is used to determine the geographic coordinates of at least one feature point in the video data according to the internal parameters and external parameters of the surveillance camera corresponding to the target image in the video data, determine the matching degree between the video data and the GIS data according to the geographic coordinates of the at least one feature point, and determine the availability of the GIS data according to the matching degree; A data fusion module is used to determine the geographic coordinates of each pixel in each frame of the video data based on the internal and external parameters of the surveillance camera corresponding to each frame of the video data when the GIS data is available, and to fuse the video data with the GIS data based on the geographic coordinates of each pixel in each frame of the video data.

12. An electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1-10.

13. A computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the method according to any one of claims 1 to 10.