Video fusion presentation method based on CIM

Abstract

The invention discloses a CIM-based video fusion presentation method and device, and a storage medium. The method comprises the steps of obtaining a swing video and a three-dimensional geographic scene of a current position; respectively extracting video frames and digital model data from the swing video and the three-dimensional geographic scene; performing feature matching on the video frames and the digital model data through a preset feature matching algorithm to obtain effective feature points; determining three-dimensional coordinates corresponding to the effective feature points, and determining camera pose and angular point information according to the three-dimensional coordinates; and interpolating the camera pose and the angular point information to obtain camera poses and angular point positions of all frames in the swing video, thereby fusing the swing video and the three-dimensional geographic scene. Effective feature points are screened through the preset feature matching algorithm, pixel errors during matching of the video and the three-dimensional scene are reduced, and the technical effect of improving visual association of fusion of the swing video and the three-dimensional geographic scene is achieved by improving feature analysis and matching precision in the swing video and the three-dimensional geographic scene.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a CIM-based video fusion presentation method, device and storage medium. Background Art

[0002] In traditional methods of combining videos with 3D models, videos are usually viewed as a series of textures or image sequences, and various technical means are used to map these image sequences onto the surface of the 3D model. Specifically, this process involves aligning and associating the video content with specific objects or surfaces in the 3D scene in order to visually merge the video with the 3D scene.

[0003] The matching process between relevant videos and 3D scenes often relies on fixed matching rules and algorithms. These rules and algorithms are prone to low matching rates between video content and 3D scenes when dealing with complex scenes and dynamic changes, which ultimately affects the visual association effect of the combination of video and 3D models.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a CIM-based video fusion presentation method, device and storage medium, aiming to solve the technical problem of poor visual correlation effect when combining existing videos with three-dimensional models.

[0006] To achieve the above objectives, the present application proposes a CIM-based video fusion presentation method, the method comprising: Get the swing video and 3D geographic scene of the current location; extracting video frames and digital model data from the swing video and the three-dimensional geographic scene respectively; Perform feature matching on the video frame and the digital model data using a preset feature matching algorithm, and obtain effective feature points based on the matching results; Determine the three-dimensional coordinates corresponding to the valid feature points, and determine the camera pose and corner point information according to the three-dimensional coordinates; The camera pose and corner point information are interpolated to obtain the camera pose and corner point positions of all frames in the swing video, thereby fusing the swing video with the three-dimensional geographic scene.

[0007] In one embodiment, the step of performing feature matching on the video frame and the digital model data using a preset feature matching algorithm and obtaining valid feature points according to the matching results includes: Extracting feature identifiers from the video frame and matching the feature identifiers with feature identifiers in the digital model data; The successfully matched feature identifier is used as a valid feature point.

[0008] In one embodiment, the step of performing feature matching on the video frame and the digital model data using a preset feature matching algorithm and obtaining valid feature points according to the matching results includes: When the valid feature points cannot be obtained according to the matching results, extracting corresponding feature identifiers from the video frame and the digital model data respectively; The feature identifier is input into a preset interactive algorithm to calculate the effective feature point.

[0009] In one embodiment, the step of determining the camera pose and corner point information using the three-dimensional coordinates includes: Matching the current shooting parameters of the camera with the three-dimensional coordinates, and correcting the camera posture according to the matching result; The video frame is perspective-transformed according to the three-dimensional coordinates to obtain corner point information.

[0010] In one embodiment, the step of matching the current shooting parameters of the camera with the three-dimensional coordinates and correcting the camera posture according to the matching result includes: Determining shooting coordinates corresponding to the shooting parameters, and comparing coordinate deviations between the shooting coordinates and the three-dimensional coordinates; The camera pose is corrected based on the coordinate deviation.

[0011] In one embodiment, before the step of determining the three-dimensional coordinates corresponding to the valid feature points, the method further includes: Taking the locations of the effective feature points as samples, and determining abnormal values ​​that distinguish the samples; Noise processing is performed on the effective feature points to remove outliers of the effective feature points.

[0012] In one embodiment, the step of interpolating the camera pose and corner point information to fuse the swing video with the three-dimensional geographic scene includes: Determine the camera pose and corner point positions of all video frames in the swing video according to the interpolation processing results of the camera pose and corner point information; The camera pose and corner point positions are deployed to the three-dimensional geographic scene to fuse the swinging video with the three-dimensional geographic scene.

[0013] In one embodiment, the step of extracting video frames and digital model data from the swing video and the three-dimensional geographic scene comprises: Access the video stream data through the RTMP service to obtain the swing video of the current position; The shooting location of the swing video is determined, and a three-dimensional geographical scene of the shooting location is obtained.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a CIM-based video fusion presentation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the CIM-based video fusion presentation method as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the CIM-based video fusion presentation method as described above are implemented.

[0016] The technical solution proposed in this application has at least the following technical effects: The data identifiers extracted from the video frame and the digital model data of the three-dimensional geographic scene are matched by a preset feature matching algorithm to obtain effective feature points, and the camera posture and corner point information are determined according to the three-dimensional coordinates of the effective feature points. In this way, the camera posture and corner point positions of all frames in the swing video are obtained by interpolating the camera posture and corner point information, so as to realize the fusion of the swing video and the three-dimensional geographic scene. The preset feature matching algorithm can well screen effective feature points, reduce the pixel error in the matching of the video and the three-dimensional scene, and improve the feature analysis and matching accuracy in the swing video and the three-dimensional geographic scene, thereby realizing the technical effect of improving the visual association of the fusion of the swing video and the three-dimensional geographic scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A flowchart of the first embodiment of the video fusion presentation method based on CIM provided in this application; Figure 2 A flowchart of the second embodiment of the video fusion presentation method based on CIM provided in this application; Figure 3Schematic diagram of the device structure of the hardware operating environment involved in the CIM-based video fusion presentation method in the embodiment of the present application.

[0020] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: obtain the swinging video and three-dimensional geographic scene of the current position; extract video frames and digital model data from the swinging video and the three-dimensional geographic scene; perform feature matching on the video frames and the digital model data through a preset feature matching algorithm, and obtain valid feature points according to the matching results; determine the three-dimensional coordinates corresponding to the valid feature points; determine the camera posture and corner point information with the three-dimensional coordinates; interpolate the camera posture and corner point information to fuse the swinging video with the three-dimensional geographic scene.

[0024] Since the existing video and 3D scene matching process often relies on fixed matching rules and algorithms, these rules and algorithms are prone to low matching rates between video content and 3D scenes when dealing with complex scenes and dynamic changes, ultimately affecting the visual association effect of the video and 3D model.

[0025] This application provides a solution that can well screen effective feature points through a preset feature matching algorithm, reduce the error in video and three-dimensional scene matching, and improve the feature analysis and matching accuracy in swinging video and three-dimensional geographic scene. Therefore, in the process of projecting real-time video to a three-dimensional model, one or more video sequences of camera images and the three-dimensional virtual scene related thereto can be matched and fused to generate a new dynamic virtual scene or model of this scene, thereby realizing the fusion of virtual scene and real-time video.

[0026] Based on this, the embodiment of the present application provides a video fusion presentation method based on CIM, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the video fusion presentation method based on CIM of the present application. In this embodiment, the video fusion presentation method based on CIM includes steps S10 to S60: Step S10, obtaining a swing video and a three-dimensional geographic scene of the current position; Step S20, extracting video frames and digital model data from the swing video and the three-dimensional geographic scene respectively; Step S30, performing feature matching on the video frame and the digital model data using a preset feature matching algorithm, and obtaining valid feature points according to the matching results; Step S40, determining the three-dimensional coordinates corresponding to the valid feature points, and determining the camera posture and corner point information according to the three-dimensional coordinates; Step S50, interpolating the camera pose and corner point information to obtain the camera pose and corner point positions of all frames in the swing video, thereby fusing the swing video with the three-dimensional geographic scene.

[0027] In this embodiment, according to the current video fusion requirements, a swinging video of the current position is shot by a shooting device and a fusion process is performed based on the three-dimensional geographic scene corresponding to the swinging video. Among them, when the environmental information of the current position is shot by a video device, its shooting action should be real-time, that is, the video information of the current position shot should also be real-time, which involves the fusion operation of the video and the three-dimensional geographic scene. The fused video is essentially a swinging video obtained after processing the acquired real-time video, that is, it is necessary to process the acquired real-time video to obtain the corresponding swinging video, so as to be fused with the three-dimensional geographic scene through the swinging video. The swinging video is essentially a video with a swinging effect, and its swinging effect is generally manifested in that a certain element in the video (such as an image, text or the entire picture) swings according to a specific rhythm or path, which is used to reflect the dynamic feeling of the current video or the authenticity of the shot object.

[0028] Since the shooting device is a hardware device fixedly and / or loosely installed at different locations, when it is necessary to obtain the real-time video of the shooting device, the real-time video needs to be obtained by accessing the video stream data, so as to obtain the corresponding swing video after processing the captured real-time video. Therefore, the step of extracting video frames and digital model data from the swing video and the three-dimensional geographic scene includes: Access the video stream data through the RTMP service to obtain the swing video of the current position; Determine the shooting location of the swing video and obtain the three-dimensional geographical scene of the shooting location Based on the real-time video shooting operation of the currently installed video device, the service where the shooting device is located is accessed through the RTMP service to obtain the video stream of the real-time video. The RTMP service is the Real-Time Messaging Protocol service, which is defined as a real-time messaging transmission protocol service. It is a real-time streaming transmission protocol service based on TCP, and its application scenarios include audio and video live broadcast, real-time communication, etc. Therefore, the video stream data is accessed through the RTMP service to obtain the real-time video of the current position, and the real-time video is processed to obtain the corresponding swing video.

[0029] In practical applications, the process of obtaining the swinging video based on the real-time video processing is to screen and process the swinging image represented in the real-time video, that is, to extract the video that meets the fusion requirements from the real-time video as the swinging video.

[0030] Afterwards, based on the shooting location of the swinging video, a three-dimensional geographic scene based on the shooting location is constructed, and the three-dimensional geographic scene is represented by DSM data, or DEM data and DOM data. Wherein, when the three-dimensional geographic scene is represented by DSM data, the DSM (Digital Surface Model) is digital surface model data, which contains surface elevation information of elements such as buildings, bridges, and trees on the surface, and is used to comprehensively reflect the elevation data of the surface and all objects on it. The elevation data is obtained by laser radar LiDAR or photogrammetry, and the elevation data is processed and analyzed by related software or tools, and the DSM data is generated based on the processing results. The DSM data is usually stored in the form of a raster data set, and a customized projection coordinate system is used.

[0031] In addition, the three-dimensional geographic scene can also be characterized as DEM data and DOM data, wherein the DEM (Digital Elevation Model) can also be defined as a digital elevation model, which is used to describe surface elevation data. However, it should be clear that the DEM data can only be used to characterize surface elevation information, and does not include height information of buildings or other structures. Therefore, in this embodiment, the object data used to characterize the three-dimensional geographic scene is a collection of DEM data and DOM data, and the DOM data (digital orthophoto map) is a digital elevation model that uses a digital elevation model to perform pixel-by-pixel projection difference correction on the scanned digitized aerial photographs or remote sensing images (monochrome or color), and then mosaics the images and clips the generated image data according to the map range, which can accurately reflect the position and shape of surface objects. That is, the data set based on the DEM data and the DOM data is used as the characterization data of the three-dimensional geographic scene corresponding to the swinging video.

[0032] According to the obtained swinging video and the corresponding three-dimensional geographic scene, a plurality of video frames are respectively extracted from the swinging video, and digital model data are extracted from the three-dimensional geographic scene, and the video frames and the digital model data are feature matched with each other using a preset feature matching algorithm. The basis of the feature matching is that the data sources of the video frames and the digital model data are different data forms at the same location. Therefore, based on the feature matching, it is essentially to extract the data identifiers in the video frames and the digital model data for matching, so as to determine whether the objects represented by the data identifiers are consistent.

[0033] Thus, according to the matching result of the data title, when it is determined that the object represented by the data identifier is consistent, the data identifier is determined to be a valid feature point. Since the valid feature point can be respectively reflected in the video frame and the data model data, and the video frame and the data model data both have three-dimensional coordinates reflecting the data identifier. Considering the data accuracy, in order to improve the data features of the valid feature points and thus improve the accuracy of determining the corresponding three-dimensional coordinates, the outliers of the valid feature points are eliminated, that is, before the step of determining the three-dimensional coordinates corresponding to the valid feature points, it also includes: Taking the locations of the effective feature points as samples, and determining abnormal values ​​that distinguish the samples; Noise processing is performed on the effective feature points to remove outliers of the effective feature points.

[0034] According to the effective feature points obtained from the feature matching results of the video frame and the data model data, the location of the effective feature points is used as a sample, and the action of removing individual values ​​in the sample is defined as noise processing. In this embodiment, the noise processing is also defined as outlier removal. Outlier is an outlier, also known as abnormal data or outliers, which refers to the individual remaining observations in the sample whose values ​​are obviously deviated from the sample to which it (or they) belong.

[0035] To this end, when performing the noise removal, a confidence probability is given to the sample with the corresponding data identifier of the effective feature point, and a confidence limit is determined based on the confidence probability. The error exceeding the confidence limit is judged as a value that does not belong to the range of random errors, and it is regarded as an outlier and removed. In addition, before removing the outlier, it is necessary to determine the relative distribution of the outlier and the location of the sample to ensure that the removed outlier is truly erroneous data or unreasonable data. In addition, due to the sample setting, the removal of the outlier may affect the overall distribution of the data identifier corresponding to the effective feature point. Therefore, it is necessary to set further removal conditions for the outlier, which can be calculated based on the confidence probability set for the sample.

[0036] After removing the noise points of the determined valid feature points, the three-dimensional coordinates of the valid feature points in the three-dimensional geographic scene are determined. Since the valid feature points can be represented as data identifiers in the three-dimensional geographic scene, the three-dimensional coordinates of the valid feature points in the three-dimensional geographic scene are determined based on the three-dimensional geographic scene as a three-dimensional model.

[0037] Furthermore, the effective feature points are also characterized as data identifiers of the video frames. Therefore, based on the location of the effective feature points at the data identifiers in the video, the camera pose and corner point information are processed. Wherein, when the real-time video is captured by a shooting device and the swinging video is obtained, the shooting device is based on its own shooting parameters as a reference, and the three-dimensional geographic scene created based on the current position of the shooting device may have an angle deviation from the shooting device when shooting the swinging video. Therefore, based on the effective feature points obtained by matching the video frame with the digital model data, the data morphological parameters of the shooting device can be corrected based on the comparison of the three-dimensional coordinates of the effective feature points in the three-dimensional geographic scene and the position coordinates of the effective feature points in the video frame. The corrected data morphological parameters are made for the fusion of the video and the three-dimensional geographic scene, and have nothing to do with the actual shooting parameters of the shooting device.

[0038] In this embodiment, it is also necessary to obtain corner point information based on the perspective transformation between the three-dimensional coordinates and the swinging video. The perspective transformation process is essentially a process of detecting corner points based on the coordinate transformation in the three-dimensional space combined with the perspective projection of the video frame image. The result of the corner point detection is the corner point information, which is characterized as the connection point of the object contour line in the video frame, or the highest point of the change rate of the gradient value and the gradient direction in the video frame.

[0039] According to the obtained camera pose and corner point information, the camera pose and the corner point information are processed by an interpolation algorithm so as to fuse the swinging video with the three-dimensional map into a real-scene three-dimensional model. The step of interpolating the camera pose and the corner point information to fuse the swinging video with the three-dimensional geographic scene includes: Determine the camera pose and corner point positions of all video frames in the swing video according to the interpolation processing results of the camera pose and corner point information; The camera pose and corner point positions are deployed to the three-dimensional geographic scene to fuse the swinging video with the three-dimensional geographic scene.

[0040] According to the determined camera pose and corner point information, an interpolation algorithm is used as a reference to determine the camera pose and corner point position of all video frames in the swinging video. Wherein, the interpolation algorithm is also a proximity interpolation method, which uses the camera pose and corner point information as a reference, projects points in the three-dimensional space onto a two-dimensional image plane as an image frame according to the camera pose, and after scaling the image frame, uses the interpolated pixel point as a reference, and selects the known pixel point closest to the pixel point as the pixel value of the pixel point, which is the interpolation process. In the interpolation process, it is first necessary to determine the pixel point position for difference, that is, the three-dimensional coordinates of the corresponding effective feature point used to determine the camera pose are mapped to the pixel point position of the two-dimensional point in the video frame image, which is used as a known pixel point, and uses the known pixel point as a reference to find the pixel point closest to it, and assign the pixel value of the pixel point to the pixel point position of the position point, which is the interpolation process. After all pixel points are interpolated, the image frame is subjected to image enhancement, restoration, transformation and other processing to achieve the fusion of the swinging video and the three-dimensional geographic scene to form a real-scene three-dimensional model.

[0041] In this embodiment, the preset feature matching algorithm can effectively screen effective feature points, reduce the error in the matching of video and three-dimensional scene, and improve the feature analysis and matching accuracy in swinging video and three-dimensional geographic scene. In the process of projecting real-time video onto the three-dimensional model, one or more camera image sequence videos and the related three-dimensional virtual scenes can be matched and fused to generate a new dynamic virtual scene or model of this scene, thereby realizing the fusion of virtual scene and real-time video.

[0042] Optionally, in one embodiment, in step S30, the step of performing feature matching on the video frame and the digital model data by using a preset feature matching algorithm, and obtaining valid feature points according to the matching results includes: Extracting feature identifiers from the video frame and matching the feature identifiers with feature identifiers in the digital model data; The successfully matched feature identifier is used as a valid feature point.

[0043] In this embodiment, multiple video frames are extracted from the swinging video, and the digital model data is extracted from the three-dimensional geographic model. The number of the multiple video frames is the number of video frames that can reflect the real scene, that is, the activity scene of the objects in the real scene. The digital model data is the digital model data for the location of the video frame based on the corresponding location range in the three-dimensional geographic model.

[0044] Since the video frame and the digital model data are both used to represent the scene information of the same area, data identifiers that can represent the same specific scene content are extracted from the video frame and the digital model data respectively for feature matching, and then it is determined whether the data identifiers are the same features based on the matching degree obtained by the feature matching, and the data identifiers with the same features are defined as the valid feature points.

[0045] Based on the matching of the data identifiers, a preset feature matching algorithm is provided in this embodiment, and the data identifiers extracted from the video frame and the digital model data are respectively input as parameters into the preset feature matching algorithm for data processing.

[0046] In practical applications, the preset feature matching algorithm essentially takes the characteristic attributes or descriptive parameters possessed by the data identifier as matching entities, and calculates the similarity measure between the matching entities to achieve conjugate entity registration. The features in the preset feature matching algorithm of the present application are generally divided into local features and global features, wherein the local features include points, edges, line segments, small faces or local energy, etc. They are usually invariant to rotation, scale and illumination changes, so that they can be accurately detected at different positions and angles of the image; the global features include polygonal image structures, etc., which are usually used to describe the overall shape and structure of the image, but in practical applications, local features have better robustness and accuracy, so in the embodiment of the present application, the data identifier is the local feature.

[0047] Furthermore, before applying the preset feature matching algorithm, the data identifiers extracted from the video frame and the digital model data must comply with the feature extraction requirements of the preset feature matching algorithm, that is, the data identifiers are extracted as local features according to the preset feature extraction algorithm for matching of the feature matching algorithm. The feature extraction algorithms include but are not limited to SIFT (Scale Invariant Feature Transform), SURF (Speeded Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc. Furthermore, after the corresponding data identifiers are extracted based on the feature extraction algorithm, the data identifiers are described for subsequent matching operations. The feature description generally involves converting feature points or feature areas into a discriminative representation, such as a feature vector or feature descriptor.

[0048] In this way, feature matching is performed based on the extracted data identifiers, and the feature matching process is essentially to correspond the data identifiers in two or more images or data to achieve association and matching between images. Moreover, in the matching process, it is usually involved to calculate the similarity measure between the feature descriptions of the data identifiers, such as Euclidean distance, Manhattan distance, cosine similarity, etc. According to the result of the similarity measure, the most similar data identifier pair can be selected as the matching result. Moreover, the data identifier pair whose matching result meets the feature point requirements is taken as a valid feature point. The feature point requirements can be defined as a matching threshold or a similarity threshold, which is customized with the representation type of the matching result.

[0049] Furthermore, considering the accuracy of the preset feature matching algorithm, a training sample based on the preset feature matching algorithm can be set to train the preset feature matching algorithm, thereby improving the accuracy of the calculation. Among them, the training sample can be created based on the video frame at the same position and the relevant data information extracted from the corresponding three-dimensional data model, and the preset feature matching algorithm can be trained based on the training sample, so that the preset feature matching algorithm can improve the feature recognition of three-dimensional objects through the training process.

[0050] In addition, the step of performing feature matching on the video frame and the digital model data by using a preset feature matching algorithm and obtaining effective feature points according to the matching results includes: When the valid feature points cannot be obtained according to the matching results, extracting corresponding feature identifiers from the video frame and the digital model data respectively; The feature identifier is input into a preset interactive algorithm to calculate the effective feature point.

[0051] Considering the deviation between the video frame and the corresponding three-dimensional geographic scene, there is a possibility that effective feature points cannot be obtained through the preset feature matching algorithm. Therefore, the video frame and the digital model data are matched with features through a preset interactive algorithm to obtain effective feature points. The interactive algorithm is essentially an algorithm based on human-computer interaction. The user manually detects the data identifier and processes it through the interactive algorithm to determine the corresponding data identifier, thereby determining the effective feature point based on the similarity between the data identifiers.

[0052] Optionally, in one embodiment, in step S50, the step of determining the camera pose and corner point information using the three-dimensional coordinates includes: Matching the current shooting parameters of the camera with the three-dimensional coordinates, and correcting the camera posture according to the matching result; The video frame is perspective-transformed according to the three-dimensional coordinates to obtain corner point information.

[0053] In this embodiment, the process of determining the camera posture and corner point information by using the three-dimensional coordinates of the effective feature points involves two processes: camera posture correction and corner point information determination.

[0054] When correcting the camera posture with three-dimensional coordinates, the concept of the camera posture needs to be clarified, that is, the camera posture refers to the position and posture of the camera in three-dimensional space, including the rotation angle and translation vector of the camera. In this embodiment, it can also be defined as the rotation angle and translation vector represented by the shooting parameters when the shooting device shoots the current real-scene video. The three-dimensional coordinates of the effective feature points are used as a reference to adjust and correct the camera posture to ensure that the image or three-dimensional model data obtained by the camera has higher precision and accuracy.

[0055] According to the coordinates of the three-dimensional coordinates of the effective feature points and the data identification position on the video frame, the external orientation elements of the camera are calculated, and the external orientation elements include the rotation angle (azimuth, pitch angle, etc.) and translation vector of the camera; then, according to the internal and external parameters of the camera (including the internal parameter matrix, distortion parameters, etc.), the camera internal parameters and camera external parameters can be obtained through the calibration method, and the calibration method can also obtain the distortion and other parameters of the corresponding camera, and perform image space coordinate transformation on the photogrammetry data. The coordinates of the points on the image are converted to the coordinates of the midpoints in the real world to obtain the initial three-dimensional model data.

[0056] Based on the spatial features represented by the initial three-dimensional model data, determining relevant coordinate deviations and other contents, that is, matching the current shooting parameters of the camera with the three-dimensional coordinates, and correcting the camera posture according to the matching results, comprises: Determining shooting coordinates corresponding to the shooting parameters, and comparing coordinate deviations between the shooting coordinates and the three-dimensional coordinates; The camera pose is corrected based on the coordinate deviation.

[0057] According to the coordinates of the three-dimensional coordinates of the effective feature points and the coordinates of the data identification position on the video frame, the coordinate deviation between the two coordinates is determined. When correcting the camera posture based on the coordinate deviation, the initial three-dimensional model data obtained above can be corrected for accuracy by the least square method or other optimization algorithms to obtain more accurate three-dimensional model data and camera posture. Furthermore, professional three-dimensional model geometric coordinate post-correction processing software (such as 3D Factory K3DMaker, etc.) can be used to assist in completing this process, and the accuracy assessment report and corrected three-dimensional model data can be exported based on the correction results.

[0058] Furthermore, in the process of obtaining corner point information by perspective transformation based on the three-dimensional coordinates and the swinging video, the contents such as coordinate transformation in three-dimensional space, perspective projection of video images and corner point detection are involved.

[0059] Among them, in the application based on the three-dimensional coordinates and perspective projection, the three-dimensional coordinates are the position representation of the effective feature points in the three-dimensional geographic model. After the three-dimensional coordinates of the effective feature points are projected as spatial points onto the two-dimensional plane, the position and angle of the camera in the swinging video will change greatly. Therefore, it is necessary to transform the points in the three-dimensional space into the two-dimensional image plane through perspective projection.

[0060] The perspective projection transformation process involves a perspective transformation matrix. The perspective is a matrix used to convert three-dimensional coordinates into two-dimensional coordinates. When this matrix converts a point (X, Y, Z) in a three-dimensional space into a point (x, y) on a two-dimensional image plane, it converts the three-dimensional coordinates of the effective feature point into a two-dimensional point position on the corresponding video frame image plane. Based on its data representation, the data representation of the perspective transformation matrix is ​​as follows: xyz1=f / Z00000f / Z00000AC00BDXYZ1.

[0061] Among them, f is the focal length of the camera, Z_0 is the depth of the reference plane in the video frame, and A, B, C, and D are parameters related to the camera intrinsic parameters and the projection plane position.

[0062] When the three-dimensional coordinates of the effective feature points are perspectively transformed into two-dimensional points on the plane of the video frame image, the corner point information of the object in the video frame is determined based on the two-dimensional points. The corner point information is essentially the data information possessed by multiple corner points, and the corner point is usually the intersection of two edges, or a point with a significant gradient change. In the swinging video indicated in this embodiment, the corner point may be a significant feature point such as the edge of an object, a texture change, etc. Moreover, in this embodiment, the corner point can also be detected based on a preset corner point detection scheme. Commonly used corner point detection methods include Moravec corner point detection, Harris corner point detection, Kitchen-Rosenfeld corner point detection, etc. Then, the coordinate information of the corner point is determined from the image of the video frame through the preset corner point detection method, which is the corner point information.

[0063] For further information, please refer to Figure 2 , Figure 2 Another embodiment of the video fusion presentation method based on CIM of the present application is based on an embodiment. The video fusion presentation method based on CIM includes the following contents: In this embodiment, the camera obtains real-time video information within the current shooting range according to the current initial positioning information. The camera can be fixed or not fixed at the current position, and shoots real-time video in the current environment according to the shooting range of the camera. Therefore, the video stream of the real-time video is obtained by accessing the service where the shooting device is located through the RTMP service as the video information at the current position, and the swing video to be fused is obtained after processing the video information.

[0064] Furthermore, based on the real-time video captured by the current camera at the current position, it is determined whether there is a corresponding DSM digital surface model for the shooting environment of the real-time video, that is, whether the pre-built DSM digital surface model contains relevant model content of the current position. If so, the DSM of the current position is extracted; if not, the DEM (digital elevation model) and DOM (digital orthophoto) of the current position are obtained from the network platform. Generally, the DEM and DEM of the current position can be captured through the Google platform.

[0065] As shown above, the swing video and digital surface model obtained after real-time video processing are both based on the current position. Therefore, the swing video and the digital surface model both record the feature information of the current position. Therefore, the swing video and the digital surface model are feature matched, and the effective feature points are determined according to the matching results. Furthermore, for the feature information represented by the swing video and the digital surface model, in order to improve the matching accuracy of the effective feature points, a preset feature matching algorithm is provided to match the feature information represented by the swing video and the digital surface model. The preset feature matching algorithm is a pre-trained data structure that can be used for feature matching. In this embodiment, the preset feature matching algorithm is trained using video frame images and digital surface model data as training samples. Based on the training process, the feature matching algorithm can be improved to reduce the pixel deviation in the feature matching process, thereby achieving the purpose of improving the accuracy of feature matching.

[0066] In addition, when it is impossible to obtain enough matching points based on the preset feature matching algorithm, feature information in the video frame and the digital surface model can be manually extracted, and the feature information can be matched with a preset interactive algorithm to obtain valid feature points that meet the requirements and quantity.

[0067] Based on this, according to the calculated effective feature points, after removing the noise points of the effective feature points, the three-dimensional feature points corresponding to the effective feature points can be clarified, thereby obtaining the three-dimensional coordinate information corresponding to the effective feature points. In this way, the camera posture of the camera can be corrected based on the three-dimensional coordinates. The correction of the camera posture defined here is essentially the correction of the camera posture in the data form, that is, adjusting the shooting parameters of the camera in the data form, including the camera position and the camera angle, etc., which has nothing to do with the actual camera shooting parameters. In addition, the swinging video is processed with perspective transformation through the three-dimensional coordinates, so as to obtain the corner point information of the video frame in the swinging video. The perspective transformation process is essentially a process of detecting corner points based on the coordinate transformation in the three-dimensional space combined with the perspective projection of the video frame image.

[0068] Through the determined camera pose and corner point information, the camera pose and corner point position of all video frames in the swing video are obtained by interpolating the corner point information and the camera pose, and the interpolation result is deployed in the digital map model to achieve the fusion of the swing video and the digital map model to obtain a real-life map model.

[0069] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the CIM-based video fusion presentation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0070] The present application provides a CIM-based video fusion presentation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the CIM-based video fusion presentation method in the above-mentioned embodiment 1.

[0071] Reference below Figure 3 , which shows a schematic diagram of the structure of a CIM-based video fusion presentation device suitable for implementing the embodiment of the present application. The CIM-based video fusion presentation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3The CIM-based video fusion presentation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0072] like Figure 3 As shown, the CIM-based video fusion presentation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of the CIM-based video fusion presentation device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the CIM-based video fusion presentation device to communicate with other devices wirelessly or by wire to exchange data. Although the CIM-based video fusion presentation device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0073] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0074] The CIM-based video fusion presentation device provided by the present application adopts the CIM-based video fusion presentation method in the above-mentioned embodiment, which can solve the technical problem that the matching rate between the existing video and the three-dimensional scene is not high, which ultimately affects the visual association effect of the combination of the video and the three-dimensional model. Compared with the prior art, the beneficial effects of the CIM-based video fusion presentation device provided by the present application are the same as the beneficial effects of the CIM-based video fusion presentation method provided by the above-mentioned embodiment, and the other technical features of the CIM-based video fusion presentation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0075] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0076] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0077] The present application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the CIM-based video fusion presentation method in the above-mentioned embodiment.

[0078] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0079] The computer-readable storage medium may be included in the CIM-based video fusion presentation device; or may exist independently without being assembled into the CIM-based video fusion presentation device.

[0080] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the CIM-based video fusion presentation device, the CIM-based video fusion presentation device implements the technical content of the CIM-based video fusion presentation method embodiment shown above.

[0081] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0084] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned CIM-based video fusion presentation method, and can solve the technical problem that the matching rate between existing videos and three-dimensional scenes is not high, which ultimately affects the visual association effect of the combination of videos and three-dimensional models. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the CIM-based video fusion presentation method provided by the above-mentioned embodiment, and will not be repeated here.

Claims

1. A video fusion presentation method based on CIM, characterized in that: The method includes: Get the swing video and 3D geographic scene of the current location; extracting video frames and digital model data from the swing video and the three-dimensional geographic scene respectively; Perform feature matching on the video frame and the digital model data using a preset feature matching algorithm, and obtain effective feature points based on the matching results; Determine the three-dimensional coordinates corresponding to the valid feature points, and determine the camera pose and corner point information according to the three-dimensional coordinates; The camera pose and corner point information are interpolated to obtain the camera pose and corner point positions of all frames in the swing video, thereby fusing the swing video with the three-dimensional geographic scene.

2. The CIM-based video fusion presentation method according to claim 1, characterized in that: The step of performing feature matching on the video frame and the digital model data by using a preset feature matching algorithm and obtaining effective feature points according to the matching results includes: Extracting feature identifiers from the video frame and matching the feature identifiers with feature identifiers in the digital model data; The successfully matched feature identifier is used as a valid feature point.

3. The CIM-based video fusion presentation method according to claim 1, characterized in that: The step of performing feature matching on the video frame and the digital model data by using a preset feature matching algorithm and obtaining effective feature points according to the matching results comprises: When the valid feature points cannot be obtained according to the matching results, extracting corresponding feature identifiers from the video frame and the digital model data respectively; The feature identifier is input into a preset interactive algorithm to calculate the effective feature point.

4. The CIM-based video fusion presentation method according to claim 1, characterized in that: The step of determining the camera position and corner point information using the three-dimensional coordinates includes: Matching the current shooting parameters of the camera with the three-dimensional coordinates, and correcting the camera posture according to the matching result; The video frame is perspective-transformed according to the three-dimensional coordinates to obtain corner point information.

5. The CIM-based video fusion presentation method according to claim 4, characterized in that: The step of matching the current shooting parameters of the camera with the three-dimensional coordinates and correcting the camera posture according to the matching result includes: Determining shooting coordinates corresponding to the shooting parameters, and comparing coordinate deviations between the shooting coordinates and the three-dimensional coordinates; The camera pose is corrected based on the coordinate deviation.

6. The CIM-based video fusion presentation method according to claim 1, characterized in that: Before the step of determining the three-dimensional coordinates corresponding to the effective feature points, the method further includes: Taking the locations of the effective feature points as samples, and determining abnormal values ​​that distinguish the samples; Noise processing is performed on the effective feature points to remove outliers of the effective feature points.

7. The CIM-based video fusion presentation method according to claim 1, characterized in that: The step of interpolating the camera position and corner point information to fuse the swing video with the three-dimensional geographic scene comprises: Determine the camera pose and corner point positions of all video frames in the swing video according to the interpolation processing results of the camera pose and corner point information; The camera pose and corner point positions are deployed to the three-dimensional geographic scene to fuse the swinging video with the three-dimensional geographic scene.

8. The CIM-based video fusion presentation method according to claim 1, characterized in that: The step of extracting video frames and digital model data from the swing video and the three-dimensional geographic scene comprises: Access the video stream data through the RTMP service to obtain the swing video of the current position; The shooting location of the swing video is determined, and a three-dimensional geographical scene of the shooting location is obtained.

9. A video fusion presentation device based on CIM, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the CIM-based video fusion presentation method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the CIM-based video fusion presentation method according to any one of claims 1 to 8 are implemented.

Images