Method and system applied to map video fusion based on video cutting and splicing technology

Abstract

The invention discloses a method and a system for applying a video cutting and splicing technology to map video fusion, belongs to the technical field of safety monitoring, and solves the problems of difficulty in effectively processing complex scenes, low processing efficiency and complex operation in a traditional method, and the problem of how to improve information extraction and presentation effects of monitoring videos through technical means. The method comprises the following steps: acquiring a front-end monitoring video; performing screenshot on the front-end monitoring video to obtain a plurality of screenshots, and generating a corresponding canvas; intercepting a to-be-spliced area on each canvas, and encoding the plurality of to-be-spliced areas to obtain a plurality of images; the method comprises the following steps: acquiring position information and attitude information of a monitoring camera, and establishing a three-dimensional earth coordinate system and a reverse view matrix; processing the plurality of to-be-spliced areas after the coding processing to obtain a plurality of pictures; and cutting off the overlapped parts of the plurality of pictures, and splicing to form a continuous video monitoring picture. The method is suitable for map application and virtual environment simulation application scenes.

Description

Technical Field

[0001] The invention belongs to the technical field of security monitoring, and in particular relates to a method for fusing video processing with a three-dimensional map. Background Art

[0002] Nowadays, surveillance video is increasingly used in a wide range of fields, especially in security monitoring, urban planning, and environmental management. However, traditional surveillance video processing technology faces the following major challenges:

[0003] 1. Surveillance video generation for complex scenes:

[0004] Surveillance videos often involve complex environments, such as traffic intersections, commercial areas, and public buildings in cities. Traditional video processing methods often have difficulty effectively processing these complex scenes, especially in video generation and editing in multi-angle and multi-camera situations.

[0005] 2. Accurate fusion of multi-camera videos:

[0006] Videos captured by multiple surveillance cameras need to be precisely time-synchronized and spatially calibrated to ensure accurate matching of positions and angles when splicing the videos.

[0007] Current surveillance video stitching technology usually suffers from low efficiency and complex operation when processing complex scenes.

[0008] 3. Improvement of video quality and information integrity:

[0009] Users' demands for surveillance videos are not limited to visual effects, but also include the accuracy and completeness of information. How to improve the information extraction and presentation effects of surveillance videos through technical means is a key issue that needs to be solved at present. Summary of the invention

[0010] In response to the above technical challenges, the present invention aims to provide a method and system for map video fusion based on video cropping and stitching technology, so as to solve the problems that traditional video processing methods are often difficult to effectively process complex scenes, the current surveillance video stitching technology is usually inefficient and complex to operate when processing complex scenes, and how to improve the information extraction and presentation effects of surveillance videos through technical means.

[0011] The present invention proposes a method for applying video cropping and splicing technology to map video fusion, comprising:

[0012] S1: collect front-end surveillance video;

[0013] S2: Taking screenshots of the front-end surveillance video, obtaining multiple screenshots, and generating a corresponding canvas according to each screenshot;

[0014] S3: intercepting a region to be stitched on each canvas to obtain a plurality of regions to be stitched, and encoding the plurality of regions to be stitched to obtain a plurality of images;

[0015] S4: Collect the location and attitude information of the surveillance camera, and use Cesium to establish a three-dimensional earth coordinate system and reverse view matrix;

[0016] S5: Use Cesium's material system to process multiple areas to be spliced ​​after encoding to obtain multiple images;

[0017] S6: cutting off the overlapping parts of the multiple images and splicing them to form a continuous video surveillance image, thereby completing the method.

[0018] Furthermore, a preferred solution is provided: in S2, the front-end monitoring video is screenshotted using the Canvas element of HTML5 and the drawImage function of JavaScript.

[0019] Furthermore, a preferred solution is provided: the S3 comprises: extracting each of the areas to be spliced, generating an image on a canvas with a black background and the areas to be spliced ​​being white; and encoding the image by base64 and storing it in a database.

[0020] Furthermore, a preferred solution is provided: S4 includes:

[0021] S4.1: Using Cesium to build GIS map Figure 3 Dimension scene, load map data;

[0022] S4.2: converting the location information and attitude information of the surveillance camera into a coordinate system on the three-dimensional earth;

[0023] S4.3: Generate a reverse view matrix based on the position information and Euler angle parameters of the surveillance camera and the rotation matrix;

[0024] S4.4: Create a perspective projection volume to define the visible range of the surveillance video in three-dimensional space; add the inverse view matrix, projection volume, and video elements to the Cesium scene.

[0025] Furthermore, a preferred solution is provided: S4.3 includes:

[0026] S4.3.1: Use the Cesium.Transforms.eastNorthUpToFixedFrame function to define the reverse view matrix of the surveillance camera;

[0027] S4.3.2: Determine the inverse view matrix using the rotation matrix.

[0028] Furthermore, a preferred solution is provided: S4.3.2 includes:

[0029] First rotation: Apply a matrix that rotates π / 2 radians around the X axis to the generated inverse view matrix;

[0030] Second rotation: rotate the heading arc around the Y axis, where heading is the yaw angle of the surveillance camera;

[0031] The third rotation: rotate the pitch radian around the X axis, where pitch is the pitch angle of the surveillance camera;

[0032] The fourth rotation: rotate the roll arc around the Z axis, where roll is the roll angle of the surveillance camera.

[0033] Furthermore, a preferred solution is provided: S5 includes:

[0034] S5.1: Pass the multiple images into the alphaImage attribute of uniforms;

[0035] S5.2: Obtain color information of the multiple images, set the areas to be retained in the multiple images to white, and set the areas to be discarded to black;

[0036] S5.3: Obtain the red channels of the multiple images as transparency values; adjust the transparency values ​​of white areas to 1, and adjust the transparency values ​​of black areas to 0.

[0037] The present invention also proposes a system for map video fusion based on video cropping and splicing technology, comprising:

[0038] Acquisition module: used to collect front-end monitoring videos;

[0039] Screenshot module: used to take screenshots of the front-end surveillance video, obtain multiple screenshots, and generate a corresponding canvas according to each screenshot;

[0040] Coding module: used for intercepting the area to be spliced ​​on each canvas to obtain multiple areas to be spliced, and performing coding processing on the multiple areas to be spliced ​​to obtain multiple images;

[0041] Coordinate establishment module: used to collect the location information and posture information of the surveillance camera, and use Cesium to establish a three-dimensional earth coordinate system and reverse view matrix;

[0042] Area processing module: used to process multiple areas to be spliced ​​after encoding using Cesium's material system to obtain multiple images;

[0043] Splicing module: used for cutting off the overlapping parts of the multiple images and splicing them into a continuous video surveillance image.

[0044] The present invention also proposes a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for applying video cropping and splicing technology to map video fusion according to any combination of the above-mentioned solutions.

[0045] The present invention also proposes a computer-readable storage medium, which is used to store a computer program, and the computer program executes a method for map video fusion based on video cropping and splicing technology described in any combination of the above solutions.

[0046] Compared with the prior art, the advantages of the present invention are:

[0047] The present invention provides a method for applying map video fusion based on video cropping and splicing technology, which realizes efficient and accurate coherent fusion of multiple videos in complex monitoring scenes through video cropping and splicing. Through innovative algorithms and technical solutions, the present invention aims to solve the technical bottlenecks existing in traditional methods and bring new technical breakthroughs and application prospects to the field of monitoring video processing.

[0048] The map video fusion method based on video cutting and splicing technology provides new possibilities for map applications through efficient video integration, enhanced interactive experience, a wide range of application scenarios and technological innovation. The introduction of this technology not only improves the visualization effect and user experience of the map, but also creates more opportunities for future map applications and virtual environments.

[0049] The present invention is applicable to map applications and virtual environment simulation application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 This is a cutting schematic diagram of the first specific implementation mode of the present invention;

[0052] Figure 2 The original image and the cropping effect image described in the first specific implementation mode of the present invention;

[0053] Figure 3The original picture and the splicing and fusion effect picture described in the first specific implementation mode of the present invention;

[0054] Figure 4 This is a diagram of monitoring coverage distribution and splicing effects described in the first specific implementation mode of the present invention. DETAILED DESCRIPTION

[0055] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0056] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0057] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0058] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific implementation methods disclosed below.

[0060] Implementation method 1:

[0061] Reference Figure 1 , Figure 2 , Figure 3 , Figure 4 This embodiment will be described.

[0062] A method for map video fusion based on video cropping and splicing technology, comprising:

[0063] S1: Collect front-end monitoring video; Specifically: This step uses webrtc-streamer.js to play the front-end monitoring video, and uses WebRTC technology to support the transmission and playback of real-time video streams, which can ensure the real-time and high efficiency of the video.

[0064] S2: Take a screenshot of the front-end monitoring video, obtain multiple screenshots, and generate a corresponding canvas according to each screenshot; further, a preferred solution is provided: in S2, the Canvas element of HTML5 and the drawImage function of JavaScript are used to take a screenshot of the front-end monitoring video; this step is used to generate a static image on the client for subsequent processing.

[0065] S3: intercepting the area to be spliced ​​on each canvas to obtain multiple areas to be spliced, and encoding the multiple areas to be spliced ​​to obtain multiple images; further, S3 includes: extracting each area to be spliced, generating an image on the canvas with a black background as the base and the area to be spliced ​​as white; encoding the image in base64 and storing it in a database for subsequent use;

[0066] S4: Collect the location and posture information of the surveillance camera, and use Cesium to establish a three-dimensional earth coordinate system and reverse view matrix. Specifically:

[0067] S4.1: Using Cesium to build GIS map Figure 3 Dimension scene, load map data;

[0068] S4.2: converting the location information and attitude information of the surveillance camera into a coordinate system on the three-dimensional earth;

[0069] S4.3: Generate a reverse view matrix based on the position information and Euler angle parameters of the surveillance camera and the rotation matrix;

[0070] S4.3.1: Use the Cesium.Transforms.eastNorthUpToFixedFrame function to define the reverse view matrix of the surveillance camera;

[0071] S4.3.2: Determine the reverse view matrix using the rotation matrix;

[0072] First rotation: Apply a matrix that rotates π / 2 radians around the X axis to the generated inverse view matrix;

[0073] Second rotation: rotate the heading arc around the Y axis, where heading is the yaw angle of the surveillance camera;

[0074] The third rotation: rotate the pitch radian around the X axis, where pitch is the pitch angle of the surveillance camera;

[0075] The fourth rotation: rotate the roll arc around the Z axis, where roll is the roll angle of the surveillance camera;

[0076] S4.4: Create a perspective projection volume to define the visible range of the surveillance video in three-dimensional space; add the inverse view matrix, projection volume, and video elements to the Cesium scene;

[0077] S5: Use Cesium's material system to process multiple areas to be spliced ​​after encoding to obtain multiple images;

[0078] S5.1: Pass the multiple images into the alphaImage attribute of uniforms;

[0079] S5.2: Obtain color information of the multiple images, set the areas to be retained in the multiple images to white, and set the areas to be discarded to black;

[0080] S5.3: Obtain the red channels of the multiple images as a transparency value; adjust the transparency value of the white area to 1, and adjust the transparency value of the black area to 0;

[0081] S6: cutting off the overlapping parts of the multiple images and splicing them to form a continuous video surveillance image, thereby completing the method.

[0082] The S4 is further described in detail below:

[0083] The S4 uses Cesium to implement video fusion, as follows:

[0084] The web front end uses CesiumJS engine to render and build GIS map Figure 3 Dimension the scene and load related map data;

[0085] Use the functions in the Cesium library to convert the projected surveillance camera's position and attitude information into a coordinate system on the three-dimensional earth. This includes converting longitude, latitude, and altitude information into Cartesian3 type objects;

[0086] Using the position and Euler angle (heading, pitch, roll) parameters of the surveillance camera, generate an inverse view matrix suitable for use in the Cesium scene. Specifically: First use the Cesium.Transforms.eastNorthUpToFixedFrame function to define the inverse view matrix (inverseViewMatrix) of the surveillance camera, and then apply the rotation matrix to determine the inverse view matrix, including: The first rotation (X-axis direction): Apply a matrix that rotates π / 2 radians around the X-axis to the generated inverseViewMatrix. This rotation ensures the correct orientation of the camera's perspective and is usually used Convert the camera's view from the geocentric coordinate system to the local coordinate system with the camera position as the origin; Second rotation (Y-axis direction): Rotate the heading radian around the Y-axis, and the heading is the camera's yaw angle (horizontal rotation angle); Third rotation (X-axis direction): Rotate the pitch radian around the X-axis, and the pitch is the camera's pitch angle (vertical rotation angle); Fourth rotation (Z-axis direction): Rotate the roll radian around the Z-axis, and the roll is the camera's roll angle (rotation angle around the forward axis); This step is to ensure that the video can be correctly mapped to the 3D map, taking into account the influence of the camera's position, direction (heading, pitch, roll) and other parameters;

[0087] After defining the inverse view matrix, create a perspective projection body (frustum) to define the visible range of the surveillance video in three-dimensional space, including the field of view angle (fov), aspect ratio (aspectRatio), and the distance of the near and far clipping planes; add the inverse view matrix, perspective projection body, and video element to the Cesium scene. The video element is HTML5 <video>The video element is used to embed a video in a web page and can play video files or video streams in multiple formats.

[0088] The S5 is further described in detail below:

[0089] Through Cesium's material system, the material type and corresponding uniforms and components are defined in the fabric properties, as follows:

[0090] 1. Pass a base64-encoded black and white image to the alphaImage property of uniforms. This image is used to define the transparent area. Usually, this image will set the area you want to display transparent to black (or other dark colors) and keep the area you want to display white (or other bright colors).

[0091] 2. The diffuse component in components defines the color of the material. Use texture2D(image,fract(materialInput.st)).rgb to get the color information of the video, which is the basic diffuse color of the video.

[0092] 3. The alpha component uses texture2D(alphaImage,fract(materialInput.st)).r to get the red channel in the alpha image as the transparency value. If the pixel in the alphaImage is black (the red channel value is close to 0), then the corresponding pixel will be completely transparent; if it is white (the red channel value is close to 1), then the corresponding pixel will be opaque. Therefore, when the alphaImage is generated, make sure that the red channel value of the black background part should be close to 0 to achieve a completely transparent effect. In this way, the black background part will be completely invisible, while the white or other colored parts will be displayed.

[0093] The map video fusion method based on video cutting and splicing technology described in this embodiment provides new possibilities for map applications through efficient video integration, enhanced interactive experience, wide application scenarios and technological innovation. The introduction of this technology not only improves the visualization effect and user experience of the map, but also creates more opportunities for future map applications and virtual environments.

[0094] Implementation method 2:

[0095] A system for map video fusion based on video cropping and splicing technology, comprising:

[0096] Acquisition module: used to collect front-end monitoring videos;

[0097] Screenshot module: used to take screenshots of the front-end surveillance video, obtain multiple screenshots, and generate a corresponding canvas according to each screenshot;

[0098] Coding module: used for intercepting the area to be spliced ​​on each canvas to obtain multiple areas to be spliced, and performing coding processing on the multiple areas to be spliced ​​to obtain multiple images;

[0099] Coordinate establishment module: used to collect the location information and posture information of the surveillance camera, and use Cesium to establish a three-dimensional earth coordinate system and reverse view matrix;

[0100] Area processing module: used to process multiple areas to be spliced ​​after encoding using Cesium's material system to obtain multiple images;

[0101] Splicing module: used for cutting off the overlapping parts of the multiple images and splicing them into a continuous video surveillance image.

[0102] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0103] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0104] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0105] It will be appreciated by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may be provided in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Furthermore, the present disclosure may be provided in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0108] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above implementation modes, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation modes of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.< / video>

Claims

1. A method for map video fusion based on video cropping and splicing technology, characterized in that: The method comprises: S1: collect front-end surveillance video; S2: Taking screenshots of the front-end surveillance video, obtaining multiple screenshots, and generating a corresponding canvas according to each screenshot; S3: intercepting a region to be stitched on each canvas to obtain a plurality of regions to be stitched, and encoding the plurality of regions to be stitched to obtain a plurality of images; S4: Collect the location and attitude information of the surveillance camera, and use Cesium to establish a three-dimensional earth coordinate system and reverse view matrix; S5: Use Cesium's material system to process multiple areas to be spliced ​​after encoding to obtain multiple images; S6: cutting off the overlapping parts of the multiple images and splicing them to form a continuous video surveillance image, thereby completing the method.

2. The method for applying map video fusion based on video cutting and splicing technology according to claim 1 is characterized in that: In S2, the front-end monitoring video is captured using the Canvas element of HTML5 and the drawImage function of JavaScript.

3. The method for applying map video fusion based on video cutting and splicing technology according to claim 1 is characterized in that: The S3 includes: extracting each of the to-be-joined areas, generating an image on a canvas with a black background as the base and the to-be-joined areas as white; and encoding the image by base64 and storing it in a database.

4. The method for applying map video fusion based on video cutting and splicing technology according to claim 1 is characterized in that: The S4 includes: S4.1: Use Cesium to build a GIS map 3D scene and load map data; S4.2: converting the location information and attitude information of the surveillance camera into a coordinate system on the three-dimensional earth; S4.3: Generate a reverse view matrix based on the position information and Euler angle parameters of the surveillance camera and the rotation matrix; S4.4: Create a perspective projection volume to define the visible range of the surveillance video in three-dimensional space; add the inverse view matrix, projection volume, and video elements to the Cesium scene.

5. The method for applying map video fusion based on video cutting and splicing technology according to claim 4 is characterized in that: The S4.3 includes: S4.3.1: Use the Cesium.Transforms.eastNorthUpToFixedFrame function to define the reverse view matrix of the surveillance camera; S4.3.2: Determine the inverse view matrix using the rotation matrix.

6. The method for applying map video fusion based on video cutting and splicing technology according to claim 5 is characterized in that: The S4.3.2 includes: First rotation: Apply a matrix that rotates π / 2 radians around the X axis to the generated inverse view matrix; Second rotation: rotate the heading arc around the Y axis, where heading is the yaw angle of the surveillance camera; The third rotation: rotate the pitch radian around the X axis, where pitch is the pitch angle of the surveillance camera; The fourth rotation: rotate the roll arc around the Z axis, where roll is the roll angle of the surveillance camera.

7. The method for applying map video fusion based on video cutting and splicing technology according to claim 1 is characterized in that: The S5 includes: S5.1: Pass the multiple images into the alphaImage attribute of uniforms; S5.2: Obtain color information of the multiple images, set the areas to be retained in the multiple images to white, and set the areas to be discarded to black; S5.3: Obtain the red channels of the multiple images as transparency values; adjust the transparency values ​​of white areas to 1, and adjust the transparency values ​​of black areas to 0.

8. A system for map video fusion based on video cropping and splicing technology, characterized in that: The system comprises: Acquisition module: used to collect front-end monitoring videos; Screenshot module: used to take screenshots of the front-end surveillance video, obtain multiple screenshots, and generate a corresponding canvas according to each screenshot; Coding module: used for intercepting the area to be spliced ​​on each canvas to obtain multiple areas to be spliced, and performing coding processing on the multiple areas to be spliced ​​to obtain multiple images; Coordinate establishment module: used to collect the location information and posture information of the surveillance camera, and use Cesium to establish a three-dimensional earth coordinate system and reverse view matrix; Area processing module: used to process multiple areas to be spliced ​​after encoding using Cesium's material system to obtain multiple images; Splicing module: used for cutting off the overlapping parts of the multiple images and splicing them into a continuous video surveillance image.

9. Computer device, characterized in that The method comprises a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a method for applying map video fusion based on video cropping and splicing technology according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program executes a method for applying video cropping and splicing technology to map video fusion according to any one of claims 1 to 7.

Images