A web-based two-dimensional panoramic image stitching and 360° panoramic inspection method and system

Abstract

The present invention discloses a web-based two-dimensional panoramic image stitching and 360° panoramic inspection method and system, comprising the following steps: collecting elevation images and depression images of a power transmission line through a web terminal, and transmitting the collected images to a server terminal; the server terminal stitches the received elevation images to form an elevation panoramic image; the server terminal stitches the received depression images to form a depression panoramic image; the server terminal stitches the elevation panoramic image and the depression panoramic image up and down to obtain a wide-angle panoramic two-dimensional image, and feeds the wide-angle panoramic two-dimensional image back to the web terminal; the web terminal generates a panoramic scene according to the received wide-angle panoramic two-dimensional image; wherein the panoramic scene performs corresponding roaming operations according to the monitored mouse events, and completes a 360° panoramic inspection of the power transmission line. The present invention realizes the purpose of establishing a full-view scene of the power transmission line, remote 3D roaming, and enhancing the immersiveness of the visualization of the power transmission line monitoring.

Description

Technical Field

[0001] The present invention relates to the field of power system transmission line monitoring, and in particular to a web-based two-dimensional panoramic image stitching and 360-degree panoramic inspection method and system. Background Art

[0002] Since the second industrial revolution, mankind has entered the electrical age. Electric energy is of great significance to a country. As the carrier of electric energy, the quality of power transmission lines is closely related to whether the transmission lines can operate safely. Transmission line failures are an important cause of large-scale power outages in the power grid. Transmission lines are widely distributed, have long transmission distances, and are mostly located in remote areas. The safe operation of transmission lines cannot be guaranteed. Therefore, it is necessary to inspect the transmission lines regularly. The inspection method is mostly manual inspection, but due to rugged roads, complex and changeable weather, lack of personnel and other reasons, it is difficult to inspect the transmission lines every day or even every month. The emergence of transmission line monitoring technology provides a new solution. Transmission line monitoring can monitor the status of transmission lines in real time. For example, ice, birds and insects, and trees that are too high are all potential hidden dangers to the stable operation of transmission lines. The emergence of transmission line monitoring technology can eliminate potential hidden dangers to transmission lines in a timely manner, thereby ensuring the reliability of users' electricity use. However, the existing transmission line monitoring technology still has the following problems:

[0003] (1) The current transmission line monitoring technology lacks the panoramic stitching technology of multi-angle monitoring images, which cannot fully understand the overall environment around the transmission line and cannot provide continuous scene pictures of the surrounding environment, resulting in problems such as localized information and incomplete information on the power production site;

[0004] (2) Due to the lack of multi-angle monitoring image panoramic stitching technology, most of the monitoring devices used in current transmission line monitoring technology only have cameras with a single viewing angle and can only monitor at a single angle, resulting in blind spots in monitoring;

[0005] (3) The current power transmission line monitoring technology is limited to one-dimensional monitoring and cannot perform 3D scene roaming, making it impossible to gain a further comprehensive understanding of the surrounding environment, nor can it allow the monitor to have a sense of immersion and participation during on-site inspections. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a web-based two-dimensional panoramic image stitching and 360° panoramic inspection method and system, which are used to solve the technical problems that the existing transmission line monitoring technology cannot perform panoramic stitching of multi-angle monitoring images, can only perform single-angle monitoring, and cannot perform 3D scene roaming, thereby achieving the purpose of establishing all-round perspective scenes of transmission lines, remote 3D roaming, and enhancing the immersiveness of transmission line monitoring visualization.

[0007] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0008] A web-based two-dimensional panoramic image stitching and 360° panoramic inspection method includes the following steps:

[0009] Start the monitoring device through the web end to collect the elevation image and the depression image of the transmission line, and transmit the collected elevation image and depression image to the server end;

[0010] The server forms an elevation panoramic image by naturally stitching the received elevation images based on depth mapping;

[0011] The server forms a low-angle panoramic image by stitching the received low-angle images through pixel-level depth images;

[0012] The server end stitches the elevation angle panoramic image and the depression angle panoramic image up and down to obtain a wide angle panoramic two-dimensional image, and feeds the wide angle panoramic two-dimensional image back to the web end;

[0013] The web end constructs an overall framework to generate a panoramic scene using a 3D engine in the web end according to the received wide-angle panoramic two-dimensional image;

[0014] The panoramic scene performs corresponding roaming operations according to the monitored mouse events to complete a 360° panoramic inspection of the transmission line.

[0015] As a preferred embodiment of the present invention, when the overall framework is constructed by using the 3D engine in the web terminal to generate a panoramic scene, it includes:

[0016] Call the Three.js scene object to create a scene and instantiate the scene;

[0017] Select and create a perspective projection camera in the scene, and place the perspective projection camera above the origin of the scene;

[0018] Create a renderer in the scene, set the size of the renderer, and add the renderer to the scene;

[0019] Creating a sphere with the origin as the center in the scene, and using the wide-angle panoramic two-dimensional image as the texture of the sphere;

[0020] Using a rendering function to render the sphere in real time to obtain a panoramic scene;

[0021] In the panoramic scene, Three.js is used to create scene interaction points at set points and set click listening events. When the interaction point is clicked, the viewing angle will jump to the corresponding adjacent scene.

[0022] Wherein, the 3D engine is Three.js.

[0023] As a preferred embodiment of the present invention, when creating a perspective projection camera, it includes: setting the vertical angle of the viewing volume, the ratio of the horizontal and vertical lengths of the perspective projection camera, and the shortest distance and the farthest distance from the perspective projection camera to the viewing volume.

[0024] As a preferred embodiment of the present invention, after creating a perspective projection camera, the method includes: setting a controller for the perspective projection camera to control the rotation speed and the zooming speed of the panoramic scene.

[0025] As a preferred embodiment of the present invention, after creating the perspective projection camera, the method further includes: setting lights, wherein the lights use parallel light sources, and after setting the positions of the parallel light sources, the parallel light sources are added to the perspective projection camera.

[0026] As a preferred embodiment of the present invention, when creating a renderer in a scene, it also includes: using the WebGLRenderer rendering method, setting the background color and height and width of the renderer, and then adding the renderer to the scene where the perspective projection camera is placed.

[0027] As a preferred embodiment of the present invention, when forming an elevation angle panoramic image, it includes:

[0028] According to the feature matching and depth value of the acquired elevation image, the pixel-to-pixel transformation of the elevation image is estimated to generate a target image;

[0029] triangulate the target image and estimate a plurality of local homographies based on the positions of the vertices and the corrected depth values;

[0030] Elevation panoramic images are synthesized by inverse mapping of the piecewise homography.

[0031] As a preferred embodiment of the present invention, when estimating the pixel-to-pixel transformation of the elevation image, it includes:

[0032] Obtaining an infinite homography between the elevation image and the preset reference image, and an epipolar in the preset reference image, according to the matched feature points and corresponding depth values ​​between the elevation image and the preset reference image;

[0033] A pixel-to-pixel transformation of the elevation image to the preset reference image is established based on the infinite homography and the epipolar.

[0034] As a preferred embodiment of the present invention, when forming a panoramic image at a depression angle, the method includes:

[0035] After repositioning each pixel in the acquired depression-angle image, a pixel-level warping module is used to generate a warped target image;

[0036] After the pixel-level warping module feeds back the reference image and the generated warped target image to the stitched image production module, the stitched image production module generates a depression-angle panoramic image.

[0037] A web-based two-dimensional panoramic image stitching and 360° panoramic inspection system, comprising:

[0038] Two-dimensional image acquisition unit: used to start multiple monitoring devices through the web end to collect elevation images and depression images of the transmission line, and transmit the collected elevation images and depression images to the server end;

[0039] An elevation image stitching unit: used for the server to stitch the received elevation images naturally based on depth mapping to form an elevation panoramic image;

[0040] A depression image stitching unit: used for the server to stitch the received depression images through pixel-level depth images to form a depression panoramic image;

[0041] A panoramic two-dimensional image stitching unit: used for the server to stitch the elevation angle panoramic image and the depression angle panoramic image up and down to obtain a wide-angle panoramic two-dimensional image, and feed the wide-angle panoramic two-dimensional image back to the web end;

[0042] Panoramic scene generation unit: used for the web end to generate a panoramic scene by constructing an overall framework using the 3D engine in the web end according to the received wide-angle panoramic two-dimensional image;

[0043] The panoramic scene performs corresponding roaming operations according to the monitored mouse events to complete a 360° panoramic inspection of the power transmission line.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) The present invention can enrich the on-site information of power transmission line monitoring and provide remote, reliable and comprehensive on-site information for power transmission line inspectors;

[0046] (2) The present invention can provide a panoramic view of the monitoring information of the power transmission line, providing the inspection personnel with a continuous scene reality on site;

[0047] (3) The present invention can provide remote 3D scene roaming, providing an immersive online inspection experience for inspection personnel, so that they can experience the feeling of standing on a tower and inspecting power transmission lines from multiple perspectives without leaving home;

[0048] (4) The method and system provided by the present invention can promote the upgrading and technological improvement of the power monitoring industry, have broad application prospects, and have high practical application value.

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 - is a step diagram of a web-based two-dimensional panoramic image stitching and 360° panoramic inspection method according to an embodiment of the present invention;

[0051] Figure 2 - is a schematic diagram of a stitching process of a wide-angle panoramic two-dimensional image according to an embodiment of the present invention;

[0052] Figure 3 - is a schematic diagram of a stitching process of an elevation angle panoramic image according to an embodiment of the present invention;

[0053] Figure 4 - is a schematic diagram of a web-based two-dimensional panoramic image stitching and 360° panoramic inspection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The web-based two-dimensional panoramic image stitching and 360° panoramic inspection method provided by the present invention is as follows: Figure 1 As shown, the following steps are included:

[0055] Step S1: Start multiple monitoring devices through the web end to collect elevation images and depression images of the transmission line, and transmit the collected elevation images and depression images to the server end;

[0056] Step S2: the server forms an elevation panoramic image by naturally stitching the received elevation images based on depth mapping;

[0057] Step S3: the server forms a low-angle panoramic image by stitching the received low-angle images with pixel-level depth images;

[0058] Step S4: the server end stitches the elevation angle panoramic image and the depression angle panoramic image up and down to obtain a wide angle panoramic two-dimensional image, and feeds the wide angle panoramic two-dimensional image back to the web end;

[0059] Step S5: the web end constructs an overall framework to generate a panoramic scene using a 3D engine in the web end according to the received wide-angle panoramic image;

[0060] Among them, the panoramic scene performs corresponding roaming operations according to the monitored mouse events to complete a 360° panoramic inspection of the transmission line.

[0061] Furthermore, the monitoring device is a camera. The camera is installed as follows:

[0062] It is equipped with 8 wide-angle cameras, 4 of which form a group, two groups up and down, to collect 8 high-definition images. The upper 4 are arranged at an elevation angle to capture environmental images from the sky to the distant view; the lower 4 are arranged at a depression angle to capture images from the distant view to the ground. The 4 elevation images are stitched together to form an elevation panoramic image, and the 4 depression images are stitched together to form a depression panoramic image. After stitching the elevation panoramic image and the depression panoramic image up and down, the wide-angle panoramic 2D image is acquired. Figure 2 shown.

[0063] Furthermore, the camera is installed on the iron tower in the transmission line, with a height of about 50 meters. A camera is installed in each of the four directions of the iron tower. The cameras are fixed in height and installed on a horizontal plane with complementary viewing angles. Figure 4 shown.

[0064] Furthermore, when performing the corresponding roaming operation, it includes: after monitoring the mouse event sent by the user in the web terminal, generating a corresponding panoramic roaming event, and the panoramic scene performs corresponding operations according to the panoramic roaming event.

[0065] When the present invention is actually used, a web browser is opened, a URL is input, a wide-angle panoramic two-dimensional image is opened through a control, and a 3D roaming special effect viewing frame is entered. The panoramic scene is dragged and rotated by a mouse, so as to realize the on-site experience of looking around the iron tower.

[0066] In the above step S1, when collecting elevation images and depression images, it includes: starting the monitoring device through the web end, the monitoring device encodes and transcodes the collected elevation images and depression images into binary data streams, and transmits them to the server through the established socket communication.

[0067] In the above step S5, when the 3D engine in the web terminal is used to construct the overall framework to generate the panoramic scene, it includes:

[0068] Call the Three.js scene object to create a scene and instantiate the scene;

[0069] Select and create a perspective projection camera in the scene, and place the perspective projection camera above the origin of the scene;

[0070] Create a renderer in the scene, set the size of the renderer, and add the renderer to the scene;

[0071] Create a sphere with the origin as the center in the scene, and use the wide-angle panoramic 2D image as the texture of the sphere;

[0072] Use the rendering function to render the sphere in real time to obtain a panoramic scene;

[0073] In the panoramic scene, use Three.js to create scene interaction points at set points and set click listening events. When the interaction point is clicked, the view will jump to the corresponding adjacent scene.

[0074] Among them, the 3D engine is Three.js.

[0075] Furthermore, the rendering function is a requestAnimationFrame() function.

[0076] In the above step S5, when creating the perspective projection camera, it includes: setting the vertical angle of the viewing volume, the ratio of the horizontal and vertical lengths of the perspective projection camera, and the shortest distance and the farthest distance from the perspective projection camera to the viewing volume.

[0077] In the above step S5, after creating the perspective projection camera, the following steps include: setting a controller for the perspective projection camera to control the rotation speed and the zooming speed of the panoramic scene.

[0078] Furthermore, the controller is a TrackballControls controller.

[0079] In the above step S5, after creating the perspective projection camera, the method further includes: setting the light, using a parallel light source, and adding the parallel light source to the perspective projection camera after setting the position of the parallel light source.

[0080] Furthermore, the parallel light source is Three's parallel light source DirectionalLight.

[0081] In the above step S5, when creating a renderer in the scene, it also includes: using the WebGLRenderer rendering method, setting the background color and height and width of the renderer, and then adding the renderer to the scene where the perspective projection camera is placed.

[0082] In the above step S2, when forming the elevation panoramic image, Figure 3 As shown, including:

[0083] According to the feature matching and depth value of the acquired elevation image, the pixel-to-pixel transformation of the elevation image is estimated to generate the target image;

[0084] Triangulate the target image and estimate multiple local homographies based on the positions of the vertices and the corrected depth values;

[0085] Elevation panoramic images are synthesized by inverse mapping of the piecewise homography.

[0086] There are many sky and white cloud images in the elevation high-definition images, and the image features are relatively few. In order to improve the stitching effect and speed, a natural stitching algorithm based on depth mapping is adopted. First, the pixel-to-pixel transformation is estimated according to the feature matching and its depth value; then, the target image is triangulated, and multiple local homographies are estimated according to the positions of the vertices and the corrected depth values; finally, the distorted image is synthesized by reverse mapping of the piecewise homographies. The present invention adopts a natural stitching algorithm based on depth mapping to stitch the elevation images, thereby improving the stitching accuracy and efficiency of the elevation images with a lot of sky and white cloud overlap.

[0087] In the above step S2, when estimating the pixel-to-pixel transformation of the elevation image, it includes:

[0088] According to the matched feature points and corresponding depth values ​​between the elevation image and the preset reference image, an infinite homography between the elevation image and the preset reference image and an epipolar in the preset reference image are obtained;

[0089] A pixel-to-pixel transformation of the elevation image to a preset reference image is established based on infinite homography and epipolars.

[0090] In the above step S3, when forming the depression angle panoramic image, it includes:

[0091] After repositioning each pixel in the acquired depression-angle image, a pixel-level warping module is used to generate a warped target image;

[0092] After the pixel-level warping module feeds the reference image and the generated warped target image back to the stitched image production module, the stitched image production module generates a low-angle panoramic image.

[0093] The high-definition images with low angles mainly include mountains, trees, tall buildings, and the ground. The image background is complex, with many features, and the image has relatively large parallax. A pixel-level deep image stitching algorithm is used to first reposition each pixel in the target image to obtain a distorted target image using a pixel-level warping module (PWM). Then, the distorted target image and the reference image from the pixel-level warping module (PWM) are fed back to the stitched image production module (SIGMo) to generate the final stitched image. The present invention uses a pixel-level deep image stitching algorithm to stitch the low-angle images, thereby improving the stitching accuracy and efficiency of the low-angle images with many features and large parallax.

[0094] The web-based two-dimensional panoramic image stitching and 360° panoramic inspection system provided by the present invention includes:

[0095] Two-dimensional image acquisition unit: used to start multiple monitoring devices through the web end to collect elevation images and depression images of the transmission line, and transmit the collected elevation images and depression images to the server end;

[0096] Elevation image stitching unit: used for the server to stitch the received elevation images into an elevation panoramic image through natural stitching based on depth mapping;

[0097] The depression image stitching unit is used for the server to stitch the received depression images through pixel-level depth images to form a depression panoramic image;

[0098] Panoramic 2D image stitching unit: used for the server to stitch the elevation angle panoramic image and the depression angle panoramic image up and down to obtain a wide angle panoramic 2D image, and feed the wide angle panoramic 2D image back to the web end;

[0099] Panoramic scene generation unit: used by the web end to generate a panoramic scene by building an overall framework using the 3D engine in the web end according to the received wide-angle panoramic two-dimensional image;

[0100] Among them, the panoramic scene performs corresponding roaming operations according to the monitored mouse events to complete a 360° panoramic inspection of the transmission line.

[0101] Compared with the prior art, the present invention has the following beneficial effects:

[0102] (1) The present invention can enrich the on-site information of power transmission line monitoring and provide remote, reliable and comprehensive on-site information for power transmission line inspectors;

[0103] (2) The present invention can provide a panoramic view of the monitoring information of the power transmission line, providing the inspection personnel with a continuous scene reality on site;

[0104] (3) The present invention can provide remote 3D scene roaming, providing an immersive online inspection experience for inspection personnel, so that they can experience the feeling of standing on a tower and inspecting power transmission lines from multiple perspectives without leaving home;

[0105] (4) The method and system provided by the present invention can promote the upgrading and technological improvement of the power monitoring industry, have broad application prospects, and have high practical application value.

[0106] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A web-based two-dimensional panoramic image stitching and 360° panoramic inspection method, characterized in that: The following steps are involved: Start multiple monitoring devices through the web end to collect elevation images and depression images of the transmission line, and transmit the collected elevation images and depression images to the server end; The server forms an elevation panoramic image by naturally stitching the received elevation images based on depth mapping; The server forms a low-angle panoramic image by stitching the received low-angle images through pixel-level depth images; The server end stitches the elevation angle panoramic image and the depression angle panoramic image up and down to obtain a wide angle panoramic two-dimensional image, and feeds the wide angle panoramic two-dimensional image back to the web end; The web end constructs an overall framework to generate a panoramic scene using a 3D engine in the web end according to the received wide-angle panoramic two-dimensional image; The panoramic scene performs corresponding roaming operations according to the monitored mouse events to complete a 360° panoramic inspection of the power transmission line; Wherein, when forming the elevation angle panoramic image, it includes: According to the feature matching and depth value of the acquired elevation image, the pixel-to-pixel transformation of the elevation image is estimated to generate a target image; triangulate the target image and estimate a plurality of local homographies based on the positions of the vertices and the corrected depth values; Synthesize elevation panoramic images through reverse mapping of the piecewise homography; Wherein, when estimating the pixel-to-pixel transformation of the elevation image, it includes: Obtaining an infinite homography between the elevation image and the preset reference image, and an epipolar in the preset reference image, according to the matched feature points and corresponding depth values ​​between the elevation image and the preset reference image; Establishing a pixel-to-pixel transformation of the elevation image to the preset reference image according to the infinite homography and the epipolar; Wherein, when forming a panoramic image at a low angle, it includes: After repositioning each pixel in the acquired depression-angle image, a pixel-level warping module is used to generate a warped target image; After the pixel-level warping module feeds back the reference image and the generated warped target image to the stitched image production module, the stitched image production module generates a depression-angle panoramic image.

2. The web-based two-dimensional panoramic image stitching and 360° panoramic inspection method according to claim 1, characterized in that: When using the 3D engine in the web terminal to build the overall framework to generate a panoramic scene, it includes: Call the Three.js scene object to create a scene and instantiate the scene; Select and create a perspective projection camera in the scene, and place the perspective projection camera above the origin of the scene; Create a renderer in the scene, set the size of the renderer, and add the renderer to the scene; Creating a sphere with the origin as the center in the scene, and using the wide-angle panoramic two-dimensional image as the texture of the sphere; Using a rendering function to render the sphere in real time to obtain a panoramic scene; In the panoramic scene, Three.js is used to create scene interaction points at set points and set click listening events. When the interaction point is clicked, the viewing angle will jump to the corresponding adjacent scene. Wherein, the 3D engine is Three.js.

3. The web-based two-dimensional panoramic image stitching and 360° panoramic inspection method according to claim 2, characterized in that: When creating a perspective projection camera, the steps include setting the vertical angle of the viewing volume, the ratio of the horizontal and vertical lengths of the perspective projection camera, and the shortest distance and the farthest distance from the perspective projection camera to the viewing volume.

4. The web-based two-dimensional panoramic image stitching and 360° panoramic inspection method according to claim 2, characterized in that: After creating the perspective projection camera, the method includes: setting a controller for the perspective projection camera to control the rotation speed and the zooming speed of the panoramic scene.

5. The web-based two-dimensional panoramic image stitching and 360° panoramic inspection method according to claim 2, characterized in that: After creating the perspective projection camera, the method further includes: setting lights, wherein the lights use parallel light sources, and after setting the positions of the parallel light sources, the parallel light sources are added to the perspective projection camera.

6. The web-based two-dimensional panoramic image stitching and 360° panoramic inspection method according to claim 2, characterized in that: When creating a renderer in a scene, the method further includes: using a WebGLRenderer rendering method, setting the background color and height and width of the renderer, and then adding the renderer to the scene where the perspective projection camera is placed.

7. A web-based two-dimensional panoramic image stitching and 360° panoramic inspection system, characterized in that: include: Two-dimensional image acquisition unit: used to start multiple monitoring devices through the web end to collect elevation images and depression images of the transmission line, and transmit the collected elevation images and depression images to the server end; An elevation image stitching unit: used for the server to stitch the received elevation images naturally based on depth mapping to form an elevation panoramic image; A depression image stitching unit: used for the server to stitch the received depression images through pixel-level depth images to form a depression panoramic image; A panoramic two-dimensional image stitching unit: used for the server to stitch the elevation angle panoramic image and the depression angle panoramic image up and down to obtain a wide-angle panoramic two-dimensional image, and feed the wide-angle panoramic two-dimensional image back to the web end; Panoramic scene generation unit: used for the web end to generate a panoramic scene by constructing an overall framework using the 3D engine in the web end according to the received wide-angle panoramic two-dimensional image; The panoramic scene performs corresponding roaming operations according to the monitored mouse events to complete a 360° panoramic inspection of the power transmission line; Wherein, when forming the elevation angle panoramic image, it includes: According to the feature matching and depth value of the acquired elevation image, the pixel-to-pixel transformation of the elevation image is estimated to generate a target image; triangulate the target image and estimate a plurality of local homographies based on the positions of the vertices and the corrected depth values; Synthesize elevation panoramic images through reverse mapping of the piecewise homography; Wherein, when estimating the pixel-to-pixel transformation of the elevation image, it includes: Obtaining an infinite homography between the elevation image and the preset reference image, and an epipolar in the preset reference image, according to the matched feature points and corresponding depth values ​​between the elevation image and the preset reference image; Establishing a pixel-to-pixel transformation of the elevation image to the preset reference image according to the infinite homography and the epipolar; Wherein, when forming a panoramic image at a low angle, it includes: After repositioning each pixel in the acquired depression-angle image, a pixel-level warping module is used to generate a warped target image; After the pixel-level warping module feeds back the reference image and the generated warped target image to the stitched image production module, the stitched image production module generates a depression-angle panoramic image.

Images