A method for establishing VR model of power survey site

By extracting and matching the image feature points of the power survey site, building a panoramic image and controlling the camera perspective with the user's head attitude, the problem of insufficient information supervision capabilities in the power survey site is solved, and a VR model with high integration and real-time performance is achieved. Improves regulatory efficiency and 3D interactive experience.

CN114565743BActive Publication Date: 2024-12-06STATE GRID SHANGHAI MUNICIPAL ELECTRIC POWER CO +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210173254.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-24
Publication Date
2024-12-06
Estimated Expiration
2042-02-24

AI Technical Summary

Technical Problem

The unified supervision capability of power survey site informationization is insufficient, and it is difficult to control the progress of the project's real scene, and there is a lack of real and friendly VR real scene scenarios to improve supervision efficiency.

Method used

By obtaining power survey site pictures, extracting descriptors and feature points, building SIFT objects, performing feature matching and perspective transformation, creating panoramic images, and realizing camera interaction and information display on the VR platform, combining the user's head attitude to control the camera's viewing angle. Implement data integration and real-time monitoring.

Benefits of technology

It has achieved high data integration and real-time VR model of power surveying, and regulators can remotely monitor the on-site status in real time, improving the real-life supervision capabilities of the project and 3D interactive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114565743B_ABST
    Figure CN114565743B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for establishing a VR model of a power survey site, comprising: step 1: obtaining construction pictures of the power survey site; step 2: performing panoramic image stitching; and step 3: constructing a VR model. Compared with the prior art, the present invention has the advantages of high data integration and good real-time performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of power survey site VR model reconstruction, and in particular to a power survey site VR model establishment method. Background Art

[0002] At present, power construction and reconstruction have the characteristics of many distribution points, large line spans, and wide construction areas, which directly lead to the difficulties of unified information supervision and settlement capabilities of the mid-term work involved in power construction companies, and difficulty in controlling the real progress of the project. Therefore, the early planning system of power construction needs to have the function of combining real-life pictures of the project, specified requirements of the specified project, and mid-term data of each settlement stage to provide real and friendly VR real-life scenes for decision makers, designers, and supervisors of each project of the power company, so that supervisors can quickly control the progress of the project, experience an immersive supervision experience remotely, and improve the efficiency of the company's supervision process. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for establishing a VR model of an electric power survey site with high data integration and good real-time performance.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A method for establishing a VR model of an electric power survey site, wherein the VR model establishment method is more preferably preferably, the step 2 is specifically:

[0006] Process the images obtained in step 1 in pairs by performing the following steps:

[0007] Step 2-1: Extract the descriptors and feature points of the image and construct the SIFT object;

[0008] Step 2-2: After obtaining the feature vectors and key points of the two images, use FLANN feature matching to perform homography matching, obtain the perspective transformation matrix and draw matching points;

[0009] Step 2-3: Perform perspective transformation on the second input image based on the transformation matrix H obtained by homography matching to unify the perspectives of the two images;

[0010] Step 2-4: Perform image stitching to obtain a panoramic stitching image.

[0011] More preferably, the step 2-1 is specifically as follows:

[0012] By performing scale space extremum detection to search image positions at all scales, Gaussian differential functions are used to identify potential interest points that are invariant to scale and rotation. At each candidate position, the Gaussian difference image pyramid method is used to determine the position and scale of the key point based on the stability of the interest point. Then, based on the local gradient direction of the image, one or more directions are assigned to each key point position, and the feature vector is calculated in the neighborhood around each key point using the Hessian algorithm.

[0013] Preferably, the step 3 is specifically:

[0014] Step 3-1: Use the panoramic image obtained in step 2 to create a parent scene;

[0015] Step 3-2: Set the camera angle;

[0016] Step 3-3: Create a child interaction object.

[0017] More preferably, the step 3-3 is specifically as follows:

[0018] Create sub-interaction objects, including dragging the mouse to move the VR camera lens and adjusting the image size with the mouse wheel.

[0019] More preferably, the step 3 further comprises:

[0020] Step 3-4: Add a VR scene switching point to the scene and bind the scene switching point to another VR scene.

[0021] More preferably, the step 3 further comprises:

[0022] Step 3-5: Add information interaction points to the scene and bind the information interaction points to the corresponding VR scene information notes.

[0023] Preferably, the VR model building method further comprises:

[0024] Step 4: Calculate the user's head posture and control the camera viewing angle based on the user's head posture.

[0025] More preferably, the method for calculating the user's head posture is:

[0026] Facial feature values ​​are calculated based on the user's face image, and the user's facial feature points are identified. Based on the preset feature points, the coordinates of these feature points in the world coordinate system are compared with their coordinates in the image coordinate system. The head rotation matrix and rotation vector can be calculated through the N-point perspective pose solution method, and the Euler angle is calculated based on the equivalent transformation of the rotation vector, rotation matrix, quaternion and Euler angle.

[0027] More preferably, the step 4 further comprises:

[0028] After obtaining the user's head posture, the calculated Euler angle is compared with the preset threshold. If the calculated value is greater than the preset threshold, the camera viewing angle deflection is triggered.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. High data integration: The method for establishing a VR model of a power survey site in the present invention can complete VR modeling of the survey site based on the images collected from the power survey site, and integrate various information of the power survey site into the VR model. The supervisory user can display detailed information of the point through the motor information interaction point, which effectively solves the problems of insufficient real-scene supervision capabilities of the project during the power construction process and low integration of various aspects of data in the settlement stage.

[0031] 2. Good real-time performance: The method for establishing a VR model of the power survey site in the present invention performs VR modeling of the power survey site through real-time images collected on-site. Supervisors can remotely monitor the status of the power survey site in real time. It has good real-time performance, strong practicality, and a strong sense of interactivity in the 3D real-scene experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the process of establishing a VR model of a power survey site in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] A method for establishing a VR model of an electric power survey site, comprising:

[0035] Step 1: Obtain construction pictures of the power survey site;

[0036] Step 2: Stitch the panorama, specifically:

[0037] Process the images obtained in step 1 in pairs by performing the following steps:

[0038] Step 2-1: Extract the descriptors and feature points of the image and construct the SIFT object;

[0039] By performing scale space extremum detection to search for image positions at all scales, Gaussian differential functions are used to identify potential interest points that are invariant to scale and rotation. At each candidate position, the Gaussian difference image pyramid method is used to determine the position and scale of the key point based on the stability of the interest point. Then, based on the local gradient direction of the image, one or more directions are assigned to each key point position, and the Hessian algorithm is used to calculate the feature vector in the neighborhood around each key point.

[0040] Step 2-2: After obtaining the feature vectors and key points of the two images, use FLANN feature matching to perform homography matching, obtain the perspective transformation matrix and draw matching points;

[0041] Step 2-3: Perform perspective transformation on the second input image based on the transformation matrix H obtained by homography matching to unify the perspectives of the two images;

[0042] Step 2-4: perform image stitching to obtain a panoramic stitching image;

[0043] Step 3: Build VR model;

[0044] Step 3-1: Use the panoramic image obtained in step 2 to create a parent scene;

[0045] Step 3-2: Set the camera angle;

[0046] Step 3-3: Create sub-interaction objects, including dragging the mouse to move the VR camera lens and adjusting the image size with the mouse wheel;

[0047] Step 3-4: Add a VR scene switching point in the scene and bind the scene switching point to another VR scene;

[0048] Step 3-5: Add information interaction points in the scene and bind the information interaction points to the corresponding VR scene information notes;

[0049] Step 4: Calculate the user's head posture and control the camera viewing angle based on the user's head posture.

[0050] The calculation method of the user's head posture is:

[0051] Facial feature values ​​are calculated based on the user's face image, and the user's facial feature points are identified. Based on the preset feature points, the coordinates of these feature points in the world coordinate system are compared with their coordinates in the image coordinate system. The head rotation matrix and rotation vector can be calculated through the N-point perspective pose solution method, and the Euler angle is calculated based on the equivalent transformation of the rotation vector, rotation matrix, quaternion and Euler angle.

[0052] Step 4 also includes:

[0053] After obtaining the user's head posture, the calculated Euler angle is compared with the preset threshold. If the calculated value is greater than the preset threshold, the camera viewing angle deflection is triggered.

[0054] The VR platform in this embodiment uses the webVR platform. The WebVR specification is an open specification that can experience VR real scenes in a browser, and is designed to provide an easy and friendly VR experience for multiple terminals and multiple devices. WebGL is a Web-oriented graphics standard designed based on OpenGL, the most commonly used cross-platform graphics library. It provides a series of JavaScript APIs that can realize the rendering and processing of complex images on the Web side, and can efficiently perform 3D graphics, lighting models and rendering. At the same time, it can also realize the rendering of 2D images without any browser plug-ins. In terms of performance, it relies on hardware accelerated rendering, and its performance is also stronger than other Web graphics technologies.

[0055] In the aspect of image and video processing, the present invention relates to image registration technology. Commonly used image feature extraction and processing algorithms based on coarse registration include two-dimensional scale-invariant feature transform (SIFT) algorithm and accelerated robust feature algorithm (UFT). SIFT is a computer vision algorithm used to detect and describe local features in images. It finds extreme points in the spatial scale and extracts its position, scale and rotation invariant. Because this algorithm is based on the local appearance interest points of objects rather than the impact size and rotation angle, and has good tolerance for image noise, ambient light and viewing angle deviation, it can often help identify objects and extract image feature values. SIFT features have a large amount of information and are suitable for fast and accurate matching in massive databases. SUFT is similar to SIFT in thought. The main difference is that the Hessian matrix is ​​used to determine candidate points, and then the feature points are preliminarily determined by the method of non-maximum suppression, and then the image feature points are comprehensively constructed by determining the main direction of the candidate feature points. In order to keep the feature points invariant in rotation and scale, SURF gives the feature points a descriptor so that they can remain invariant and can be significantly distinguished from each other. A very important function of SURF in image recognition is that it can pair feature points through descriptors, which can help effectively identify the common areas of two images and pair them in panoramic image synthesis.

[0056] The following is a specific application example, the process is as follows Figure 1 As shown, including:

[0057] Step S1: Input four or fewer pictures of the construction site. If the user only inputs one image, it is assumed to be a panoramic image.

[0058] Step S2: Panorama stitching.

[0059] The images are processed in pairs. For two input images, the following operations are performed:

[0060] S2-1: First, extract the descriptors and feature points of the image and construct the SIFT object. The specific method is to perform scale space extreme value detection, that is, search for image positions at all scales, use Gaussian differential functions to identify potential interest points that are invariant to scale and rotation, and at each candidate position, use the Gaussian difference image pyramid method to determine the position and scale of the key points based on their stability. Then, based on the local gradient direction of the image, assign one or more directions to each key point position. All subsequent operations on the image data are transformed relative to the direction, scale, and position of the key points, thereby providing invariance to these transformations. In the neighborhood around each key point, the feature vector is calculated using the Hessian algorithm.

[0061] S2-2: After obtaining the feature vectors and key points of the two images, use FLANN fast matching to perform fast homography matching, obtain the perspective transformation matrix and draw the matching points.

[0062] S2-3: Perform perspective transformation on the second input image based on the transformation matrix H obtained by homography matching to unify the perspectives of the two images.

[0063] S2-4: The last step is image stitching. The first image is added to the panoramic frame obtained in the previous step. Weighted smoothing is performed on the overlapping area to prevent obvious stitching gaps, and a panoramic stitching image after weighted smoothing is obtained.

[0064] Step S3: Basic display and interaction of VR scenes, to achieve true high-definition engineering real-scene VR display and friendly and convenient interactive effects.

[0065] S2-1: Create a parent scene using the panoramic image obtained in step S1.

[0066] S2-2: Set the camera angle (supports head gesture control of the lens).

[0067] S2-3: Create sub-interaction objects, including basic mouse dragging to move the VR camera lens and mouse wheel adjustment to complete basic scene display and interaction.

[0068] S2-4: Add a VR scene switching point in the scene and bind it to another VR scene. The supervisor user can click the interaction point to complete the interface switching.

[0069] S2-5: Add information interaction points in the scene and bind the corresponding VR scene information notes. The supervisor user clicks the interaction point to display the detailed information of the point.

[0070] Step S3: head posture estimation, calling the camera to calculate the head Euler angle.

[0071] Head posture estimation mainly involves obtaining several key orientation data of the face.

[0072] Generally, four equivalent values ​​are used: rotation vector, rotation matrix, quaternion and Euler angle. The present invention uses the RPY Euler angle format to measure the head posture, where Roll represents the rotation angle, that is, the angle γ of the object rotating around the X axis; Pitch represents the pitch angle β, that is, the angle of the object rotating along the Y axis; Yaw represents the yaw angle, that is, the angle α of the object rotating along the Z axis.

[0073] The rotation vector can be calculated based on the control point coordinates in the world coordinate system, the corresponding control point coordinates in the image coordinate system, the camera intrinsic parameter matrix, the camera distortion coefficient and other parameters, and the corresponding quaternion can be calculated according to the definition. The present invention uses the following method to convert the quaternion into Euler angles with stronger semantics and engineering usability.

[0074] Define two quaternions as follows:

[0075]

[0076]

[0077] in, Represents vector<b,c,d> , Represents vector<x,y,z> .

[0078] Quaternions are like complex numbers, vectors, and matrices; the sum of two quaternions requires adding up the different elements.

[0079]

[0080] Addition obeys all the commutative and associative properties of real and complex numbers.

[0081] The meaning of quaternion multiplication is similar to matrix multiplication, which can represent the synthesis of rotations. When there are multiple rotation operations, using quaternions can achieve higher computational efficiency.

[0082]

[0083] pq=(at-bx-cy-dz)+(ax+bt+cz+dy)i+(ay-bz+ct+dx)j+(az+dt-cx+by)k

[0084] Due to the non-commutativity of quaternion multiplication, pq is not equal to qp. The vector part of the qp product is:

[0085] Then convert the ZYX Euler angle (or RPY angle: rotate the XYZ of the fixed coordinate system by angles a, β, γ in sequence) to a quaternion:

[0086]

[0087] According to the above formula, the inverse solution can be obtained, that is, the conversion from quaternion q = (q0, q1, q2, q3) or q = (w, x, y, z) to Euler angle is:

[0088]

[0089] Since the value range of arctan and arcsin is between -π / 2 and π / 2, which is only 180°, and the range is 360° when rotating around a certain axis, the atan2 function should be used instead of the arctan function:

[0090]

[0091] For tanθ=y / x:

[0092] θ=ATan(y / x) The value range of θ is [-π / 2,π / 2];

[0093] θ=ATan(y,x) The value range of θ is [-π / 2,π / 2];

[0094] When (x, y) is in the first quadrant, 0<θ<π / 2;

[0095] When (x, y) is in the second quadrant, π / 2<θ≤π;

[0096] When (x, y) is in the third quadrant, -π<θ≤-π / 2;

[0097] When (x, y) is in the fourth quadrant, -π / 2<θ<0.

[0098] There is a problem with the above situation, that is, the singularity is not considered. Consider the following special case: Assume that an aircraft rotates 90° around the Y axis (pitch angle pitch = 90), and the nose is vertically upward.

[0099] At this time, the degree of freedom will be lost, that is, Yaw and Roll will become one degree of freedom. At this time, pay attention to the classification and special calculation.

[0100] Head posture estimation is to calculate the 468 facial feature points through the model, and compare the coordinates of the five points of nose tip, two outer corners of eyes, and two corners of mouth in the world coordinate system with the coordinates of these five points in the image coordinate system. The head rotation matrix and rotation vector can be calculated through the N-point perspective posture solution method, and then the Euler angle is calculated based on the equivalent transformation of the four values ​​of rotation vector, rotation matrix, quaternion and Euler angle. At the same time, the corresponding facial topology map is drawn according to the 468 feature points, and data visualization and graphical demonstration are given according to the calculated key values ​​of head posture estimation. Two methods of visual modeling of facial recognition feature values ​​are given, one is to draw a two-dimensional facial topology map based on the feature value, and the other is point cloud 3D face modeling.

[0101] The specific method is:

[0102] The server side calculates facial feature values ​​based on the face image or video stream or real-time camera recording images;

[0103] The server side calculates key attributes based on facial feature values;

[0104] The front-end interface visualizes the facial recognition feature values ​​and displays a facial orientation indicator frame;

[0105] The key results of head posture estimation are visualized on the front-end interface, and the key results of head posture estimation are displayed in real time, namely the Euler angle of head deflection and whether the eyes are closed.

[0106] 3D point cloud modeling of faces in the front-end interface as another way to visualize the feature values ​​of face recognition;

[0107] When the front-end performance is low or the browser is not compatible with WebGL, start the middle-end service to realize head posture estimation (simulation);

[0108] Verification of the result returned by the front-end head posture estimation;

[0109] Change the VR view angle according to the facial recognition results, including triggering the VR view lens deflection when the face recognition Euler angle exceeds a certain threshold, and desensitizing to closed eyes and undetectable head deflection states to prevent excessive interaction.

[0110] The VR model building method in this embodiment provides a convenient and friendly VR view and interactive data integration model, supports high-precision reproduction of the construction site and provides a novel and friendly remote interaction function, solving the pain points of difficult real-scene supervision and insufficient 3D interactive experience during power construction.

[0111] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for establishing a VR model for power survey site, It is characterized in that The VR model establishment method comprises: Step 1: Obtain construction pictures of the power survey site; Step 2: Stitch the panorama; Step 3: Build VR model; The step 2 is specifically as follows: Process the images obtained in step 1 in pairs by performing the following steps: Step 2-1: Extract the descriptors and feature points of the image and construct the SIFT object; Step 2-2: After obtaining the feature vectors and key points of the two images, use FLANN feature matching to perform homography matching, obtain the perspective transformation matrix and draw matching points; Step 2-3: Perform perspective transformation on the second input image based on the transformation matrix H obtained by homography matching to unify the perspectives of the two images; Step 2-4: perform image stitching to obtain a panoramic stitching image; The step 3 is specifically as follows: Step 3-1: Use the panoramic image obtained in step 2 to create a parent scene; Step 3-2: Set the camera angle; Step 3-3: Create a child interaction object; The step 3-3 is specifically as follows: Create sub-interaction objects, including dragging the mouse to move the VR camera lens and adjusting the frame size with the mouse wheel; The step 3 also includes: Step 3-4: Add a VR scene switching point in the scene and bind the scene switching point to another VR scene; The step 3 also includes: Step 3-5: Add information interaction points to the scene and bind the information interaction points to the corresponding VR scene information notes.

2. A method for establishing a VR model for electric power survey site according to claim 1, It is characterized in that The step 2-1 is specifically as follows: By performing scale space extremum detection to search image positions at all scales, Gaussian differential functions are used to identify potential interest points that are invariant to scale and rotation. At each candidate position, the Gaussian difference image pyramid method is used to determine the position and scale of the key point based on the stability of the interest point. Then, based on the local gradient direction of the image, one or more directions are assigned to each key point position, and the feature vector is calculated in the neighborhood around each key point using the Hessian algorithm.

3. A method for establishing a VR model for electric power survey site according to claim 1, It is characterized in that The VR model building method further includes: Step 4: Calculate the user's head posture and control the camera viewing angle based on the user's head posture.

4. A method for establishing a VR model for electric power survey site according to claim 3, It is characterized in that The calculation method of the user's head posture is: Facial feature values ​​are calculated based on the user's face image, and the user's facial feature points are identified. Based on the preset feature points, the coordinates of these feature points in the world coordinate system are compared with their coordinates in the image coordinate system. The head rotation matrix and rotation vector can be calculated through the N-point perspective pose solution method, and the Euler angle is calculated based on the equivalent transformation of the rotation vector, rotation matrix, quaternion and Euler angle.

5. A method for establishing a VR model for electric power survey site according to claim 3, It is characterized in that The step 4 further comprises: After obtaining the user's head posture, the calculated Euler angle is compared with the preset threshold. If the calculated value is greater than the preset threshold, the camera viewing angle deflection is triggered.

Citation Information

Patent Citations

  • Power equipment three-dimensional modeling method

    CN107492147A

  • Three-dimensional scene reconstruction method based on picture

    CN108921939A