Spatial perception positioning method and system for multi-interval complex environment

Abstract

The invention belongs to the technical field of spatial perception positioning, and discloses a spatial perception positioning method and system for a multi-interval complex environment, and the method comprises the steps: obtaining a multi-interval environment video stream from a transformer substation monitoring camera, and synchronously collecting an environment scanning point cloud model; establishing a space coordinate transformation relation between the 2D image and the 3D environment model; acquiring a local area updating point cloud, splicing the local point cloud and the initial point cloud under the condition of no initial pose, identifying an overlapping area, and realizing dynamic map updating of equipment state change; projecting two-dimensional image data of a visible light / infrared / ultraviolet camera video stream to a digital twin three-dimensional space through perspective transformation based on a space coordinate transformation relation to obtain a three-dimensional perception result of multispectral information; and based on a three-dimensional perception result, after multi-camera target tracks are fused, space coordinates are optimized and output. Rapid updating of the local three-dimensional point cloud data and high-precision registration updating of the monitoring video stream and the digital twin space are realized, and the virtual-real consistency is ensured.

Description

Technical Field

[0001] This invention belongs to the field of spatial sensing and positioning technology, and specifically relates to a spatial sensing and positioning method and system for complex environments with multiple intervals. Background Technology

[0002] Safety accidents in power production scenarios have a significant impact, causing not only personal injury and death but also potentially triggering widespread power outages. Current production safety management relies primarily on manual methods, with automation and artificial intelligence technologies playing a limited role. To achieve comprehensive production safety management of substations, precise spatial perception and identification of the location, status, and changes of personnel and equipment are urgently needed. Positioning technology based on spatial perception units has already been applied in industries such as surveying and entertainment: Tesla vehicles utilize multi-source information, including laser, vision, and microwave, to achieve real-time environmental perception and spatial positioning-assisted autonomous driving; the Eagle Eye system uses multiple high-speed cameras to help referees make accurate and fair judgments. Various spatial positioning and perception solutions vary in real-time performance, versatility, accuracy, complexity, and scalability. Currently, in substation safety operation scenarios, ordinary positioning solutions face difficulties in spatial perception and positioning in complex, multi-interval environments. Summary of the Invention

[0003] The purpose of this invention is to provide a spatial sensing and positioning method and system for complex environments with multiple intervals, so as to solve the problem of difficult positioning in spatial sensing and positioning in complex environments with multiple intervals.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a spatial sensing and positioning method for complex environments with multiple intervals, comprising:

[0006] Multi-interval environmental video streams are acquired from substation monitoring cameras, and environmental scanning point cloud models are collected simultaneously. Two-dimensional and three-dimensional matching points are selected from the images and point clouds by manual point selection, the camera extrinsic parameters are solved, and the spatial coordinate transformation relationship between the 2D images and the 3D environmental models is established.

[0007] The local area updated point cloud is acquired, and the local point cloud is stitched with the initial point cloud without initial pose. The overlapping area is identified to realize dynamic map updates for changes in device status.

[0008] Based on the spatial coordinate transformation relationship, the two-dimensional image data of the visible light / infrared / ultraviolet camera video stream is projected onto the three-dimensional space of the digital twin through perspective transformation to obtain the three-dimensional perception result of multispectral information;

[0009] Based on the results of 3D perception, human targets are detected, key human body nodes are identified, and the output spatial coordinates are optimized after fusing target trajectories from multiple cameras.

[0010] Furthermore, the process of acquiring multi-interval environmental video streams from substation monitoring cameras and simultaneously collecting environmental scanning point cloud models; selecting two-dimensional and three-dimensional matching points in the images and point clouds through manual point selection, solving for camera extrinsic parameters, and establishing the spatial coordinate transformation relationship between the 2D images and the 3D environmental model includes:

[0011] The system acquires video footage of complex environmental conditions across multiple intervals from monitoring cameras in a substation. Based on the images captured by the monitoring cameras and the environmental scanning point cloud model, 3D and 2D points are manually selected and used to calculate camera extrinsic parameters. The EPNP algorithm is used to calibrate the 2D images and 3D point cloud model, and the accuracy of the reprojection error is evaluated. The calibration results are output to a YAML file, and the error between the reprojected points and the original points is plotted. Four control points are selected to simplify the mapping relationship between 3D matching points and 2D image points. The positions of these control points in the camera coordinate system are found by solving matrix equations. The distance difference between control points in the two coordinate systems is minimized using the Gauss-Newton method to determine the precise positions of these control points. Based on the obtained control point positions, the camera rotation matrix R and translation matrix t are obtained using SVD decomposition or nonlinear optimization methods to complete camera pose estimation and establish the spatial coordinate transformation relationship between the 2D images and the 3D environmental model.

[0012] Furthermore, four control points are selected to simplify the mapping relationship from 3D matching points to 2D image points. The specific steps are as follows:

[0013] Select control points

[0014] Let the coordinates of the 3D matching point be p. i (i = 1, 2, ..., n), c j (j=1,2,3,4) represents the coordinates of four virtual control points. c1 is taken as the centroid of the 3D matching point set, and the other control points are taken as the three principal components after the principal component decomposition of the 3D matching point set. The world coordinates of the 3D matching points are expressed as a weighted sum of the control points:

[0015] and

[0016] Calculate the coordinates of the control points in the camera coordinate system.

[0017] In the camera coordinate system, the 3D matching points and control points also satisfy the relationship described above:

[0018] and

[0019] Control points:

[0020]

[0021] Where (u i ,v i )for The coordinates in the camera pixel coordinate system, simplified to the above formula, yield:

[0022]

[0023] Let the camera coordinate system spatial coordinates of the four control points be... Solving the system of equations for the four control points (c1, c2, c3, c4), we obtain a homogeneous matrix equation of the following form:

[0024] MX=0

[0025] Then the kernel space of matrix M is the solution to the unknown X, and the solution to the unknown X can be written as:

[0026]

[0027] Where v i For M T The eigenvectors of M when the eigenvalue is 0, and N is the eigenvector of M. T The dimension of the M-kernel space; β i The distance between the two control points is unknown; according to the distance-preserving property of Euclidean transformation, the distance between the two control points is the same in both the camera coordinate system and the world coordinate system, i.e. Solving for β by minimizing the distance difference between control points in two coordinate systems using the Gauss-Newton method. i :

[0028]

[0029] Calculate R,t

[0030] β i Substituting into equation (7) yields the coordinates of the control point in the camera coordinate system. The problem of solving the 3D to 2D PnP problem is transformed into solving the classic 3D to 3D rigid body motion problem, and [Rt] is obtained by using SVD decomposition or nonlinear optimization methods.

[0031] Furthermore, the step of acquiring a local area updated point cloud, stitching the local point cloud with the initial point cloud without an initial pose, identifying overlapping areas, and realizing dynamic map updates in response to changes in device state includes:

[0032] A local area update point cloud is acquired using a handheld Livox mobile scanning device. An ICP registration method based on Anderson acceleration and Welsch equation is adopted to stitch the local point cloud with the initial point cloud without initial pose. A kdTree is constructed to perform neighborhood search to identify overlapping areas. Combined with position-weighted fusion technology, redundant data is reduced and memory usage is optimized to achieve dynamic map updates for device status changes.

[0033] Furthermore, based on spatial coordinate transformation, the two-dimensional image data of the visible light / infrared / ultraviolet camera video stream is projected onto the digital twin three-dimensional space through perspective transformation to obtain a three-dimensional perception result with multispectral information, including:

[0034] Perspective transformation is used to project point data from a two-dimensional image to a three-dimensional space. The coordinate transformation is calculated by using a pinhole imaging model combined with the camera's intrinsic and extrinsic parameters. Surface data projection is completed by aligning the selected three-dimensional rectangular area and performing image perspective transformation to achieve pixel-level projection. For infrared cameras, intrinsic parameter calibration with a set resolution is performed, and extrinsic parameters are solved using the EPnP algorithm for accurate registration.

[0035] Furthermore, specifically including:

[0036] Perspective transformation is used to project point data from a two-dimensional image to a three-dimensional space. The general transformation formula is as follows:

[0037]

[0038] Where the perspective transformation matrix is The original point coordinates, The coordinates of the transformed point are given; the image is in a two-dimensional plane, divided by Z to obtain:

[0039]

[0040] Let a 33 =1, expanding the above equation, we get the equation of a point:

[0041]

[0042] Four points yield four equations, resulting in eight equations. Solve for the remaining eight unknowns in A:

[0043]

[0044] Surface data projection achieves pixel-level projection by aligning a selected 3D rectangular region and performing image perspective transformation. The rectangular region to be mapped is determined by three selected points; the coordinates of the other two vertices p3 and p4 of the selected rectangle are calculated; first, the two side length parameters of the selected rectangle, and the width W, are determined. rect High Hrect W rect =|p0p1|, Among them W rect H is the length of line segment p0p1. rect Let p2 be the distance from point p2 to line p0p1; let vector p2 be the distance from point p2 to line p0p1. The normal vector of the selected rectangle can be obtained using the right-hand rule. Where norm represents normalization; the direction vector of the selected rectangle's width is... Final result Simultaneously obtain the center of the selected rectangle: center = (p0 + p4) / 2;

[0045] To rotate and translate the initial rectangle, first, rotate the initial rectangle to the same plane as the selected rectangle by rotating it about an axis. Rotation angle is Let mat be the rotation matrix corresponding to this rotation; after rotating about the axis aixs, the initial rectangle and the selected rectangle have the same normal vector. Secondly, around Rotate by angle θ to make the poses completely aligned; after rotating around axis aixs, the vertex coordinates corresponding to p′0 are p″0=mat*p′0, the vector from the center of the selected rectangle center to p0 is v0=norm(p0-center), and the vector from the initial rectangle center′ to p″0 after rotating around axis aixs is v1=norm(p″0-center′).

[0046]

[0047] After solving the equations, we get θ = atan2(sinθ,cosθ);

[0048] By combining the camera's intrinsic and extrinsic parameters, the transformation from world coordinates to pixel coordinates is obtained:

[0049]

[0050] For any point P in three-dimensional space, its image point p on camera C is obtained through observation by camera C;

[0051] In a flat terrain positioning scenario, the ground plane is set as the z-axis. w Plane, set z w =H, representing the planar space at a height H above the horizontal plane, called elevation; if the pixel coordinate system (u,v) and the camera's intrinsic and extrinsic parameter values ​​are known, then z w Substituting H into the formula, we get z. c =r 31 x w +r 32 yw +r 33 H+t3, let:

[0052] A = r 31 uf x r 11 -u0r 31

[0053] B = r 32 uf x r 12 -u0r 32

[0054] C = f x r 31 +u0r 33 -ur 33

[0055] D = f x t1+(u0-u)t3

[0056] E = r 31 vf y r 21 -v0r 31

[0057] F = r 32 uf y r 22 -v0r 32

[0058] G = f y r 23 +v0r 33 -vr 33

[0059] I = f y t2+(v0-v)t3

[0060] Then we have:

[0061]

[0062] Solving the system of two linear equations in two variables, we get:

[0063]

[0064] For the infrared camera, intrinsic parameter calibration of the resolution was performed using Matlab. Let the intrinsic parameter matrix of the infrared camera be K:

[0065]

[0066] Simultaneously, the distortion parameter RadialDistortion was obtained as [-0.4218-1.2254 0 0 0];

[0067] Six pairs of matching points were selected for camera-to-scan point cloud registration. The extrinsic parameters of the camera facing the air conditioner were obtained using the EPnP algorithm.

[0068]

[0069] External parameters of the camera viewed from the chassis perspective:

[0070]

[0071] Device-level temperature coloring was implemented based on infrared images and point cloud data to display the temperature distribution of devices such as air conditioners and chassis from multiple perspectives.

[0072] Furthermore, the process of detecting human targets, identifying key human body nodes, and optimizing the output spatial coordinates after fusing target trajectories from multiple cameras based on 3D perception results includes:

[0073] The intrinsic, extrinsic, and fundamental matrices of the cameras are obtained through the calibration technology of the binocular vision system. Stereo alignment of the two cameras' image planes is corrected. Based on the epipolar geometry principle, the eigenvalue and fundamental matrices are calculated. In the stereo matching stage, a block-based matching algorithm is used to identify corresponding points in the left and right images, generate a disparity map, and further convert it into a depth map. For target tracking in multi-camera space, the target detection algorithm is first used to identify objects in a single camera. Then, data from multiple cameras are fused using methods such as Kalman filtering to track the target's motion trajectory in real time, achieving accurate target tracking in complex scenes.

[0074] By employing the YOLOX model for efficient and accurate human detection and utilizing OpenPose for key node identification in human pose estimation, this study integrates image super-resolution reconstruction technology with low-light image enhancement methods. GAN-based and Transformer-based models are used to improve image quality, and the Zero-DCE algorithm is employed to enhance image visibility under low-light conditions. For occlusion scenarios, a human localization algorithm for non-calibrated scenes is developed. Candidate foot points are weighted based on cluster analysis to optimize the accuracy of human position determination, ensuring precise localization and tracking of human targets in various environments.

[0075] Secondly, the present invention provides a spatial sensing and positioning system for complex environments with multiple intervals, comprising:

[0076] The data acquisition module is used to acquire multi-interval environmental video streams from substation monitoring cameras and simultaneously collect environmental scanning point cloud models; it selects two-dimensional and three-dimensional matching points in the image and point cloud by manual point selection, solves the camera extrinsic parameters, and establishes the spatial coordinate transformation relationship between the 2D image and the 3D environmental model.

[0077] The update module is used to acquire the updated point cloud of a local area, stitch the local point cloud with the initial point cloud without an initial pose, identify overlapping areas, and realize dynamic map updates for changes in device status.

[0078] The projection module is used to project two-dimensional image data from visible light / infrared / ultraviolet camera video streams onto the digital twin three-dimensional space based on spatial coordinate transformation relationships and through perspective transformation, so as to obtain three-dimensional perception results with multispectral information;

[0079] The perception output module is used to detect human targets based on the 3D perception results, identify key human body nodes, and optimize the output spatial coordinates after fusing target trajectories from multiple cameras.

[0080] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the spatial perception and positioning method for complex multi-segment environments.

[0081] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the spatial perception and positioning method for complex multi-interval environments.

[0082] Compared with the prior art, the present invention has the following technical effects:

[0083] This invention studies a dynamic calibration and spatial transformation analysis method for video cameras in plant environments, and a gridded mapping algorithm for visible light / infrared / ultraviolet camera video data in a digital twin 3D space. It proposes a high-precision gridded mapping algorithm for digital twin 3D space, achieving gridded mapping of video data in the digital twin space through a set of mapping methods for point data, surface data, and volume data. Furthermore, it studies a high-precision registration and compensation optimization algorithm between plant monitoring video streams and the digital twin 3D space, proposing an algorithm to achieve rapid updates of local 3D point cloud data and high-precision registration updates between the monitoring video stream and the digital twin space, ensuring consistency between the virtual and real worlds.

[0084] This invention studies perception and tracking algorithms based on multi-camera space and spatial perception and positioning algorithms for complex environments with multiple intervals. It proposes a personnel perception and positioning tracking algorithm for complex environments with multiple cameras and multiple intervals, realizing a seamless personnel positioning and tracking algorithm applicable to real-world production scenarios. The project develops a basic algorithm package for 2D / 3D spatial fusion positioning and perception, integrating high-precision 2D / 3D spatial conversion calculation of video pixels, error control calculation during continuous positioning, personnel key node and target recognition, high-precision mapping of digital twin 3D spatial meshing, high-precision registration and compensation optimization of video stream and digital twin space, super-resolution, and image illumination enhancement for personnel perception and positioning tracking in complex environments with multiple cameras and multiple intervals. Attached Figure Description

[0085] Figure 1 It is a three-dimensional mapped rectangle.

[0086] Figure 2 This is a model for optical axis convergence.

[0087] Figure 3 This is a diagram of the YOLOX network structure.

[0088] Figure 4 This is a diagram of the CSPDarknet53 architecture.

[0089] Figure 5 For ResNet and CSPResNet.

[0090] Figure 6 This is the PAF workflow.

[0091] Figure 7 This is an example of the OpenPose workflow.

[0092] Figure 8 The effect of the parameter on the brightness enhancement curve.

[0093] Figure 9 The framework of the basic algorithm package for substation two / three-dimensional spatial fusion positioning and perception and its laboratory environment verification.

[0094] Figure 10 This is a flowchart of the present invention. Detailed Implementation

[0095] The present invention will be further described below with reference to the accompanying drawings:

[0096] Example 1, please refer to Figure 10 This invention provides a spatial perception and positioning method for complex environments with multiple intervals, comprising:

[0097] Multi-interval environmental video streams are acquired from substation monitoring cameras, and environmental scanning point cloud models are collected simultaneously. Two-dimensional and three-dimensional matching points are selected from the images and point clouds by manual point selection, the camera extrinsic parameters are solved, and the spatial coordinate transformation relationship between the 2D images and the 3D environmental models is established.

[0098] The local area updated point cloud is acquired, and the local point cloud is stitched with the initial point cloud without initial pose. The overlapping area is identified to realize dynamic map updates for changes in device status.

[0099] Based on the spatial coordinate transformation relationship, the two-dimensional image data of the visible light / infrared / ultraviolet camera video stream is projected onto the three-dimensional space of the digital twin through perspective transformation to obtain the three-dimensional perception result of multispectral information;

[0100] Based on the results of 3D perception, human targets are detected, key human body nodes are identified, and the output spatial coordinates are optimized after fusing target trajectories from multiple cameras.

[0101] Example 2: This invention provides a spatial perception and positioning method for complex environments with multiple intervals, comprising:

[0102] (1) Dynamic calibration and spatial transformation analysis of video cameras for plant environments

[0103] The EPNP algorithm is used to calibrate between 2D images and 3D point cloud models. 3D and 2D matching points are obtained through manual selection and used to calculate camera extrinsic parameters. Four control points are selected to simplify the mapping from 3D matching points to 2D image points. The positions of these control points in the camera coordinate system are found by solving matrix equations. The Gauss-Newton method is used to minimize the distance difference between control points in the two coordinate systems, thus determining the precise positions of these control points. Based on the obtained control point positions, the camera's rotation matrix R and translation matrix t are obtained using SVD decomposition or nonlinear optimization methods to complete camera pose estimation and establish the spatial coordinate transformation relationship between the 2D image and the 3D environment model.

[0104] (2) Digital twin 3D spatial meshing mapping algorithm for visible light / infrared / ultraviolet camera video data

[0105] Perspective transformation is used to project point data from a 2D image to 3D space, and the coordinate transformation is calculated using a pinhole imaging model combined with the camera's intrinsic and extrinsic parameters. Surface data projection is achieved at the pixel level by aligning selected 3D rectangular regions and performing image perspective transformation. For the infrared camera, intrinsic parameter calibration at a resolution of 320×240 was performed, and the extrinsic parameters were solved using the EPnP algorithm for accurate registration. Based on infrared images and point cloud data, device-level temperature coloring was implemented, demonstrating the temperature distribution of equipment such as air conditioners and chassis from multiple perspectives. This series of steps supports functions such as real-time monitoring and hazard warning.

[0106] (3) The surveillance video images are synchronously updated to the three-dimensional surveillance scene through perspective transformation.

[0107] An ICP registration method based on Anderson acceleration and the Welsch equation is employed to accurately register the original point cloud and the secondary scanned local point cloud. A kdTree is constructed to perform point cloud neighborhood search, identifying the matching relationship between the source and target point clouds. A location-weighted overlapping point cloud fusion technique is used to reduce redundant data and optimize memory usage. This series of operations effectively updates the device status changes in the local map, such as tilt, movement, or additions, enhancing the secondary utilization value of the existing map.

[0108] (4) Research on perception and tracking algorithms based on multi-camera space

[0109] The intrinsic, extrinsic, and fundamental matrices of the cameras are obtained through binocular vision system calibration techniques, ensuring accurate geometric relationships between the two cameras. The stereo calibration step aligns the image planes of the two cameras in a coplanar manner, significantly improving stereo matching efficiency. During distortion optimization, Zhang Zhengyou's distortion model is used to calculate and correct image distortion, guaranteeing image quality.

[0110] Based on epipolar geometry principles, the eigenvalue matrix and fundamental matrix are calculated to provide necessary parameters for stereo matching. In the stereo matching stage, a block-based matching algorithm is used to identify corresponding points in the left and right images, generating a disparity map, which is then further converted into a depth map. For target tracking in multi-camera spaces, object detection algorithms such as Faster R-CNN or SSD are first used to identify objects in a single camera. Then, data from multiple cameras are fused using methods such as Kalman filtering to track the target's trajectory in real time. This effectively solves problems such as occlusion and lighting changes, achieving accurate target tracking in complex scenes. Each step is closely interconnected, forming a complete process from calibration to target tracking.

[0111] (5) Spatial perception and localization algorithm for complex multi-interval environments

[0112] This algorithm encompasses personnel detection, key node recognition, and spatial awareness enhancement. It employs the YOLOX model for efficient and accurate personnel detection and utilizes OpenPose for key node recognition in human pose estimation. To address the impact of complex environments, it integrates image super-resolution reconstruction and low-light image enhancement methods, using GAN-based and Transformer-based models to improve image quality and the Zero-DCE algorithm to enhance image visibility under low-light conditions. For occlusion scenarios, a human localization algorithm for non-calibrated scenes is developed. Based on cluster analysis, candidate footpoints are weighted to optimize human position determination accuracy, ensuring precise localization and tracking of human targets in various environments.

[0113] (6) Basic algorithm package for two / three-dimensional spatial fusion positioning and perception

[0114] Camera calibration and digital twin creation were carried out in typical substation scenarios. Personnel movement routes were determined within the selected scenarios, and algorithm verification and parameter fine-tuning were performed based on this. Support for personnel positioning area timeout alarm and electronic safety fence functions was realized through long-term operation demonstration application.

[0115] Example 3: This invention provides a spatial sensing and positioning method for complex environments with multiple intervals, comprising:

[0116] (1) Dynamic calibration and spatial transformation analysis of video cameras for plant environments

[0117] The EPNP algorithm is used to calibrate between 2D images and 3D point cloud models. Three-dimensional matching points and two-dimensional matching points are obtained by manual point selection, and these points are used to calculate camera extrinsic parameters.

[0118] The principle of the EPnP algorithm is as follows: Any point in the world coordinate system or the camera coordinate system can be represented by four control points c:

[0119]

[0120] because The system of equations is underdetermined, so there must exist a solution α. ij The selection of control points is theoretically unconstrained, but choosing the origin of the camera coordinate system as a reference point and selecting other reference points at the unit length of the principal axis obtained by PCA can improve the stability of the algorithm.

[0121] Let w be a projection parameter scalar, then there exists a camera intrinsic parameter equation:

[0122]

[0123] Substituting into the above equation, we get:

[0124]

[0125] Substituting the third row of the equation into the first and second rows, we get:

[0126]

[0127] Rewrite the above equation in matrix form:

[0128] Mx = 0 (6)

[0129] In the above formula It is a vector consisting of the coordinates of the four control points in the camera coordinate system; M is a 2n×12 matrix. X is the right null space of M, let V i Let M be the right singular vector (which can be obtained by finding M) T (If the null space eigenvalue of M is obtained, then:)

[0130]

[0131] Depending on the location of the reference point, M T The null space dimension of M can be 1 to 4, i.e., N = 1, 2, 3, 4. Since the distance between each pair of control points is fixed, there are a total of 6 constraints. By discussing N = 1, 2, 3, 4 respectively, the parameter β can be solved, and the coordinates of the control points in the camera coordinate system can be recovered. Then, based on the centroid coordinate coefficients, the coordinates of the feature points in the camera coordinate system are obtained. Since the coordinates of the feature points in the world coordinate system are also known, the transformation matrix between the two coordinate systems can be calculated.

[0132] Four control points are selected to simplify the mapping relationship from 3D matching points to 2D image points. The specific steps are as follows:

[0133] Select control points

[0134] Let the coordinates of the 3D matching point be p. i (i = 1, 2, ..., n), c j (j = 1, 2, 3, 4) represent the coordinates of four virtual control points. c1 is taken as the centroid of the 3D matching point set, and the other control points are taken as the three principal components after the principal component decomposition of the 3D matching point set. The world coordinates of the 3D matching points are expressed as a weighted sum of the control points:

[0135]

[0136] Calculate the coordinates of the control points in the camera coordinate system.

[0137] Based on the linear invariance of Euclidean space, it can be seen that the 3D matching points and control points in the camera coordinate system also satisfy the above relationship:

[0138]

[0139] Then, substitute the control points into equation (1):

[0140]

[0141] Where (u i ,v i )for The coordinates in the camera pixel coordinate system, simplified from equation (10), are:

[0142]

[0143] Let the camera coordinate system spatial coordinates of the four control points be... Combining equation (11) with the four control points (c1, c2, c3, c4), we obtain a homogeneous matrix equation of the following form:

[0144] MX = 0(12)

[0145] Then the kernel space of matrix M is the solution to the unknown X, since M T The kernel space of M is equal to the kernel space of M, therefore the solution to the unknown X can be written as:

[0146]

[0147] Where v i For M T The eigenvectors of M when the eigenvalue is 0, and N is the eigenvector of M. T The dimension of the M-kernel space. β i The distance between the two control points is unknown. According to the distance-preserving property of Euclidean transformation, the distance between them is the same in both the camera coordinate system and the world coordinate system. Solving for β by minimizing the distance difference between control points in two coordinate systems using the Gauss-Newton method. i :

[0148]

[0149] Calculate R,t

[0150] β i Substituting into equation (7) yields the coordinates of the control point in the camera coordinate system. Thus, solving the 3D to 2D PnP problem is transformed into solving the classic 3D to 3D rigid body motion problem, which can be solved using SVD decomposition or nonlinear optimization methods to obtain [R t].

[0151] (2) Digital twin 3D spatial meshing mapping algorithm for visible light / infrared / ultraviolet camera video data

[0152] Perspective transformation is used to project point data from a two-dimensional image to a three-dimensional space. The general transformation formula is as follows:

[0153]

[0154] Where the perspective transformation matrix is The original point coordinates, These are the coordinates of the transformed points. Since the image is in a two-dimensional plane, dividing by Z gives:

[0155]

[0156] Let a 33 =1, expanding the above equation, we get the equation of a point:

[0157]

[0158] Four points can yield four equations, resulting in eight equations, which can be used to solve for the remaining eight unknowns in A.

[0159]

[0160] Surface data projection achieves pixel-level projection by aligning selected 3D rectangular regions and performing image perspective transformation. In this study, the image mapping regions are all 3D rectangles, such as... Figure 1 The 3D mapped rectangle is shown. Three points p0, p1, and p2 are selected clockwise, where p0 and p1 are two vertices of the rectangle. These two vertices determine one side e0 of the rectangle, and p2 is located on the parallel side e1 of e0. Thus, the rectangular region of the image mapping can be determined by selecting these three points.

[0161] The coordinates of the other two vertices p3 and p4 of the selected rectangle can be obtained through calculation. First, determine the two side length parameters of the selected rectangle, the width W. rect High H rect W rect =|p0p1|, Among them W rect H is the length of line segment p0p1. rect Let be the distance from point p2 to line p0p1. Let vector be... The normal vector of the selected rectangle can be obtained using the right-hand rule. Where norm represents normalization. The direction vector of the selected rectangle width is... Finally, we can obtain At the same time, obtain the center of the selected rectangle: center = (p0 + p4) / 2.

[0162] 3D engines typically provide an interface for creating planar geometry. This interface is used to initialize a plane with a width of W. rect The height is H rect The initial rectangle, whose normal direction is The vertex of the initial rectangle corresponding to p0 is p′0 = [-W rect / 2,H rect [ / 2,0], with center′ = [0,0,0]. To align the initial rectangle to the point selection pose, it is also necessary to rotate and translate the initial rectangle.

[0163] First, rotate the initial rectangle to the same plane as the selected rectangle by rotating it around an axis. Rotation angle is Let mat be the rotation matrix corresponding to this rotation. After rotating about the axis aixs, the initial rectangle and the selected rectangle have the same normal vector. However, the poses are not perfectly aligned. Secondly, it is necessary to go around... Rotate by a certain angle θ to make the poses completely aligned. After rotating around the axis aixs, the vertex coordinates corresponding to p′0 are p″0=mat*p′0, the vector from the center of the selected rectangle center to p0 is v0=norm(p0-center), and the vector from the center of the initial rectangle center′ to p″0 after rotating around the axis aixs is v1=norm(p″0-center′).

[0164]

[0165] Combining equations (19), we can obtain θ = atan2(sinθ,cosθ).

[0166] The camera's extrinsic parameters are obtained using the method in part (1). Based on formula (9), the projection points of the selected rectangle vertices p0, p1, p3, p4 onto the two-dimensional image can be calculated.

[0167] Achieving point-level 2D-3D coordinate mapping requires establishing a model of the camera imaging system; the most common camera imaging model is the "pinhole model." By combining the camera's intrinsic and extrinsic parameters, the transformation from world coordinates to pixel coordinates can be obtained:

[0168]

[0169] At any point P in three-dimensional space, observed through camera C, its image point p on camera C can be obtained. Since the conversion from the camera coordinate system to the image coordinate system involves perspective projection, and perspective projection has a one-to-many relationship (meaning any point on the perspective line OP corresponds to point p), knowing the pixel position of point p in the image only determines the location of the real three-dimensional point Р on the ray OP, but does not provide the real three-dimensional coordinates of the point. To achieve the conversion from the pixel coordinate system to the world coordinate system, it is necessary to know not only the camera's intrinsic parameters... External reference We also need to know z c value.

[0170] In scenarios involving positioning on flat ground, the ground plane can be set as the z-axis. w Plane, set z w =H, representing the planar space at a height H above the horizontal plane, called elevation. If the pixel coordinate system (u,v) and the camera's intrinsic and extrinsic parameter values ​​are known, then z... w Substituting H into the formula, we can see that z c =r 31 xw +r 32 y w +r 33 H+t3, let:

[0171] A = r 31 uf x r 11 -u0r 31 ,

[0172] B = r 32 uf x r 12 -u0r 32 ,

[0173] C = f x r 31 +u0r 33 -ur 33 ,

[0174] D = f x t1+(u0-u)t3,

[0175] E = r 31 vf y r 21 -v0r 31 ,

[0176] F = r 32 uf y r 22 -v0r 32 ,

[0177] G = f y r 23 +v0r 33 -vr 33 ,

[0178] I = f y t2+(v0-v)t3, (21)

[0179] Then we have:

[0180]

[0181] Solving the system of two linear equations in two variables, we get:

[0182]

[0183] Therefore, if the ground elevation information is known, the conversion from pixel coordinates to world coordinates can be completed.

[0184] For the infrared camera, intrinsic parameter calibration was performed at a resolution of 320×240 using Matlab. Let the intrinsic parameter matrix of the infrared camera be K.

[0185]

[0186] Meanwhile, the distortion parameter RadialDistortion is obtained as [-0.4218-1.2254 0 0 0].

[0187] Six pairs of matching points were selected for camera-to-scan point cloud registration. The extrinsic parameters of the camera facing the air conditioner were obtained using the EPnP algorithm.

[0188]

[0189] External parameters of the camera viewed from the chassis perspective:

[0190]

[0191] Device-level temperature coloring was implemented based on infrared images and point cloud data, showcasing the temperature distribution of devices such as air conditioners and chassis from multiple perspectives. This series of steps supports functions such as real-time monitoring and hazard warnings.

[0192] (3) The surveillance video images are synchronously updated to the three-dimensional surveillance scene through perspective transformation.

[0193] An ICP registration method based on Anderson acceleration and Welsch equations is used to accurately register the original point cloud and the secondary scan local point cloud. This study is based on the idea of ​​Anderson acceleration (an iterative process for finding the fixed point of the shrinking mapping), and introduces the Anderson iterative framework into the ICP algorithm, which improves the convergence rate of the ICP algorithm and is generally faster than the standard Picard iterative method commonly used in ICP.

[0194] Anderson's main idea is to utilize the iteration history information generated during fixed-point iteration to construct an iterative sequence by combining fixed-point function values. Consider general fixed-point iteration.

[0195] T k+1 =G(T) k ),k=0,1,...(26)

[0196] Anderson's accelerated algorithm is in the form of: in, This is obtained by solving the following least squares problem:

[0197]

[0198] In the formula: f(T) = G(T) - T. β k >0 is the relaxation parameter, m kLet β be the minimum of m and k. Specifically, β... k =1 is a commonly used selection method, and β is assumed to be used in subsequent studies. k =1. In the first m iterations of Anderson's accelerated algorithm, the residual vector corresponding to each iteration is saved and used to construct matrix R. k When the number of iterations exceeds m steps, in matrix R k While retaining the latest residual vector, the oldest residual vector is deleted. The number of residual vectors retained in the k-th iteration is m. k and called m k Let m be the Anderson depth. During algorithm execution, the maximum Anderson depth is represented by the parameter m. When m = 0, Anderson Accelerator degenerates into fixed-point iteration. The result of Anderson Accelerator iteration n+1 is related not only to the result of the previous nth iteration but also to the results of all previous iterations. Due to its "jump property," it can achieve a faster convergence speed.

[0199] A kdTree is constructed to perform neighborhood search of point clouds in order to identify the matching relationship between the source point cloud and the target point cloud. The steps are as follows:

[0200]

[0201] Two point sets, Ps and Qs, are obtained through point cloud search, remaining unchanged before and after comparison. To prevent redundancy in overlapping point clouds and unnecessary memory consumption, a point set weighted fusion method with the source point cloud as the trust center in the current state is adopted. This involves statistically analyzing the neighboring point sets of points within the overlapping region to identify unfused point markers within those neighboring sets. These markers represent the point cloud containing a given point within the neighboring set. The number of points in the neighboring set belonging to different point clouds is then counted. Next, the weights of different points within the neighboring set are calculated. The weight is the ratio of the number of points in the neighboring set that are in the same point cloud as the given point to the total number of points in the neighboring set. The product of all unfused points in the neighboring set and their weights is summed to obtain the new fused point. Finally, this new point and all points in the neighboring set are marked as fused points. This process is iterated until all points within the overlapping region have been fused.

[0202] (4) Research on perception and tracking algorithms based on multi-camera space

[0203] In the calibration technique of binocular vision systems, the optical axis convergence model is selected to convert two-dimensional pixel coordinates into three-dimensional world coordinates.

[0204] for Figure 2 The two camera feeds show:

[0205]

[0206] Multiply the left side of the above two equations by Zc1 and Zc2 respectively.

[0207]

[0208] Based on the determinant above, X, Y, and Z can be solved using the least squares method. However, it seems possible to achieve the transformation from pixel coordinates to 3D spatial coordinates. The intrinsic, extrinsic, and fundamental matrices of the camera are obtained through methods such as Zhang's calibration method to ensure the accurate geometric relationship between the two cameras.

[0209] Using Bouguet's method for stereo correction involves decomposing the rotation matrix R and translation matrix T into rotation and translation matrices R that rotate halfway between the left and right cameras. l ,T l R r ,T r The principle of decomposition is to minimize the distortion caused by the reprojection of the left and right images and maximize the common area of ​​the left and right views.

[0210] The rotation matrix of the right image plane relative to the left image plane relative to the left image plane is decomposed into two matrices R. l and R r , is called the composite rotation matrix of the left and right cameras. R l =R 1 / 2 R r =R -1 / 2 .

[0211] Where R 1 / 2 R 1 / 2 =R,R -1 / 2 It is R 1 / 2 The inverse matrix.

[0212] Rotate each of the left and right cameras by half until their optical axes are parallel. At this point, the imaging planes of the left and right cameras are parallel, but the baselines are not parallel to the imaging plane.

[0213] Construct the transformation moment matrix R rect This ensures the baseline is parallel to the imaging plane. This is achieved using the offset matrix T of the right camera relative to the left camera.

[0214] Construct e1. The transformation matrix transforms the pole of the left view to infinity, making the epipolar line horizontal. Therefore, the translation vector between the projection centers of the left and right cameras is the direction of the left pole:

[0215] The direction e2 is orthogonal to the principal optical axis, along the image direction, and perpendicular to e1. ​​Therefore, the direction e2 can be obtained by taking the cross product of e1 and the principal optical axis and normalizing the result.

[0216] After obtaining e1 and e2, e3 is orthogonal to both e1 and e2, so e1 is naturally the cross product of the two:

[0217] This allows the pole of the left camera to be transformed to matrix R at infinity. rect ,as follows:

[0218] The overall rotation matrix of the left and right cameras is obtained by multiplying the composite rotation matrix by the transformation matrix. Multiplying the coordinate systems of the left and right cameras by their respective overall rotation matrices ensures that the principal optical axes of the left and right cameras are parallel, and that the image plane is parallel to the baseline.

[0219] By using the two overall rotation matrices described above, an ideally parallel binocular stereo image can be obtained. After correction, the image needs to be cropped as needed, requiring the selection of a new image center and image edges to maximize the overlap between the left and right sides.

[0220] During distortion optimization, Zhang Zhengyou's distortion model is used for distortion correction. The specific implementation process is as follows:

[0221] Step 0: Calculate the projection matrix Q based on the given camera internal and external parameters. This is the mapping matrix that transforms 3D points in the camera coordinate system to the pixel coordinate system.

[0222] Step 1: Calculate the correction coefficients for radial and tangential distortion in the camera coordinate system based on the Q matrix and distortion coefficients.

[0223] Step 2: Perform distortion correction on the input image using correction coefficients.

[0224] Based on epipolar geometry principles, the eigenvalue matrix and fundamental matrix are calculated to provide the necessary parameters for stereo matching. In the stereo matching stage, a block-based matching algorithm is used to divide the image into small blocks, and then the region corresponding to the block with the minimum difference is searched in the two camera images. Common block matching algorithms include greedy matching algorithms (e.g., minimum absolute difference, minimum mean square error) and adaptive window matching algorithms (e.g., adaptive support window).

[0225] For multi-camera spatial target tracking, Faster R-CNN is first used to identify objects in a single camera. Faster R-CNN primarily uses the Region Proposal Network (RPN) to directly generate bounding boxes, with only 300 proposed boxes. The RPN and Fast R-CNN share some components, significantly improving target detection speed. The Faster R-CNN target detection process is divided into four parts: candidate region generation, feature extraction, classification, and bounding box regression. Region selection is handled by the RPN, while the other three parts utilize Fast R-CNN.

[0226] Then, data from multiple cameras are fused using methods such as Kalman filtering. The Kalman filtering algorithm is a Linear Minimum Variance Estimation (LVME) method. In moving target tracking, the Kalman filtering method is used to determine the target model, parameters, and position in the current frame based on the motion state of previous video frames, thereby accurately estimating the target's location in real time. The Kalman filter (KF) is a highly efficient recursive filter with strong noise immunity, capable of accurately estimating the state of moving targets. The core of this filtering method is to iteratively estimate the current state value based on measurement results from the actual scene and the state value from the previous moment.

[0227] (5) Spatial perception and localization algorithm for complex multi-interval environments

[0228] This algorithm encompasses personnel detection, key node identification, and spatial awareness enhancement. It utilizes the YOLOX model for efficient and accurate personnel detection. The YOLOX algorithm is based on previous versions of the YOLO series, with a series of optimizations and improvements. The structure of YOLOX is as follows: Figure 3 As shown.

[0229] like Figure 3 As shown, the YOLOX network mainly consists of a backbone feature extraction network, a feature extraction part, and a detection head.

[0230] The main function of the backbone network is to extract features from the input image, transforming the information in the image into a higher-level representation. These features are typically extracted at multiple levels or scales to capture semantic information at different levels, and spatial information in the input image is encoded into feature maps through operations such as convolution and pooling. This helps the model understand information such as the location, size, and shape of objects in the image.

[0231] In this study, CSPDarknet53 was used as the backbone network, and the network structure is shown in the figure below. The backbone network uses multiple stacked CSPResblockBody to extract and encode image information, and extracts three main semantic feature maps, thereby enabling the detection head to correctly predict the target, such as... Figure 4 As shown.

[0232] CSPResblockBody is a residual network structure, such as Figure 5As shown, the left half is the baseline ResNet structure, and the right half is the CSPNet structure, which is essentially an application of CSPNet on ResNet. The network structure can be roughly divided into two parts: the right half continues to stack residual blocks according to the original approach; the left half is directly connected to the output after minimal processing, like a large residual edge. This structure enhances the learning ability of CNNs, enabling them to maintain sufficient accuracy while being lightweight; it evenly distributes computational load across each layer of the CNN, effectively improving the utilization of each computational unit and reducing unnecessary computational consumption.

[0233] This paper describes how to identify key nodes in human pose estimation using OpenPose. Its main workflow is as follows:

[0234] a) Input an image, extract features through a convolutional network to obtain a set of feature maps, and then split into two paths, using a CNN network to extract Part Confidence Maps and Part Affinity Fields respectively;

[0235] b) After obtaining these two pieces of information, we use Bipartite Matching in graph theory to find PartAssociation, connecting the joints of the same person. Due to the vector nature of PAF itself, the generated bipartite matches are correct and are finally merged into the overall skeleton of a person.

[0236] c) Finally, based on PAFs, find the Multi-Person Parsing problem —> transform the Multi-Person Parsing problem into a graph problem —> Hungarian Algorithm (The Hungarian Algorithm is a bipartite graph matching algorithm. Its core is finding augmenting paths; it's an algorithm that uses augmenting paths to find the maximum matching in a bipartite graph, such as...) Figure 6 As shown.

[0237] The network consists of two branches, predicting keypoint heatmaps and PAF maps respectively. Each branch has t stages, representing increasingly refined levels, and each stage fuses the feature maps. Here, ρφ represents the network. During training, each stage incurs a loss to avoid gradient vanishing; during prediction, only the output of the last layer is used.

[0238] PAF (Part Affinity Fields) encodes 2D vectors representing the position and orientation of limbs in the image domain. Simultaneously, it uses CMP (Part Detection Confidence Maps) to label the confidence level of each keypoint (commonly known as a "heatmap"). Through these two branches, it jointly learns the keypoint locations and the relationships between them. It also infers these bottom-up detections and associations using a greedy parsing algorithm, which sufficiently encodes the global context to obtain high-quality results while consuming only a small portion of the computational cost. In parallel scenarios, it achieves near real-time performance, and the processing time is not strongly correlated with the number of people in the image.

[0239] To address the impact of complex environments, this study integrates image super-resolution reconstruction techniques with low-light image enhancement methods, using GAN-based and Transformer-based models respectively to improve image quality. Super-resolution imaging (SR) is a technique for improving video resolution. Image super-resolution research can be divided into three main categories: interpolation-based, reconstruction-based, and learning-based methods. SR can be viewed as an image restoration problem. The process of obtaining low resolution can be considered as an image degradation problem, and the main mathematical formulas can be summarized as follows:

[0240] Y (LR) =HX (HR) +n (31)

[0241] Y represents the low-resolution image (LR), X represents the high-resolution image (HR), and H represents the degradation function, which commonly includes interpolation methods. H can also be simply understood as downsampling. n represents additional noise. The downsampling factor can be arbitrary, but as the downsampling factor increases, a large amount of HR information contained in LR is lost, which increases the complexity of the algorithm and the quality of the recovered high-resolution image. Common downsampling factors are 2, 4, 8, and 16 times.

[0242] The Zero-DCE algorithm is used to improve image visibility under low-light conditions. The Zero-DCE uses the following formula to describe the brightness improvement curve.

[0243] LE(I(x);α)=I(x)+αI(x)(1-I(x)) (32)

[0244] In the formula: x is the pixel coordinate, α is the learnable parameter, and LE(I(x); α) is the enhanced image of the input I(x).

[0245] Different α parameters will produce curves of different shapes, as follows: Figure 8 As shown.

[0246] While α can adjust the curve's vertical movement, the curve is essentially a quadratic function, resulting in insufficient variation. To fit more complex brightness changes, Zero-DCE iterates and nests this function to achieve the desired effect. The general form of the final brightness enhancement curve is shown in the following equation.

[0247] LE n (x)=LE n-1 (x)+αLE n-1 (x)(1-LE n-1 (x)) (33)

[0248] After nested iterations, it becomes possible to represent a more complex function of a higher order, with each iteration having a new parameter α. n In the specific implementation of the algorithm, n is at most 8, and the highest possible fit is 2. 8 a quadratic function.

[0249] To address occlusion scenarios, a human body localization algorithm for non-calibrated scenes was developed. The algorithm's core idea is to use the detected human body bounding box to replace the foreground information of the human body. The algorithm then samples and projects the homography onto the top and bottom edges of the bounding box to obtain candidate foot points. Latitude clustering is performed, and within each cluster, weights are assigned based on the distance to the cluster center. Valid candidate foot points are then filtered based on weight thresholds to finally obtain the location of the human body's foot points.

[0250] The rule for weighting candidate target footpoints is as follows: First, calculate all target footpoints, then cluster them according to the number of people in the scene. Candidate footpoints farther from the cluster center have lower weights, and candidate footpoints closer to the cluster center have higher weights. Let i represent the i-th camera, and G represent the set of all candidate human target footpoints. M i K represents the number of people present in the i-th camera, and its value is defined as:

[0251] K = argmax(M) i (34)

[0252] K is defined as the maximum number of human figures in the camera's field of view because human figures may obstruct the view in the scene. K-means++ is used to cluster G, resulting in K clusters. Each candidate target foot point is assigned a weight W. j The greater the distance between the candidate foot point and the cluster center, the better r jThe smaller the value, the greater the weight w of the j-th human candidate point. j The weights are defined as follows:

[0253]

[0254] Since noise is inevitably generated during the process of introducing human candidate foot points, a threshold W needs to be set. H Remove some candidate human target footpoints. After obtaining a threshold, discard candidate human target footpoints whose weights are less than the threshold. If the weight is greater than the threshold, retain them. Assuming the final number of remaining candidate target footpoints is v, then the final human footpoint position, i.e., the final human location in the scene, is O.

[0255]

[0256] weight w j The value of w actually represents the degree of matching between candidate footpoints and cluster centers; the larger the weight, the higher the degree of matching. When there is no occlusion between human targets, w... j Smaller candidate foot points refer to interference points near the head region at the top of the human foreground bounding box and the foot region at the bottom of the human foreground bounding box. When there is occlusion between human targets, w j Smaller candidate foot points typically consist of two types of points: one type is interference points near the head region at the top of the human foreground rectangle and the foot region at the bottom of the human foreground region; the other type is foreground points of other human figures. Foreground points of other human figures have a lower degree of matching with the centroid of the foot point compared to the outline points of the human head and feet, and therefore have smaller weights. Setting a threshold for the weights and then discarding candidate foot points below the threshold helps reduce noise interference and decrease human figure localization errors.

[0257] (6) Basic algorithm package for two / three-dimensional spatial fusion positioning and perception

[0258] This paper proposes a holistic solution for 2D / 3D spatial fusion positioning and sensing in power grid substations. A basic algorithm package for 2D / 3D spatial fusion positioning and sensing in substation environments was developed. The intelligent front-end device utilizes the Eclipse development platform, with C and Java programming languages ​​used for terminal function development. The intelligent mobile terminal development environment is built using the Eclipse platform, with C and Java programming languages ​​employed. The intelligent mobile device communicates with the backend server via a web service interface using JSON structured data format.

[0259] The main site system uses the Eclipse development platform, and the development of main site services is carried out using programming languages ​​such as C and Java. External service interfaces are published uniformly using Tomcat, the database uses the open-source MySQL database technology, and unstructured data is stored using CSTOR distributed storage.

[0260] The algorithm package module based on the mobile hardware and software platform can be logically divided into an application layer, a service interface layer, and a database service. The application layer mainly includes seamless positioning of power workers based on 2D / 3D spatial fusion positioning perception, intelligent identification of potential hazards in power equipment based on 2D / 3D spatial fusion positioning perception, and perimeter risk warning for power facilities based on 2D / 3D spatial fusion positioning perception. The service interface layer includes a communication client running in the browser and a service access point running on the server. The service interaction layer provides functions such as formatting, transmitting, encapsulating and unpacking user request data, and URL processing dispatch. Internal service calls within the platform support RESTful and BlazeDS, and the platform provides Web Service support for system integration. Internal service call data is serialized in JSON format. The server uses DispatchServlet in Spring MVC for request dispatch. The business logic layer consists of the business logic modules of the business application system, as well as the standard logic modules and public service proxies provided by the platform. The platform services expose public service interfaces in the business system runtime environment. These interfaces can be referenced by the business logic and can also be exposed to clients through the service interaction layer (solving the cross-domain issue of browser HTTP requests). The database service layer uses MySQL to perform unified analysis and backup of structured and unstructured data, and adopts CSTOR distributed storage for data management to ensure data efficiency and security.

[0261] The basic algorithm package will be demonstrated and applied in typical substation safety management scenarios.

[0262] This invention overcomes the problem of non-intrusive personnel positioning in actual power production environments, solving difficulties such as low light, low resolution, partial personnel obstruction, electromagnetic interference of traditional sensors, high positioning deployment costs, low accuracy of visual positioning, and low real-time performance. It has been successfully implemented in experimental and substation settings, including the Zhejiang Yunqi Digital Security Management Technology Verification Base, the Tianjin Huashan Road 110kV substation, the Tianjin Ningchegu 35kV substation, and the Tianjin Yongjiqu 220kV substation, through on-site data collection and technical verification. This demonstrates the feasibility of achieving high-precision three-dimensional spatial positioning and perception of personnel / equipment based solely on existing video surveillance systems, supporting a practical solution for real-time, accurate, lightweight, and flexible non-intrusive spatial positioning.

[0263] In another embodiment of the present invention, a spatial sensing and positioning system for complex environments with multiple intervals is provided, which can be used to implement the above-mentioned spatial sensing and positioning method for complex environments with multiple intervals. Specifically, the system includes:

[0264] The data acquisition module is used to acquire multi-interval environmental video streams from substation monitoring cameras and simultaneously collect environmental scanning point cloud models; it selects two-dimensional and three-dimensional matching points in the image and point cloud by manual point selection, solves the camera extrinsic parameters, and establishes the spatial coordinate transformation relationship between the 2D image and the 3D environmental model.

[0265] The update module is used to acquire the updated point cloud of a local area, stitch the local point cloud with the initial point cloud without an initial pose, identify overlapping areas, and realize dynamic map updates for changes in device status.

[0266] The projection module is used to project two-dimensional image data from visible light / infrared / ultraviolet camera video streams onto the digital twin three-dimensional space based on spatial coordinate transformation relationships and through perspective transformation, so as to obtain three-dimensional perception results with multispectral information;

[0267] The perception output module is used to detect human targets based on the 3D perception results, identify key human body nodes, and optimize the output spatial coordinates after fusing target trajectories from multiple cameras.

[0268] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0269] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used in the operation of a spatial perception and positioning method for complex multi-interval environments.

[0270] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the spatial awareness and localization method for multi-segment complex environments described in the above embodiments.

[0271] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0272] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0273] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0274] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0275] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A spatial sensing and positioning method for complex environments with multiple intervals, characterized in that, include: Multi-interval environmental video streams are acquired from substation monitoring cameras, and environmental scanning point cloud models are collected simultaneously. Two-dimensional and three-dimensional matching points are selected from the image and point cloud by manual point selection, the camera extrinsic parameters are solved, and the spatial coordinate transformation relationship between the 2D image and the 3D environment model is established. The local area updated point cloud is acquired, and the local point cloud is stitched with the initial point cloud without initial pose. The overlapping area is identified to realize dynamic map updates for changes in device status. Based on the spatial coordinate transformation relationship, the two-dimensional image data of the visible light / infrared / ultraviolet camera video stream is projected onto the three-dimensional space of the digital twin through perspective transformation to obtain the three-dimensional perception result of multispectral information; Based on the results of 3D perception, human targets are detected, key human body nodes are identified, and the output spatial coordinates are optimized after fusing target trajectories from multiple cameras.

2. The spatial perception and positioning method for complex environments with multiple intervals according to claim 1, characterized in that, The process involves acquiring multi-interval environmental video streams from substation monitoring cameras and simultaneously collecting environmental scanning point cloud models; selecting 2D and 3D matching points in the images and point clouds through manual point selection; solving for camera extrinsic parameters; and establishing the spatial coordinate transformation relationship between the 2D images and the 3D environmental model, including: The system acquires video footage of complex environmental conditions across multiple intervals from monitoring cameras in a substation. Based on the images captured by the monitoring cameras and the environmental scanning point cloud model, 3D and 2D points are manually selected and used to calculate camera extrinsic parameters. The EPNP algorithm is used to calibrate the 2D images and 3D point cloud model, and the accuracy of the reprojection error is evaluated. The calibration results are output to a YAML file, and the error between the reprojected points and the original points is plotted. Four control points are selected to simplify the mapping relationship between 3D matching points and 2D image points. The positions of these control points in the camera coordinate system are found by solving matrix equations. The distance difference between control points in the two coordinate systems is minimized using the Gauss-Newton method to determine the precise positions of these control points. Based on the obtained control point positions, the camera rotation matrix R and translation matrix t are obtained using SVD decomposition or nonlinear optimization methods to complete camera pose estimation and establish the spatial coordinate transformation relationship between the 2D images and the 3D environmental model.

3. The spatial perception and positioning method for complex environments with multiple intervals according to claim 2, characterized in that, Four control points are selected to simplify the mapping relationship from 3D matching points to 2D image points. The specific steps are as follows: Select control points Let the coordinates of the 3D matching point be p. i (i = 1, 2, ..., n), c j (j=1,2,3,4) represents the coordinates of four virtual control points. c1 is taken as the centroid of the 3D matching point set, and the other control points are taken as the three principal components after the principal component decomposition of the 3D matching point set. The world coordinates of the 3D matching points are expressed as a weighted sum of the control points: Calculate the coordinates of the control points in the camera coordinate system. In the camera coordinate system, the 3D matching points and control points also satisfy the relationship described above: and Control points: Where (u i ,v i )for The coordinates in the camera pixel coordinate system, simplified to the above formula, yield: Let the camera coordinate system spatial coordinates of the four control points be... Solving the system of equations for the four control points (c1, c2, c3, c4), we obtain a homogeneous matrix equation of the following form: MX=0 Then the kernel space of matrix M is the solution to the unknown X, and the solution to the unknown X can be written as: Where v i For M T The eigenvectors of M when the eigenvalue is 0, and N is the eigenvector of M. T The dimension of the M-kernel space; β i The distance between the two control points is unknown; according to the distance-preserving property of Euclidean transformation, the distance between the two control points is the same in both the camera coordinate system and the world coordinate system, i.e. Solving for β by minimizing the distance difference between control points in two coordinate systems using the Gauss-Newton method. i : Calculate R,t β i Substituting into equation (7) yields the coordinates of the control point in the camera coordinate system. The problem of solving the 3D to 2D PnP problem is transformed into solving the classic 3D to 3D rigid body motion problem, and [Rt] is obtained by using SVD decomposition or nonlinear optimization methods.

4. The spatial perception and positioning method for complex environments with multiple intervals as described in claim 1, characterized in that, The process of acquiring a local area updated point cloud, stitching the local point cloud with the initial point cloud without an initial pose, identifying overlapping areas, and realizing dynamic map updates for changes in device state includes: A handheld Livox mobile scanning device is used to acquire local area updated point clouds. An ICP registration method based on Anderson acceleration and Welsch equation is adopted to stitch the local point cloud with the initial point cloud without initial pose. A kdTree is constructed to perform neighborhood search to identify overlapping areas. Combined with position-weighted fusion technology, redundant data is reduced and memory usage is optimized to achieve dynamic map updates for device status changes.

5. The spatial perception and positioning method for complex environments with multiple intervals according to claim 1, characterized in that, The method, based on spatial coordinate transformation, projects two-dimensional image data from visible light / infrared / ultraviolet camera video streams onto a digital twin three-dimensional space through perspective transformation to obtain a three-dimensional perception result with multispectral information, including: Perspective transformation is used to project point data from a two-dimensional image to a three-dimensional space. The coordinate transformation is calculated by using a pinhole imaging model combined with the camera's intrinsic and extrinsic parameters. Surface data projection is completed by aligning the selected three-dimensional rectangular area and performing image perspective transformation to achieve pixel-level projection. For infrared cameras, intrinsic parameter calibration with a set resolution is performed, and extrinsic parameters are solved using the EPnP algorithm for accurate registration.

6. The spatial sensing and positioning method for complex multi-interval environments according to claim 5, characterized in that, Specifically, it includes: Perspective transformation is used to project point data from a two-dimensional image to a three-dimensional space. The general transformation formula is as follows: Where the perspective transformation matrix is The original point coordinates, The coordinates of the transformed point are given; the image is in a two-dimensional plane, divided by Z to obtain: Let a 33 =1, expanding the above equation, we get the equation of a point: Four points yield four equations, resulting in eight equations. Solve for the remaining eight unknowns in A: Surface data projection achieves pixel-level projection by aligning a selected 3D rectangular region and performing image perspective transformation. The rectangular region to be mapped is determined by three selected points; the coordinates of the other two vertices p3 and p4 of the selected rectangle are calculated; first, the two side length parameters of the selected rectangle, and the width W, are determined. rect High H rect W rect =|p0p1|, Among them W rect H is the length of line segment p0p1. rect Let p2 be the distance from point p2 to line p0p1; let vector p2 be the distance from point p2 to line p0p1. The normal vector of the selected rectangle can be obtained using the right-hand rule. Where norm represents normalization; the direction vector of the selected rectangle's width is... Final result Simultaneously obtain the center of the selected rectangle: center = (p0 + p4) / 2; To rotate and translate the initial rectangle, first, rotate the initial rectangle to the same plane as the selected rectangle by rotating it about an axis. Rotation angle is Let mat be the rotation matrix corresponding to this rotation; after rotating about the axis aixs, the initial rectangle and the selected rectangle have the same normal vector. Secondly, around Rotate by angle θ to make the poses completely aligned; after rotating around axis aixs, the vertex coordinates corresponding to p′0 are p″0=mat*p′0, the vector from the center of the selected rectangle center to p0 is v0=norm(p0-center), and the vector from the initial rectangle center′ to p″0 after rotating around axis aixs is v1=norm(p″0-center′). After solving the equations, we get θ = atan2(sinθ,cosθ); By combining the camera's intrinsic and extrinsic parameters, the transformation from world coordinates to pixel coordinates is obtained: For any point P in three-dimensional space, its image point p on camera C is obtained through observation by camera C; In a flat terrain positioning scenario, the ground plane is set as z. w Plane, set z w =H, representing the planar space at a height H above the horizontal plane, called elevation; if the pixel coordinate system (u,v) and the camera's intrinsic and extrinsic parameter values ​​are known, then z w Substituting H into the formula, we get z. c =r 31 x w +r 32 y w +r 33 H+t3, let: A=r 31 u-f x r 11 -u0r 31 B=r 32 u-f x r 12 -u0r 32 C=f x r 31 +u0r 33 -ur 33 D=f x t1+(u0-u)t3 E=r 31 vf y r 21 -v0r 31 F=r 32 ugh y r 22 -v0r 32 G=f y r 23 +v0r 33 -vr 33 I=f y t2+(v0-v)t3 Then we have: Solving the system of two linear equations in two variables, we get: For the infrared camera, intrinsic parameter calibration of the resolution was performed using Matlab. Let the intrinsic parameter matrix of the infrared camera be K: Simultaneously, the distortion parameter RadialDistortion was obtained as [-0.4218-1.2254 0 0 0]; Six pairs of matching points were selected for camera-to-scan point cloud registration. The extrinsic parameters of the camera facing the air conditioner were obtained using the EPnP algorithm. External parameters of the camera viewed from the chassis perspective: Device-level temperature coloring was implemented based on infrared images and point cloud data to display the temperature distribution of devices such as air conditioners and chassis from multiple perspectives.

7. The spatial perception and positioning method for complex multi-interval environments according to claim 1, characterized in that, The process of detecting human targets, identifying key human body nodes, and optimizing the output spatial coordinates after fusing target trajectories from multiple cameras, based on 3D perception results, includes: The intrinsic, extrinsic, and fundamental matrices of the cameras are obtained through the calibration technology of the binocular vision system. Stereo alignment of the two cameras' image planes is corrected. Based on the epipolar geometry principle, the eigenvalue and fundamental matrices are calculated. In the stereo matching stage, a block-based matching algorithm is used to identify corresponding points in the left and right images, generate a disparity map, and further convert it into a depth map. For target tracking in multi-camera space, the target detection algorithm is first used to identify objects in a single camera. Then, data from multiple cameras are fused using methods such as Kalman filtering to track the target's motion trajectory in real time, achieving accurate target tracking in complex scenes. By employing the YOLOX model for efficient and accurate human detection and utilizing OpenPose for key node identification in human pose estimation, this study integrates image super-resolution reconstruction technology with low-light image enhancement methods. GAN-based and Transformer-based models are used to improve image quality, and the Zero-DCE algorithm is employed to enhance image visibility under low-light conditions. For occlusion scenarios, a human localization algorithm for non-calibrated scenes is developed. Candidate foot points are weighted based on cluster analysis to optimize the accuracy of human position determination, ensuring precise localization and tracking of human targets in various environments.

8. A spatial sensing and positioning system for complex environments with multiple intervals, characterized in that, include: The data acquisition module is used to acquire multi-interval environmental video streams from substation monitoring cameras and simultaneously collect environmental scanning point cloud models. Two-dimensional and three-dimensional matching points are selected from the image and point cloud by manual point selection, the camera extrinsic parameters are solved, and the spatial coordinate transformation relationship between the 2D image and the 3D environment model is established. The update module is used to acquire the updated point cloud of a local area, stitch the local point cloud with the initial point cloud without an initial pose, identify overlapping areas, and realize dynamic map updates for changes in device status. The projection module is used to project two-dimensional image data from visible light / infrared / ultraviolet camera video streams onto the digital twin three-dimensional space based on spatial coordinate transformation relationships and through perspective transformation, so as to obtain three-dimensional perception results with multispectral information; The perception output module is used to detect human targets based on the 3D perception results, identify key human body nodes, and optimize the output spatial coordinates after fusing target trajectories from multiple cameras.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the spatial perception and positioning method for complex multi-segment environments as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the spatial perception and positioning method for complex multi-segment environments as described in any one of claims 1 to 7.

Images