A method and system for multimodal identification of power distribution network equipment

Abstract

This invention relates to the field of distribution network equipment identification, and discloses a multimodal identification method and system for distribution network equipment. The method acquires multi-view inspection images and camera parameters, outputs and divides the sets of anchored and auxiliary components using a multimodal visual language model; establishes a joint optimization objective function based on the distribution network equipment assembly topology map, and solves to output optimized camera extrinsic parameters; fits the local assembly coordinate system using anchored components, calculates the 3D theoretical bounding box of the expected component, and projects it to generate expected candidate regions; verifies and eliminates auxiliary components using optimized extrinsic parameters; emits detection rays to the 3D bounding box, and determines whether the component is occluded or missing based on the spatial intersection state; for suspicious targets, extracts local patches and combines them with prompt words input into the multimodal model for directional verification, and finally maps them back to the global coordinate system. This invention introduces topological constraints to improve pose calculation accuracy, accurately distinguishes between occlusion and actual missing components through 3D ray detection, and reduces false alarm and false negative rates.

Description

Technical Field

[0001] This invention relates to the field of power distribution equipment identification technology, specifically to a multimodal identification method and system for power distribution equipment. Background Technology

[0002] With the expansion of power grid scale, computer vision technology based on multi-view inspection images has been widely applied to the condition monitoring of distribution network equipment. Existing visual recognition schemes typically rely on single two-dimensional image features for target detection and camera pose estimation. However, in complex field inspection environments, due to limitations such as lighting variations and background interference, pure visual feature matching is prone to reprojection divergence. Furthermore, the lack of constraints on the rigid connection relationships between distribution network equipment components leads to insufficient accuracy and robustness in camera pose calculation.

[0003] Meanwhile, existing methods often remain at the level of two-dimensional image correlation when processing multi-view data, without establishing a rigorous three-dimensional spatial geometric volume model. This makes it difficult for the system to distinguish, from a physical space perspective, whether the invisibility of a target component is due to viewpoint obstruction or a genuine physical absence when faced with changes in viewpoint or tree-like interference, thus easily generating false missed detection alarms.

[0004] Furthermore, distribution network equipment contains numerous tiny hardware components. After the global inspection image is input into the recognition network, the texture features of these tiny components are easily lost during downsampling, leading to low confidence in the initial recognition. Existing technologies struggle to balance the preservation of local high-resolution details with the overall computational load of the model, and lack a mechanism for targeted error correction by incorporating prior spatial topological information, resulting in limited recall capabilities for questionable tiny targets. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal identification method and system for distribution network equipment, which solves the problems of low accuracy, difficulty in accurately distinguishing viewpoint occlusion from actual physical defects, and high missed detection rate of small suspicious components in the pose calculation of existing distribution network equipment due to the lack of physical topological constraints.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a multimodal identification method for distribution network equipment, comprising the following steps: Acquire multi-view inspection images and initial camera intrinsic and extrinsic parameter data, output an initial candidate target set through a multimodal visual language model, and divide it into an anchoring component set and an auxiliary component set; Based on the topology map of the distribution network equipment, a joint optimization objective function is established and solved, and the optimized camera extrinsic parameter data is output. The local assembly coordinate system is fitted based on the anchored component set to calculate the three-dimensional theoretical bounding box of the expected component, and the expected candidate region is generated by projecting the optimized camera extrinsic data. The set of auxiliary components is verified using the optimized camera extrinsic data, and targets that fail the verification are removed. A detection ray is emitted toward the three-dimensional theoretical bounding box, and the expected component is determined to be either occluded or missing based on the intersection state. For questionable targets, local patches are cropped based on the expected candidate regions and prompt words are generated. These prompt words are then input into the multimodal visual language model to obtain verification results, which are then mapped to the global image coordinate system.

[0007] In one optional implementation, the step of establishing and solving a joint optimization objective function based on the distribution network equipment assembly topology map, and outputting optimized camera extrinsic parameter data, includes: Through multi-view Figure 3 An initial three-dimensional spatial point set of the anchoring component set is obtained by angularization, and the initial three-dimensional spatial point set is aligned with the topology nodes in the distribution network equipment assembly topology map by coordinate mapping. Extract the connection attribute of two nodes marked as rigidly connected to form a rigid constraint node pair, and obtain the physical space distance constant corresponding to the rigid constraint node pair; The quantized geometric distance deviation is used as a visual reprojection error term, and the difference between the spatial distance deformation and the physical spatial distance constant is calculated as a topological rigidity constraint penalty term. The joint optimization objective function is constructed by combining the visual reprojection error term and the topological rigidity constraint penalty term, and then iteratively solved to output the optimized camera extrinsic data.

[0008] In one optional implementation, the step of fitting a local assembly coordinate system based on the anchored component set to calculate the three-dimensional theoretical bounding box of the expected component, and generating the expected candidate region by projecting the optimized camera extrinsic data, includes: Extract the pole main axis direction vector and crossarm main axis direction vector corresponding to the anchoring component set respectively, extract the three-dimensional coordinates of the midpoint of the common perpendicular segment, and generate the orthogonal local assembly coordinate system axial matrix; Generate a global 3D theoretical bounding box for the expected component to be detected, and calculate the actual 2D pixel coordinates of the vertices of the global 3D theoretical bounding box using the transformation matrix and the optimized camera extrinsic data; The extreme values ​​of the horizontal and vertical coordinates of the actual two-dimensional pixel coordinates are extracted to construct the minimum bounding orthogonal rectangle, which is then used as the expected candidate region.

[0009] In one optional implementation, the step of verifying the set of auxiliary components using the optimized camera extrinsic data and removing targets that fail the verification includes: Using the optimized camera extrinsic data, a multi-view fundamental matrix is ​​constructed, which is then multiplied by the homogeneous coordinates of the target center point to generate an epipolar vector; Calculate the vertical pixel distance from the center pixel coordinates of the candidate target of the same category to the epipolar vector. If all the vertical pixel distances are greater than the set epipolar tolerance threshold, then the current candidate target is determined to be isolated noise from a single viewpoint and is removed from the set of auxiliary components.

[0010] Furthermore, the step of using the optimized camera extrinsic data to verify the set of auxiliary components and removing targets that fail the verification also includes: The actual detected 3D coordinates are obtained by solving the combination of candidate targets for multi-view matching retained through epipolar geometry verification; Calculate the actual detection space Euclidean distance between the actual detected 3D coordinates and the queried parent node 3D coordinates; If the actual detection space Euclidean distance is greater than the maximum spatial distance threshold obtained from the query, the multi-view matching candidate target combination is determined to violate the physical space assembly connection condition and is removed.

[0011] In one alternative implementation, emitting a detection ray toward the three-dimensional theoretical bounding box includes: Perform a Boolean union operation on the bounding boxes corresponding to the set of anchoring components to generate the bounding volume of the known entity reconstruction. Starting from the three-dimensional coordinates of the camera's optical center, the detection ray is constructed toward the geometric center coordinates of the three-dimensional theoretical bounding box; Calculate the set of path parameters for the detected ray as it passes through the known entity and reconstructs the boundary surface of the enclosing volume. Generate a visibility indicator variable based on the comparison between the set of path parameters and the target path length. Finally, summarize the total number of unobstructed visible viewpoints.

[0012] Furthermore, determining whether the expected component is in an occluded or missing state based on the intersection state includes: If the total number of unobstructed visible viewpoints is greater than or equal to the set threshold for the number of unobstructed viewpoints, the state of the expected component is output as a true physical missing state. If the total number of unobstructed visible viewpoints is less than the threshold number of unobstructed viewpoints, the state of the expected component is output as viewpoint occlusion and invisible.

[0013] In one optional implementation, the step of cropping local map tiles based on the expected candidate regions for questionable targets includes: Targets with confidence scores lower than the set secondary review confidence threshold are extracted into the suspicious target set as the suspicious targets; The buffer margin value obtained by expanding the boundary of the expected candidate region is used to generate the boundary coordinates of the local clipping window. A matrix slicing operation is performed on the original high-resolution image data matrix based on the boundary coordinates of the local cropping window to extract the local patch and record the coordinate offset constant.

[0014] Further, the generation of prompt words, inputting the multimodal visual language model to obtain the verification result, and mapping it to the global image coordinate system includes: The target theoretical three-dimensional direction vector and the expected included angle value are converted into a structured natural language character sequence and combined to generate the prompt word; Extract the relative pixel coordinate parameters of the local two-dimensional bounding box in the verification result output by the multimodal visual language model; Perform a vector addition operation between the relative pixel coordinate parameters and the coordinate offset constant, and then backmap them to the absolute bounding box corner coordinates.

[0015] A second aspect of the present invention provides a multimodal identification system for distribution network equipment, applying the multimodal identification method for distribution network equipment described in the first aspect above. The system includes: The data acquisition module is used to acquire multi-view inspection images and initial camera intrinsic and extrinsic parameter data. It outputs an initial candidate target set through a multimodal visual language model and divides it into an anchoring component set and an auxiliary component set. The joint optimization module is used to establish and solve the joint optimization objective function based on the topology map of the distribution network equipment, and output the optimized camera extrinsic data. The region generation module is used to fit a local assembly coordinate system based on the set of anchored components to calculate the three-dimensional theoretical bounding box of the expected component, and to generate the expected candidate region by projecting the optimized camera extrinsic data. The collaborative error correction module is used to verify the set of auxiliary components using the optimized camera extrinsic data and remove targets that fail the verification. The state determination module is used to emit detection rays to the three-dimensional theoretical bounding box and determine whether the expected component is occluded or missing based on the intersection state. The secondary recognition module is used to crop local patches based on the expected candidate regions for suspicious targets and generate prompt words, input them into the multimodal visual language model to obtain verification results, and map them to the global image coordinate system.

[0016] This invention provides a method and system for multimodal identification of distribution network equipment. It has the following beneficial effects: 1. This invention extracts the physical spatial distance constant of rigid constraint node pairs in the topology map of distribution network equipment assembly, uses it as a topological rigid constraint penalty term and combines it with the visual reprojection error term to construct a joint optimization objective function. This mechanism overcomes the limitations of traditional single-reliance image feature-based pose estimation, and uses the inherent physical topology connection relationship of the equipment itself to geometrically constrain multi-view triangulated data, thereby improving the accuracy and robustness of camera extrinsic data calculation in complex field inspection backgrounds.

[0017] 2. This invention utilizes a fitted local assembly coordinate system to generate a three-dimensional theoretical bounding box for the expected component. Combined with the known entity of the anchored component, the bounding volume is reconstructed and a detection ray is emitted into space. By calculating the path parameters of the detection ray on the surface of the entity boundary, the total number of unobstructed visible viewpoints is counted. This transforms the conventional two-dimensional image-level target omission detection into a rigorous three-dimensional spatial geometric intersection analysis, thereby distinguishing whether the expected component is in a state of viewpoint occlusion and invisibility or a true physical missing state, reducing the false alarm rate of the inspection system.

[0018] 3. This invention establishes a recall mechanism for low-confidence suspicious targets by local patch cropping and multimodal verification. By calculating the boundary coordinates of the local cropping window, local patches are cropped from the original high-resolution image. The structured prompts are generated by combining the three-dimensional direction vector and the included angle parameter for targeted verification. This processing logic reduces the overall computational load of the multimodal visual language model while preserving the original texture details of the small parts. Furthermore, it achieves accurate back-mapping of the verification results to the absolute coordinates of the global image through a pre-recorded coordinate offset constant. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the system modules of the present invention; Figure 3 This is a schematic diagram of the local pose and geometry joint optimization process of the present invention; Figure 4 This is a schematic diagram illustrating the process of constructing the local coordinate system and generating the expected candidate region according to the present invention; Figure 5 This is a schematic diagram of the occlusion and missingness determination process based on visible ray detection of the present invention; Figure 6 This is a schematic diagram of the topology-guided high-resolution local secondary orientation recognition process of the present invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention provides a multimodal identification system for distribution network equipment, which may include a data acquisition module, a joint optimization module, a region generation module, a collaborative error correction module, a status determination module, and a secondary identification module.

[0022] The multimodal recognition system for power distribution network equipment operates within a computer device. This computer device is communicatively connected to image acquisition and pose acquisition devices.

[0023] The image acquisition device is configured to acquire multi-view inspection images of the same power distribution network device under different shooting poses. The multi-view inspection images include the original high-resolution image without downsampling.

[0024] The pose acquisition device is configured to simultaneously acquire initial camera intrinsic and extrinsic data when capturing multi-view inspection images. The initial camera extrinsic data includes the initial rotation matrix and the initial translation vector.

[0025] Computer equipment includes a processor and memory. The memory stores an assembly topology map of the distribution network equipment. The assembly topology map includes physical spatial distance constants and topological connection vectors for each component of the distribution network equipment.

[0026] The data acquisition module is configured to acquire multi-view inspection images, as well as initial camera intrinsic and extrinsic parameter data. The module also inputs the multi-view inspection images into a multimodal visual language model, outputting an initial candidate target set. This initial candidate target set is divided into an anchoring component set and an auxiliary component set.

[0027] The joint optimization module is connected to the data acquisition module. The joint optimization module is configured to extract rigid constraint node pairs based on the topology map of the distribution network equipment. The module establishes a joint optimization objective function that includes a visual reprojection error term and a topological rigid constraint penalty term, solves the joint optimization objective function, and outputs the optimized camera extrinsic parameter data.

[0028] The region generation module is connected to the joint optimization module. The region generation module is configured to fit a local assembly coordinate system based on the principal axes of the components in the anchored component set. The region generation module calculates the three-dimensional theoretical bounding box of the expected components based on the topology map of the distribution network equipment assembly, and projects the three-dimensional theoretical bounding box onto the two-dimensional image plane using the optimized camera extrinsic data to generate the expected candidate regions.

[0029] The collaborative error correction module is connected to the region generation module. The collaborative error correction module is configured to construct a multi-view fundamental matrix using optimized camera extrinsic data and perform epipolar geometry verification on targets in the auxiliary component set. The collaborative error correction module also performs topological spatial verification based on the comparison between the actual detected spatial distance of the target and a set spatial distance threshold, removing targets that fail the verification from the initial candidate target set.

[0030] The state determination module is connected to the collaborative error correction module. The state determination module is configured to emit a detection ray from the camera's optical center to the 3D theoretical bounding box for expected components not detected in the initial candidate target set. Based on the intersection status of the detection ray with known entities in the environment, and multi-view observation records, the state determination module determines whether the expected component is currently occluded or missing.

[0031] The secondary recognition module is connected to the region generation module. The secondary recognition module is configured to identify suspicious targets that fail verification or have a confidence level below a set threshold. Based on the expected candidate regions, the secondary recognition module crops local patches from the original high-resolution image and generates prompts containing spatial feature parameters of the target. The secondary recognition module inputs the local patches and prompts into a multimodal visual language model to obtain verification results, and then inversely maps the coordinate parameters of the verification results to the global image coordinate system.

[0032] See attached document Figure 1 This invention provides a method for multimodal identification of distribution network equipment. In this embodiment, the specific steps of the data acquisition module in performing multi-view data acquisition and initial multimodal identification are as follows.

[0033] The data acquisition module receives inspection task instructions for the target distribution network equipment. The data acquisition module acquires multi-view inspection image sequences captured by image acquisition equipment at a set sampling frequency. .in, The total number of camera viewpoints. Indicates the first Images captured from the perspective of each camera, among which Indicates 1 to The data acquisition module obtains the sensor timestamp sequence from the pose acquisition device. The data acquisition module then aligns and matches the sensor timestamp sequence with the acquisition timestamps of the multi-view inspection image sequence.

[0034] Based on the alignment matching results, the data acquisition module extracts and... Images captured from the perspective of a camera The corresponding initial camera parameters. The initial camera parameters include the initial camera intrinsic parameter matrix. Initial rotation matrix and the initial translation vector .

[0035] The data acquisition module constructs basic natural language prompts. These prompts contain predefined instruction text that requests the output of the target bounding box and category label.

[0036] The data acquisition module will be the first Images captured from the perspective of a camera And a basic natural language prompt word input multimodal visual language model. The multimodal visual language model processes images... Feature extraction and image-text alignment are performed.

[0037] The multimodal visual language model outputs the first... The initial two-dimensional object detection set corresponding to images acquired from each camera's perspective. .in, This is the initial camera intrinsic parameter matrix. Indicates the first The pixel coordinates of the two-dimensional bounding box of each target. Indicates the first Category tags for each target, Indicates the first The confidence score of each target.

[0038] The data acquisition module reads a preset component attribute mapping table. The component attribute mapping table records the category attribute tags of various components of the distribution network equipment.

[0039] The data acquisition module performs initial two-dimensional target detection set based on the component attribute mapping table. The targets are categorized. The data acquisition module will then assign category labels. Targets belonging to the benchmark support category are extracted and stored in the anchoring component set. The basic support categories include utility poles and crossarms.

[0040] The data acquisition module will classify the tags. The remaining targets that do not belong to the benchmark support category are extracted and stored in the auxiliary component set. Auxiliary component set The targets include insulators and clamps. The data acquisition module will collect the anchoring components. Auxiliary component collection The corresponding initial camera parameters are output to the joint optimization module.

[0041] See attached document Figure 3 In this embodiment of the invention, the specific steps of the joint optimization module in performing joint optimization of local pose and geometry are as follows.

[0042] The joint optimization module receives the set of anchoring components and initial camera parameters output by the data acquisition module. The joint optimization module then extracts image feature points from the two-dimensional target region within the anchoring component set, obtaining the corresponding two-dimensional observation pixel coordinate sequence.

[0043] The joint optimization module establishes corresponding feature matching pairs among the two-dimensional observation pixel coordinate sequences from multiple views using a local feature descriptor matching algorithm. The module utilizes initial camera parameters, including the initial camera intrinsic matrix, initial rotation matrix, and initial translation vector, to perform multi-view optimization on the corresponding feature matching pairs. Figure 3 Angularity calculation. The joint optimization module outputs the initial set of three-dimensional spatial points for the anchoring component.

[0044] The joint optimization module parses the topology map of the distribution network equipment assembly. It reads the topology nodes and connection attributes corresponding to the target distribution network equipment model. Finally, it aligns the spatial points within the initial 3D spatial point set with the topology nodes through coordinate mapping.

[0045] The joint optimization module extracts two nodes marked as rigidly connected from the mapped and aligned nodes, forming rigidly constrained node pairs. It then reads the standard three-dimensional physical distance of the corresponding rigidly constrained node pairs from the distribution network equipment assembly topology map, using it as a physical space distance constant. Finally, the joint optimization module summarizes the constraints from various perspectives to construct a set of rigidly constrained pairs. .

[0046] The joint optimization module constructs the visual reprojection error term. The visual reprojection error term quantifies the geometric distance deviation between a 3D point mapped to a 2D plane by the camera projection model and the actual observed 2D pixel coordinates. The specific formula for calculating the visual reprojection error term is: ; in, Represents the set of camera viewpoints. Represents the initial set of points in three-dimensional space. Indicates the first A three-dimensional spatial point In the Two-dimensional observation pixel coordinates from the perspective of a camera; For the indicator variable mask matrix, if the three-dimensional space points In the If there are matching observations from each camera's perspective, then ,otherwise ; This represents the camera pinhole projection mapping function; This is the initial camera intrinsic parameter matrix. and These are the rotation matrix and translation vector variables to be iteratively optimized, respectively. This represents a robust kernel function that suppresses the effects of outlier mismatches.

[0047] The joint optimization module constructs topological rigid constraint penalty terms. The topological rigidity constraint penalty term is used to calculate the difference between the spatial distance deformation between 3D points and the constant specified in the graph during the iteration process. The specific formula for calculating the topological rigidity constraint penalty term is as follows: in, Represents rigid constraints on a set Any pair of nodes in the array, and These represent the current coordinate variables of the corresponding nodes in the three-dimensional coordinate system; This represents the physical spatial distance constant extracted from the topology map of the distribution network equipment assembly.

[0048] The joint optimization module linearly combines the visual reprojection error term with the topological rigidity constraint penalty term to construct the joint optimization objective function. : in, This indicates the setting of constant weighting coefficients.

[0049] The joint optimization module calls a nonlinear least squares solver to minimize the joint optimization objective function. During the solution process, the joint optimization module uses the initial rotation matrix, initial translation vector, and coordinates of the initial set of points in 3D space as the initial input values ​​for the iteration variables.

[0050] The solver performs synchronous gradient descent updates on the rotation matrix variables, translation vector variables, and 3D coordinate variables by calculating the Jacobian matrix of the objective function relative to each iterative variable. When the difference in single-step updates of each iterative variable falls below a preset iteration termination threshold, the joint optimization module stops the solution loop.

[0051] The joint optimization module extracts the converged camera extrinsic variable values ​​and outputs them as optimized camera extrinsic data to the region generation module. The optimized camera extrinsic data includes the optimized rotation matrix. With the optimized translation vector .

[0052] See attached document Figure 4 In this embodiment of the invention, the specific steps of the region generation module in constructing the local coordinate system and generating the expected candidate region are as follows.

[0053] The region generation module receives the converged set of 3D spatial points output by the joint optimization module. From this set of 3D spatial points, the region generation module separates the first point cloud subset corresponding to the utility pole and the second point cloud subset corresponding to the crossarm.

[0054] The region generation module performs spatial line fitting on the first subset of point clouds using a random sampling consensus algorithm to extract the principal axis direction vector of the utility pole. The same algorithm is then used to perform spatial line fitting on the second subset of point clouds to extract the principal axis direction vector of the crossarm.

[0055] The region generation module calculates the common perpendicular segment in three-dimensional space between the main axis direction vector of the utility pole and the main axis direction vector of the crossarm. The module then extracts the three-dimensional coordinates of the midpoint of this common perpendicular segment and sets it as the origin of the local assembly coordinate system.

[0056] The region generation module defines the crossarm principal axis direction vector as the X-axis reference vector of the local assembly coordinate system, and the pole principal axis direction vector as the Z-axis reference vector of the local assembly coordinate system. The module calculates the cross product of the X-axis and Z-axis reference vectors to generate the Y-axis reference vector of the local assembly coordinate system. The module then performs Schmitt orthogonalization on the X, Y, and Z-axis reference vectors, outputting an orthogonal axial matrix for the local assembly coordinate system.

[0057] The region generation module reads the distribution network equipment assembly topology map and obtains the geometric assembly parameters of the expected components to be inspected recorded in the map. The geometric assembly parameters include the standard translational displacement, standard Euler angles of rotation, and the length, width, and height dimensions of the expected components relative to the origin of the local assembly coordinate system.

[0058] The region generation module constructs a 3D spatial cube model of the expected component to be inspected within the local assembly coordinate system based on geometric assembly parameters. The module extracts the eight corner points of the 3D spatial cube model as a local 3D vertex set. Using the transformation matrix between the local assembly coordinate system and the global spatial coordinate system, the module transforms the local 3D vertex set to the global coordinate system, generating the global 3D theoretical bounding box of the expected component to be inspected.

[0059] The region generation module acquires the initial camera intrinsic parameter matrix provided by the data acquisition module, as well as the optimized rotation matrix and optimized translation vector output by the joint optimization module.

[0060] The region generation module calculates the pixel coordinates of the eight vertices of the global 3D theoretical bounding box mapped to the 2D image plane for each shooting viewpoint. The specific vertex projection mapping calculation formula is as follows: in, Indicates the first The vertex at the th Homogeneous two-dimensional pixel coordinates on the image plane of a camera's viewpoint This represents the initial camera intrinsic parameter matrix; and They represent the first Optimized rotation matrix and optimized translation vector for each camera viewpoint; Represents the global three-dimensional theoretical bounding box. The global 3D homogeneous coordinates of each vertex are obtained. After transforming the obtained homogeneous coordinates, their actual 2D pixel coordinates are obtained.

[0061] The region generation module summarizes all eight actual 2D pixel coordinates corresponding to the global 3D theoretical bounding box from a single viewpoint. The module iterates through these eight coordinates, extracting the extreme values ​​of the x-coordinate and y-coordinate. Using these extreme values ​​as boundaries, the module constructs a minimum bounding orthogonal rectangle. This minimum bounding orthogonal rectangle is then marked as a candidate region for the expected component to be detected from that viewpoint.

[0062] The region generation module outputs the calculated set of all expected candidate regions and the global 3D theoretical bounding box data to the collaborative error correction module and the secondary recognition module.

[0063] In this embodiment of the invention, the specific steps of the collaborative error correction module in performing topological and geometric consistency collaborative error correction for multimodal output are as follows.

[0064] The collaborative error correction module receives the set of auxiliary components output by the data acquisition module and the optimized camera extrinsic parameter data output by the joint optimization module. The set of auxiliary components contains multiple initial two-dimensional candidate targets and their center point pixel coordinates.

[0065] The collaborative error correction module establishes view association pairs for any two shooting perspectives and constructs a multi-view fundamental matrix. Using the optimized rotation matrix and optimized translation vector from the optimized camera extrinsic data, the module calculates the relative rotation matrix and relative translation vector from the first perspective to the second perspective. The collaborative error correction module converts the relative translation vector into an antisymmetric matrix and, combined with the camera's intrinsic matrix, calculates the corresponding fundamental matrix. .

[0066] The collaborative error correction module extracts the homogeneous coordinates of the center point of the current candidate target in the first-person perspective from the set of auxiliary components. The collaborative error correction module will use the basic matrix. Homogeneous coordinates of the center point Multiplying them generates an epipolar parameter vector corresponding to the second-view image plane. : in, The fundamental matrix representing the transition from first-person to second-person perspectives. This represents the homogeneous coordinates of the center point from a first-person perspective. This represents the epipolar parameter vector mapped onto the second-view image plane.

[0067] The collaborative error correction module calculates the epipolar parameter vector from the center pixel coordinates of candidate targets of the same class in the second view. The vertical pixel distance is calculated. The collaborative error correction module compares the calculated vertical pixel distance with the set epipolar tolerance threshold. If the vertical pixel distances corresponding to all candidate targets of the same category in the second view are greater than the epipolar tolerance threshold, the collaborative error correction module determines that the current candidate target is isolated noise in a single view. The collaborative error correction module removes the current candidate target determined to be isolated noise in a single view from the auxiliary component set, completing the epipolar geometry verification.

[0068] For the multi-view matching candidate target combinations retained through epipolar geometry verification, the collaborative error correction module performs topological space verification. The collaborative error correction module utilizes the optimized camera extrinsic data and camera intrinsic matrix for the corresponding viewpoints to establish a multi-view... Figure 3 Angularized equations. The collaborative error correction module solves the multi-view equations using singular value decomposition. Figure 3 The angular equation system is used to obtain the actual detected 3D coordinates of the matching candidate target combination in the global coordinate system.

[0069] The collaborative error correction module reads the distribution network equipment assembly topology map and queries the parent component category label corresponding to the current verification target. The module then extracts the 3D coordinates of the parent node corresponding to the parent component category label from the determined 3D spatial data. Finally, the module calculates the actual detection spatial Euclidean distance between the actual detection 3D coordinates and the parent node 3D coordinates.

[0070] The collaborative error correction module reads the maximum spatial distance threshold set between the current verification target and its parent component from the distribution network equipment assembly topology map. The collaborative error correction module then compares the actual detected spatial Euclidean distance with the maximum spatial distance threshold.

[0071] If the actual detection space Euclidean distance is greater than the maximum spatial distance threshold, the collaborative error correction module determines that the matching candidate target combination violates the physical space assembly and connection conditions, and classifies it as a false target output by the multimodal visual language model. The collaborative error correction module removes the matching candidate target combination that violates the physical space assembly and connection conditions from the auxiliary component set.

[0072] The collaborative error correction module records the candidate targets retained after epipolar geometric verification and topological space verification, and outputs the filtered target state data to the state determination module and the secondary recognition module.

[0073] See attached document Figure 5In this embodiment of the invention, the specific steps of the state determination module in performing the occlusion and missingness determination based on visible ray detection are as follows.

[0074] The state determination module receives the global 3D theoretical bounding box data of each expected component from the region generation module, as well as the filtered target detection state data from the collaborative error correction module. The state determination module compares the list of expected components defined in the distribution network equipment assembly topology map with the target detection state data, extracts target components not detected in the target detection state data, and marks them as expected components to be determined.

[0075] The state determination module acquires the 3D spatial geometric parameters of the anchoring component. Based on these parameters, it constructs the geometric bounding volume of the environmental entity structure in the global coordinate system. Specifically, the module fits the point cloud model of the utility pole to a 3D cylindrical bounding box and the point cloud model of the crossarm to a 3D cuboid bounding box. The module then performs a Boolean union operation on the constructed bounding boxes to generate the reconstructed bounding volume of the known entity. .

[0076] For any image captured from any viewpoint, the state determination module reads the optimized translation vector of the corresponding viewpoint as the three-dimensional coordinates of the camera's optical center for that viewpoint. The state determination module extracts the geometric center coordinates of the global 3D theoretical bounding box corresponding to the expected component to be determined. .

[0077] The state determination module uses the three-dimensional coordinates of the camera optical center. Starting from the coordinates pointing to the geometric center The direction is the unit direction vector. A detection ray is constructed in three-dimensional space. The parametric equation of the detection ray is defined as: in, Indicates that the ray is at the parameter The three-dimensional spatial coordinates of the location. A non-negative path length scalar greater than or equal to zero, and a unit direction vector. The calculation formula is .

[0078] The state determination module calculates the three-dimensional coordinates of the camera's optical center. coordinates to the geometric center The physical distance between them is recorded as the target path length. .

[0079] The state determination module performs the detection ray and reconstructs the bounding volume with known entities. Spatial intersection test. The state determination module calculates the ray equation to reconstruct the enclosing volume through a known entity. The set of path parameters corresponding to the intersection points of the boundary surfaces.

[0080] The status determination module determines the path based on the path parameter set and the target path length. Based on the comparison results, the visibility indicator variable for the expected component to be determined is calculated from the current viewpoint. If there is a positive real number in the path parameter set. satisfy This indicates that the detection ray was blocked by the solid structure before reaching the geometric center of the target, and the state determination module assigns a value to this visibility indicator variable. If the path parameter set does not contain anything less than... A positive real number indicates that the detection ray passes through space unimpeded and reaches the target area; the state determination module assigns this visibility indicator variable a value. .

[0081] The state determination module iterates through all shooting angles, executes the intersection test process described above, and summarizes the visibility indicator variables output by each angle. The state determination module then sums the visibility indicator variables of all angles to obtain the total number of unobstructed visible angles in space for the component to be determined.

[0082] The status determination module reads the preset threshold for the number of unobstructed viewpoints in the system. The status determination module then performs a logical comparison between the total number of unobstructed visible viewpoints and the threshold for the number of unobstructed viewpoints.

[0083] If the total number of unobstructed visible viewpoints is greater than or equal to the threshold number of unobstructed viewpoints, the state determination module determines that the current multi-view data can fully cover the expected area. If no target is detected under the premise of no physical obstruction, the state of the expected component to be determined is output as the true physical missing state.

[0084] If the total number of unobstructed visible viewpoints is less than the threshold number of unobstructed viewpoints, the state determination module determines that the target expected area is blocked by environmental entities in most shooting viewpoints. The undetected phenomenon is due to geometric observation limitations. Therefore, the state of the expected component to be determined is output as viewpoint occlusion and invisible.

[0085] The status determination module records the generated actual physical missing status or viewpoint occlusion and invisible status to the equipment defect detection log.

[0086] See attached document Figure 6 In this embodiment of the invention, the specific steps of the secondary identification module performing topology-guided high-resolution local secondary orientation identification are as follows.

[0087] The secondary identification module receives the filtered target detection status data output by the collaborative error correction module, as well as the initial confidence score recorded by the data acquisition module. The secondary identification module compares the initial confidence score with a set secondary review confidence threshold. Targets with confidence scores below the secondary review confidence threshold, as well as targets marked as topologically questionable in the collaborative error correction step but not directly removed, are merged and extracted into a questionable target set.

[0088] For each current target in the set of questionable targets, the secondary recognition module reads the two-dimensional expected candidate region of the target output by the region generation module under each corresponding shooting view. The expected candidate region is a two-dimensional orthogonal rectangle.

[0089] The secondary recognition module reads the preset pixel expansion coefficient. It then multiplies the side length of the expected two-dimensional candidate region by the pixel expansion coefficient to calculate the buffer margin value. Finally, the secondary recognition module extends the top, bottom, left, and right boundary parameters of the expected candidate region outwards by the buffer margin value to generate the local clipping window boundary coordinates.

[0090] The secondary recognition module acquires the original high-resolution image data matrix from the multi-view inspection image sequence before downsampling. Based on the coordinates of the local cropping window boundaries, the secondary recognition module performs matrix slicing on the original high-resolution image data matrix, outputting local patches containing local pixel features. The secondary recognition module records a constant coordinate offset of the top-left vertex of the local cropping window relative to the origin of the original high-resolution image.

[0091] The secondary identification module reads the assembly topology map of the distribution network equipment and obtains the geometric topology prior parameters corresponding to the current target. The geometric topology prior parameters include the three-dimensional direction vector of the theoretical three-dimensional center of the current target relative to the center of the anchoring component, and the expected angle between the principal axis of the current target's bounding box and the ambient horizontal plane.

[0092] The secondary recognition module is equipped with a text conversion mapping table. By querying the text conversion mapping table, the secondary recognition module converts the three-dimensional direction vector and the expected angle value into a structured natural language character sequence. The secondary recognition module combines and concatenates the instruction text, the current target category name, and the structured natural language character sequence according to a set format to generate a dynamic prompt word text sequence carrying spatial feature constraints.

[0093] The secondary recognition module inputs local image patches and dynamic prompt text sequences into the multimodal visual language model. The multimodal visual language model performs multimodal feature fusion operations and outputs a verified classification label for the local image patch and a verified local two-dimensional bounding box.

[0094] The secondary recognition module extracts the relative pixel coordinate parameters of the local 2D bounding box for verification. The secondary recognition module then performs a vector addition operation on the relative pixel coordinate parameters and a coordinate offset constant to achieve a back-mapping transformation in coordinate space. The specific formula for the back-mapping transformation is as follows: in, This represents the coordinates of the absolute bounding box corners in the global two-dimensional coordinate system of the original high-resolution image after transformation; This represents the relative corner coordinates of the local 2D bounding box output by the multimodal visual language model in the local tile coordinate system. This represents the coordinate offset constant corresponding to the top-left vertex of the local clipping window.

[0095] The secondary recognition module acquires the coordinates of multiple absolute bounding box corner points of the current target from multiple viewpoints. The secondary recognition module then calls the optimized camera extrinsic parameter data output by the joint optimization module, utilizing multi-view... Figure 3 The angularization algorithm calculates the updated three-dimensional spatial coordinates of the target.

[0096] The secondary recognition module will overwrite the verification classification labels, absolute bounding box corner coordinates, and updated 3D spatial coordinates into the device target recognition record library, completing the secondary orientation recognition and data update process.

[0097] See attached document Figure 2 The present invention also provides a multimodal identification system for distribution network equipment, which may include: a data acquisition module, a joint optimization module, a region generation module, a collaborative error correction module, a status determination module, and a secondary identification module.

[0098] The data acquisition module is configured to communicate with external image acquisition devices and pose sensors. It includes a data receiving unit and an image-text inference unit. The data receiving unit synchronously acquires multi-view inspection image sequences of the target device and initial camera intrinsic and extrinsic parameter data with timestamps. The image-text inference unit inputs the multi-view inspection image sequences and preset command text into a multimodal visual language model to extract an initial two-dimensional target detection set containing bounding boxes and classification labels. The data acquisition module queries the system's preset component attribute mapping table to classify the detected targets into a set of anchoring components including utility poles and crossarms, and a set of auxiliary components including other connecting parts.

[0099] The joint optimization module is connected to the data acquisition module. The joint optimization module includes a feature matching unit and a constraint solving unit. The feature matching unit performs pixel feature matching on the two-dimensional target region in the anchoring component set and uses camera parameters for multi-view calculations. Figure 3Angle calculation is performed, outputting an initial set of 3D spatial points. The constraint solving unit reads the topology map of the distribution network equipment assembly and extracts rigid constraint node pairs with constant physical distance. The constraint solving unit establishes a joint optimization objective function that includes a 3D point visual reprojection error term and a topological rigid constraint penalty term between nodes. The constraint solving unit calls the numerical optimization solver to iteratively update the camera pose matrix variables and 3D coordinate variables until the convergence condition is met, and outputs the optimized camera extrinsic parameter data.

[0100] The region generation module is connected to the joint optimization module. The region generation module includes a coordinate system construction unit and a projection mapping unit. The coordinate system construction unit extracts multiple principal axis direction vectors of the anchoring components using a spatial straight-line fitting algorithm, calculates the common perpendicular segment of the principal axes, performs vector orthogonalization processing, and establishes a local assembly coordinate system with its origin located at the intersection of the equipment. The projection mapping unit, based on the translation and rotation parameters set in the atlas, calculates the global 3D theoretical bounding box of the undetected expected components within the local assembly coordinate system. Using the optimized camera extrinsic data, the projection mapping unit projects the vertices of the global 3D theoretical bounding box onto the 2D image planes of each viewpoint, calculates the extreme values ​​of the horizontal and vertical coordinates to generate 2D expected candidate regions for the corresponding viewpoints.

[0101] The collaborative error correction module is connected to the data acquisition module and the region generation module. The collaborative error correction module includes an epipolar verification unit and a spatial verification unit. The epipolar verification unit generates a fundamental matrix using optimized camera extrinsic data from both perspectives and calculates the epipolar vector corresponding to the target center point in the second perspective plane. The epipolar verification unit removes targets deemed as isolated noise by comparing the vertical pixel distance with the epipolar tolerance threshold. The spatial verification unit numerically compares the actual detected spatial distance of the multi-view matched target with the maximum spatial distance threshold recorded in the map to determine whether the target violates the physical assembly connection conditions, and removes targets that violate the conditions from the auxiliary component set.

[0102] The state determination module is connected to the region generation module and the collaborative error correction module. The state determination module includes a bounding volume construction unit and a ray intersection unit. The bounding volume construction unit fits spatial cylindrical and cuboid models based on anchored 3D point cloud coordinates, and performs a model space union calculation to generate the reconstructed bounding volume of the known entity. The ray intersection unit constructs detection rays with parametric equations from the optical center coordinates of each camera viewpoint towards the geometric center of the bounding box of the expected component. The ray intersection unit calculates the spatial intersection path parameters between the detection rays and the reconstructed bounding volume of the known entity, and outputs the actual physical missing state or the viewpoint occlusion invisible state of the expected component based on the total number of unobstructed intersections from all views and a preset threshold.

[0103] The secondary recognition module is connected to the collaborative error correction module and the region generation module. The secondary recognition module includes a patch cropping unit and a verification mapping unit. The patch cropping unit extracts suspicious targets with confidence levels below a threshold, calculates local cropping boundaries based on the two-dimensional expected candidate regions and a set expansion coefficient, and extracts local pixel patches from the original un-downsampled high-resolution image matrix. The verification mapping unit generates a text character sequence carrying spatial feature constraints based on the target's theoretical three-dimensional direction vector and angle parameters. The verification mapping unit inputs the local pixel patches and text character sequence into a multimodal visual language model to generate a verification two-dimensional bounding box, and uses the constant coordinate offset between the local cropping boundary and the global image origin to inversely convert the verification two-dimensional bounding box into absolute bounding box corner coordinates.

Claims

1. A method for multimodal identification of distribution network equipment, characterized in that, Includes the following steps: Acquire multi-view inspection images and initial camera intrinsic and extrinsic parameter data, output an initial candidate target set through a multimodal visual language model, and divide it into an anchoring component set and an auxiliary component set; Based on the topology map of the distribution network equipment, a joint optimization objective function is established and solved, and the optimized camera extrinsic parameter data is output. The local assembly coordinate system is fitted based on the anchored component set to calculate the three-dimensional theoretical bounding box of the expected component, and the expected candidate region is generated by projecting the optimized camera extrinsic data. The set of auxiliary components is verified using the optimized camera extrinsic data, and targets that fail the verification are removed. A detection ray is emitted toward the three-dimensional theoretical bounding box, and the expected component is determined to be either occluded or missing based on the intersection state. For questionable targets, local patches are cropped based on the expected candidate regions and prompt words are generated. These prompt words are then input into the multimodal visual language model to obtain verification results, which are then mapped to the global image coordinate system.

2. The method for multimodal identification of distribution network equipment according to claim 1, characterized in that, The steps of establishing and solving a joint optimization objective function based on the distribution network equipment assembly topology map, and outputting optimized camera extrinsic parameter data include: The initial three-dimensional spatial point set of the anchoring component set is obtained by multi-view triangulation, and the initial three-dimensional spatial point set is aligned with the topology nodes in the distribution network equipment assembly topology map by coordinate mapping. Extract the connection attribute of two nodes marked as rigidly connected to form a rigid constraint node pair, and obtain the physical space distance constant corresponding to the rigid constraint node pair; The quantized geometric distance deviation is used as a visual reprojection error term, and the difference between the spatial distance deformation and the physical spatial distance constant is calculated as a topological rigidity constraint penalty term. The joint optimization objective function is constructed by combining the visual reprojection error term and the topological rigidity constraint penalty term, and then iteratively solved to output the optimized camera extrinsic data.

3. The method for multimodal identification of distribution network equipment according to claim 1, characterized in that, The steps of fitting a local assembly coordinate system based on the anchored component set to calculate the three-dimensional theoretical bounding box of the expected component, and generating the expected candidate region by projecting the optimized camera extrinsic data, include: Extract the pole main axis direction vector and crossarm main axis direction vector corresponding to the anchoring component set respectively, extract the three-dimensional coordinates of the midpoint of the common perpendicular segment, and generate the orthogonal local assembly coordinate system axial matrix; Generate a global 3D theoretical bounding box for the expected component to be detected, and calculate the actual 2D pixel coordinates of the vertices of the global 3D theoretical bounding box using the transformation matrix and the optimized camera extrinsic data; The extreme values ​​of the horizontal and vertical coordinates of the actual two-dimensional pixel coordinates are extracted to construct the minimum bounding orthogonal rectangle, which is then used as the expected candidate region.

4. The method for multimodal identification of distribution network equipment according to claim 1, characterized in that, The step of verifying the set of auxiliary components using the optimized camera extrinsic data and removing targets that fail the verification includes: Using the optimized camera extrinsic data, a multi-view fundamental matrix is ​​constructed, which is then multiplied by the homogeneous coordinates of the target center point to generate an epipolar vector; Calculate the vertical pixel distance from the center pixel coordinates of the candidate target of the same category to the epipolar vector. If all the vertical pixel distances are greater than the set epipolar tolerance threshold, then the current candidate target is determined to be isolated noise from a single viewpoint and is removed from the set of auxiliary components.

5. The method for multimodal identification of distribution network equipment according to claim 4, characterized in that, The set of auxiliary components is validated using the optimized camera extrinsic data, and targets that fail the validation are removed. This also includes: The actual detected 3D coordinates are obtained by solving the combination of candidate targets for multi-view matching retained through epipolar geometry verification; Calculate the actual detection space Euclidean distance between the actual detected 3D coordinates and the queried parent node 3D coordinates; If the actual detection space Euclidean distance is greater than the maximum spatial distance threshold obtained from the query, the multi-view matching candidate target combination is determined to violate the physical space assembly connection condition and is removed.

6. The method for multimodal identification of distribution network equipment according to claim 1, characterized in that, The steps of emitting detection rays into the three-dimensional theoretical bounding box include: Perform a Boolean union operation on the bounding boxes corresponding to the set of anchoring components to generate the bounding volume of the known entity reconstruction. Starting from the three-dimensional coordinates of the camera's optical center, the detection ray is constructed toward the geometric center coordinates of the three-dimensional theoretical bounding box; Calculate the set of path parameters for the detected ray as it passes through the known entity and reconstructs the boundary surface of the enclosing volume. Generate a visibility indicator variable based on the comparison between the set of path parameters and the target path length. Finally, summarize the total number of unobstructed visible viewpoints.

7. The method for multimodal identification of distribution network equipment according to claim 6, characterized in that, The step of determining whether the expected component is in an occluded or missing state based on the intersection state includes: If the total number of unobstructed visible viewpoints is greater than or equal to the set threshold for the number of unobstructed viewpoints, the state of the expected component is output as a true physical missing state. If the total number of unobstructed visible viewpoints is less than the threshold number of unobstructed viewpoints, the state of the expected component is output as viewpoint occlusion and invisible.

8. The method for multimodal identification of distribution network equipment according to claim 1, characterized in that, The step of cropping local map patches based on the expected candidate regions for questionable targets includes: Targets with confidence scores lower than the set secondary review confidence threshold are extracted into the suspicious target set as the suspicious targets; The buffer margin value obtained by expanding the boundary of the expected candidate region is used to generate the boundary coordinates of the local clipping window. A matrix slicing operation is performed on the original high-resolution image data matrix based on the boundary coordinates of the local cropping window to extract the local patch and record the coordinate offset constant.

9. The method for multimodal identification of distribution network equipment according to claim 8, characterized in that, The steps of generating prompt words, inputting them into the multimodal visual language model to obtain verification results, and mapping them to the global image coordinate system include: The target theoretical three-dimensional direction vector and the expected included angle value are converted into a structured natural language character sequence and combined to generate the prompt word; Extract the relative pixel coordinate parameters of the local two-dimensional bounding box in the verification result output by the multimodal visual language model; Perform a vector addition operation between the relative pixel coordinate parameters and the coordinate offset constant, and then backmap them to the absolute bounding box corner coordinates.

10. A multimodal identification system for power distribution network equipment, characterized in that, The method for multimodal identification of distribution network equipment according to any one of claims 1-9 includes: The data acquisition module is used to acquire multi-view inspection images and initial camera intrinsic and extrinsic parameter data. It outputs an initial candidate target set through a multimodal visual language model and divides it into an anchoring component set and an auxiliary component set. The joint optimization module is used to establish and solve the joint optimization objective function based on the topology map of the distribution network equipment, and output the optimized camera extrinsic data. The region generation module is used to fit a local assembly coordinate system based on the set of anchored components to calculate the three-dimensional theoretical bounding box of the expected component, and to generate the expected candidate region by projecting the optimized camera extrinsic data. The collaborative error correction module is used to verify the set of auxiliary components using the optimized camera extrinsic data and remove targets that fail the verification. The state determination module is used to emit detection rays to the three-dimensional theoretical bounding box and determine whether the expected component is occluded or missing based on the intersection state. The secondary recognition module is used to crop local patches based on the expected candidate regions for suspicious targets and generate prompt words, input them into the multimodal visual language model to obtain verification results, and map them to the global image coordinate system.

Images