A path planning method and system for a patrol robot

Abstract

The application relates to the technical field of path planning, in particular to a path planning method and system for a patrol robot, and the application proposes the following scheme: an initial electronic map is used to perform a patrol task, first image data and pose information are collected, a patrol target is identified and image registration is completed, and second image data is generated; based on the structure sensitivity distribution of local visual features and the illumination change trend, an optimal observation direction is inverted and calibrated, a shooting position and an angle are determined to form a path node; a multi-objective optimization model is used to comprehensively consider path length, visual angle continuity and illumination interference, and an optimal patrol path is generated; and the application is suitable for outdoor patrol scenes with complex structures and uneven target distribution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, in particular to a path planning method and system for a patrol robot. BACKGROUND

[0002] In the prior art, patrol robots have been widely used in power, petrochemical, transportation and other industrial scenarios to replace manual execution of equipment state detection and troubleshooting tasks. However, most of the existing path planning methods rely on manually preset patrol target coordinates or static task area division methods, which are difficult to adapt to actual environments with unclear target boundaries, complex spatial structures and non-uniform target distribution. In addition, traditional image acquisition usually uses fixed angles and distances for shooting, without fully considering the influence of target structure perception sensitivity and environmental lighting conditions on imaging quality, which easily leads to key details being blurred or blocked, thereby affecting the accuracy of subsequent identification and discrimination.

[0003] For example, the Chinese patent with the authorization announcement number CN111781936B provides a robot path planning method, device, robot and computer readable storage medium, applied to a robot, the method comprises: determining a positioning map of a current working area; receiving forbidden area data sent by an upper computer; planning a path according to the positioning map and the forbidden area data; moving according to the path. This technical solution does not need to increase additional sensors, can make the robot bypass the forbidden area, at the same time, increasing the forbidden map layer will not affect the original positioning system, maintains the original positioning accuracy.

[0004] The above patent has the problems proposed in the background art: under the condition of a track-type mobile platform and other limited mobile paths, how to realize effective modeling and information acquisition of the target in the limited pose selection is still a technical difficulty in current patrol path planning. In order to solve the above problems, the present application designs a path planning method and system for a patrol robot. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a path planning method and system for a patrol robot, which executes a patrol task through an initial electronic map, collects first image data and pose information, identifies a patrol target and completes image registration, and generates second image data. Based on the structure sensitivity distribution of local visual features and the lighting change trend, the optimal observation direction is inverted and calibrated to determine the shooting position and angle to form path nodes. A multi-objective optimization model is used to consider the path length, visual angle continuity and lighting interference to generate an optimal patrol path.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] A path planning method for an inspection robot, the method comprising:

[0008] driving the inspection robot to perform a first stage inspection according to a preset initial electronic map, obtaining first image data and corresponding pose data;

[0009] processing the first image data, identifying inspection targets, and registering first image data obtained from multiple shooting positions for each inspection target, obtaining second image data;

[0010] based on the second image data, determining a shooting position and a shooting angle for obtaining inspection target information as a shooting configuration for the corresponding inspection target;

[0011] inputting the shooting configurations of all inspection targets as path nodes into a preset path optimization model, outputting an inspection path of the inspection robot, wherein the path optimization model generates an inspection path through a multi-objective optimization algorithm, and the multi-objective optimization algorithm includes kinematic constraints of the inspection robot;

[0012] generating a motion control instruction according to the inspection path to control the inspection robot to perform an inspection task according to the inspection path.

[0013] inputting the shooting configurations of all inspection targets as path nodes into a preset path optimization model, outputting an inspection path of the inspection robot, comprising:

[0014] constituting path nodes with the shooting position and the shooting angle of each inspection target, and constructing a cost function between nodes, the cost function including spatial movement distance between nodes, view angle switching amplitude, and shooting illumination condition change degree;

[0015] inputting all path nodes into a multi-objective optimization algorithm to comprehensively minimize inspection total path length, view angle switching cost, and illumination change interference influence, outputting an inspection path satisfying shooting continuity constraints, wherein the constraint conditions of the multi-objective optimization algorithm include kinematic constraints, dynamic constraints of the inspection robot, and time constraints of the inspection task.

[0016] generating a motion control instruction according to the inspection path, comprising:

[0017] generating a control parameter set including a speed curve, a steering angle instruction, and a torque instruction according to the path nodes of the inspection path in combination with a kinematic model of the inspection robot;

[0018] converting the control parameter set into an electrical signal and sending it to the inspection robot.

[0019] processing the first image data, identifying inspection targets, comprising:

[0020] processing the first image data by a pre-trained target detection model, wherein the target detection model is a convolutional neural network structure trained on a power equipment image dataset;

[0021] According to the inspection target candidate region, the corresponding contour feature and texture feature are calculated by edge detection calculation, and the contour feature and the texture feature are matched with a preset template library to identify the inspection target.

[0022] The first image data obtained from multiple shooting positions for each inspection target is registered to obtain second image data, including:

[0023] According to the timestamp and pose information when the first image data is collected, an image sequence for the same inspection target is determined, and each first image data in the image sequence corresponds to a different shooting position;

[0024] The first image data in the image sequence is registered by feature point matching to obtain second image data, wherein the feature point matching includes corner point extraction on the first image data, construction of a local descriptor, and acquisition of corresponding point pairs between images according to the local descriptor through bidirectional matching.

[0025] Determine the shooting position and shooting angle for obtaining the inspection target information, including:

[0026] For each inspection target, local visual features of the corresponding inspection target are identified in the second image data, wherein the local visual features include nameplate characters, connection terminals, and equipment joints.

[0027] Perform structure sensitivity analysis on the local visual features to obtain a structure sensitivity distribution.

[0028] According to the structure sensitivity distribution, the first observation direction of the local visual features is inverted, wherein the shooting angle corresponding to the first observation direction is the shooting direction that makes the comprehensive value of the structure sensitivity of the multiple local visual features reach a maximum value.

[0029] According to the light information in the second image data, the first observation direction is calibrated to determine a second observation direction, the second observation direction is taken as the shooting angle, and the intersection point of the second observation direction and the connection line of the initial electronic map is taken as the shooting position, wherein the light information is the brightness variation range of the second image data under different viewing angles.

[0030] Perform structure sensitivity analysis on the local visual features to obtain a structure sensitivity distribution, including:

[0031] calculating pixel gradient and spatial arrangement features of the local visual feature;

[0032] comparing image structure changes of the pixel gradient and spatial arrangement features frame by frame to obtain a structure change quantity, wherein the structure change quantity includes definition change, structure integrity change and image detail retention;

[0033] constructing a corresponding sensitivity curve according to the change trend of the structure change quantity in the view angle dimension, and generating a structure sensitivity distribution according to the slope of the sensitivity curve.

[0034] inverting the first observation direction of the local visual feature according to the structure sensitivity distribution, including:

[0035] combining the sensitivity values corresponding to each shooting direction in the structure sensitivity distribution into a spatial sensitivity point set, and calculating the structure sensitivity change rate between each shooting direction according to the spatial sensitivity point set to generate a spatial sensitivity gradient field;

[0036] calculating a structure response path based on the gradient rising direction in the spatial sensitivity gradient field, wherein the structure response path is a sequence of directions with the largest response change in the gradient rising path;

[0037] performing change rate analysis on the structure response path to obtain an extreme point of the structure sensitivity change rate, and taking the direction corresponding to the extreme point as the first observation direction of the local visual feature.

[0038] calibrating the first observation direction according to the illumination information in the second image data to determine a second observation direction, including:

[0039] extracting the luminance gradient values of the corresponding illumination information according to a plurality of adjacent view angles of the first observation direction to obtain a local luminance change sequence;

[0040] calculating the luminance fluctuation amplitude corresponding to the first observation direction, and taking the first observation direction as the second observation direction if the luminance fluctuation amplitude is less than the average amplitude of the local luminance change sequence;

[0041] if the luminance fluctuation amplitude is greater than or equal to the average amplitude of the local luminance change sequence, constructing a luminance fluctuation trend vector field based on the local luminance change sequence;

[0042] calculating the direction distribution trend with the smallest luminance disturbance amplitude in the luminance fluctuation trend vector field, and calculating a calibration vector according to the direction distribution trend, wherein the calibration vector represents the minimum illumination interference adjustment direction relative to the first observation direction;

[0043] The first observation direction is directionally calibrated according to the calibration vector to obtain a second observation direction.

[0044] A path planning system for a patrol robot, the system comprising:

[0045] An image acquisition module configured to control the patrol robot to perform a first stage of patrol according to an initial electronic map, and to acquire first image data and corresponding pose data;

[0046] An image processing module configured to process the first image data, to identify a patrol target, and to register first image data acquired from multiple shooting positions of the same patrol target to obtain second image data;

[0047] A shooting configuration generation module configured to perform structural sensitivity analysis and observation direction inversion based on local visual features in the second image data, and to calibrate a first observation direction in combination with illumination information to determine a shooting position and a shooting angle of each patrol target as a shooting configuration;

[0048] A path planning module configured to construct a cost function using the shooting configuration of each patrol target as a path node, and to generate a patrol path through a multi-objective optimization algorithm;

[0049] A control execution module configured to send the patrol path to the patrol robot, and to control the patrol robot to sequentially complete target information acquisition according to the patrol path.

[0050] Compared with the prior art, the application has the following beneficial effects:

[0051] The application can dynamically identify patrol targets and their optimal observation configurations without target prior information, and automatically generate a patrol path with shooting continuity and perception stability. By constructing a structural sensitivity distribution and inverting a first observation direction, the most clear observation point of structural information can be selected from multiple direction perspectives, and further combined with illumination disturbance trend to calibrate the shooting direction, thereby improving the effectiveness and consistency of image data. BRIEF DESCRIPTION OF DRAWINGS

[0052] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0053] Figure 1 An exemplary patrol scene schematic diagram for an embodiment of the application;

[0054] Figure 2 A flowchart of a path planning method for a patrol robot according to an embodiment of the application;

[0055] Figure 3 A first image data processing flowchart according to an embodiment of the application;

[0056] Figure 4 Fig. 2 is a schematic diagram of the principle of the second image processing of the embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0058] In this document, the term “embodiment” means that a specific feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive or alternative embodiments to other embodiments. It can be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] The present application is applicable to an outdoor autonomous inspection system with dual-stage image acquisition and path optimization capability based on free navigation control. The inspection system faces the following scene conditions, and the application environment characteristics include but are not limited to:

[0060] The wheeled or tracked inspection robot with autonomous navigation capability can complete initial path exploration and secondary planning execution in unstructured or semi-structured outdoor environments;

[0061] The target inspection objects are irregularly arranged along the outdoor inspection path, and most of them lack clear positioning marks. There are differences between the targets in spatial position, direction, orientation, and occlusion state;

[0062] The target is a power equipment component with typical outdoor installation characteristics, which can specifically include pole-mounted transformers, switchgear, outdoor distribution boxes, and photovoltaic components, etc. The common characteristics include but are not limited to: the surface structure details are the key basis for judgment, and the image acquisition is limited by the angle, lighting, and occlusion conditions;

[0063] It should be noted that the present application does not rely on the preset coordinates or static acquisition position information of all inspection targets, but dynamically identifies the target and its structure contour information through the image and pose data collected in the first stage, and then reverses the most suitable shooting direction and shooting position;

[0064] It can be understood that the inspection path planning method proposed in the present application is applicable to both the first inspection of a certain outdoor scene and the periodic daily inspection of the scene. In an optional embodiment, the inspection robot can move through a fixed track.

[0065] Please refer to Figure 1 ,Figure 1 An exemplary inspection scene schematic diagram provided for the embodiments of the present application.

[0066] Figure 1 It is shown that the inspection robot moves along the preset inspection track. In the first inspection shooting stage, image collection is performed according to a fixed step distance, and first image data and corresponding pose information are obtained in the process. In this stage, the spatial information of the inspection target is not clear, and the system relies on the first image data for target recognition and contour extraction.

[0067] Figure 1 It is further shown that based on the results obtained after the first stage registration, the system identifies the inspection target (such as the equipment cabinet panel in the figure), and reverses the optimal observation direction by combining image structure sensitivity analysis and light condition deduction, to determine the appropriate second inspection shooting position and shooting angle.

[0068] It should be noted that the inspection target of the present application is usually a device unit installed in an open space, which lacks uniform size, standardized anchor points or explicit boundary markers, and has structural features of coexistence of extensibility and occlusion in space.

[0069] The accompanying drawings provided by the present application are only illustrative examples, and the specific target type, number and track configuration can be flexibly adjusted according to the actual deployment environment, and do not constitute a limitation on the device type, identification method or installation scheme.

[0070] In the present embodiment, the inspection robot performs first-stage inspection image collection along the track based on the preset initial electronic map. The electronic map provides basic path information such as track structure, relative spatial position and collection step distance, but does not contain specific spatial coordinates or boundary configuration of the inspection target.

[0071] In the actual inspection environment, the inspection target is usually installed on the side of the track, such as switch cabinets, electrical control boxes or equipment panels, etc., and the characteristics of its spatial arrangement include but are not limited to:

[0072] There is no uniform size standard between targets, the spatial distance is uneven, and there is a lack of uniform identification;

[0073] The shooting direction of the inspection robot in the collection process is deflected, and the presentation structure of the target in different frames of images is significantly different;

[0074] The target boundary in the image may be connected with the background or interfered by occlusion, and it is difficult to statically determine the target image belonging to;

[0075] It can be understood that the embodiment realizes dynamic identification of the inspection target without presetting target coordinates or labels by performing structural feature analysis and spatial position alignment on the first-stage collected image. The system clusters and attributes local visual structures (including nameplate characters, terminal rows, structural joints, etc.) extracted from the image, and classifies images with consistent structures in consecutive frames into the same inspection target image set.

[0076] Further, based on the image set, the optimal shooting configuration for obtaining target information is derived through structural sensitivity analysis and observation direction inversion; combined with illumination gradient analysis, the shooting angle is calibrated, and the shooting angle is input as a node into a path optimization module to generate the second-stage inspection path.

[0077] Next, a path planning method for an inspection robot provided by the embodiment of the application will be introduced in combination with the drawings. Figure 2 The method shown includes the following S1-S4, and the specific steps are as follows:

[0078] S1: performing first-stage inspection according to a preset initial electronic map to obtain first image data and corresponding pose data;

[0079] In the embodiment, the inspection robot performs image collection tasks according to a set stepping rule in a track path or a specified movement area based on an electronic map configured before task deployment, and records the spatial pose information of each image frame synchronously in the collection process. The first image data is a sequence of images obtained at a stepping interval along the inspection track, and the pose data includes information such as displacement, attitude and shooting direction corresponding to the shooting time, which is used for subsequent image alignment and target attribution judgment.

[0080] It can be understood that the purpose of the first-stage inspection is to construct a set of image data with spatial reference under the premise of unknown inspection target position, to provide an original basis for target identification and information inversion. The electronic map only provides a navigation path framework, and the application does not rely on preset target coordinates for subsequent modeling.

[0081] S2: processing the first image data, identifying the inspection target, and registering the first image data obtained from multiple shooting positions for each inspection target to obtain second image data;

[0082] In the embodiment, the target detection network trained on the power equipment image data set is used to extract the first image data for the inspection target candidate region; and the edge feature and texture matching method are combined to construct the judgment standard of the inspection target, to exclude background interference and non-structural feature region. For each identified inspection target, image segments thereof are obtained from the corresponding multiple shooting positions, and registration is performed by using the pose relationship between the images to form a unified image set of the target under different viewing angles as the second image data.

[0083] S3: determining, based on the second image data, a shooting position and a shooting angle for acquiring inspection target information as a shooting configuration of a corresponding inspection target;

[0084] In the embodiment, first, typical local visual structure features such as nameplate characters, terminal rows or device corner gaps are extracted in the second image data, and pixel gradients and structure definition in different view images are calculated; the spatial direction response relationship of each structure is derived in combination with the pose information, and a structure sensitivity distribution model is established. Subsequently, in the spatial sensitivity distribution field, the structure response peak direction is identified based on the gradient change path, and the light interference level is judged in combination with the brightness change trend, and the automatic calibration of the shooting view angle is realized. The finally output shooting angle and the corresponding spatial position constitute the shooting configuration of the target.

[0085] S4: inputting the shooting configurations of all inspection targets as path nodes into a preset path optimization model, and outputting an inspection path of the inspection robot;

[0086] In the embodiment, the shooting position and angle corresponding to each inspection target are taken as path nodes, and a cost function is constructed based on the spatial movement distance between the nodes, the view angle switching amplitude and the light change degree; through a multi-objective optimization method, the total length of the path, the view angle continuity and the stability of the shooting conditions are comprehensively considered, and the optimal inspection sequence and movement path are solved, and a task trajectory for the second stage execution is generated. The path planning model can select an optimization algorithm such as genetic algorithm, ant colony optimization or local search, and the running data can be collected according to the actual scene setting, which is not limited in the present application.

[0087] It can be understood that the inspection path in the present application can be an initial planning path autonomously generated by the robot before the first task execution, or a daily maintenance path dynamically updated based on existing task data, and the specific path planning strategy can be flexibly configured according to factors such as deployment frequency, target change frequency and scene stability.

[0088] As can be appreciated by those skilled in the art, the final formation of the inspection path not only depends on the spatial distribution of the shooting configurations of the inspection targets, but also is affected by non-structural factors such as dynamic environmental conditions, so the path optimization strategy should have certain dynamic adaptability and configurability to adapt to the diversified actual inspection scene requirements.

[0089] In an optional embodiment, the method further comprises:

[0090] S5: generating a motion control instruction according to the inspection path to control the inspection robot to perform an inspection task according to the inspection path.

[0091] In actual outdoor power inspection applications, target devices are usually distributed in irregular terrains or non-standard erection environments, and the inspection path and shooting strategy often cannot completely rely on the pre-constructed high-precision map or static target coordinate system. Traditional path planning methods mostly arrange inspection points based on pre-set target coordinates or structured region division, which requires detailed modeling of the scene before system operation. Obviously, this has limitations in the initial deployment or scenarios where device locations frequently change. In addition, in conventional image acquisition methods, the shooting configuration for a certain inspection target usually uses fixed parameters or empirical presets, which fail to fully consider external disturbance factors such as spatial perspective differences formed by actual image acquisition, target surface structure characteristics, and changes in lighting, resulting in unstable reuse of shooting results over a long period, frequent adjustment of retakes, and difficulty in balancing inspection efficiency and image quality.

[0092] Taking outdoor busbar inspection in a substation as an example, such targets generally have characteristics such as surface reflection, structure cable shielding, and dense spatial layout. If the shooting point selection logic based on geometric models is directly used, it is often difficult to balance image quality and feasibility of passing. However, the present application automatically adjusts shooting parameters and poses by starting from multi-view registration images, reversely judging image structure integrity and brightness disturbance trends under different perspectives, without external human annotation or fixed coordinate system support, to realize autonomous modeling of target structure levels and generation of optimal shooting configurations.

[0093] Next, the part of the method of the present application about the first image data processing is further expanded.

[0094] Please refer to Figure 3 , Figure 3 The first image data processing flowchart of the embodiment of the present application is shown. Figure 3 The processing of the first image data of the present application includes two parts of inspection target recognition and data registration.

[0095] It can be understood that the first image data of the present application refers to all images obtained by the inspection robot during the first inspection stage, and is sorted according to the shooting timestamp. When the same inspection target is identified, the data registration part defaults to registering the first image data of the same inspection target.

[0096] In one example, the specific steps of inspection target recognition are as follows:

[0097] S2.1: Process the first image data by a pre-trained target detection model to extract inspection target candidate regions, wherein the target detection model is a convolutional neural network structure trained on a power equipment image dataset;

[0098] In this embodiment, the target detection model adopted is a pre-trained convolutional neural network structure, and the training data thereof comes from an image dataset with power equipment features. The dataset covers multiple common power equipment forms, including but not limited to switch cabinets, cable boxes and terminal blocks, and has multiple angles, different illuminations, occlusion interference and other diversities, so that the model can adapt to complex actual inspection scenes. The network model can be selected from YOLOv5, Faster R-CNN or RetinaNet, etc. with target positioning capability, and through end-to-end regression detection, the rectangular frame information of all possible electrical targets in the image is output as a candidate region.

[0099] S2.2: According to the inspection target candidate region, the corresponding contour feature and texture feature are calculated through edge detection, and the contour feature and texture feature are matched with a preset template library to identify the inspection target.

[0100] Specifically, edge feature extraction is performed inside the candidate region. In an optional embodiment, feature extraction is performed through a Canny operator or a Sobel operator to construct the contour boundary shape of the target.

[0101] Further, the texture feature of the candidate region is extracted through a gray level co-occurrence matrix to describe the surface structure characteristics of the target region in the image space.

[0102] Further, the contour feature and the texture feature are compared with a locally maintained template library. The template library pre-collects and organizes multi-view image features of standard target equipment, and establishes a corresponding structure feature archive for each type of equipment.

[0103] Further, in the comparison process, the similarity between the candidate region and the template can be evaluated based on multiple indexes such as shape matching coefficient, texture direction consistency score and edge closure degree through multi-dimensional similarity matching logic. If the similarity exceeds a preset similarity threshold, the candidate region is confirmed as an effective inspection target region, and is marked as a specific target number. The similarity threshold can be determined by experiment by those skilled in the art.

[0104] It can be understood that the number of inspection targets is determined by the specific application scenario, and the identification of the inspection target in the present application is based on the existing related technology, and the number, type or appearance structure of the inspection target is not limited.

[0105] In one example, the specific steps of data registration are as follows:

[0106] S2.3: According to the time stamp and pose information when the first image data is collected, an image sequence for the same inspection target is determined, and each first image data in the image sequence corresponds to a different shooting position.

[0107] S2.4: registering each first image data in the image sequence by feature point matching to obtain second image data, wherein the feature point matching comprises corner point extraction on the first image data, construction of local descriptors, and acquisition of corresponding point pairs between images according to the local descriptors through bidirectional matching;

[0108] In this embodiment, the first step of registration is to preliminarily aggregate image frames with similar shooting times, high consistency in shooting directions, and pose differences within an acceptable threshold range to form a candidate image sequence through time stamps and camera pose data accompanying acquisition. The aggregation process can exclude image frames in which the target deviates from the center of the field of view or completely moves out of the field of view due to robot movement, thereby improving the quality of subsequent image matching while ensuring target consistency. Next, for each image in the image sequence, a stable corner point feature is extracted using a Harris corner point or FAST feature point extraction operator, and a scale-invariant local descriptor is constructed in the neighborhood of the feature point, such as a lightweight descriptor like ORB, BRIEF, or AKAZE to adapt to edge computing scenarios. A bidirectional matching strategy is used to match image frames in the image sequence two by two.

[0109] Further, after completing the feature point matching, a RANSAC algorithm is used to estimate the homography matrix between images, and a unified coordinate transformation is performed on all image frames to project them into the reference frame coordinate system, thereby constructing second image data with consistent viewing angles and high boundary overlap.

[0110] As a preferred option, the estimated results can be optimized in combination with pose information to make the transformation parameters more consistent with the actual shooting state and avoid matching drift problems caused by weak texture or repeated structures in the image content.

[0111] Next, the part of the method of the present application related to shooting configuration is further expanded.

[0112] It can be understood that the second image data obtained by the present application refers to the corresponding image of each inspection target after inspection target identification and registration. The second image data can be a single image after image stitching or multiple images, which is not limited by the present application.

[0113] Further, the present application obtains the shooting configuration of all inspection targets by traversing the second image data, and the present embodiment is expanded with the shooting configuration of a single inspection target.

[0114] As can be appreciated by those skilled in the art, in the process of applying the inspection robot to an outdoor scene, the spatial layout of the inspection target often has complexity and irregularity. Especially in the engineering site lacking unified installation specifications and explicit identification, the distribution between the inspection targets not only has the problems of unequal distance and misalignment, but also may have different degrees of missing or noise interference of visual information due to external occlusion, background interference or natural aging. Under this background, if only relying on the initial electronic map or static pose information to preset the shooting position and angle, it is easy to cause the problems of out-of-focus, local missing of the target, insufficient information, etc., thereby affecting the accuracy and usability of the inspection data.

[0115] Based on the image sequence with completed registration in the second image data, without relying on external three-dimensional modeling equipment, the most information valuable observation direction and shooting position are autonomously determined by extracting the local visual features of the inspection target, analyzing the structural sensitivity, and inversely inferring the lighting information. The local visual features include but are not limited to typical structural units such as nameplate characters, terminal interfaces, and boundary gaps. These units usually have strong task relevance and image expression sensitivity in actual use, and are important basis for determining the equipment state and structural changes. By introducing the combination of structural sensitivity distribution inversion and lighting interference calibration, not only the effectiveness of the shooting angle is improved, but also the potential impact of the actual lighting conditions on the image quality is taken into account, thereby realizing the automatic reasoning and precise positioning of the optimal shooting configuration without complex three-dimensional modeling or laser scanning.

[0116] For example, the second image data can be understood with reference to Figure 4 Figure 4 The principle diagram of the second image processing of the embodiments of the present application is shown.

[0117] Figure 4 The extraction of the corresponding structural characteristics of the local visual feature region in the second image data is shown. The structural sensitivity distribution takes the viewing angle as the horizontal axis and the structural sensitivity in the structural response index as the vertical axis, which can be represented as a one-dimensional curve (as shown in Figure 4 ) or a two-dimensional / three-dimensional distribution graph. The two-dimensional form is used to represent the response change of the local feature under the two angle dimensions of pitch angle and azimuth angle, and the three-dimensional structural sensitivity distribution further introduces the distance index to form a complete observation space sensitivity map, which is not limited by the present application.

[0118] Figure 4 The inversion process of the structural sensitivity distribution is further shown. As can be understood, in the inversion process, the structural response path represents the optimal region of the imaging response of the current local visual feature under a specific shooting angle. In a typical case, it should at least include two extreme points, such as Figure 4 ​The extreme point one and the extreme point two in the gradient field correspond to structural clear peak values in different spatial directions. Embodiments identify the sequence with the maximum structural change rate in the gradient rising path through structural response path analysis, and select the direction connected with the extreme point as the first observation direction in combination with the connectivity of the local feature position and the observation direction.

[0119] Figure 4 Further, the distribution of brightness fluctuation in the adjacent area of the first observation direction is shown. The adjustment vector with small light disturbance and high direction consistency is identified by constructing a brightness fluctuation trend vector field, and is combined with the first observation direction to calibrate the final second observation direction. Under the premise of maintaining the structural response advantage, the problem of image quality degradation caused by outdoor natural light interference or high-contrast shadow area is avoided to the greatest extent, thereby improving the stability and practicality of the shooting angle.

[0120] In one example, the specific steps of S3 are as follows:

[0121] S3.1: For each inspection target, local visual features corresponding to the inspection target are identified in the second image data, wherein the local visual features include nameplate characters, connection terminals, and device seams.

[0122] Specifically, due to the complex surface structure of the device, it is difficult to locate the key inspection area only by relying on the overall contour, and it is necessary to accurately extract small visual features such as nameplate characters, connection terminals, and device seams to ensure subsequent positioning and image clarity judgment.

[0123] As understood by those skilled in the art, the identification method of the local visual features can be realized by existing technologies, for example, by a joint method of feature point detection based on a convolutional neural network and traditional edge detection, the CNN trained on power equipment data is used to extract candidate regions, and then Canny edge and texture filtering operations are performed on each region to generate a fine local visual feature mask. This application does not repeat here.

[0124] S3.2: Perform structural sensitivity analysis on the local visual features to obtain a structural sensitivity distribution.

[0125] Specifically, the image performance (clarity, boundary integrity, detail contrast) of the local visual features under different shooting angles is significantly different, and only typical angles will cause missed detection or false alarm, and the response changes need to be quantified from multiple perspectives.

[0126] In the embodiment, for the positioned feature region, pixel-level gradient and texture arrangement analysis is performed within a preset view angle window, image detail retention, edge connectivity and confidence index are compared frame by frame, and a structure sensitivity curve is generated. Each structure sensitivity curve corresponds to a feature, and the structure sensitivity distribution is further normalized by the slope change in the view angle dimension to generate a structure sensitivity distribution.

[0127] Further, the structure sensitivity distribution can be expressed in two-dimensional or three-dimensional space, fully characterizing the observability of device details in spatial angle, and providing data support for the inversion of stable and effective observation directions.

[0128] In one example, structure sensitivity analysis is performed on the local visual features to obtain a structure sensitivity distribution, including:

[0129] Calculating the pixel gradient and spatial arrangement features of the local visual features;

[0130] Comparing the image structure changes of the pixel gradient and spatial arrangement features frame by frame to obtain a structure change amount, wherein the structure change amount includes a definition change, a structure integrity change, and an image detail retention;

[0131] According to the change trend of the structure change amount in the view angle dimension, a corresponding sensitivity curve is constructed, and a structure sensitivity distribution is generated according to the slope of the sensitivity curve.

[0132] S3.3: Inverting a first observation direction of the local visual features according to the structure sensitivity distribution, wherein the first observation direction corresponds to a shooting angle that maximizes the comprehensive value of the structure sensitivity of a plurality of local visual features;

[0133] As can be appreciated by those skilled in the art, if the maximum value of the structure sensitivity distribution is simply selected, it is easy to be affected by local noise. The purpose of the present application is to find the angle that optimally combines structure definition and integrity through path continuity and gradient ascent analysis, to ensure that the inverted direction is realizable.

[0134] It can be understood that path continuity ensures that the direction selection does not deviate from the optimal shooting interval due to accidental feature enhancement, and the gradient ascent mechanism excavates the stable growth area from the overall change trend, so that the inverted direction has both global optimality of visual definition and continuous feasibility of shooting posture.

[0135] In one example, according to the structure sensitivity distribution, a first observation direction of the local visual features is inverted, including:

[0136] S3.3.1: combine the sensitivity values corresponding to each shooting direction in the structure sensitivity distribution into a set of spatial sensitivity points, and calculate the structure sensitivity change rate between each shooting direction according to the set of spatial sensitivity points to generate a spatial sensitivity gradient field;

[0137] Specifically, a single extreme value may be derived from noise or local microstructure differences, and if the highest point is directly selected, it is easy to cause inversion direction deviation or fracture, which is not conducive to the next step of observation stability. By mapping the discrete sensitivity values into a spatial gradient field, irrelevant floating can be smoothed and the overall trend can be retained, and a continuous search basis is provided for subsequent path tracking.

[0138] In this embodiment, for each perspective direction containing the target in the second image data, a structure sensitivity value is calculated by uniformly sampling at a preset angle step; each sampling perspective and its sensitivity value are mapped to a three-dimensional direction coordinate system to form a set of spatial sensitivity points; then the sensitivity difference between adjacent directions is divided by the angle difference to form a discrete change rate, and a spatial sensitivity gradient field is obtained through a three-dimensional interpolation algorithm. Each point in the spatial sensitivity gradient field has both absolute sensitivity and gradient change trend, which serves as an input for structure response path search.

[0139] It can be understood that the spatial sensitivity gradient field can shield isolated noise and take into account the overall trend of the structure, retaining not only the sensitivity change trend between angles, but also forming a smooth and traceable field, effectively supporting multi-directional parallel search and reducing inversion deviation caused by local data anomalies.

[0140] S3.3.2: in the spatial sensitivity gradient field, calculate a structure response path based on a gradient ascent direction, wherein the structure response path is a sequence of directions with the largest response change in the gradient ascent path;

[0141] Specifically, there may be multiple local ascent channels on different paths, and only relying on local growth rate cannot comprehensively consider all structure features. By constructing a response path and selecting the optimal sequence, it can be ensured that the selected direction is the path endpoint with the best comprehensive performance of structure edge definition and integrity.

[0142] In this embodiment, multiple initial perspectives are randomly selected as starting points, and the next direction with the largest sensitivity gain is selected in the neighborhood of each step along the direction of the largest gradient.

[0143] It can be understood that the present application accumulates the total sensitivity gain value along each path and tracks the path continuity; finally, one path that is optimal in overall sensitivity gain, path continuity and gradient curve smoothness is selected as the structure response path.

[0144] S3.3.3: performing rate analysis on the structural response path to obtain an extreme point of a structural sensitivity rate of change, and taking a direction corresponding to the extreme point as the first observation direction of the local visual feature;

[0145] Specifically, although the structural response path as a whole shows an upward trend, in practice, the rate of improvement of structural definition and integrity often increases sharply at certain local angles, rather than continuously and linearly. Therefore, selecting the path endpoint based only on the total amount of sensitivity may result in the shooting angle falling into a slow but smooth segment, losing the high response viewing angle with clear structural boundaries and significant texture contrast. The present application accurately excavates the sensitivity mutation position in the path through rate analysis, to reflect the real non-linear relationship between the observation angle and the image structure response.

[0146] In the present embodiment, for the obtained structural response path, the path is divided into several angle stepping segments, and the structural sensitivity increment and the angle increment ratio between adjacent segments are calculated in turn to form a structural sensitivity rate of change curve. Subsequently, first derivative analysis is performed on the curve to extract all local extreme points, and effective extreme points with a rate increase exceeding a preset threshold are filtered out. It should be noted that, in view of structural continuity and feature diversity, a single extreme value is not taken as the basis for the first observation direction, but the direction vectors corresponding to all effective extreme points are subjected to weighted average processing. The weighting process can be based on the sensitivity gain amplitude, the structural detail improvement rate and the gradient field continuity stability corresponding to each extreme point to generate the final first observation direction.

[0147] Further, if the included angle between the direction vectors corresponding to multiple local extreme points deviates significantly, exceeding a set tolerance range, then the isolated extreme points with inconsistent directions are preferentially removed to avoid disturbing the stability of the direction synthesis. The final first observation direction not only integrates the local advantages of multiple structural response peaks, but also retains the structural information continuity and the realizability of the shooting viewing angle, ensuring that the selected direction has the best recognition conditions and higher path robustness during execution.

[0148] S3.4: calibrating the first observation direction according to light information in the second image data to determine a second observation direction, taking the second observation direction as the shooting angle, and taking an intersection point of the second observation direction and the initial electronic map as the shooting position, wherein the light information is a luminance variation range of the second image data under different viewing angles;

[0149] Specifically, the first observation direction is usually the optimal angle of structural response, and in outdoor variable lighting, it may also be blurred due to strong light or shadow, so the angle needs to be adjusted based on the luminance variation statistics to improve the image quality and stability.

[0150] In the embodiment, a plurality of view angle frames adjacent to the first observation direction are sampled, a corresponding luminance gradient sequence is extracted, a fluctuation amplitude is counted and compared with a sequence mean value, and when the fluctuation is lower than the mean value, the original vector is retained; otherwise, a luminance disturbance trend vector field is constructed from the adjacent view angles, a distribution trend with the smallest fluctuation and the highest consistency with the structural sensitivity direction is identified, and a calibration vector is calculated according to the trend.

[0151] Further, the calibration vector and the first observation direction are linearly combined by a preset offset coefficient, and a second observation direction is obtained after normalization of the result; the second observation direction takes into account the strongest structural sensitivity and the smallest light disturbance, not only improves the detail clarity, but also avoids the visual occlusion caused by light and shadow, thereby improving the stability of subsequent intelligent inspection and maintenance work.

[0152] In one example, the first observation direction is calibrated according to the light information in the second image data to determine the second observation direction, including:

[0153] S3.4.1: Extract luminance gradient values corresponding to the light information from a plurality of adjacent view angles of the first observation direction to obtain a local luminance change sequence;

[0154] S3.4.2: Calculate the luminance fluctuation amplitude corresponding to the first observation direction, and if the luminance fluctuation amplitude is less than the amplitude mean value of the local luminance change sequence, the first observation direction is taken as the second observation direction;

[0155] S3.4.3: If the luminance fluctuation amplitude is greater than or equal to the amplitude mean value of the local luminance change sequence, a luminance fluctuation trend vector field is constructed based on the luminance change sequence;

[0156] S3.4.3: Calculate the direction distribution trend with the smallest luminance disturbance amplitude in the luminance fluctuation trend vector field, and calculate a calibration vector according to the direction distribution trend, wherein the calibration vector represents the smallest light interference adjustment direction relative to the first observation direction;

[0157] S3.4.4: Calibrate the first observation direction according to the calibration vector to obtain a second observation direction;

[0158] Specifically, in the actual inspection environment, the external natural light condition is difficult to keep stable, and the target surface material, angle and reflection characteristic difference may also cause brightness gradient mutation, which directly affects the recognizability of the structural features in the image. If the shooting angle and the light direction exist high reflection interference or shadow superposition, it is easy to cause the local visual feature boundary to be blurred and the texture to be distorted, thereby interfering with the subsequent information extraction and recognition accuracy. Therefore, although the first observation direction has the optimal performance in the structural information sensitivity, it still needs to be combined with the light distribution for disturbance analysis and direction calibration to avoid the perception misjudgment caused by the light deviation and ensure the stability and practicality of the inspection image.

[0159] In the embodiment, for the selected first observation direction, a plurality of adjacent shooting viewing angles with equidistant angular offset are set around it, and the brightness gradient information of the corresponding image region is extracted from the second image data, and then a one-to-one correspondence relationship between the viewing angle and the brightness change is formed. Each adjacent viewing angle outputs a brightness gradient value sequence, and the brightness change amplitude obtained from the sequence is recorded as a local brightness change sequence. Then, the brightness fluctuation amplitude of the first observation direction itself in the sequence is calculated as a quantitative indicator of the light stability under the direction.

[0160] Further, if the brightness fluctuation amplitude of the first observation direction is less than the average amplitude of the local brightness change sequence, it indicates that the direction has a relatively stable brightness response under the current light condition, and no additional direction adjustment is needed, which can be directly used as the final shooting angle. If the detection result is the opposite, it indicates that although the direction is excellent in structural sensitivity, it has fluctuations in light response, which may pose a risk to the image clarity, and at this time the light interference calibration mechanism needs to be started.

[0161] Therefore, in the embodiment, a brightness fluctuation trend vector field is constructed based on the entire local brightness change sequence. The vector field takes the viewing angle change as the coordinate axis, maps the brightness gradient fluctuation of each direction into a continuous vector state, and forms a light stability spatial distribution model with the disturbance amplitude as the amplitude. On this basis, directional regression analysis of the vector field is performed to search for a continuous trend area with the smallest direction disturbance, and the main direction of the area is subjected to vector average calculation to output a calibration vector as a reference direction for light adjustment.

[0162] The calibration vector represents the adjustment direction and degree relative to the first observation direction. In actual execution, the calibration vector and the direction vector of the original first observation direction are subjected to linear fusion processing, and the fusion weight can be adaptively allocated according to the disturbance intensity, so as to form a direction fine-tuning vector. The finally output direction vector is normalized and used as the second observation direction.

[0163] In one example, the specific steps of S4 are as follows:

[0164] S4.1: constructing path nodes with shooting positions and shooting angles of each inspection target, and building a cost function between nodes, the cost function including spatial movement distance between nodes, view switching amplitude and shooting light condition change degree;

[0165] S4.2: inputting all path nodes into a multi-objective optimization algorithm to comprehensively minimize total path length, view switching cost and light change interference, and outputting an inspection path meeting shooting continuity constraints, wherein the constraint conditions of the multi-objective optimization algorithm include kinematics constraints, dynamics constraints of the inspection robot and time constraints of the inspection task.

[0166] In one example, motion control instructions are generated according to the inspection path, including:

[0167] According to the path nodes of the inspection path, a control parameter set including a speed curve, a steering angle instruction and a torque instruction is generated in combination with a kinematics model of the inspection robot;

[0168] The control parameter set is converted into an electrical signal and sent to the inspection robot.

[0169] In one example, the present application provides a path planning system for an inspection robot, the system including:

[0170] An image acquisition module for controlling the inspection robot to perform first stage inspection according to an initial electronic map, and acquiring first image data and corresponding pose data;

[0171] An image processing module for processing the first image data, identifying inspection targets, and registering first image data obtained from multiple shooting positions of the same inspection target to obtain second image data;

[0172] A shooting configuration generation module for performing structure sensitivity analysis and observation direction inversion based on local visual features in the second image data, and calibrating the first observation direction in combination with light information to determine shooting positions and shooting angles of each inspection target as shooting configurations;

[0173] A path planning module for constructing a cost function with shooting configurations of each inspection target as path nodes, and generating an inspection path through a multi-objective optimization algorithm;

[0174] A control execution module for sending the inspection path to the inspection robot and controlling it to sequentially complete target information acquisition according to the inspection path.

[0175] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A path planning method for a patrol robot, characterized by, The method comprises: According to the preset initial electronic map, the first stage inspection is driven to obtain the first image data and the corresponding pose data; The first image data is processed to identify the inspection target, and the first image data obtained from multiple shooting positions for each inspection target is registered to obtain the second image data; Based on the second image data, the shooting position and the shooting angle for obtaining the inspection target information are determined as the shooting configuration of the corresponding inspection target; The shooting configuration of all inspection targets is input into the preset path optimization model as the path node, and the inspection path of the inspection robot is output, wherein the path optimization model generates the inspection path through a multi-objective optimization algorithm, and the multi-objective optimization algorithm includes the kinematic constraint of the inspection robot; According to the inspection path, the motion control instruction is generated to control the inspection robot to perform the inspection task according to the inspection path; Determination of the shooting position and the shooting angle for obtaining the inspection target information comprises: For each inspection target, the local visual features of the corresponding inspection target are identified in the second image data, wherein the local visual features include nameplate characters, connection terminals and device joints; The structure sensitivity analysis is performed on the local visual features to obtain the structure sensitivity distribution; According to the structure sensitivity distribution, the first observation direction of the local visual features is reversed, wherein the shooting angle corresponding to the first observation direction is the shooting direction that makes the comprehensive value of the structure sensitivity of the multiple local visual features reach the maximum value; According to the illumination information in the second image data, the first observation direction is calibrated to determine the second observation direction, the second observation direction is taken as the shooting angle, and the intersection of the second observation direction and the connection line of the initial electronic map is taken as the shooting position, wherein the illumination information is the brightness change range of the second image data under different angles of view; The structure sensitivity analysis on the local visual features comprises: The pixel gradient and the spatial arrangement features of the local visual features are calculated; The image structure changes of the pixel gradient and the spatial arrangement features are compared frame by frame to obtain the structure change amount, wherein the structure change amount includes the definition change, the structure integrity change and the image detail retention degree; According to the change trend of the structure change amount in the angle of view dimension, the corresponding sensitivity curve is constructed, and the structure sensitivity distribution is generated according to the slope of the sensitivity curve; According to the structure sensitivity distribution, the first observation direction of the local visual features is reversed, comprising: The sensitivity values corresponding to each shooting direction in the structure sensitivity distribution are combined into a spatial sensitivity point set, and the structure sensitivity change rate between each shooting direction is calculated according to the spatial sensitivity point set to generate a spatial sensitivity gradient field; In the spatial sensitivity gradient field, the structure response path is calculated based on the gradient rising direction, wherein the structure response path is the direction sequence with the maximum response change in the gradient rising path. Performing rate of change analysis on the structural response path to obtain an extreme point of a structural sensitivity rate of change, and taking a direction corresponding to the extreme point as a first observation direction of the local visual feature; Calibrating the first observation direction according to illumination information in the second image data to determine a second observation direction, including: Extracting a brightness gradient value of the corresponding illumination information according to a plurality of adjacent visual angles of the first observation direction to obtain a local brightness change sequence; Calculating a brightness fluctuation amplitude corresponding to the first observation direction, and taking the first observation direction as the second observation direction if the brightness fluctuation amplitude is less than an average amplitude of the local brightness change sequence; If the brightness fluctuation amplitude is greater than or equal to the average amplitude of the local brightness change sequence, constructing a brightness fluctuation trend vector field based on the local brightness change sequence; Calculating a direction distribution trend of a minimum brightness disturbance amplitude in the brightness fluctuation trend vector field, and calculating a calibration vector according to the direction distribution trend, wherein the calibration vector represents a minimum illumination interference adjustment direction relative to the first observation direction; Calibrating the first observation direction according to the calibration vector to obtain the second observation direction. 2.The path planning method for a patrol robot according to claim 1, wherein, Inputting shooting configurations of all inspection targets to a preset path optimization model as path nodes to output an inspection path of the inspection robot, including: Constructing path nodes from shooting positions and angles of each inspection target, and constructing a cost function between nodes, the cost function including a spatial movement distance between nodes, a visual angle switching amplitude, and a shooting illumination condition change degree; Inputting all path nodes to a multi-objective optimization algorithm to comprehensively minimize an inspection total path length, a visual angle switching cost, and an illumination change interference influence, and outputting an inspection path satisfying a shooting continuity constraint, wherein constraint conditions of the multi-objective optimization algorithm include kinematics constraints, dynamics constraints of the inspection robot, and time constraints of the inspection task. 3.The path planning method for a patrol robot according to claim 1, wherein, Generating a motion control instruction according to the inspection path, including: Generating a control parameter set including a speed curve, a steering angle instruction, and a torque instruction according to path nodes of the inspection path in combination with a kinematics model of the inspection robot; Converting the control parameter set into an electrical signal and sending the electrical signal to the inspection robot. 4.The path planning method for a patrol robot according to claim 1, wherein, Processing the first image data to identify an inspection target, including: Processing the first image data through a pre-trained target detection model to extract an inspection target candidate region, wherein the target detection model is a convolutional neural network structure trained on a power equipment image data set; According to the inspection target candidate region, calculating corresponding contour features and texture features through edge detection, and performing similarity matching on the contour features and the texture features with a preset template library to identify an inspection target.

5. The path planning method for a patrol robot according to claim 4, wherein, Registering first image data obtained from a plurality of shooting positions for each inspection target to obtain second image data, including: Determining an image sequence for a same inspection target according to a timestamp and pose information when the first image data is collected, each first image data in the image sequence corresponding to a different shooting position; The first image data in the image sequence are registered to obtain the second image data by feature point matching, wherein the feature point matching comprises corner point extraction on the first image data, construction of local descriptors, and acquisition of corresponding point pairs between images according to the local descriptors through bidirectional matching.

6. A path planning system for a patrol robot, for implementing a path planning method for a patrol robot according to any one of claims 1-5, characterized in that, The system comprises: An image acquisition module configured to control the inspection robot to perform first-stage inspection according to the initial electronic map, and acquire first image data and corresponding pose data; An image processing module configured to process the first image data, identify inspection targets, and register first image data of the same inspection target acquired from multiple shooting positions to obtain second image data; A shooting configuration generation module configured to perform structure sensitivity analysis and observation direction inversion based on local visual features in the second image data, calibrate the first observation direction in combination with illumination information, and determine shooting positions and shooting angles of each inspection target as shooting configurations; A path planning module configured to construct a cost function by taking the shooting configurations of the inspection targets as path nodes, and generate an inspection path through a multi-objective optimization algorithm; A control execution module configured to send the inspection path to the inspection robot, and control the inspection robot to sequentially complete target information acquisition according to the inspection path.

Images