Inspection robot inspection action planning method and device, electronic equipment and medium

Abstract

This application provides a method, apparatus, electronic device, and medium for planning the inspection actions of an inspection robot. The method involves determining the robot's degrees of freedom description information; determining the camera's feasible area based on the robot's inspection path, base mobility information, and camera mobility information; determining the ideal camera position based on the camera's feasible area and the target object information of the target object to be captured; determining at least one pose information of the robot based on the ideal camera position; determining corresponding load state information and camera state information for each pose information; and determining at least one combination of action planning information. From the at least one combination of action planning information, a target action planning combination is selected, and the robot is adjusted based on the target action planning combination to capture the target image of the target object. This improves the efficiency of the inspection robot's action planning and the accuracy of image acquisition.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to inspection action planning methods, devices, electronic equipment and media for inspection robots. Background Technology

[0002] Inspection robots can be applied in various industrial or smart fields. For example, for industrial facilities with a large number of factories, power substations, etc., inspection robots can be used to collect images of various targets that require visual inspection, such as dashboards, indicator lights, valves, nameplates, insulators, etc., so as to analyze the operation status of industrial facilities based on the collected images.

[0003] Inspection action planning for inspection robots refers to planning the robot's position, load status, and camera status as it moves along a given inspection path, thereby accurately acquiring images of the target object. In existing technologies, inspection action planning for inspection robots is generally pre-set by staff based on their planning experience. This method of setting up inspection robots cannot adapt to different acquisition scenarios and acquisition types, requiring different inspection action planning schemes for different types of inspection robots. This results in low efficiency in inspection action planning. Furthermore, due to the varying experience of staff, the quality of the target object acquired by the robot is inconsistent, further affecting the accuracy of image acquisition by the inspection robot. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, device, electronic device and medium for planning the inspection action of an inspection robot. By constructing the degree-of-freedom description information of the inspection robot, combining the target object information of the target object in the image to be acquired and the inspection path of the inspection robot, the feasible area of ​​the camera of the inspection robot is automatically determined. The ideal camera position with the best acquisition effect is determined through the camera feasible area. Then, the pose information, load state information and camera state information of the inspection robot are determined according to the ideal camera position. Finally, the optimal target action planning combination is determined. This reduces the error of manual planning that cannot be combined with the specific acquisition scene and the low efficiency of manual planning for different types of inspection robots, thereby improving the efficiency of inspection robot inspection action planning and the accuracy of image acquisition.

[0005] In a first aspect, embodiments of this application provide an inspection action planning method for an inspection robot, the inspection action planning method comprising:

[0006] Determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters of the camera;

[0007] Determine the target object information of the target object to be captured in the image and the inspection path of the inspection robot.

[0008] Based on the inspection path, the base mobility information, and the camera mobility information, the camera feasible area of ​​the inspection robot is determined;

[0009] Based on the camera's feasible area and the target object information, the ideal camera position of the inspection robot is determined;

[0010] Based on the ideal camera position, at least one pose information of the inspection robot is determined. For each pose information, corresponding load state information and camera state information are determined. The pose information, load state information and camera state information are combined to determine at least one action planning information combination.

[0011] From at least one combination of action planning information, a target action planning combination is selected, and the pose, load state, and camera state of the inspection robot are adjusted based on the target action planning combination before acquiring a target image of the target object.

[0012] In one possible implementation, determining the camera-feasible area of ​​the inspection robot based on the inspection path, the base mobility information, and the camera mobility information includes:

[0013] Based on the position information of the camera of the inspection robot on the inspection path, the inspection path, and the movable range of the camera as represented by the camera's movable information, the feasible area of ​​the camera of the inspection robot is determined.

[0014] In one possible implementation, the target object information includes the target object center and the target object's directional attribute information; determining the ideal camera position of the inspection robot based on the camera's feasible area and the target object information includes:

[0015] If the directional attribute information indicates that the target object is a target object with directional attributes, the normal ray is determined based on the center of the target object and the normal vector of the main surface;

[0016] Detect whether there is an intersection between the normal ray and the feasible region of the camera;

[0017] If there is an intersection between the normal ray and the feasible area of ​​the camera, the location of the intersection between the normal ray and the feasible area of ​​the camera is determined as the ideal camera position;

[0018] If there is no intersection between the normal ray and the camera's feasible region, the ideal camera position is determined in the camera's feasible region based on the incident angle.

[0019] In one possible implementation, determining the ideal camera position for the inspection robot based on the camera's feasible area and the target object information further includes:

[0020] If the directional attribute information indicates that the target object is a target object without directional attributes, the location closest to the target object in the camera's feasible area is determined as the ideal camera position; or...

[0021] Determine at least one projection position of the target object within the camera's feasible area, and determine the projection position with the largest area among the projection positions as the ideal camera position.

[0022] In one possible implementation, determining at least one pose information of the inspection robot based on the ideal camera position includes:

[0023] Within a preset range of the ideal camera position, based on the inspection path and the base mobility information, the path position and base orientation are discretely sampled to determine at least one pose information of the inspection robot.

[0024] In one possible implementation, determining the corresponding load state information and camera state information for each of the pose information includes:

[0025] For each of the aforementioned pose information, the load state information is determined by solving based on the structure of the inspection robot;

[0026] Based on the target object's center and the camera's position, determine the adjustment position vector;

[0027] The horizontal rotation angle and vertical pitch angle of the camera are determined based on the adjusted position vector.

[0028] Based on the camera imaging parameters, the optical zoom value of the camera is determined.

[0029] In one possible implementation, selecting the target action plan combination from at least one of the action planning information combinations includes:

[0030] For each action planning information combination, determine the base deviation from the inspection path value, the camera position offset value between the camera position and the ideal camera position, and the incident angle cost value corresponding to that action planning information combination;

[0031] For each action planning information combination, the candidate evaluation value corresponding to the action planning information combination is calculated according to the preset weighting coefficients for the base deviation from the inspection path value, the camera position offset value, and the incident angle value.

[0032] The action planning information combination with the lowest corresponding candidate evaluation value among at least one of the action planning information combinations is determined as the target action planning combination.

[0033] Secondly, embodiments of this application also provide an inspection action planning device for an inspection robot, the inspection action planning device comprising:

[0034] A degree-of-freedom description information determination module is used to determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters of the camera;

[0035] The inspection target determination module is used to determine the target object information of the target object in the image to be collected and the inspection path of the inspection robot.

[0036] The camera feasible area determination module is used to determine the camera feasible area of ​​the inspection robot based on the inspection path, the base mobility information and the camera mobility information;

[0037] An ideal camera position determination module is used to determine the ideal camera position of the inspection robot based on the camera's feasible area and the target object information.

[0038] The action planning information combination determination module is used to determine at least one pose information of the inspection robot based on the ideal camera position, and for each pose information, determine the corresponding load state information and camera state information, and combine the pose information, the load state information and the camera state information to determine at least one action planning information combination.

[0039] The action planning information combination filtering module is used to filter out a target action planning combination from at least one action planning information combination, and after adjusting the pose, load state and camera state of the inspection robot based on the target action planning combination, acquire a target image of the target object.

[0040] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the inspection action planning method for the inspection robot as described in any of the first aspects.

[0041] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the inspection action planning method for the inspection robot as described in any of the first aspects.

[0042] The inspection robot action planning method, apparatus, electronic device, and medium provided in this application embodiment determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters; determine the target object information of the target object to be captured and the inspection path of the inspection robot; determine the camera feasible area of ​​the inspection robot based on the inspection path, base mobility information, and camera mobility information; determine the ideal camera position of the inspection robot based on the camera feasible area and target object information; determine at least one pose information of the inspection robot based on the ideal camera position; for each pose information, determine the corresponding load state information and camera state information, and combine the pose information, load state information, and camera state information to determine at least one action planning information combination; select a target action planning combination from the at least one action planning information combination, and after adjusting the pose, load state, and camera state of the inspection robot based on the target action planning combination, acquire the target image of the target object. In this way, by constructing the degree-of-freedom description information of the inspection robot, combining the target object information of the image to be acquired with the inspection path of the inspection robot, the camera feasible area of ​​the inspection robot is automatically determined. The ideal camera position with the best acquisition effect is determined through the camera feasible area. Then, based on the ideal camera position, the pose information, load status information and camera status information of the inspection robot are determined. Finally, the optimal target action planning combination is determined, reducing the error of manual planning that cannot be combined with the specific acquisition scenario and the low efficiency of manual planning for different types of inspection robots. This improves the efficiency of inspection robot inspection action planning and the accuracy of image acquisition.

[0043] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating an inspection action planning method for an inspection robot provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram illustrating the camera mounting offset effect provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram of the camera feasible area of ​​the wheeled robot provided in an embodiment of this application;

[0048] Figure 4 A schematic diagram of the structure of an inspection robot action planning device provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0051] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of robot control technology.

[0052] Inspection robots can be applied in various industrial or smart fields. For example, for industrial facilities with a large number of factories, power substations, etc., inspection robots can be used to collect images of various targets that require visual inspection, such as dashboards, indicator lights, valves, nameplates, insulators, etc., so as to analyze the operation status of industrial facilities based on the collected images.

[0053] In the existing technology, there are inspection path planning methods developed separately for different types of robots, such as two-dimensional grid path planning methods for wheeled mobile chassis and three-dimensional obstacle avoidance path planning methods for aircraft. However, these methods usually only focus on the motion planning of the robot chassis or body, and lack a unified planning mechanism for how to automatically determine the image state (including robot pose, load state and camera state) on the inspection path.

[0054] Specifically, currently, parameters such as the rotation / tilt angle / zoom magnification of the PTZ camera, and the height or position of movable mechanisms during inspection image acquisition are typically set manually by operators based on prior information such as the robot's structure, camera mounting method, target object location, and site environment. This manual setting method has the following typical problems:

[0055] 1. Reliance on human experience makes standardization and reuse difficult. Different operators have different understandings and settings of parameters such as camera posture and load height, as well as different preferences for image acquisition, resulting in unstable inspection image quality.

[0056] 2. Preset results are highly sensitive to the robot's structure and difficult to transfer. When the inspection robot is changed from one model to another (e.g., from wheeled to legged, or from robotic arm mounting to lifting rod mounting), the original photo-taking posture settings are usually no longer applicable, requiring manual adjustment of a large number of state parameters.

[0057] 3. Lack of automation and optimization capabilities. Existing methods generally lack the ability to automatically evaluate factors such as camera visibility, incident angle, relative position, and imaging scale, which cannot guarantee the stability of inspection quality, nor can they efficiently generate optimal or near-optimal robot inspection poses and load / camera states.

[0058] 4. Difficulty in compatibility with multiple types of robots, resulting in severe coupling between algorithms and hardware. The industry generally adopts the approach of "specific algorithms for specific robots," which makes the system difficult to expand and adapt to the current trend of multi-platform collaborative inspection.

[0059] Therefore, although inspection paths (robot movement trajectories) can be automatically generated using existing technologies, a unified method is still lacking for the specific inspection behaviors along the path—especially the automatic planning of robot pose, load status, and camera status. This deficiency significantly hinders the cross-platform use, batch deployment, and automation improvement of inspection robot systems.

[0060] Based on this, this application provides a method for planning inspection actions of an inspection robot to improve the efficiency of inspection action planning and the accuracy of image acquisition.

[0061] Please see Figure 1 , Figure 1 This is a flowchart illustrating an inspection action planning method for an inspection robot provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the inspection action planning method for the inspection robot includes:

[0062] S101. Determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters of the camera.

[0063] S102. Determine the target object information of the target object in the image to be acquired and the inspection path of the inspection robot.

[0064] S103. Based on the inspection path, the base mobility information, and the camera mobility information, determine the camera feasible area of ​​the inspection robot.

[0065] S104. Based on the feasible area of ​​the camera and the target object information, determine the ideal camera position of the inspection robot.

[0066] S105. Based on the ideal camera position, determine at least one pose information of the inspection robot. For each pose information, determine the corresponding load state information and camera state information, and combine the pose information, the load state information and the camera state information to determine at least one action planning information combination.

[0067] S106. From at least one combination of action planning information, select a target action planning combination, and after adjusting the pose, load state and camera state of the inspection robot based on the target action planning combination, acquire a target image of the target object.

[0068] The inspection action planning method for inspection robots provided in this application constructs the degree-of-freedom description information of the inspection robot, combines the target object information of the target object in the image to be acquired with the inspection path of the inspection robot, automatically determines the camera feasible area of ​​the inspection robot, and determines the ideal camera position with the best acquisition effect through the camera feasible area. Then, based on the ideal camera position, the pose information, load state information and camera state information of the inspection robot are determined, and finally the optimal target action planning combination is determined. This reduces the error of manual planning that cannot be combined with the specific acquisition scene and the low efficiency of manual planning for different types of inspection robots, thereby improving the efficiency of inspection robot inspection action planning and the accuracy of image acquisition.

[0069] The exemplary steps of the embodiments of this application are described below:

[0070] S101. Determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters of the camera.

[0071] Here, the inspection robot is a drone / wheeled / legged movable autonomous robot equipped with image acquisition equipment (pan-tilt camera) to perform inspection tasks. Controllable equipment can be mounted on the inspection robot, which is called the inspection robot's payload. For example, the payload may include a pan-tilt, lifting / translation platform, robotic arm, etc., and may have different degrees of freedom and structures.

[0072] Among them, the pan-tilt camera can be a PTZ (Pan-Tilt-Zoom) camera, which can be mounted on the inspection robot body or load and has controllable rotation angles in both horizontal and vertical directions and variable zoom.

[0073] Furthermore, the inspection robot can perform inspection actions along a given inspection route. These actions are specific actions taken by the robot to achieve the task of acquiring inspection images. Examples include chassis movement and rotation, load movement / behavior, such as PTZ gimbal rotation and image capture, lifting and lowering of the lifting platform, and robotic arm movements.

[0074] Here, the inspection path is the movement path executed by the inspection robot in order to accomplish the inspection image acquisition task.

[0075] In this application embodiment, the inspection robot can be applied in different industrial fields or smart fields. For example, for industrial facilities with a large number of factory workshops, power substations and other facilities, the inspection robot can be used to collect images of various targets that require visual inspection, such as instrument panels, indicator lights, valves, nameplates, insulators and so on, so as to analyze the operation status of the industrial facilities based on the collected images.

[0076] Inspection action planning for inspection robots refers to planning the robot's position, load status, and camera status as it moves along a given inspection path, thereby accurately acquiring images of the target object. In existing technologies, inspection action planning for inspection robots is generally pre-set by staff based on their planning experience. This method of setting up inspection robots cannot adapt to different acquisition scenarios and acquisition types, requiring different inspection action planning schemes for different types of inspection robots. This results in low efficiency in inspection action planning. Furthermore, due to the varying experience of staff, the quality of the target object acquired by the robot is inconsistent, further affecting the accuracy of image acquisition by the inspection robot.

[0077] Based on this, in this embodiment, by constructing the degree-of-freedom description information of the inspection robot, combining the target object information of the target object in the image to be acquired with the inspection path of the inspection robot, the camera feasible area of ​​the inspection robot is automatically determined, and the ideal camera position with the best acquisition effect is determined through the camera feasible area. Then, based on the ideal camera position, the pose information, load status information and camera status information of the inspection robot are determined, and finally the optimal target action planning combination is determined. This reduces the error of manual planning that cannot be combined with the specific acquisition scenario and the low efficiency of manual planning for different types of inspection robots, thereby improving the efficiency of inspection robot inspection action planning and the accuracy of image acquisition.

[0078] In one possible implementation, after determining the type of inspection robot that needs to be controlled, the degree-of-freedom description information of the inspection robot is determined according to the type of inspection robot.

[0079] The degree-of-freedom description information includes the mobility information of the inspection robot's base, the mobility information of the inspection robot's camera, and the camera imaging parameters of the camera.

[0080] Specifically, in the first aspect, when the inspection robot moves along a given inspection path, the robot's base is confined to the given inspection path, can stop at any point on the inspection path, and can rotate freely around the vertical axis. The position of the inspection robot's base can be represented by the inspection path parameter s∈[0,1], and the attitude can be represented by the yaw angle ψ. When the inspection robot moves along the inspection path, the position of the inspection robot from p to p+1 on the inspection path can be represented as: |p+1-p|*s.

[0081] Secondly, the camera mobility information of the inspection robot includes the camera installation offset and range of motion, as well as the range of motion of the PTZ camera gimbal.

[0082] In one possible implementation, the camera mounting offset and range of motion can be determined based on the deviation of the camera center relative to the navigation center of the inspection robot.

[0083] Specifically, the deviation of the camera center from the navigation center of the inspection robot can be determined using the following formula:

[0084] ;

[0085] in, This represents the deviation of the camera center relative to the navigation center of the inspection robot; let the origin of the robot's coordinate system be the robot's navigation center. This represents the offset of the camera center relative to the robot navigation center in each axis within the body coordinate system.

[0086] For example, please refer to Figure 2 , Figure 2 This is a schematic diagram of the camera mounting offset effect provided in the embodiments of this application, such as... Figure 2 As shown, the camera is mounted on the lifting rod, with the origin of the robot's coordinate system as the robot's navigation center. The camera center moves along the lifting rod, causing changes in the Z-axis value. However, the Z-axis movement range of the camera center is within (Z... min Z max Within this framework, based on the different positions of the camera on the lifting rod, the offset of the camera center relative to the robot navigation center in each axis within the body coordinate system is determined. .

[0087] In one possible implementation, for a robot with the camera mounted on a stationary structure, The value is set to a fixed value; for robots with structures such as lifting mechanisms, sliding tables, and robotic arms, the offset of the camera mounted on these movable structures is... It can have a corresponding range of values, and multiple offsets constitute the camera's movable range.

[0088] Here, the camera's movable range Rpayload is all The set of possible values ​​for can be specifically determined using the following formula:

[0089] ;

[0090] Where Rpayload is the camera's movable range; This represents the deviation of the camera center from the navigation center of the inspection robot.

[0091] For example, depending on the structure of the inspection robot, the Rpayload can be a one-dimensional region, a two-dimensional region, or a three-dimensional region, or a subset of the reachable space of the robotic arm's end effector. Specifically, for an inspection robot with the camera mounted on a lifting rod, the one-dimensional region can be a line segment formed by the camera's translation and sliding.

[0092] In another possible implementation, the movable range of the PTZ camera gimbal can be defined based on the horizontal rotation angle range and the vertical pitch angle range of the PTZ camera gimbal; specifically, the movable range of the PTZ camera gimbal can be characterized by the following formula:

[0093] ;

[0094] ;

[0095] in, The horizontal rotation angle of the PTZ camera gimbal; This represents the minimum horizontal rotation angle of the PTZ camera gimbal. This represents the maximum horizontal rotation angle of the PTZ camera gimbal. The vertical pitch angle of the PTZ camera gimbal; This is the minimum vertical pitch angle of the PTZ camera gimbal; This represents the maximum vertical pitch angle of the PTZ camera gimbal.

[0096] Thirdly, camera imaging parameters can be determined based on the camera's horizontal field of view, camera resolution, and aspect ratio.

[0097] Specifically, the camera imaging parameters can be determined using the following formula:

[0098] ;

[0099] in, These are the camera imaging parameters; This represents the minimum horizontal field of view. This represents the maximum horizontal field of view. For camera resolution, The aspect ratio is 1.

[0100] In one possible implementation, once the type of inspection robot is determined, the base mobility information, camera mounting offset and range of motion, PTZ camera gimbal range of motion, and camera imaging parameters of the corresponding inspection robot are all fixed. Therefore, the degree of freedom description information of the inspection robot can be determined.

[0101] Specifically, the degree of freedom description information of the inspection robot can be described by the following formula:

[0102] ;

[0103] in, Describing information for the degrees of freedom of the inspection robot; Information on the movable base of the inspection robot; Install offset and range of motion for the camera; This refers to the movable range of the PTZ camera gimbal. These are the camera imaging parameters.

[0104] Furthermore, after determining the degree of freedom description information of the inspection robot, in order to control the inspection robot to move and determine the target object to be collected and thus carry out an accurate image acquisition process, it is also necessary to determine the target object information of the image to be collected and the inspection path of the inspection robot.

[0105] S102. Determine the target object information of the target object in the image to be acquired and the inspection path of the inspection robot.

[0106] In the context of industrial facilities, target objects can include factory equipment dashboards, power grid insulators, nameplates, and other objects that require visual confirmation of abnormalities.

[0107] Specifically, target objects can include target objects with directional attributes, that is, target objects with a clear viewing angle, such as instrument panels, nameplates, valve scales, flat objects, and structural surfaces with orientation. These objects need to be viewed from the front to collect corresponding numerical information. Target objects can also include target objects without directional attributes, that is, targets without a clear orientation, such as insulators, rotating bodies, cable joints, and pipe connectors. Most of these target objects require observation of their structural features and do not have a clear viewing direction requirement.

[0108] In one implementation, an Oriented Bounding Box (OBB) oriented towards industrial facilities may be used to describe the target object information of the inspected target object.

[0109] In one possible implementation, the target envelope can be calculated based on the eight corner points of the OBB, and used to solve for the target object information of the inspected target object by considering the view frustum constraint and scaling factor.

[0110] Specifically, the target object information can be described using the following formula:

[0111] ;

[0112] in, The target object information; The center of the target object; A quaternion describing the target's pose; A represents the size of the target object; A represents the directional attribute information of the target object.

[0113] Here, when the directional attribute information represents the target object as a target object with directional attributes, it can be described by the principal surface normal vector n. Specifically, n is defined in the world coordinate system; if given in the local coordinate system, it is transformed to the world coordinate system through the attitude quaternion q.

[0114] In one possible implementation, for a target object with directional properties, the directional property information A of the target object can be characterized by the following formula:

[0115] ;

[0116] Where A represents the directional attribute information of the target object; n is the normal vector of the main surface.

[0117] In another possible implementation, for a target object that does not have directional attributes, the directional attribute information A of the target object can be characterized by the following formula:

[0118] ;

[0119] Where A represents the directional attribute information of the target object.

[0120] Furthermore, the inspection path for the inspection robot can be determined through path planning algorithms, manual input by staff, or a high-level inspection task system. The specific determination method is not limited here.

[0121] Specifically, the position of the inspection robot's base is constrained by a known inspection path, which consists of several consecutive path points and can be described by the following formula:

[0122] ;

[0123] Where p is the inspection path; p i Let i be the i-th path point on the inspection path; "For" refers to three-dimensional Euclidean space, which is the set of all ordered arrays (x, y, z) consisting of three real numbers, used to describe and model the three-dimensional spatial environment in the real world.

[0124] Here, the base of the inspection robot can stop at any point on a given path and rotate freely around the vertical axis.

[0125] Furthermore, after determining the inspection path and the degree-of-freedom description information of the inspection robot, the feasible area of ​​the inspection robot's camera can be determined based on the inspection path and the base mobility information and camera mobility information in the degree-of-freedom description information.

[0126] S103. Based on the inspection path, the base mobility information, and the camera mobility information, determine the camera feasible area of ​​the inspection robot.

[0127] Here, the camera's feasible region, also known as the camera-capable space or camera-reachable space, represents the set of possible camera locations under the constraints of the inspection robot's structure, the movable range of the load mechanism, and environmental limitations. Since different inspection robots have different load structures, the camera's reachable region relative to the inspection path also varies.

[0128] Specifically, the step "determining the camera-feasible area of ​​the inspection robot based on the inspection path, the base mobility information, and the camera mobility information" includes:

[0129] a1: Based on the position information of the camera of the inspection robot on the inspection path, the inspection path, and the movable range of the camera as represented in the camera movable information, determine the camera feasible area of ​​the inspection robot.

[0130] In one possible implementation, the camera-feasible area of ​​the inspection robot can be described by the following formula:

[0131] ;

[0132] in, The camera's feasible area for the inspection robot; is the position on the inspection path; s is the inspection path parameter; ψ is the attitude, specifically the yaw angle. Rpayload represents the rotation matrix of the base orientation; Rpayload represents the camera's movable range, which is determined by the different inspection robot's payload structure. This represents the deviation of the camera center from the navigation center of the inspection robot.

[0133] For example, in an inspection robot with a PTZ camera mounted on a single lifting rod, the camera's feasible area W cam It degenerates into a curved surface with thickness, i.e., a path curtain; for a legged inspection robot equipped with a robotic arm and a PTZ camera mounted on the robotic arm, W cam The inspection path is defined as the three-dimensional reachable volume distribution; for UAVs equipped with PTZ cameras, W cam It can be the free space area near the inspection path.

[0134] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of the feasible camera area for a wheeled robot provided in an embodiment of this application; as shown... Figure 3 As shown, based on the determined camera movement range 310 of the wheeled robot and the inspection path 320, the camera feasible area 330 along the inspection path is determined.

[0135] Here, by determining the feasible area of ​​the camera, the load degrees of freedom of the inspection robot are mapped to the reachable set in the camera position space. This allows for the solution of the ideal image acquisition position in a unified manner without relying on the specific structure of the inspection robot, thereby improving the uniformity and portability of the inspection robot's inspection action planning.

[0136] Furthermore, after determining the feasible area for the camera, the ideal camera position can be determined based on the feasible area.

[0137] S104. Based on the feasible area of ​​the camera and the target object information, determine the ideal camera position of the inspection robot.

[0138] Here, after identifying the target object, the ideal camera position for the target object can be determined by combining the camera's feasible area.

[0139] Specifically, the ideal camera state can be described by the following formula:

[0140] ;

[0141] in, Ideal camera condition; Ideal camera position; This is the ideal camera optical axis angle.

[0142] In one possible implementation, the solution principle for the ideal camera state may include at least one of the following: the preferred viewing angle is as close as possible to the target object; the target object is completely within the camera's imaging field of view; and imaging ratio, distance constraints, etc. are taken into account.

[0143] Ideal camera states include, but are not limited to: extreme points, local optima, and the set of all feasible solutions that satisfy a preset threshold condition.

[0144] Specifically, when screening action planning information combinations, the optimal solution can be selected by using the calculation formula corresponding to the candidate evaluation value of the action planning information combination or by using a function that can obtain the same calculation result. The ideal camera state specifically includes the extreme point, local optimal solution, and all feasible solution sets that meet the preset threshold conditions, calculated by the calculation formula corresponding to the candidate evaluation value of the action planning information combination or by using a function that can obtain the same calculation result.

[0145] In the embodiments of this application, there are different methods for determining the ideal phase position for target objects with different directional attributes, which will be described separately below.

[0146] Firstly, when the directional attribute information indicates that the target object is a target object with directional attributes, the step "determining the ideal camera position of the inspection robot based on the camera's feasible area and the target object information" includes:

[0147] b1: If the directional attribute information indicates that the target object is a target object with directional attributes, determine the normal ray based on the center of the target object and the normal vector of the main surface.

[0148] b2: Detect whether there is an intersection between the normal ray and the feasible area of ​​the camera.

[0149] b3: If there is an intersection between the normal ray and the feasible area of ​​the camera, the location of the intersection between the normal ray and the feasible area of ​​the camera is determined as the ideal camera position.

[0150] b4: If there is no intersection between the normal ray and the camera feasible region, determine the ideal camera position in the camera feasible region according to the incident angle.

[0151] In one possible implementation, if the target object is determined to be a target object with directional properties, the normal ray can be determined based on the center of the target object and the normal vector of the main surface.

[0152] Specifically, the normal ray can be determined using the following formula;

[0153] ;

[0154] in, It is a normal ray; is the center of the target object; n is the normal vector of the principal surface; t is a parameter representing the direction of the normal ray.

[0155] Furthermore, after determining the normal ray, it is determined whether there is an intersection between the normal ray and the feasible area of ​​the camera. If it is determined that there is an intersection between the normal ray and the feasible area of ​​the camera, the location of the intersection between the normal ray and the feasible area of ​​the camera is determined as the ideal camera position.

[0156] In another possible implementation, if there is no intersection between the normal ray and the camera's feasible area, the position with the best incident angle can be searched within the camera's feasible area, and then the position with the best incident angle can be determined as the ideal camera position.

[0157] In another possible implementation, the ideal camera position can be directly solved in the feasible region based on numerical optimization or by using a pre-trained ideal camera position model (e.g., a neural network) based on the geometric relationship of the inspection target object represented by an explicit model or implicit form.

[0158] Secondly, when the directional attribute information indicates that the target object is a target object without directional attributes, the step "determining the ideal camera position of the inspection robot based on the camera's feasible area and the target object information" includes:

[0159] c1: If the directional attribute information indicates that the target object is a target object without directional attributes, the location closest to the target object in the feasible area of ​​the camera is determined as the ideal camera position; or...

[0160] c2: Determine at least one projection position of the target object in the camera's feasible area, and determine the projection position with the largest area among the projection positions as the ideal camera position.

[0161] In one possible implementation, if the target object is a target object that does not have directional attributes, the location of the target object can be used to find the closest position to the target object within the camera's feasible area, and then the closest position to the target object within the camera's feasible area can be determined as the ideal camera position.

[0162] Specifically, the ideal camera position can be determined using the following formula:

[0163] ;

[0164] in, Ideal camera position; X represents the camera's feasible area for the inspection robot; X represents any location within the camera's feasible area. The center of the target object.

[0165] In another possible implementation, if the target object is a target object that does not have directional properties, the projection of the target object within the camera's feasible area can be determined, and then the projection position with the largest area among the target object's projection positions can be determined as the ideal camera position.

[0166] Furthermore, after determining the ideal camera position, a search is performed in the local state space of the robot's pose / load state to determine the robot pose, load state, and camera state of the inspection robot at or near the ideal camera position.

[0167] S105. Based on the ideal camera position, determine at least one pose information of the inspection robot. For each pose information, determine the corresponding load state information and camera state information, and combine the pose information, the load state information and the camera state information to determine at least one action planning information combination.

[0168] In one possible implementation, the path position and base orientation can be sampled based on the inspection path and the base mobility information within a preset range of the ideal camera position, thereby determining at least one pose information of the inspection robot.

[0169] Specifically, the step "determining at least one pose information of the inspection robot based on the ideal camera position" includes:

[0170] d1: Within a preset range of the ideal camera position, based on the inspection path and the base mobility information, the path position and base orientation are discretely sampled to determine at least one pose information of the inspection robot.

[0171] Furthermore, after sampling the path position and the orientation of the base, and combining this with the load's movable range, at least one pose information is determined.

[0172] In one possible implementation, the position of the inspection robot on the inspection path can be determined using the following formula:

[0173] ;

[0174] in, This indicates the location of the inspection robot along the inspection path. Let i be the i-th path point of the inspection path.

[0175] In one possible implementation, the search range for the base orientation of the inspection robot can be determined based on the minimum and maximum horizontal rotation angles of the PTZ camera gimbal.

[0176] Specifically, the search range for the base orientation can be determined using the following formula:

[0177] ;

[0178] ;

[0179] Where, ψ min ψ is the minimum angle of orientation of the base. max This represents the maximum angle of the base orientation. This represents the minimum horizontal rotation angle of the PTZ camera gimbal. ψ is the maximum horizontal rotation angle of the PTZ camera gimbal; rob->obb The angle of the camera relative to the front of the target object.

[0180] Furthermore, after determining the pose information of at least one inspection robot, load status information and camera status information can be determined for each pose information.

[0181] Specifically, the step "for each of the pose information, determine the corresponding load state information and camera state information" includes:

[0182] e1: For each of the pose information, the load state information is determined by solving based on the structure of the inspection robot.

[0183] e2: Determine the adjustment position vector based on the target object center and the camera position;

[0184] e3: Determine the horizontal rotation angle and vertical pitch angle of the camera based on the adjusted position vector.

[0185] e4: Determine the optical zoom value of the camera based on the camera imaging parameters.

[0186] Here, the load state refers to the controllable load carried by the inspection robot. A load state uniquely defines the state (value) of each controllable degree of freedom, such as lifting height and robotic arm joint angle. The specific load classification is determined according to the type of inspection robot. For example, for a one-degree-of-freedom lifting platform, the load state may be: lifting height = 60cm.

[0187] Specifically, for inspection robots with simple structures such as lifting rods, the load state can be determined through geometric solutions; for inspection robots with structures such as robotic arms, the load state can be solved through forward / inverse kinematics; for inspection robots with non-analytical structures, the load state can be solved through numerical optimization methods; furthermore, for all types of inspection robots, the load state can also be solved based on a pre-trained deep learning network. For example, the basic attribute information of the inspection robot can be input into a pre-trained deep learning network, and the deep learning network can output the corresponding load state.

[0188] Here, the camera state uniquely defines the state of all controllable degrees of freedom of the PTZ camera. For example, the horizontal rotation angle is set to 60 degrees; the vertical pitch angle is set to 20 degrees; and the optical zoom is set to x3.

[0189] That is, when determining the camera status information, it is necessary to solve for the camera's horizontal rotation angle, vertical pitch angle, and optical zoom value.

[0190] In one possible implementation, it is necessary to determine the adjustment position vector based on the object center of the target object and the camera position, and then determine the horizontal rotation angle and the vertical pitch angle based on the adjustment position vector.

[0191] Specifically, the adjustment position vector can be determined using the following formula:

[0192] ;

[0193] in, To adjust the position vector ; The center of the target object; This indicates the camera position.

[0194] Furthermore, the horizontal rotation angle of the camera can be determined using the following formula:

[0195] ψ;

[0196] in, This refers to the horizontal rotation angle of the camera; The X and Y coordinates of the position vector are adjusted; ψ is the yaw angle of the base.

[0197] Furthermore, the camera's vertical pitch angle can be determined using the following formula:

[0198] ;

[0199] in, The vertical tilt angle of the camera; To adjust the Z-coordinate of the position vector; To adjust the X coordinate of the position vector; To adjust the Y-coordinate of the position vector.

[0200] In one possible implementation, the optical zoom value of the camera can be determined based on the camera's imaging parameters.

[0201] Specifically, the camera imaging parameters can be determined using the following formula:

[0202] ;

[0203] Where zoom is a camera imaging parameter; This represents the minimum horizontal field of view. This represents the maximum horizontal field of view angle.

[0204] Furthermore, the pose information, load status information, and camera status information are combined to obtain at least one action planning information combination. That is, each action planning information combination includes one pose information, one load status information, and one camera status information. Then, the at least one action planning information combination is filtered to determine the target action planning combination, and then the subsequent control process of the inspection robot is carried out.

[0205] S106. From at least one combination of action planning information, select a target action planning combination, and after adjusting the pose, load state and camera state of the inspection robot based on the target action planning combination, acquire a target image of the target object.

[0206] In one possible implementation, a candidate evaluation value can be calculated for each combination of action planning information, and then a target action planning combination can be selected from at least one combination of action planning information based on the calculated candidate evaluation value.

[0207] Specifically, the step "selecting target action plan combinations from at least one combination of said action planning information" includes:

[0208] f1: For each action planning information combination, determine the base deviation from the inspection path value, the camera position offset value between the camera position and the ideal camera position, and the incident angle value corresponding to the action planning information combination.

[0209] f2: For each action planning information combination, calculate the candidate evaluation value corresponding to the action planning information combination according to the preset weight coefficients for the base deviation from the inspection path value, the camera position offset value, and the incident angle value.

[0210] f3: Among at least one combination of action planning information, the action planning information combination with the lowest corresponding candidate evaluation value is determined as the target action planning combination.

[0211] In one possible implementation, the deviation of the base from the inspection path can be determined using the following formula:

[0212] ;

[0213] in, ψ represents the deviation of the base from the inspection path; ψ represents the base yaw angle; ψ tangent This represents the path direction angle.

[0214] In one possible implementation, the camera position offset between the camera position and the ideal camera position can be determined using the following formula:

[0215] ;

[0216] in, This is the camera position offset value between the camera position and the ideal camera position; For camera position; For the ideal camera position.

[0217] In one possible implementation, for a target object with directional attributes, the value of the incident angle can be determined by the following formula;

[0218] ;

[0219] in, The incident angle is the cost; n is the principal surface normal vector; To adjust the position vector .

[0220] In another possible implementation, for target objects that do not have directional attributes, the incident angle value can be set to a constant of 0.

[0221] Furthermore, for each action planning information combination, after determining the base deviation from the inspection path value, the camera position offset value between the camera position and the ideal camera position, and the incident angle value, it can be weighted according to preset coefficients to determine the candidate evaluation value of the action planning information combination.

[0222] Specifically, the candidate evaluation value of the action planning information combination can be determined using the following formula:

[0223] ;

[0224] Where J represents the candidate evaluation value; The value representing the deviation of the base from the inspection path; This is the camera position offset value between the camera position and the ideal camera position; The value of the angle of incidence; , as well as The preset weighting coefficients can be set according to the type of cruise robot, the target object information, etc.

[0225] Here, in the process of determining the ideal camera state and / or the target action plan combination, a line-of-sight constraint can also be added. Specifically, the line-of-sight constraint determines whether the line of sight between the camera and the inspected target is blocked by other structures or equipment in the environment. If there is obstruction, the corresponding camera state is considered an infeasible solution or is given a large penalty cost. Then, after processing according to the line-of-sight constraint, the ideal camera state and / or target action plan combination that meets the conditions is determined.

[0226] In one possible implementation, a single site pose candidate can be expanded into multiple candidate poses, and a global optimization module such as site set selection, pose combination selection, or site sequence optimization can be introduced at the inspection task level to select action planning information. Furthermore, a load motion cost can be introduced to describe the adjustment cost of the load mechanism (such as a lifting rod, translation mechanism, or robotic arm) between different poses. This cost can be calculated based on the change in load degrees of freedom, the spatial distance of the robotic arm joints, the load motion time, or its energy consumption model. The load motion cost can be used in multi-pose selection, site set optimization, or overall path optimization of the inspection task to achieve globally optimal scheduling of the inspection robot's overall behavior. This type of global optimization represents a further combination and scheduling of the solution results from the embodiments of this application, to improve the accuracy of the combined screening of action planning information.

[0227] Furthermore, after calculating the candidate evaluation value of each action planning information combination, the action planning information combination with the smallest candidate evaluation value is determined as the target action planning combination.

[0228] Furthermore, the system outputs the base position and orientation, load status information (load degrees of freedom: lifting, joint angle, slide, etc.) and camera status (horizontal rotation angle, vertical pitch angle, optical zoom) of the target action planning combination, adjusts the pose, load status and camera status of the inspection robot, and captures target images of the target object.

[0229] In another possible implementation, in this embodiment of the application, the inspection robot is controlled to move to the location of each target object to be imaged, and the pose, load state, and camera state of the inspection robot for capturing images of the target objects are planned in real time. Alternatively, the photographing pose of each target object can be expanded into multiple candidates, and tools such as integer programming, graph optimization, dynamic programming, or OR-Tools can be used to minimize the overall cost of the inspection robot's overall movement distance, load movement, and number of start-stops while ensuring that all target objects are covered, thus planning the overall image acquisition task.

[0230] In another possible implementation, the action plans for multiple identified targets can be merged to generate multiple candidate action plan combinations that can simultaneously control the inspection robot to collect information from multiple targets. These candidate action plan combinations may include the same robot pose, the same load state, or the same camera state. A candidate evaluation cost is then calculated for each candidate action plan combination. Based on the calculated candidate evaluation costs, the target action plan combination with the lowest overall cost is selected from at least one action plan combination, thus determining the target action plan combination capable of simultaneously collecting information from multiple targets. The process of calculating the candidate evaluation costs for each candidate action plan combination and then selecting the target action plan combination with the lowest overall cost from at least one action plan combination is consistent with the process of selecting the corresponding target action plan combination for each target, and will not be elaborated further here.

[0231] In another possible implementation, for multiple targets to be collected, a unique target action plan combination can be determined based on the target information of each target. Based on the pose information, load status information, and camera status information in the target action plan combination, the inspection robot is controlled to capture images of each target in one go. The following specific examples will illustrate the inspection action planning process of the inspection robot in this embodiment of the disclosure:

[0232] Example 1: When the inspection robot is a ground-based inspection robot, it moves along a given three-dimensional path. Its base can stop at any point on the path and is allowed to rotate freely about a vertical axis. The robot's load is a camera mounting pole with lifting capabilities, and the PTZ camera is mounted on top of the pole. This structure makes the camera appear as a "curtain" relative to the accessible area of ​​the path.

[0233] Assume the inspection path of the inspection robot is as follows:

[0234] ;

[0235] Where p is the inspection path; p i Let i be the i-th path point on the inspection path; It is a three-dimensional Euclidean space, which is the set of all ordered arrays (x, y, z) consisting of three real numbers, used to describe and model the three-dimensional spatial environment in the real world.

[0236] The parameterization of the inspection robot base along the inspection path is set as follows:

[0237] ;

[0238] in, Parameters for the inspection robot base along the inspection path; is the kth path point of the inspection path; s is the inspection path parameter.

[0239] Furthermore, the camera's degrees of freedom are provided by the load-bearing lifting rod. The horizontal offset of the lifting rod's mounting position relative to the base navigation center is 0. The lifting rod allows the camera height to be within [z...]. min , z max The camera can move within a certain range. At this time, the camera's movable range Rpayload is a line segment.

[0240] The deviation of the camera center from the navigation center of the inspection robot can be calculated using the following formula:

[0241] ;

[0242] in, This represents the deviation of the camera center relative to the navigation center of the inspection robot; let the origin of the robot's coordinate system be the robot's navigation center. This represents the offset of the camera center relative to the robot navigation center in each axis within the body coordinate system.

[0243] Furthermore, the camera's feasible region can be calculated accordingly. Here, the camera's feasible region degenerates into a two-dimensional curved surface "path curtain," which can be calculated using the following formula:

[0244] ;

[0245] in, Example 1: camera feasible area (path curtain); s represents the inspection path parameters; This is the horizontal projection curve of the inspection path.

[0246] Furthermore, the target object information is described using the following formula:

[0247] ;

[0248] in, The target object information; The center of the target object; A quaternion describing the target's pose; Let A represent the size of the target object; let A represent the directional attribute information of the target object. Here, the target object is an object with directional attributes, and each target object has a normal vector n (preferably the shooting angle), which is included in A.

[0249] Furthermore, taking the center of the target object as the starting point, the normal ray is determined using the following formula;

[0250] ;

[0251] in, It is a normal ray; is the center of the target object; n is the normal vector of the principal surface; t is a parameter representing the direction of the normal ray.

[0252] Furthermore, the nearest intersection point between the ray and the path curtain is found and used as the ideal camera position. The ideal camera position is then determined using the following formula:

[0253] ;

[0254] in, This is the ideal camera position in Example 1; This is the camera's feasible area (path curtain).

[0255] In another possible implementation, if there is no intersection between the ray and the path curtain, a position with the optimal incident angle is searched within the curtain area as an alternative ideal position.

[0256] Furthermore, taking the path segment where the ideal camera is located as the center, the path parameters s are sampled in several path segments before and after it.

[0257] Specifically, the position of the robot base is calculated for each sampling point using the following formula:

[0258] ;

[0259] in, This indicates the location of the inspection robot along the inspection path. Let i be the i-th path point of the inspection path.

[0260] Furthermore, the range of the base's orientation angle can be determined using the following formula:

[0261] ;

[0262] ;

[0263] ;

[0264] in, ψ is the yaw angle of the base. min ψ is the minimum angle of orientation of the base. max This represents the maximum angle of the base orientation. This represents the minimum horizontal rotation angle of the PTZ camera gimbal. ψ is the maximum horizontal rotation angle of the PTZ camera gimbal; rob->obb The angle of the camera relative to the front of the target object.

[0265] Furthermore, for each candidate base pose, the camera position in the world coordinate system is calculated based on the inspection robot's degree-of-freedom description information:

[0266] ;

[0267] in, This indicates the camera position and the inspection robot position. The base is oriented towards the rotation matrix; This represents the deviation of the camera center from the navigation center of the inspection robot.

[0268] Furthermore, the adjustment position vector can be determined using the following formula:

[0269] ;

[0270] in, To adjust the position vector ; The center of the target object; This indicates the camera position.

[0271] Furthermore, the horizontal rotation angle of the camera can be determined using the following formula:

[0272] ψ;

[0273] in, This refers to the horizontal rotation angle of the camera; The X and Y coordinates of the position vector are adjusted; ψ is the yaw angle of the base.

[0274] Furthermore, the camera's vertical pitch angle can be determined using the following formula:

[0275] ;

[0276] in, The vertical tilt angle of the camera; To adjust the Z-coordinate of the position vector; To adjust the X coordinate of the position vector; To adjust the Y-coordinate of the position vector.

[0277] Here, if , If all targets fall within the gimbal's range of motion, the pose is considered feasible. The required field of view and zoom ratio are calculated to determine whether the target can completely fall within the camera's field of view.

[0278] Furthermore, the candidate evaluation value of the action planning information combination is determined using the following formula:

[0279] ;

[0280] Where J represents the candidate evaluation value; The value representing the deviation of the base from the inspection path; This is the camera position offset value between the camera position and the ideal camera position; The value of the angle of incidence; , as well as The preset weighting coefficients can be set according to the type of cruise robot, the target object information, etc.

[0281] Here, as mentioned earlier, occlusion detection can be performed on the line of sight between the camera position and the target object based on the environmental map or point cloud model. If the line of sight is blocked by other objects, the photo pose will not be considered as a valid candidate, thereby reducing the amount of subsequent data processing and improving the planning efficiency for the inspection robot's inspection action planning.

[0282] After calculating the candidate evaluation value of each action plan information combination, the action plan information combination with the smallest candidate evaluation value is determined as the target action plan combination.

[0283] Example 2: For a robot whose load is a three-dimensional reachable area (e.g., robotic arm + camera), for an inspection robot with a robotic arm, a three-dimensional sliding table and other structures, the camera's feasible area can be expanded to a three-dimensional volume area near the path.

[0284] The feasible area for the camera can be calculated using the following formula:

[0285] ;

[0286] in, Example 2: The feasible region for the camera (3D volume region); Parameters for the inspection robot base along the inspection path; The base is oriented towards the rotation matrix; Forward kinematics of the robotic arm; This refers to the joint angle of the robotic arm.

[0287] The remaining steps (ideal camera position search, inverse load state, cost function evaluation, etc.) are similar to those in Example 1 and will not be repeated here.

[0288] Example 3: When the inspection robot is a multi-rotor aircraft, the given inspection path is a spatial polyline; the base position can be directly taken as the path point; the feasible area of ​​the camera is the free space area near the path; the load is a three-axis gimbal. The remaining steps (ideal camera position search, inverse solution of load state, cost function evaluation, etc.) are similar to Example 1 and will not be repeated here.

[0289] This application embodiment unifies the modeling of the inspection robot's body degrees of freedom, load mechanisms (such as lifting platforms and robotic arms), and controllable parameters of the PTZ camera, mapping robots of different forms to a consistent state space. This eliminates the need for the planning algorithm to depend on the specific mechanical configuration of the robot platform. Therefore, when changing the inspection robot (e.g., switching from a wheeled platform to a flying platform), only a new degree of freedom description file needs to be provided to reuse the same planning process, eliminating the need to maintain differentiated algorithm logic for different platforms. This technical advantage stems from the unified degree of freedom description model and its mapping mechanism proposed in this application embodiment, rather than simple parameter replacement. This application embodiment incorporates the aforementioned degrees of freedom into the planning variables, automatically solving them during the unified solution process based on environmental geometric constraints (such as line-of-sight occlusion) and the imaging requirements of the target object (such as resolution constraints and effective incident angle range). This advantage stems from the joint planning mechanism of this application's embodiments: integrating path planning, load planning, and camera planning under the same general solution framework, rather than processing them separately based on the traditional robot platform; this application's embodiments, through a unified task description method (such as unified inspection points, unified imaging requirements, and unified constraint formats), enable multi-rotor, wheeled, and legged robots to solve their optimal inspection behaviors under the same set of task descriptions; in the same scene, different robots can each generate executable inspection behavior sequences without defining image acquisition points and component states separately for each type of robot. This effect comes from the platform-independent task modeling and its automatic mapping capability to robot degrees of freedom of this application's embodiments. Since the robot behaviors of different inspection types are all mapped to the same state space, the calculation formulas corresponding to the candidate evaluation costs (such as time, energy consumption, gimbal movement, number of lifts and drops, safety margin, etc.) can be expressed in a unified form, which brings two technical effects. First, it allows direct comparison of execution costs between different robots, which is beneficial for task allocation and scheduling. Secondly, all behaviors can be decomposed into physical costs (such as travel distance and flight power consumption) and load costs, making energy consumption and operational efficiency assessments more accurate. In this embodiment, the hardware differences of the inspection robot are reflected in the degree-of-freedom description information in the general degree-of-freedom description file and the description file construction rules. Therefore, the development work required for new platform integration is reduced to simply generating the corresponding description file according to the rules. This allows the planning method of this embodiment to be quickly migrated between multiple platforms and models of inspection robots, improving the engineering value of the system. The fundamental reason for this effect is that this embodiment abstracts the equipment differences, creating a clear interface boundary between the planning module and the hardware module.

[0290] The inspection action planning method for an inspection robot provided in this application embodiment determines the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters; determines the target object information of the target object to be captured and the inspection path of the inspection robot; determines the camera feasible area of ​​the inspection robot based on the inspection path, base mobility information, and camera mobility information; determines the ideal camera position of the inspection robot based on the camera feasible area and the target object information; determines at least one pose information of the inspection robot based on the ideal camera position; for each pose information, determines the corresponding load state information and camera state information, and combines the pose information, load state information, and camera state information to determine at least one action planning information combination; selects a target action planning combination from the at least one action planning information combination, and after adjusting the pose, load state, and camera state of the inspection robot based on the target action planning combination, acquires the target image of the target object. In this way, by constructing the degree-of-freedom description information of the inspection robot, combining the target object information of the image to be acquired with the inspection path of the inspection robot, the camera feasible area of ​​the inspection robot is automatically determined. The ideal camera position with the best acquisition effect is determined through the camera feasible area. Then, based on the ideal camera position, the pose information, load status information and camera status information of the inspection robot are determined. Finally, the optimal target action planning combination is determined, reducing the error of manual planning that cannot be combined with the specific acquisition scenario and the low efficiency of manual planning for different types of inspection robots. This improves the efficiency of inspection robot inspection action planning and the accuracy of image acquisition.

[0291] Based on the same inventive concept, this application also provides an inspection action planning device for an inspection robot corresponding to the inspection action planning method of the inspection robot. Since the principle of the device in this application is similar to the inspection action planning method of the inspection robot described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0292] Please see Figure 4 , Figure 4 This is a schematic diagram of the inspection action planning device for an inspection robot provided in an embodiment of this application. Figure 4 As shown, the inspection action planning device 400 includes:

[0293] The degree-of-freedom description information determination module 410 is used to determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters of the camera;

[0294] The inspection target determination module 420 is used to determine the target object information of the target object in the image to be acquired and the inspection path of the inspection robot.

[0295] The camera feasible area determination module 430 is used to determine the camera feasible area of ​​the inspection robot based on the inspection path, the base mobility information and the camera mobility information.

[0296] The ideal camera position determination module 440 is used to determine the ideal camera position of the inspection robot based on the camera's feasible area and the target object information.

[0297] The action planning information combination determination module 450 is used to determine at least one pose information of the inspection robot based on the ideal camera position, and for each pose information, determine the corresponding load state information and camera state information, and combine the pose information, the load state information and the camera state information to determine at least one action planning information combination.

[0298] The action planning information combination filtering module 460 is used to filter out a target action planning combination from at least one action planning information combination, and after adjusting the pose, load state and camera state of the inspection robot based on the target action planning combination, acquire a target image of the target object.

[0299] In one possible implementation, when the camera feasible area determination module 430 determines the camera feasible area of ​​the inspection robot based on the inspection path, the base mobility information, and the camera mobility information, the camera feasible area determination module 430 is used to:

[0300] Based on the position information of the camera of the inspection robot on the inspection path, the inspection path, and the movable range of the camera as represented by the camera's movable information, the feasible area of ​​the camera of the inspection robot is determined.

[0301] In one possible implementation, the target object information includes the target object center and the target object's directional attribute information; when the ideal camera position determination module 440 determines the ideal camera position of the inspection robot based on the camera's feasible area and the target object information, the ideal camera position determination module 440 is used for:

[0302] If the directional attribute information indicates that the target object is a target object with directional attributes, the normal ray is determined based on the center of the target object and the normal vector of the main surface;

[0303] Detect whether there is an intersection between the normal ray and the feasible region of the camera;

[0304] If there is an intersection between the normal ray and the feasible area of ​​the camera, the location of the intersection between the normal ray and the feasible area of ​​the camera is determined as the ideal camera position;

[0305] If there is no intersection between the normal ray and the camera's feasible region, the ideal camera position is determined in the camera's feasible region based on the incident angle.

[0306] In one possible implementation, when determining the ideal camera position of the inspection robot based on the camera's feasible area and the target information, the ideal camera position determination module 440 is further configured to:

[0307] If the directional attribute information indicates that the target object is a target object without directional attributes, the location closest to the target object in the camera's feasible area is determined as the ideal camera position; or...

[0308] Determine at least one projection position of the target object within the camera's feasible area, and determine the projection position with the largest area among the projection positions as the ideal camera position.

[0309] In one possible implementation, when the action planning information combination determination module 450 is used to determine at least one pose information of the inspection robot based on the ideal camera position, the action planning information combination determination module 450 is used to:

[0310] Within a preset range of the ideal camera position, based on the inspection path and the base mobility information, the path position and base orientation are discretely sampled to determine at least one pose information of the inspection robot.

[0311] In one possible implementation, when the action planning information combination determination module 450 determines the corresponding load state information and camera state information for each pose information, the action planning information combination determination module 450 is used to:

[0312] For each of the aforementioned pose information, the load state information is determined by solving based on the structure of the inspection robot;

[0313] Based on the target object's center and the camera's position, determine the adjustment position vector;

[0314] The horizontal rotation angle and vertical pitch angle of the camera are determined based on the adjusted position vector.

[0315] Based on the camera imaging parameters, the optical zoom value of the camera is determined.

[0316] In one possible implementation, when the action planning information combination filtering module 460 is used to filter out a target action planning combination from at least one of the action planning information combinations, the action planning information combination filtering module 460 is configured to:

[0317] For each action planning information combination, determine the base deviation from the inspection path value, the camera position offset value between the camera position and the ideal camera position, and the incident angle cost value corresponding to that action planning information combination;

[0318] For each action planning information combination, the candidate evaluation value corresponding to the action planning information combination is calculated according to the preset weighting coefficients for the base deviation from the inspection path value, the camera position offset value, and the incident angle value.

[0319] The action planning information combination with the lowest corresponding candidate evaluation value among at least one of the action planning information combinations is determined as the target action planning combination.

[0320] The inspection action planning device for the inspection robot provided in this application embodiment determines the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters; determines the target object information of the target object to be acquired and the inspection path of the inspection robot; determines the camera feasible area of ​​the inspection robot based on the inspection path, base mobility information, and camera mobility information; determines the ideal camera position of the inspection robot based on the camera feasible area and the target object information; determines at least one pose information of the inspection robot based on the ideal camera position, and for each pose information, determines the corresponding load state information and camera state information, and combines the pose information, load state information, and camera state information to determine at least one action planning information combination; selects a target action planning combination from the at least one action planning information combination, and after adjusting the pose, load state, and camera state of the inspection robot based on the target action planning combination, acquires the target image of the target object. In this way, by constructing the degree-of-freedom description information of the inspection robot, combining the target object information of the image to be acquired with the inspection path of the inspection robot, the camera feasible area of ​​the inspection robot is automatically determined. The ideal camera position with the best acquisition effect is determined through the camera feasible area. Then, based on the ideal camera position, the pose information, load status information and camera status information of the inspection robot are determined. Finally, the optimal target action planning combination is determined, reducing the error of manual planning that cannot be combined with the specific acquisition scenario and the low efficiency of manual planning for different types of inspection robots. This improves the efficiency of inspection robot inspection action planning and the accuracy of image acquisition.

[0321] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0322] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate via the bus 530. When the machine-readable instructions are executed by the processor 510, they can perform the operations described above. Figure 1 The steps of the inspection action planning method for the inspection robot in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0323] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1The steps of the inspection action planning method for the inspection robot in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0324] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0325] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0326] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0327] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0328] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0329] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for planning a patrol action of a patrol robot, characterized by, The inspection action planning method includes: Determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters of the camera; Determine the target object information of the target object to be captured in the image and the inspection path of the inspection robot. Based on the inspection path, the base mobility information, and the camera mobility information, the camera feasible area of ​​the inspection robot is determined; Based on the camera's feasible area and the target object information, the ideal camera position of the inspection robot is determined; Based on the ideal camera position, at least one pose information of the inspection robot is determined. For each pose information, corresponding load state information and camera state information are determined. The pose information, load state information and camera state information are combined to determine at least one action planning information combination. From at least one combination of action planning information, a target action planning combination is selected, and the pose, load state, and camera state of the inspection robot are adjusted based on the target action planning combination before acquiring a target image of the target object. The target object information includes the target object center and the target object's directional attribute information; determining the ideal camera position for the inspection robot based on the camera's feasible area and the target object information includes: If the directional attribute information indicates that the target object is a target object with directional attributes, the normal ray is determined based on the center of the target object and the normal vector of the main surface; Detect whether there is an intersection between the normal ray and the feasible region of the camera; If there is an intersection between the normal ray and the feasible area of ​​the camera, the location of the intersection between the normal ray and the feasible area of ​​the camera is determined as the ideal camera position; If there is no intersection between the normal ray and the camera's feasible region, the ideal camera position is determined in the camera's feasible region based on the incident angle.

2. The method of claim 1, wherein, The step of determining the camera-feasible area of ​​the inspection robot based on the inspection path, the base mobility information, and the camera mobility information includes: Based on the position information of the inspection robot's camera on the inspection path, the inspection path, and the camera's movable range as represented by the camera's movable information, the feasible area of ​​the inspection robot's camera is determined.

3. The inspection action planning method according to claim 1, characterized in that, The step of determining the ideal camera position for the inspection robot based on the camera's feasible area and the target object information further includes: If the directional attribute information indicates that the target object is a target object without directional attributes, the location closest to the target object in the camera's feasible area is determined as the ideal camera position; or... Determine at least one projection position of the target object within the camera's feasible area, and determine the projection position with the largest area among the projection positions as the ideal camera position.

4. The inspection action planning method according to claim 1, characterized in that, Determining at least one pose information of the inspection robot based on the ideal camera position includes: Within a preset range of the ideal camera position, based on the inspection path and the base mobility information, the path position and base orientation are discretely sampled to determine at least one pose information of the inspection robot.

5. The inspection action planning method according to claim 1, characterized in that, The step of determining the corresponding load state information and camera state information for each pose information includes: For each of the aforementioned pose information, the load state information is determined by solving based on the structure of the inspection robot; Based on the target object's center and the camera's position, determine the adjustment position vector; The horizontal rotation angle and vertical pitch angle of the camera are determined based on the adjusted position vector. Based on the camera imaging parameters, the optical zoom value of the camera is determined.

6. The inspection action planning method according to claim 1, characterized in that, The step of selecting a target action plan combination from at least one combination of action planning information includes: For each action planning information combination, determine the base deviation from the inspection path value, the camera position offset value between the camera position and the ideal camera position, and the incident angle cost value corresponding to that action planning information combination; For each action planning information combination, the candidate evaluation value corresponding to the action planning information combination is calculated according to the preset weighting coefficients for the base deviation from the inspection path value, the camera position offset value, and the incident angle value. The action planning information combination with the lowest corresponding candidate evaluation value among at least one of the action planning information combinations is determined as the target action planning combination.

7. An inspection action planning device for an inspection robot, characterized in that, The inspection action planning device includes: A degree-of-freedom description information determination module is used to determine the degree-of-freedom description information of the inspection robot; wherein, the degree-of-freedom description information includes the base mobility information of the inspection robot, the camera mobility information of the inspection robot, and the camera imaging parameters of the camera; The inspection target determination module is used to determine the target object information of the target object in the image to be collected and the inspection path of the inspection robot. The camera feasible area determination module is used to determine the camera feasible area of ​​the inspection robot based on the inspection path, the base mobility information and the camera mobility information; An ideal camera position determination module is used to determine the ideal camera position of the inspection robot based on the camera's feasible area and the target object information. The action planning information combination determination module is used to determine at least one pose information of the inspection robot based on the ideal camera position, and for each pose information, determine the corresponding load state information and camera state information, and combine the pose information, the load state information and the camera state information to determine at least one action planning information combination. The action planning information combination filtering module is used to filter out a target action planning combination from at least one action planning information combination, and after adjusting the pose, load state and camera state of the inspection robot based on the target action planning combination, acquire a target image of the target object. The target object information includes the target object center and the target object's directional attribute information; when determining the ideal camera position of the inspection robot based on the camera's feasible area and the target object information, the ideal camera position determination module is used for: If the directional attribute information indicates that the target object is a target object with directional attributes, the normal ray is determined based on the center of the target object and the normal vector of the main surface; Detect whether there is an intersection between the normal ray and the feasible region of the camera; If there is an intersection between the normal ray and the feasible area of ​​the camera, the location of the intersection between the normal ray and the feasible area of ​​the camera is determined as the ideal camera position; If there is no intersection between the normal ray and the camera's feasible region, the ideal camera position is determined in the camera's feasible region based on the incident angle.

8. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the inspection action planning method for the inspection robot as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the inspection action planning method for the inspection robot as described in any one of claims 1 to 6.

Images