Inspection point planning method and apparatus, computing device, and storage medium

Abstract

An inspection point planning method and apparatus, a computing device, and a storage medium, relating to the technical field of artificial intelligence. The method comprises: on the basis of a 3D point cloud map of a site to be inspected, a computing device determines the location and orientation of a first object among a plurality of objects to be inspected in said site (S202); on the basis of the 3D point cloud map, the computing device determines an accessible region in said site, the accessible region being a region that a device performing an inspection task can reach (S203); and the computing device determines the location of an inspection point of the first object on the basis of the accessible region and the location and orientation of the first object (S204). According to the method, the locations of inspection points can be automatically planned in the 3D point cloud map, without the need for manually and remotely controlling the device performing the inspection task to reach the vicinity of each object to be inspected and then determine the locations of the inspection points, so that inspection point planning takes less time and is not affected by extreme environments, improving the deployment efficiency of inspection points, and reducing labor costs.

Description

Inspection point planning method, device, computing equipment and storage medium Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computing device, and storage medium for planning inspection points. Background Art

[0002] Robotic inspection involves using robots to replace manual inspection of objects within an inspection area, improving inspection efficiency and reducing costs. Prior to inspection, the robot must first be programmed with inspection point information to deploy inspections of the objects. These inspection points, also known as the robot's stop points, include their location, the robot's orientation relative to the inspection point, the orientation of the robot's onboard camera's field of view, the camera's magnification, and its focal length.

[0003] In related technologies, the operator first remotely controls the robot to the vicinity of each object to be detected, and records the coordinates and orientation of the robot in the map; by remotely controlling the robot's pan / tilt orientation, the operator makes the center of the object to be detected located in the center of the camera's field of view; adjusts the camera's magnification and focal length so that the camera can clearly capture the object to be detected and the object to be detected occupies the main part of the camera's field of view; records the debugged parameters to obtain information on the inspection points of the object to be detected.

[0004] However, in the above method, manual remote control of the robot is required to reach the vicinity of the object to be detected in order to determine the inspection point corresponding to the object to be detected. In a large inspection site, there may be nearly 10,000 points. The method of determining the inspection points by manual remote control is time-consuming, and the recording efficiency is low in extreme environments, which greatly affects the deployment efficiency of the inspection points.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method, apparatus, computing device, and storage medium for patrol point planning, which can shorten the time consumption for patrol point planning, is not affected by extreme environments, and can improve the deployment efficiency of patrol points and reduce labor costs. The technical solution is as follows:

[0007] In a first aspect, a method for planning inspection points is provided for inspecting multiple objects to be inspected in an inspection site, the method comprising:

[0008] The method comprises determining the position and orientation of a first object among multiple objects to be inspected in a three-dimensional (3D) point cloud map of a site to be inspected, determining a feasible area for equipment performing an inspection task in the 3D point cloud map, and then determining the position of an inspection point of the first object in the feasible area based on the position and orientation of the first object.

[0009] Among them, the objects to be detected in the site to be inspected are also the inspection targets in the site to be inspected. Taking the substation inspection scenario as an example, the site to be inspected is the substation, and the objects to be detected include meters, insulators, voltage transformers, current transformers, grounding switches, disconnector tubes, bushings, lightning arrester rings, drain wire clamps, electrical boxes, ladder locks, insulator connections, etc., which are not limited in the embodiments of the present application. The first object can be any object to be detected in the site to be inspected. The position of the first object is described by the coordinates of the midpoint of the first object in the 3D point cloud map, and the orientation of the first object is described by the normal direction of the plane corresponding to the front view of the first object.

[0010] The device that performs the inspection task may be a device with an inspection function, such as a robot, which is not limited in this embodiment of the present application. The feasible area is an area that the device that performs the inspection task can reach.

[0011] In the above method, the locations of inspection points can be automatically planned in the 3D point cloud map of the site to be inspected, without the need for manual remote control of the robot to reach the vicinity of each object to be inspected and then determine the inspection points. The planning of inspection points takes less time and is not affected by extreme environments. It can improve the deployment efficiency of inspection points and reduce labor costs.

[0012] Optionally, the method further includes:

[0013] The direction of the line connecting the position of the inspection point of the first object and the position of the first object is used as the orientation of the inspection point of the first object, and the orientation is used to indicate the direction of the camera field of view carried by the equipment performing the inspection task.

[0014] Optionally, determining the position of the inspection point of the first object includes:

[0015] Determine an intersection of the feasible area and a first plane, where the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes a plurality of points; and determine a position of an inspection point of the first object from the plurality of points in the intersection.

[0016] The inspection point can be located at the point farthest from the center of the first object among the multiple points, thereby expanding the capture range of the camera carried by the device performing the inspection task. The inspection point can also be located at the point closest to the center of the first object among the multiple points, thereby improving the clarity of the image captured by the camera carried by the device performing the inspection task. For example, based on the midpoint xd and orientation of the first object, plane A (ax+by+cz+d=0) is determined. This plane A is also the first plane. Intersecting plane A with the feasible region yields a point equation ex+fy=0 (x0<=x<=x1), where x, y, and z represent coordinate axes, a, b, c, d, e, and f are constants, and x0 and x1 are the intersection points of plane A and the boundaries of the feasible region. Comparing the midpoint xd of the first object with x0 and x1, if xd is closer to x0, then the inspection point of the first object is located at x1; if xd is closer to x1, then the inspection point of the first object is located at x0.

[0017] Optionally, the plurality of objects to be detected include the first object and the second object, and the method further includes:

[0018] Based on the feasible area and the position and orientation of the second object, determine the position of the inspection point of the second object; determine that the distance between the inspection point of the first object and the inspection point of the second object is less than a threshold; based on the inspection point of the first object and the inspection point of the second object, determine the position of the merged inspection point; use the position of the merged inspection point as the position of the inspection point of the first object, and use the position of the merged inspection point as the position of the inspection point of the second object.

[0019] In the above method, multiple inspection points that can be merged are merged into one inspection point, and the equipment performing the inspection task can inspect multiple objects at one inspection point, which is conducive to improving the efficiency of the inspection.

[0020] Optionally, determining the position of the merged inspection point based on the inspection point of the first object and the inspection point of the second object includes:

[0021] Determine the midpoint between the first inspection point and the second inspection point, which is the location of the merged inspection point.

[0022] Optionally, the method further comprises:

[0023] The direction of the line connecting the midpoints of the first object and the second object and the position of the combined inspection point is used as the orientation of the combined inspection point.

[0024] Optionally, the method further includes:

[0025] The locations of inspection points are displayed in the 3D point cloud map.

[0026] Optionally, the method further includes:

[0027] A digital twin model is constructed for the site to be inspected to obtain a 3D point cloud map of the site to be inspected.

[0028] Optionally, the method further includes:

[0029] Determine multiple first anchor points in the first map and multiple second anchor points in the second map; determine a coordinate transformation matrix based on the multiple first anchor points and the multiple second anchor points; and transform the coordinates of at least one first inspection point of the equipment performing the inspection task into the coordinates of at least one corresponding second inspection point through the coordinate transformation relationship.

[0030] The first map is a grid map corresponding to the 3D point cloud map of the site to be inspected, the second map is a navigation map of the site to be inspected by the device performing the inspection task, and the multiple first anchor points and the multiple second anchor points are in one-to-one correspondence. The coordinate transformation matrix is ​​used to represent the coordinate transformation relationship between the coordinate system of the first map and the coordinate system of the second map. The at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

[0031] In the above method, the planned inspection points and the inspection points used for actual inspections are automatically adapted based on anchor points, which can reduce the impact of errors between different maps on actual inspections without the need to manually re-enter the inspection points, saving labor costs. Moreover, since the inspection points between different maps can be adapted, the inspection points obtained based on one inspection point planning can be adapted to the navigation maps of multiple devices performing inspection tasks, realizing the reuse of inspection points on different devices without the need to plan inspection points for each device, which can improve the deployment efficiency of inspections and further save labor costs.

[0032] Optionally, determining a plurality of first anchor points in the first map includes:

[0033] Based on a key point detection algorithm, a plurality of first anchor points are determined in the first map.

[0034] Optionally, determining a plurality of first anchor points in the first map includes:

[0035] Receive an anchor point configuration instruction from a user, where the anchor point configuration instruction is used to add at least one anchor point to a first map, delete at least one first anchor point from a plurality of first anchor points, or adjust a position of at least one first anchor point among a plurality of first anchor points; and adjust the plurality of first anchor points based on the anchor point configuration instruction.

[0036] The method for determining multiple second anchor points in the second map is the same as the method for determining multiple first anchor points in the first map, and will not be repeated here.

[0037] Optionally, the device performing the inspection task is equipped with a camera, and the position of the camera is adjustable. The method further includes:

[0038] An image captured by a camera at a second inspection point is obtained; based on the image, a posture of the camera corresponding to the second inspection point is adjusted to obtain a direction of the camera field of view at the second inspection point.

[0039] In the above method, on the basis of the coarse point position (that is, the second inspection point position) generated based on the anchor point, the coarse point position is fine-tuned based on the image taken by the equipment performing the inspection at the coarse point position to compensate for the error caused by the algorithm in the coordinate system transformation process based on the anchor point, so as to obtain a more accurate inspection point position, thereby ensuring that the equipment performing the inspection task collects high-quality images during the inspection, thereby improving the accuracy of the inspection results.

[0040] Optionally, adjusting the posture of the camera corresponding to the second inspection point based on the image to obtain the direction of the camera field of view of the second inspection point includes:

[0041] Based on the template matching algorithm, the object to be detected in the image is determined; based on the position of the center point of the object to be detected in the image, the camera's posture is adjusted to control the center point of the camera's field of view to align with the center point of the object to be detected.

[0042] In a second aspect, a patrol point planning device is provided for executing the above patrol point planning method. Specifically, the patrol point planning device includes a functional module for executing the patrol point planning method provided in the above first aspect or any optional manner of the above first aspect.

[0043] In a third aspect, a computing device is provided, which includes a processor, wherein the processor is used to execute program code so that the computing device executes to implement the operations performed by the above-mentioned inspection point planning method.

[0044] In a fourth aspect, a computing device cluster is provided, which computing device machine includes at least one computing device, each computing device includes a processor and a memory, and the processor of at least one computing device is used to execute the program code stored in the memory of at least one computing device, so that the computing device cluster executes to implement the operations performed by the above-mentioned inspection point planning method.

[0045] In a fifth aspect, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is read by a processor of a computing device to enable the computing device to perform operations such as those performed by the above-mentioned inspection point planning method.

[0046] In a sixth aspect, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is read by a processor of a computing device in a computing device cluster to enable the computing device cluster to perform operations such as those performed by the above-mentioned inspection point planning method.

[0047] In the seventh aspect, a computer program product or computer program is provided, which includes a program code, which is stored in a computer-readable storage medium. The processor of the computing device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device executes the inspection point planning method provided in the above-mentioned one aspect or various optional implementations of one aspect.

[0048] In an eighth aspect, a computer program product or a computer program is provided, which includes a program code, which is stored in a computer-readable storage medium. The processor of a computing device in a computing device cluster reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device cluster executes the inspection point planning method provided in the above-mentioned one aspect or various optional implementations of one aspect.

[0049] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] FIG1 is a schematic diagram of an implementation environment of a patrol point planning method provided in an embodiment of the present application;

[0051] FIG2 is a flow chart of a method for planning inspection points provided in an embodiment of the present application;

[0052] FIG3 is a flow chart of a method for planning inspection points according to an embodiment of the present application;

[0053] FIG4 is a flow chart of a method for generating inspection points provided in an embodiment of the present application;

[0054] FIG5 is a schematic diagram of a point adaptation module provided in an embodiment of the present application;

[0055] FIG6 is a schematic diagram of a pan / tilt self-correction module provided in an embodiment of the present application;

[0056] FIG7 is a schematic structural diagram of an inspection point planning device provided in an embodiment of the present application;

[0057] FIG8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0058] FIG9 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0059] FIG10 is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0061] In order to facilitate understanding of the technical content of this application, several technical terms involved in this application are explained below.

[0062] Robotic Inspection: Inspection involves checking and registering the environment and equipment at the inspection site to promptly identify faults and potential hazards. Inspections enable standardized, process-based error detection, enabling timely identification and resolution of issues to prevent them from occurring. Robotic inspection, on the other hand, utilizes robots to perform inspection tasks, improving efficiency, reducing costs, and minimizing casualties. Robotic inspection can be applied in a wide range of fields, including power inspection, oil pipeline inspection, highway inspection, and fire scene inspection.

[0063] Inspection point planning: Inspection point information includes location, orientation, camera pose, camera magnification, and focal length of the equipment performing the inspection. During the deployment phase, the location and orientation of inspection points for objects to be inspected within the inspection site must be determined. This is known as inspection point planning, which guides the execution of inspection tasks.

[0064] Digital twin technology: Digital twin is an enabling technology that supports the digital transformation of the economy and society. It comprehensively uses information technologies such as perception, computing, and modeling to represent, diagnose, predict, and make decisions about physical space through software definition, thereby realizing the interactive mapping of physical space and cyberspace.

[0065] The implementation environment of the embodiments of the present application is introduced below.

[0066] FIG1 is a schematic diagram of an implementation environment for a method for planning inspection points provided in an embodiment of the present application. As shown in FIG1 , the implementation environment includes a computing device 101 and a device 102 for performing inspection tasks. The computing device 101 and the device 102 for performing inspection tasks are connected to each other via a wired network or a wireless network.

[0067] Among them, the computing device 101 can be at least one of a desktop computer, a laptop, a wireless terminal and a laptop portable computer. The computing device 101 has a communication function and can access the Internet. The computing device 101 can generally refer to one of multiple computing devices. The embodiment of the present application is only illustrated by the computing device 101. Those skilled in the art will know that the number of the above-mentioned computing devices can be more or less. The computing device 101 can also be an independent physical server, or a server cluster composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (content delivery networks, CDNs), and big data and artificial intelligence platforms. The embodiment of the present application does not limit this. For example, the computing device 101 provides a patrol point planning function, which can determine the position and orientation of the object to be detected in the patrol site and the feasible area in the patrol site based on the 3D point cloud map of the patrol site, and then determine the position of the patrol point of the object to be detected in the feasible area. In some embodiments, the computing device 101 also provides a patrol point entry function, which can enter the position and orientation of each determined patrol point into the device that performs the patrol task. This embodiment of the present application does not limit this.

[0068] In some embodiments, the computing device 101 is associated with a data acquisition device, which can import the data of the site to be inspected collected by the data acquisition device into the computing device 101, and then perform digital twin modeling based on the imported data to generate a 3D point cloud map of the site to be inspected. The data acquisition device can be a backpack mapping device, which includes two 16-line laser radars and four fisheye cameras perpendicular to each other. The mapping device can collect a 3D color point cloud of the area to be inspected with an error of no more than 1 centimeter (cm). The embodiments of the present application do not limit the type of data acquisition device.

[0069] The device 102 for performing inspection tasks may be a robot or other device that can be used to perform inspection tasks, and the device for performing inspection tasks is equipped with a camera. The robot includes a robot body, a camera, and a pan-tilt head. The robot body is used to control the movement and orientation of the robot; the camera is used to capture images; and the pan-tilt head is used to control the orientation of the camera's field of view. Exemplarily, the device 102 for performing inspection tasks can, based on the device's navigation map of the inspection site and the recorded inspection points, reach the location of the inspection point and adjust the orientation of the camera's field of view to the orientation of the inspection point, thereby capturing images corresponding to the inspection point to detect the object to be detected corresponding to the inspection point.

[0070] In some embodiments, the wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, peer-to-peer (P2P) communication is implemented between the computing device 101 and the device 102 performing the inspection task based on a remote procedure call protocol (RPC). In some embodiments, technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. are used between the computing device 101 and the device 102 performing the inspection task to represent data blocks exchanged over the network. In addition, all or some links can be encrypted using conventional encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), internet protocol security (IPsec), etc. In other embodiments, customized and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0071] An embodiment of the present application provides a method for planning inspection points, wherein, based on a 3D point cloud map of the site to be inspected, the position and orientation of the object to be inspected and the feasible area in the site to be inspected are determined, and then, based on the feasible area and the position and orientation of the object to be inspected, the position of the inspection point of the object to be inspected is determined. In this method, the positions of the inspection points can be automatically planned in the 3D point cloud map, without the need for manually remotely controlling the equipment performing the inspection task to reach the vicinity of each object to be inspected and then determine the position of the inspection point. The inspection point planning process is time-saving and is not affected by extreme environments, which can improve the deployment efficiency of the inspection and reduce labor costs.

[0072] The above inspection point planning method can be applied to substation inspection point planning scenarios, campus inspection scenarios, and scenarios requiring long-term safety monitoring, such as computer rooms, tunnels, and granaries. The following describes each of these application scenarios.

[0073] (1) Planning scenario of substation inspection points: The safety of the power system plays an irreplaceable role in the development of the national economy. Any accident in any link may lead to a chain reaction, causing large-scale power outages, personal injury, damage to main equipment, and even catastrophic accidents such as the collapse of the entire network. As a key node connecting the backbone network and the distribution network, how to ensure its normal operation is directly related to the stability and safety of the entire power system. In order to ensure real-time monitoring of the operating status of major primary equipment such as main transformers, busbars, switches, etc. in the substation, it is necessary to inspect and maintain the substation. Traditional substation monitoring and inspection are mainly done manually, using human senses to make simple qualitative judgments on the equipment, mainly through methods such as seeing, touching, listening, and smelling. However, manual inspection has many shortcomings. Traditional manual inspection methods have shortcomings such as high labor intensity, low work efficiency, scattered inspection quality, and single means. The data of manual inspection cannot be accurately and timely connected to the management information system. In addition, with the promotion of unmanned mode, the inspection workload is increasing, and the inspection rate and timeliness cannot be guaranteed. Furthermore, manual inspections pose significant safety risks and lack effective inspection methods in harsh weather conditions such as high altitude, hypoxic conditions, and cold weather. Inclement weather conditions such as strong winds, fog, ice and snow, hail, and thunderstorms also make timely inspections impossible. Therefore, traditional substation inspection methods urgently need a new successor. Power inspection robots, operating autonomously or remotely, inspect outdoor high-voltage equipment in unmanned or understaffed substations. They can promptly detect abnormalities such as defects and suspended foreign objects in power equipment, automatically triggering alarms or implementing pre-configured troubleshooting procedures. Their flexible and reliable operation truly reduces staffing and increases efficiency, accelerating the transition to unmanned substations. Planning inspection points and recording information about these points are essential deployment steps for power inspection robots.

[0074] (2) Park inspection scenario: With the development of the economy, artificial intelligence has gradually come into people's sight. The construction of smart parks and intelligent communities has brought a series of conveniences to residents. Traditional patrol and security work has moved towards "smart patrol" and "three-dimensional patrol" based on artificial intelligence technology, using patrol robots to replace manual patrol. Similarly, patrol point planning and point information recording are also necessary deployment tasks for park patrol robots.

[0075] (3) Scenarios that require long-term monitoring of safety status, such as computer rooms, tunnels, and grain warehouses: Inspection scenarios such as computer rooms, tunnels, and grain warehouses all have objects to be inspected. Therefore, inspection point planning and point information recording are also necessary deployment tasks for inspection robots in these inspection scenarios.

[0076] The following describes a method for planning inspection points provided by an embodiment of the present application. FIG2 is a flowchart of a method for planning inspection points provided by an embodiment of the present application. As shown in FIG2 , taking the method executed by a computing device as an example, the method includes the following steps 201 to 205 .

[0077] 201. The computing device performs digital twin modeling of the site to be inspected based on the data of the site to be inspected collected by the data acquisition device to obtain a 3D point cloud map of the site to be inspected.

[0078] The computing device is associated with a data acquisition device that can import data collected by the data acquisition device about the site to be inspected into the computing device, and then perform digital twin modeling based on the imported data to generate a 3D point cloud map of the site to be inspected. The data acquisition device can be a backpack mapping device that includes two mutually perpendicular 16-line laser radars and four fisheye cameras. The mapping device can collect a 3D color point cloud of the area to be inspected with an error of no more than 1 centimeter (cm). The embodiments of this application do not limit the type of data acquisition device.

[0079] It should be noted that step 201 is an optional step. In some embodiments, step 201 may not be performed, that is, the computing device directly obtains the generated 3D point cloud map and executes subsequent steps 202 to 205 on the 3D point cloud map without the need for the computing device to perform digital twin modeling.

[0080] 202. The computing device determines, based on the 3D point cloud map of the site to be inspected, a position and orientation of a first object among a plurality of objects to be inspected in the site to be inspected.

[0081] Among them, the objects to be inspected in the inspection site are also the inspection targets in the inspection site. Taking the substation inspection scenario as an example, the inspection site is the substation, and the objects to be inspected include meters, insulators, voltage transformers, current transformers, grounding switches, disconnector tubes, bushings, lightning arrester rings, drain wire clamps, electrical boxes, ladder locks, insulator connections, etc., which are not limited in the embodiments of this application.

[0082] The first object is any one of the multiple objects to be detected. The position of the first object is described by the coordinates of the midpoint of the first object in the 3D point cloud map. In some embodiments, the position of the first object is the coordinates of the midpoint of the key component of the first object in the 3D point cloud map. The position of the first object can be determined according to the rules corresponding to actual needs, and this embodiment of the application does not limit this. The orientation of the first object is described by the normal direction of the plane where the front view of the first object is located.

[0083] The computing device automatically identifies the position and orientation of the object to be detected on a 3D point cloud map based on a 3D point cloud detection algorithm, such as BirdNet. In the above method, the computing device automatically identifies the position and orientation of the first object using a point cloud detection algorithm, eliminating the need to manually control a remote control robot to move near the object to be detected and then determine its position and orientation. This improves the efficiency of inspection point planning and inspection deployment, reduces labor costs, and is unaffected by extreme environments.

[0084] In some embodiments, the computing device uses human-computer interaction to determine an object selected in the 3D point cloud map of the site to be inspected as the object to be detected. The position and orientation of the selected object are also the position and orientation of the object to be detected. In the above method, the position and orientation of the object to be detected are determined through human-computer interaction, and the result is what is selected, which can ensure the accuracy and convenience of determining the object to be detected.

[0085] In other embodiments, the computing device first automatically detects the positions and orientations of multiple objects to be detected in the 3D point cloud map, and then uses human-computer interaction to adjust the multiple objects to be detected. For example, the positions and orientations of the objects to be detected that were detected incorrectly are deleted, the positions and orientations of the objects to be detected that were not detected are added, and at least one of the positions or orientations of the objects to be detected that have been detected is corrected. In the above method, an algorithm is first used for automatic detection, and then the results of the automatic detection are adjusted through human-computer interaction. This can ensure the accuracy of the position and orientation of the objects to be detected while ensuring the efficiency of determining the position and orientation of the objects to be detected, thereby reducing the impact of errors caused by automatic detection.

[0086] In some embodiments, the computing device is configured with a display module. After determining the positions and orientations of multiple objects to be detected in the site to be inspected, the computing device displays the determined positions and orientations of the objects to be detected in a 3D point cloud map.

[0087] 203. The computing device determines a feasible area in the site to be inspected based on the 3D point cloud map. The feasible area is an area that can be reached by the device performing the inspection task.

[0088] The computing device determines the convex hull range of the 3D point cloud map, and determines the ground area corresponding to the convex hull range as a feasible area in the site to be inspected.

[0089] In some embodiments, the computing device automatically identifies feasible areas on the 3D point cloud map based on a 3D point cloud segmentation algorithm, such as a point cloud convolution-based segmentation algorithm (PointNet). In the above method, the computing device uses the point cloud segmentation algorithm to identify feasible areas, which can prevent the equipment performing the inspection task from sounding when driving in the identified feasible area, thereby ensuring the accuracy and availability of the subsequently determined inspection points.

[0090] In some embodiments, the computing device uses human-computer interaction to determine the area selected in the 3D point cloud map of the site to be inspected as the feasible area. In the above method, the feasible area is determined by human-computer interaction, and the selected area is the result, which can ensure the accuracy and convenience of determining the feasible area.

[0091] In other embodiments, the computing device is configured with a display module. The computing device first determines the convex hull range in the 3D point cloud map, displays the ground area corresponding to the determined convex hull range in the 3D point cloud map as a feasible area, and then uses human-computer interaction to adjust the displayed feasible area. For example, it may delete erroneously determined feasible areas, add undetermined feasible areas, and correct at least one of the position or range of determined feasible areas. In the above method, the algorithm is first used to automatically segment the feasible area, and then the automatically segmented feasible area is adjusted through human-computer interaction. This can ensure the accuracy of the determined feasible area while ensuring the efficiency of determining the feasible area, thereby reducing the impact of errors in the automatic segmentation algorithm.

[0092] It should be noted that the above-mentioned execution of step 202 first and then step 203 is only exemplary. In some embodiments, step 203 may be executed first and then step 202, or step 202 and step 203 may be executed simultaneously. The embodiments of the present application do not limit this.

[0093] 204. The computing device determines the location of the inspection point of the first object based on the feasible area and the location and orientation of the first object.

[0094] The process of the computing device determining the position of the inspection point of the first object includes: determining the intersection of the feasible area and the first plane, the first plane being parallel to the orientation of the first object and passing through the center of the first object, the intersection including multiple points; and determining the position of the inspection point of the first object from the multiple points in the intersection. The position of the inspection point can be the position of the point farthest from the center of the first object among the multiple points, so as to expand the shooting range of the camera carried by the equipment performing the inspection task; the position of the inspection point can also be the position of the point closest to the center of the first object among the multiple points, so as to improve the clarity of the image captured by the camera on the equipment performing the inspection task. For example, the computing device is based on the midpoint x of the first object. d and orientation to determine plane A (ax+by+cz+d=0), which is also the first plane; the computing device intersects plane A with the feasible region to obtain a point equation ex+fy=0 (x0<=x<=x1), where x, y, and z represent coordinate axes, a, b, c, d, e, and f are constants, and x0 and x1 are the intersection points of plane A and the boundaries of the feasible region; the computing device compares the midpoint x of the first object with the midpoint x of the first object. d With respect to the relationship between x0 and x1, if it is closer to x0, the position of the inspection point of the first object is x1; if it is closer to x1, the position of the inspection point of the first object is x0.

[0095] In some embodiments, step 204 is implemented through a point planning algorithm, which is used to abstract the physical optimization conditions of point planning and optimization into constraint parameters of the algorithm to achieve optimized output of inspection points. The point planning algorithm includes the following conditions: inspection points can only be located in feasible areas; inspection points are as far away from the objects to be detected as possible; inspection points are located in the direction of the objects to be detected; if the distance between multiple inspection points is less than a threshold, the multiple inspection points are merged into a single point.

[0096] 205. The computing device determines a direction of the inspection point of the first object based on the position of the inspection point of the first object and the position of the first object.

[0097] The computing device uses the position of the inspection point of the first object and the direction of the line connecting the position of the first object as the orientation of the inspection point of the first object, which is used to indicate the orientation of the camera field of view carried by the device performing the inspection task.

[0098] The above steps 202 to 205 introduce the process of determining the inspection point of the first object among multiple objects to be inspected in the inspection site, wherein the multiple objects to be inspected include the first object and the second object. If the distance between the inspection point of the first object and the inspection point of the second object is less than a threshold value, the computing device merges the inspection point of the first object and the inspection point of the second object to obtain a merged inspection point, and uses the merged inspection point as the inspection point of the first object and the second object. The process includes: the computing device determines the position of the inspection point of the second object based on the feasible area and the position and orientation of the second object; determines that the distance between the inspection point of the first object and the inspection point of the second object is less than a threshold value; determines the position of the merged inspection point based on the inspection points of the first object and the second object; uses the position of the merged inspection point as the position of the inspection point of the first object, and uses the position of the merged inspection point as the position of the inspection point of the second object. Wherein, determining the position of the merged inspection point based on the inspection point of the first object and the inspection point of the second object includes: determining the midpoint of the first inspection point and the second inspection point, and the midpoint is the position of the merged inspection point. In some embodiments, the computing device uses the direction of the line connecting the midpoint of the first object and the second object and the position of the merged inspection point as the orientation of the merged inspection point. It should be noted that if the distance between the merged inspection point and the inspection points of other objects in the object to be detected is less than a threshold value, the computing device merges the merged inspection point with the inspection points of other objects in the same manner as the above process until the merger cannot be performed. In the above method, multiple inspection points that can be merged are merged into one inspection point, so that the equipment performing the inspection task does not need to shoot different objects to be detected at multiple points that are close to each other, but only needs to shoot images of multiple objects to be detected at one point, which can improve the execution efficiency of the inspection task.

[0099] The following uses Figure 3 as an example to illustrate the process shown in the above steps 201 to 205. Figure 3 is a flow chart of a patrol point planning method provided in an embodiment of the present application. As shown in Figure 3, a digital twin model is performed on the patrol site to obtain a 3D color laser point cloud map; the 3D color laser point cloud map is used to detect the object to be detected through the 3D point cloud detection algorithm to obtain the position and orientation of the first object among the multiple objects to be detected in the patrol site; the 3D color laser point cloud map is used to segment the feasible area through the 3D point cloud segmentation algorithm to obtain the coordinate information of the feasible area; based on the feasible area and the position and orientation of the first object, the position of the patrol point of the first object is determined.

[0100] In the above method, the positions of inspection points can be automatically planned in the 3D point cloud map of the site to be inspected, without the need for manual remote control of the equipment performing the inspection task to reach the vicinity of each object to be inspected and then determine the position of the inspection point. The planning of the inspection points is time-consuming and not affected by extreme environments, which can improve the deployment efficiency of robot inspections and reduce labor costs; further, multiple inspection points that can be merged are merged into one inspection point, so that the equipment performing the inspection task does not need to shoot different objects to be inspected at multiple points that are close to each other, but only needs to shoot images of multiple objects to be inspected at one point, which can improve the execution efficiency of the inspection task.

[0101] The embodiment shown in FIG2 above introduces the process of planning inspection points in a site to be inspected based on a 3D point cloud map of the site to be inspected. Since there is an error between the navigation map used by the equipment performing the inspection task (such as a robot) when inspecting the site to be inspected and the 3D point cloud map, it is necessary to adapt the planned inspection points to the navigation map. The process is introduced below. FIG4 is a flowchart of a method for generating inspection points provided in an embodiment of the present application. Taking the process performed by a computing device as an example, the process includes the following steps 401 to 405.

[0102] 401. The computing device determines a plurality of first anchor points in a first map and a plurality of second anchor points in a second map, wherein the first map is a raster map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected for the device performing the inspection task, and the plurality of first anchor points and the plurality of second anchor points are in one-to-one correspondence.

[0103] The computing device converts the 3D point cloud map of the site to be inspected into a 2D grid map (GM) by rasterization to obtain the first map. The computing device converts the 3D point cloud map into a 2D grid map using a 3D point cloud projection method. This method can retain as much 3D information as possible based on the original resolution, thereby facilitating improved accuracy of the 2D grid map. The navigation map of the site to be inspected for the device performing the inspection task is a 2D grid map.

[0104] The process of determining a plurality of first anchor points in the first map by the computing device includes any of the following methods.

[0105] The first method: Based on a key point detection algorithm, multiple anchor points are determined in the first map, and the multiple anchor points are determined as multiple first anchor points.

[0106] In some embodiments, the computing device uses the Harris corner detection algorithm and a gradient operator such as Sobel to calculate the gradient of each point in the first map in the x and y directions. and For each point in the first map, construct Find the corner response function det(M)-α·trace(M) 2 , multiple points whose corner point response function values ​​(response values) are greater than a preset threshold are determined as multiple anchor points, and these multiple anchor points are determined as multiple first anchor points. Where w is a weighting function, which can be a constant or a Gaussian weighting function; α is an empirical constant.

[0107] It should be noted that the above step 401 is described by taking the Harris corner detection algorithm as an example to determine the anchor point. In some embodiments, the anchor point is determined based on other methods, which is not limited in the embodiments of the present application.

[0108] The second method: using a human-computer interaction method, multiple anchor points are determined in the first map, and the multiple anchor points are determined as multiple first anchor points.

[0109] Among them, determining multiple anchor points in the first map using human-computer interaction includes: receiving a user's anchor configuration instruction, the anchor configuration instruction is used to determine multiple first anchor points in the first map; and determining multiple anchor points in the first map based on the anchor configuration instruction.

[0110] In some embodiments, for a computing device configured with a display module, the computing device is capable of displaying a first map and displaying multiple anchor points in the first map. Therefore, a human-computer interaction method can be used to select multiple points in the first map displayed by the computing device. The selected multiple points are anchor points, and the computing device determines the multiple anchor points as multiple first anchor points.

[0111] The third method: Based on the key point detection algorithm, multiple anchor points are determined in the first map, and at least one of the multiple anchor points is adjusted through human-computer interaction to obtain multiple first anchor points.

[0112] The process of determining multiple anchor points in the first map based on the key point detection algorithm by the computing device is the same as the first method described above and will not be repeated here. Adjusting at least one of the multiple anchor points to obtain multiple first anchor points includes at least one of the following: adjusting the position of at least one of the multiple anchor points; deleting at least one of the multiple anchor points; adding at least one anchor point to the first map. In some embodiments, the computing device receives an anchor configuration instruction from the user, the anchor configuration instruction being used to add at least one first anchor point to the first map, delete at least one first anchor point from the multiple first anchor points, or adjust the position of at least one of the multiple first anchor points; the computing device adjusts the multiple first anchor points based on the anchor configuration instruction.

[0113] It should be noted that the above three methods are all described using the first map as an example. The process of the computing device determining multiple second anchor points in the second map is the same as the process of the computing device determining multiple second anchor points in the first map, and will not be repeated.

[0114] 402. The computing device determines a coordinate transformation matrix based on the plurality of first anchor points and the plurality of second anchor points. The coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map.

[0115] The coordinate transformation relationship between the coordinate system of the first map and the coordinate system of the second map can be described by the coordinate transformation relationship between the anchor points in the first map and the second map. The computing device matches the multiple first anchor points with the multiple second anchor points one by one to obtain multiple groups of anchor points, and determines the coordinate transformation relationship between the first anchor point and the second anchor point in each group of anchor points, that is, the coordinate transformation matrix. The computing device matches the multiple first anchor points with the multiple second anchor points one by one based on a feature matching algorithm. In some embodiments, the computing device also performs anchor point matching based on other methods, for example, dividing the multiple first anchor points and the multiple second anchor points into different groups, and determining the first anchor points and the second anchor points with the same group number and sequence number within the group as a group of matching anchor points. In other embodiments, the multiple first anchor points and the multiple second anchor points are matched one by one based on human-computer interaction, for example, the matching anchor points are marked as a group, which is not limited in the embodiments of the present application.

[0116] The coordinate transformation matrix can be a homography matrix or an affine matrix, etc., which is not limited in the present embodiment. Taking the coordinate transformation matrix as a homography matrix as an example, in any set of anchor points, the coordinate transformation relationship from the first anchor point (x1, y1) to the second anchor point (x2, y2) can be expressed as the following formula (1):

[0117] In the above formula (1), H is the coordinate transformation matrix, and its value is

[0118] Expanding and arranging the above formula (1) in the form of a polynomial, we obtain the following coordinate system transformation formulas (2) and (3).

[0119] Each set of anchor points determined can list the coordinate system transformation equations corresponding to the above formulas (2) and (3). Based on multiple equations, the h in the coordinate transformation matrix can be solved. 00 ~h 22 , and then obtain the coordinate transformation matrix H.

[0120] It should be noted that the above description is based on the example of the coordinate transformation matrix being a homography matrix. The same applies when the coordinate transformation matrix is ​​an affine matrix or other matrix, and will not be repeated here.

[0121] 403. The computing device transforms at least one first inspection point of the equipment performing the inspection task into at least one corresponding second inspection point through a coordinate transformation matrix, wherein the at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

[0122] The computing device multiplies the coordinates of the first inspection point by the coordinate transformation matrix to obtain the corresponding second inspection point.

[0123] In the above steps 401 to 403, the planned first inspection point is automatically generated into the second inspection point used for actual inspection based on the anchor point, which can reduce the impact of errors between different maps on actual inspections and realize the reuse of planned inspection points on different equipment performing inspection tasks, thereby eliminating the need for multiple inspection point planning and saving labor costs.

[0124] In some embodiments, the process of generating a second inspection point based on a first inspection point includes two stages: generating a coarse inspection point and fine-tuning the coarse inspection point. The process of generating the coarse inspection point is used to determine the corresponding position of the first inspection point planned on the first map on the second map (i.e., the second inspection point). This process is similar to steps 401 to 403 above and will not be repeated here. The process of fine-tuning the coarse inspection point is used to fine-tune the direction of the camera field of view of the second inspection point to compensate for the error in the process of generating the coarse inspection point and obtain a precise inspection point. Taking the fine-tuning of any second inspection point as an example, this process includes the following steps 404 and 405.

[0125] 404. The computing device obtains an image captured at the second inspection point by a camera carried by the device performing the inspection task.

[0126] The camera's position is adjustable. The second inspection point is the coarse point obtained during the coarse point generation phase. The device performing the inspection task, based on the second map, arrives at the location indicated by the second inspection point, captures an image at the second inspection point, and transmits the captured image to the computing device.

[0127] 405. The computing device adjusts the posture of the camera corresponding to the second inspection point based on the image to obtain the direction of the camera field of view of the second inspection point.

[0128] In the image captured by the camera of the device performing the inspection task in the adjusted position, the center point of the object to be inspected is located at the center point of the image. In some embodiments, the device performing the inspection task carries the camera via a gimbal, the position of which is adjustable, and the computing device adjusts the position of the camera by adjusting the position of the gimbal.

[0129] The process in which the computing device adjusts the posture of the camera corresponding to the second inspection point based on the image to obtain the orientation of the camera field of view of the second inspection point includes the following steps 405A to 405B.

[0130] 405A. Determine the object to be detected in the image based on a template matching algorithm.

[0131] The template refers to an image of an object to be detected that is manually or pre-set at the inspection site. The computing device uses a key point detection algorithm to determine multiple object feature points in the image. Using a template matching algorithm, the computing device matches these multiple object feature points with the template to determine the object to be detected in the image.

[0132] 405B. Based on the position of the center point of the object to be detected in the image, adjust the position of the camera to control the center point of the camera's field of view to align with the center point of the object to be detected.

[0133] The center point of the camera's field of view is also the center point of the image captured by the camera. The computing device determines the relative positional relationship between the center point of the object to be detected and the center point of the camera's field of view based on the position of the center point of the object to be detected in the image, and then adjusts the camera's posture based on the relative positional relationship to control the center point of the camera's field of view to align with the center point of the object to be detected. In some embodiments, the computing device adjusts the camera's posture at least once. After each adjustment, the camera retakes an image and sends it to the computing device. The computing device performs the next adjustment based on the retaken image until the center point of the camera's field of view aligns with the center point of the object to be detected.

[0134] If the center point of the object to be detected is exactly in the center of the image, there is no need to adjust the camera's position.

[0135] In some embodiments, the position of the camera is adjusted based on the position of the center point of the object to be detected in the image, so as to control the field of view of the camera to include the object to be detected to the greatest extent.

[0136] It should be noted that the above steps 404 and 405 are described by taking one second inspection point as an example, and the same applies to other second inspection points, which will not be described in detail.

[0137] It should be noted that the process of fine-tuning the second inspection point shown in steps 404 and 405 can occur during the formal inspection process, that is, the device used to perform the inspection task arrives at each second inspection point in sequence based on the second map. Each time the device performing the inspection task arrives at a second inspection point, the computing device fine-tunes the second inspection point to obtain a fine-tuned second inspection point. The device performing the inspection task completes the current inspection task based on the fine-tuned second inspection point and then arrives at the next second inspection point. The fine-tuned second inspection point recorded in this process can be directly applied to the next inspection. The process shown in steps 404 and 405 can also occur during a pre-inspection process before the formal inspection. During this process, the computing device obtains the fine-tuned second inspection point and records the fine-tuned second inspection point. However, the device performing the inspection task does not perform the inspection task at this point at this time. Instead, it arrives at the next second inspection point and records the next fine-tuned second inspection point, until all fine-tuned second inspection points have been recorded. During the formal inspection, the pre-inspected equipment can directly perform inspections based on the recorded fine-tuned second inspection points. The computing device can also enter the fine-tuned second inspection points into the device corresponding to the same second map as the pre-inspected equipment, so that the device can directly perform inspections based on the fine-tuned second inspection points without the need for temporary point fine-tuning.

[0138] The following uses Figures 5 and 6 as examples to illustrate the process shown in steps 401 to 405 above. Figure 5 is a schematic diagram of a point adaptation module provided in an embodiment of the present application. The point adaptation module can be deployed on a computing device or on a device such as a robot that performs inspection tasks. Figure 5 corresponds to the stage of generating rough points (i.e., steps 401 to 403 above). As shown in Figure 5, Map 1 is the first map, which can be a 2D grid map or a 3D point cloud map. Map 2 is the second map, which can be a 2D grid map or a 3D point cloud map. The point adaptation module includes a map 2Dization unit, an anchor point generation unit, a map matching unit, and a point adaptation unit. If there is a 3D point cloud map in the first map and the second map, the 3D point cloud map is rasterized by the map 2Dization unit. If both the first map and the second map are 2D grid maps, no modal conversion is required. The anchor point generation unit determines multiple first anchor points in the first map and multiple second anchor points in the second map. The map matching unit performs a one-to-one matching of multiple first anchor points in the first map with multiple second anchor points in the second map to generate multiple sets of anchor points. Based on these sets of anchor points, a coordinate transformation matrix between the first map and the second map is determined. Based on the coordinate transformation matrix, the point adaptation unit transforms the first inspection point in the first map into the second inspection point in the second map. This second inspection point is then converted into the generated coarse point.

[0139] FIG6 is a schematic diagram of a pan-tilt self-correction module provided in an embodiment of the present application. The pan-tilt self-correction module can be deployed on a computing device, and can also be deployed on a device such as a robot that performs inspection tasks. FIG6 corresponds to the above-mentioned fine-tuning stage of the coarse point position (that is, the above-mentioned steps 404 to 405). As shown in FIG6 , the pan-tilt self-correction module includes a template matching unit based on image features and a pan-tilt control unit. Among them, the image captured at the coarse point position by the device that performs the inspection task is processed by the template matching module based on image features to determine the coordinates and confidence of the center point of the object to be detected in the image; the pan-tilt control unit controls the pan-tilt posture based on the relative position between the center point of the object to be detected and the center point of the image, until the center of the object to be detected is moved to the center of the camera's field of view, and the obtained point position is recorded as the precise point position (that is, the second inspection point position after fine-tuning).

[0140] It should be noted that the above steps 404 and 405 are optional steps. In some embodiments, steps 404 and 405 may not be performed, and the embodiments of the present application do not limit this.

[0141] It should be noted that the above steps 401 to 405 are explained by taking the computing device as the execution body as an example. In some embodiments, the above steps 401 to 405 are executed by a computing device cluster. In other embodiments, the above steps 401 to 405 are executed by robots and other devices that perform inspection tasks, that is, the inspection point generation function is built into the device that actually performs the inspection task. In some other embodiments, the above steps 401 to 403 are executed by a computing device, and steps 404 and 405 are executed by the device that performs the inspection task, that is, the computing device completes the generation of the rough point, and the device that performs the inspection task completes the fine adjustment of the rough point. The embodiments of the present application do not limit this.

[0142] In the above method, the computing device automatically adapts the planned inspection points and the inspection points used for actual inspections based on the anchor points, which can reduce the impact of errors between different maps on actual inspections without the need to manually re-enter the inspection points, saving labor costs. Moreover, since the inspection points between different maps can be adapted, the inspection points obtained based on one inspection point planning can be adapted to the navigation maps of multiple devices performing inspection tasks, thereby realizing the reuse of inspection points on different devices without the need to plan inspection points for each device, which can improve the deployment efficiency of inspections and further save labor costs. Furthermore, on the basis of the coarse points generated based on the anchor points, the coarse points are fine-tuned based on the images taken by the equipment performing the inspection at the coarse points to compensate for the errors caused by the algorithm in the coordinate system transformation process based on the anchor points, so as to obtain more accurate inspection points, thereby ensuring that the equipment performing the inspection tasks collects high-quality images during the inspection, thereby improving the accuracy of the inspection results.

[0143] FIG7 is a schematic structural diagram of a patrol point planning device provided in an embodiment of the present application. As shown in FIG7 , the device includes a first determination module 701 , a second determination module 702 , and a third determination module 703 .

[0144] The first determining module 701 is configured to determine the position and orientation of a first object among a plurality of objects to be detected in the site to be inspected based on a 3D point cloud map of the site to be inspected;

[0145] The second determination module is used to determine a feasible area in the site to be inspected based on the 3D point cloud map, where the feasible area is an area that can be reached by the equipment performing the inspection task;

[0146] The third determining module is configured to determine the position of the inspection point of the first object based on the feasible area and the position and orientation of the first object.

[0147] Optionally, the device further comprises:

[0148] The fourth determination module is used to use the direction of the line connecting the position of the inspection point of the first object and the position of the first object as the orientation of the inspection point of the first object, which is used to indicate the direction of the camera field of view carried by the equipment performing the inspection task.

[0149] Optionally, the third determining module 703 is configured to:

[0150] Determine an intersection of the feasible region and a first plane, where the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes a plurality of points;

[0151] The position of the inspection point of the first object is determined from the multiple points in the intersection.

[0152] Optionally, the multiple objects to be detected include a first object and a second object, and the first determining module is further configured to:

[0153] Determine the position of the inspection point of the second object based on the feasible area and the position and orientation of the second object;

[0154] The device also includes:

[0155] a distance determination module, configured to determine whether a distance between an inspection point of a first object and an inspection point of a second object is less than a threshold;

[0156] a position determination module, configured to determine a position of a combined inspection point based on the inspection point of the first object and the inspection point of the second object;

[0157] The third determining module 703 is further configured to: use the position of the merged inspection point as the position of the inspection point of the first object, and use the position of the merged inspection point as the position of the inspection point of the second object.

[0158] Optionally, the location determination module is configured to:

[0159] Determine the midpoint between the first inspection point and the second inspection point. The midpoint is the location of the merged inspection point.

[0160] Optionally, the device further comprises:

[0161] The orientation determining module is configured to use the direction of a line connecting the midpoints of the first object and the second object and the position of the combined inspection point as the orientation of the combined inspection point.

[0162] Optionally, the device further comprises:

[0163] The display module is used to display the location of inspection points in the 3D point cloud map.

[0164] Optionally, the device further comprises:

[0165] The modeling module is used to perform digital twin modeling of the inspection site and obtain a 3D point cloud map of the inspection site.

[0166] Optionally, the device further comprises:

[0167] An anchor point determination module, configured to determine a plurality of first anchor points in a first map and a plurality of second anchor points in a second map, wherein the first map is a grid map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected by the device performing the inspection task, and the plurality of first anchor points and the plurality of second anchor points are in one-to-one correspondence;

[0168] a matrix determination module, configured to determine a coordinate transformation matrix based on the plurality of first anchor points and the plurality of second anchor points, wherein the coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map;

[0169] The coordinate transformation module is used to transform the coordinates of at least one first inspection point of the equipment performing the inspection task into the coordinates of at least one corresponding second inspection point through a coordinate transformation matrix. The at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

[0170] Optionally, the anchor point determination module is used to:

[0171] Based on a key point detection algorithm, a plurality of first anchor points are determined in the first map.

[0172] Optionally, the anchor point determination module is used to:

[0173] receiving an anchor point configuration instruction from a user, where the anchor point configuration instruction is used to add at least one first anchor point to the first map, delete at least one first anchor point from a plurality of first anchor points, or adjust a position of at least one first anchor point from the plurality of first anchor points;

[0174] Based on the anchor point configuration instruction, multiple first anchor points are adjusted.

[0175] Optionally, the device performing the inspection task is equipped with a camera, the position of the camera is adjustable, and the device further includes:

[0176] An acquisition module is used to acquire images taken by the camera at the second inspection point;

[0177] The adjustment module is used to adjust the posture of the camera corresponding to the second inspection point based on the image to obtain the direction of the camera field of view of the second inspection point.

[0178] Optionally, the adjustment module is used to:

[0179] Based on the template matching algorithm, determine the object to be detected in the image;

[0180] Based on the position of the center point of the object to be detected in the image, the camera's position is adjusted to control the center point of the camera's field of view to align with the center point of the object to be detected.

[0181] The first determination module 701, the second determination module 702, and the third determination module 703 can all be implemented via software or hardware. For example, the implementation of the first determination module 701 will be described below using the first determination module 701 as an example. Similarly, the implementation of the second determination module 702 and the third determination module 703 can refer to the implementation of the first determination module 701.

[0182] As an example of a software functional unit, the first determination module 701 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the first determination module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0183] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0184] As an example of a hardware functional unit, the first determination module 701 may include at least one computing device, such as a server. Alternatively, the first determination module 701 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0185] The multiple computing devices included in the first determination module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the first determination module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first determination module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0186] It should be noted that, in other embodiments, the steps that the above modules are responsible for implementing can be specified as needed, and the above modules can respectively implement different steps in the above inspection point planning method to realize all the functions of the above device. That is, the inspection point planning device provided by the above embodiment only uses the division of the above functional modules as an example when implementing the inspection point planning method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided by the above embodiment and the corresponding method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0187] The present application also provides a computing device 800. FIG8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in FIG8, computing device 800 includes: a bus 801, a processor 802, a memory 803, and a communication interface 804. Processor 802, memory 803, and communication interface 804 communicate with each other via bus 801. Computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 800.

[0188] Bus 801 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 shows a single bus line, but this does not imply a single bus or type of bus. Bus 801 may include a path for transmitting information between various components of computing device 800 (e.g., memory 803, processor 802, and communication interface 804).

[0189] The processor 802 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0190] The memory 803 may include a volatile memory, such as a random access memory (RAM). The memory 803 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0191] Memory 803 stores executable program code, and processor 802 executes the executable program code to implement the functions of the first determination module 701, the second determination module 702, and the third determination module 703, respectively, thereby implementing the inspection point planning method. In other words, memory 803 stores program code for executing the inspection point planning method.

[0192] The communication interface 804 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0193] The present application also provides a computing device cluster. Figure 9 is a schematic diagram of a computing device cluster provided in an embodiment of the present application. As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 803 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the inspection point planning method.

[0194] In some possible implementations, the memory 803 of one or more computing devices 800 in the computing device cluster may also store a portion of the program code for executing the inspection point planning method. In other words, the combination of one or more computing devices 800 can jointly execute the program code for executing the inspection point planning method.

[0195] It should be noted that the memory 803 in different computing devices 800 in the computing device cluster can store different program codes, each for executing a portion of the functions of the inspection point planning apparatus. In other words, the program codes stored in the memory 803 in different computing devices 800 can implement the functions of one or more of the first determination module 701, the second determination module 702, and the third determination module 703.

[0196] It should be understood that the functions of the computing device 800 shown in FIG. 9 may also be performed by multiple computing devices 800 .

[0197] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), among others. FIG. 10 illustrates one possible implementation. FIG. 9 is a schematic diagram of a possible implementation of a computing device cluster provided by an embodiment of the present application. As shown in FIG. 10 , two computing devices 800 are connected via a network. Specifically, the network is connected via a communication interface in each computing device.

[0198] This application implements a computer program product containing instructions. The computer program product may be software or a program product containing program code that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device implements the inspection point planning method provided in the above embodiment.

[0199] The embodiments of the present application provide a computer-readable storage medium, which can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes program code. When the program code is executed by a computing device, the computing device executes the inspection point planning method provided in the above embodiment; when the program code is executed by a computing device cluster, the computing device cluster executes the inspection point planning method provided in the above embodiment.

[0200] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the 3D point cloud maps involved in this application were obtained with full authorization.

[0201] Those skilled in the art will appreciate that the various method steps and units described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0202] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0204] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0205] In addition, each unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software units.

[0206] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computing device (which can be a personal computer, a server, or a computing device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0207] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items that have substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on quantity and order of execution. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various examples, the first inspection point may be referred to as the second inspection point, and similarly, the second inspection point may be referred to as the first inspection point. Both the first inspection point and the second inspection point may be inspection points, and in some cases, may be separate and different inspection points.

[0208] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, "plurality of inspection points" means two or more inspection points. The terms "system" and "network" are often used interchangeably herein.

[0209] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that..." or "if [stated condition or event] is detected" may be interpreted to mean "upon determining that..." or "in response to determining that..." or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0210] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0211] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0212] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state drive)), etc.

[0213] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0214] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for planning inspection points, characterized in that: For inspecting a plurality of objects to be inspected in an inspection site, the method comprises: Determine the position and orientation of a first object among the plurality of objects to be detected in the site to be inspected based on the 3D point cloud map of the site to be inspected; Based on the 3D point cloud map, determine a feasible area in the site to be inspected, where the feasible area is an area that can be reached by equipment performing the inspection task; Based on the feasible area and the position and orientation of the first object, the position of the inspection point of the first object is determined.

2. The method according to claim 1, characterized in that The method further comprises: The direction of the line connecting the position of the inspection point of the first object and the position of the first object is used as the orientation of the inspection point of the first object, and the orientation is used to indicate the orientation of the camera field of view carried by the equipment performing the inspection task.

3. The method according to claim 1 or 2, characterized in that: The determining the position of the inspection point of the first object includes: Determine an intersection of the feasible area and a first plane, where the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes a plurality of points; The position of the inspection point of the first object is determined from the multiple points in the intersection.

4. The method according to any one of claims 1 to 3, characterized in that: The plurality of objects to be detected include the first object and the second object, and the method further includes: Determine the position of the inspection point of the second object based on the feasible area and the position and orientation of the second object; Determining that a distance between an inspection point of the first object and an inspection point of the second object is less than a threshold; Determining a position of a merged inspection point based on the inspection point of the first object and the inspection point of the second object; The position of the combined inspection point is used as the position of the inspection point of the first object, and the position of the combined inspection point is used as the position of the inspection point of the second object.

5. The method according to claim 4, characterized in that The determining the position of the merged inspection point based on the inspection point of the first object and the inspection point of the second object includes: Determine a midpoint between the first inspection point and the second inspection point, the midpoint being the position of the combined inspection point.

6. The method according to claim 4, characterized in that The method further comprises: The direction of the line connecting the midpoint of the first object and the second object and the position of the combined inspection point is used as the orientation of the combined inspection point.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The locations of inspection points are displayed in the 3D point cloud map.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Digital twin modeling is performed on the site to be inspected to obtain a 3D point cloud map of the site to be inspected.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Determine a plurality of first anchor points in a first map, and determine a plurality of second anchor points in a second map, wherein the first map is a grid map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected by a device performing an inspection task, and the plurality of first anchor points and the plurality of second anchor points are in one-to-one correspondence; Based on the plurality of first anchor points and the plurality of second anchor points, determining a coordinate transformation matrix, wherein the coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map; Through the coordinate transformation matrix, the coordinates of at least one first inspection point of the equipment performing the inspection task are transformed into the coordinates of at least one corresponding second inspection point, the at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

10. The method according to claim 9, characterized in that The determining of a plurality of first anchor points in the first map comprises: Based on a key point detection algorithm, the plurality of first anchor points are determined in the first map.

11. The method according to claim 9 or 10, characterized in that: The determining of a plurality of first anchor points in the first map comprises: receiving an anchor point configuration instruction from a user, wherein the anchor point configuration instruction is used to add at least one first anchor point in the first map, delete at least one first anchor point from the plurality of first anchor points, or adjust the position of at least one first anchor point in the plurality of first anchor points; Based on the anchor point configuration instruction, the plurality of first anchor points are adjusted.

12. The method according to claim 9, characterized in that The device for performing the inspection task is equipped with a camera, and the position and posture of the camera is adjustable. The method further includes: Acquire an image captured by the camera at the second inspection point; Based on the image, the posture of the camera corresponding to the second inspection point is adjusted to obtain the direction of the camera field of view of the second inspection point.

13. The method according to claim 12, characterized in that The adjusting, based on the image, the posture of the camera corresponding to the second inspection point to obtain the direction of the camera field of view of the second inspection point includes: Determine the object to be detected in the image based on a template matching algorithm; Based on the position of the center point of the object to be detected in the image, the position and posture of the camera are adjusted to control the center point of the field of view of the camera to align with the center point of the object to be detected.

14. A patrol point planning device, characterized in that: Used for inspecting multiple objects to be inspected in an inspection site, the device comprises: A first determination module, configured to determine the position and orientation of a first object among the plurality of objects to be detected in the site to be inspected based on a 3D point cloud map of the site to be inspected; A second determination module is used to determine a feasible area in the site to be inspected based on the 3D point cloud map, where the feasible area is an area that can be reached by equipment performing the inspection task; The third determination module is used to determine the position of the inspection point of the first object based on the feasible area and the position and orientation of the first object.

15. The device according to claim 14, characterized in that The device also includes: The fourth determination module is used to use the direction of the line connecting the position of the inspection point of the first object and the position of the first object as the orientation of the inspection point of the first object, and the orientation is used to indicate the orientation of the camera field of view carried by the equipment performing the inspection task.

16. The device according to claim 14 or 15, characterized in that The third determining module is used to: Determine an intersection of the feasible area and a first plane, where the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes a plurality of points; The position of the inspection point of the first object is determined from the multiple points in the intersection.

17. The device according to any one of claims 14 to 16, characterized in that The device also includes: The display module is used to display the locations of inspection points in the 3D point cloud map.

18. The device according to claim 14, characterized in that The device also includes: An anchor point determination module, used to determine a plurality of first anchor points in a first map, and determine a plurality of second anchor points in a second map, wherein the first map is a grid map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected by a device performing an inspection task, and the plurality of first anchor points and the plurality of second anchor points are in one-to-one correspondence; a matrix determination module, configured to determine a coordinate transformation matrix based on the plurality of first anchor points and the plurality of second anchor points, wherein the coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map; A coordinate transformation module, used to transform the coordinates of at least one first inspection point of the equipment performing the inspection task into the coordinates of at least one corresponding second inspection point through the coordinate transformation matrix, wherein the at least one first inspection point is the coordinate system of the first map. The at least one second inspection point is a point in the coordinate system of the second map.

19. The device according to claim 18, characterized in that The anchor point determination module is used to: Based on a key point detection algorithm, the plurality of first anchor points are determined in the first map.

20. The device according to claim 18 or 19, characterized in that The anchor point determination module is used to: receiving an anchor point configuration instruction from a user, wherein the anchor point configuration instruction is used to add at least one first anchor point in the first map, delete at least one first anchor point from the plurality of first anchor points, or adjust the position of at least one first anchor point in the plurality of first anchor points; Based on the anchor point configuration instruction, the plurality of first anchor points are adjusted.

21. A computing device, characterized in that The computing device comprises a processor, and the processor is used to execute program code, so that the computing device executes the inspection point planning method according to any one of claims 1 to 13.

22. A computing device cluster, characterized in that: comprising at least one computing device, each of the computing devices comprising a processor and a memory; The processor of the at least one computing device is used to execute the program code stored in the memory of the at least one computing device, so that the computing device cluster executes the inspection point planning method according to any one of claims 1 to 13.

23. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is read by a processor of a computing device to enable the computing device to execute the inspection point planning method according to any one of claims 1 to 13.

24. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is read by a processor of a computing device in a computing device cluster so that the computing device cluster executes the inspection point planning method according to any one of claims 1 to 13.

Images