Inspection point planning method and device, computing equipment and storage medium

Abstract

The invention discloses an inspection point planning method and device, computing equipment and a storage medium, and belongs to the technical field of artificial intelligence. In the method, the position and orientation of a to-be-detected object and a feasible region in a to-be-inspected site are determined based on a 3D point cloud map of the to-be-inspected site, and then the position of an inspection point location of the to-be-detected object is determined based on the feasible region and the position and orientation of the to-be-detected object. According to the method, the position of the inspection point location can be automatically planned in the 3D point cloud map without manually remotely controlling the equipment executing the inspection task to reach the vicinity of each to-be-detected object to determine the position of the inspection point location, so that the inspection point location planning consumes less time and is not influenced by an extreme environment; the deployment efficiency of inspection point locations can be improved, and the labor cost is reduced.

Description

[0001] This application claims the priority of Chinese patent application with application number 202310796728.9 filed on June 30, 2023, and invention name “A point planning method and device”, and the priority of Chinese patent application with application number 202310911738.2 filed on July 24, 2023, and invention name “A patrol point configuration method, device and system”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computing equipment and storage medium for planning inspection points. Background Art

[0003] Robot inspection refers to the use of robots to replace manual inspection of objects to be inspected in the inspection site to improve inspection efficiency and reduce inspection costs. Before the inspection, the information of the inspection points needs to be entered into the robot to deploy the inspection of the objects to be inspected. Among them, the inspection point is also the robot's stop point. The information of the inspection point includes the location of the inspection point, the direction of the robot's body at the inspection point, the direction of the camera field of view on the robot, the camera magnification and the camera focal length.

[0004] In the related technology, the operator first remotely controls the robot to the vicinity of each object to be detected, and records the coordinates and orientation of the robot in the map; by remotely controlling the orientation of the robot's pan / tilt, the operator makes the center of the object to be detected be located at the center of the camera's field of view; adjusts the camera's magnification and focal length so that the camera can clearly capture the object to be detected and the object to be detected occupies the main part of the camera's field of view; and records the debugged parameters to obtain information on the inspection points of the object to be detected.

[0005] However, in the above method, it is necessary to manually control the robot to reach the vicinity of the object to be detected in order to determine the inspection point corresponding to the object to be detected. In a large inspection site, there may be nearly 10,000 points. The method of determining the inspection points by manual remote control is time-consuming and the recording efficiency is low in extreme environments, which greatly affects the deployment efficiency of the inspection points. Summary of the invention

[0006] The embodiments of the present application provide a method, device, computing device and storage medium for planning inspection points, which can shorten the time consumption of inspection point planning, and are not affected by extreme environments, and can improve the deployment efficiency of inspection points and reduce labor costs. The technical solution is as follows:

[0007] In a first aspect, a method for planning inspection points is provided for inspecting a plurality of objects to be inspected in an inspection site, the method comprising:

[0008] The position and orientation of a first object among multiple objects to be inspected are determined in a three-dimensional (3D) point cloud map of the site to be inspected, and a feasible area for equipment to perform the inspection task is determined in the 3D point cloud map, and then based on the position and orientation of the first object, the position of the inspection point of the first object is determined in the feasible area.

[0009] Among them, the objects to be detected in the site to be inspected are also the inspection targets in the site to be inspected. Taking the substation inspection scenario as an example, the site to be inspected is the substation, and the objects to be detected include meters, insulators, voltage transformers, current transformers, grounding switches, disconnector tubes, bushings, lightning arrester rings, drainage wire clamps, electrical boxes, ladder locks, insulator connections, etc., which are not limited in the embodiments of the present application. The first object can be any object to be detected in the site to be inspected. The position of the first object is described by the coordinates of the midpoint of the first object in the 3D point cloud map, and the orientation of the first object is described by the normal direction of the plane corresponding to the front view of the first object.

[0010] The device that performs the inspection task may be a device with an inspection function such as a robot, which is not limited in the present embodiment. The feasible area is an area that the device that performs the inspection task can reach.

[0011] In the above method, the locations of inspection points can be automatically planned in the 3D point cloud map of the site to be inspected, without the need for manual remote control of the robot to reach the vicinity of each object to be inspected and then determine the inspection points. The planning of inspection points takes a short time and is not affected by extreme environments. It can improve the deployment efficiency of inspection points and reduce labor costs.

[0012] Optionally, the method further comprises:

[0013] The direction of the line connecting the position of the inspection point of the first object and the position of the first object is used as the orientation of the inspection point of the first object, and the orientation is used to indicate the orientation of the camera field of view carried by the equipment performing the inspection task.

[0014] Optionally, determining the position of the inspection point of the first object includes:

[0015] Determine an intersection of a feasible area and a first plane, the first plane is parallel to the orientation of the first object and passes through the center of the first object, the intersection includes a plurality of points; determine the position of the inspection point of the first object from the plurality of points in the intersection.

[0016] The position of the inspection point can be the position of the point farthest from the center of the first object among the multiple points, so as to expand the shooting range of the camera carried by the equipment performing the inspection task; the position of the inspection point can also be the position of the point closest to the center of the first object among the multiple points, so as to improve the clarity of the image captured by the camera carried by the equipment performing the inspection task. d and orientation to determine plane A (ax+by+cz+d=0), which is also the first plane; intersect plane A with the feasible region to obtain the point equation ex+fy=0 (x0<=x<=x1), where x, y and z represent coordinate axes, a, b, c, d, e and f are constants, and x0 and x1 are the intersection points of plane A and the boundary of the feasible region; compare the midpoint x of the first object d With respect to the relationship between x0 and x1, if it is closer to x0, the position of the inspection point of the first object is x1, and if it is closer to x1, the position of the inspection point of the first object is x0.

[0017] Optionally, the plurality of objects to be detected include the first object and the second object, and the method further includes:

[0018] Based on the feasible area and the position and orientation of the second object, determine the position of the inspection point of the second object; determine that the distance between the inspection point of the first object and the inspection point of the second object is less than a threshold; based on the inspection point of the first object and the inspection point of the second object, determine the position of the merged inspection point; use the position of the merged inspection point as the position of the inspection point of the first object, and use the position of the merged inspection point as the position of the inspection point of the second object.

[0019] In the above method, multiple inspection points that can be merged are merged into one inspection point, and the equipment performing the inspection task can inspect multiple objects at one inspection point, which is conducive to improving the efficiency of the inspection.

[0020] Optionally, determining the position of the merged inspection point based on the inspection point of the first object and the inspection point of the second object includes:

[0021] Determine the midpoint between the first inspection point and the second inspection point, the midpoint being the position of the merged inspection point.

[0022] Optionally, the method further comprises:

[0023] The direction of the line connecting the midpoint of the first object and the second object and the position of the combined inspection point is used as the orientation of the combined inspection point.

[0024] Optionally, the method further comprises:

[0025] The locations of inspection points are displayed in the 3D point cloud map.

[0026] Optionally, the method further comprises:

[0027] A digital twin model is built for the site to be inspected to obtain a 3D point cloud map of the site to be inspected.

[0028] Optionally, the method further comprises:

[0029] Determine multiple first anchor points in the first map, and determine multiple second anchor points in the second map; determine a coordinate transformation matrix based on the multiple first anchor points and the multiple second anchor points; and transform the coordinates of at least one first inspection point of the equipment performing the inspection task into the coordinates of at least one corresponding second inspection point through the coordinate transformation relationship.

[0030] Among them, the first map is a grid map corresponding to the 3D point cloud map of the site to be inspected, the second map is a navigation map of the site to be inspected by the equipment performing the inspection task, and the multiple first anchor points and the multiple second anchor points are one-to-one corresponding. The coordinate transformation matrix is ​​used to represent the coordinate transformation relationship between the coordinate system of the first map and the coordinate system of the second map. The at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

[0031] In the above method, the planned inspection points and the inspection points used for actual inspections are automatically adapted based on anchor points, which can reduce the impact of errors between different maps on actual inspections without the need to manually re-enter the inspection points, thereby saving labor costs. In addition, since the inspection points between different maps can be adapted, the inspection points planned based on one inspection point can be adapted to the navigation maps of multiple devices that perform inspection tasks, thereby realizing the reuse of inspection points on different devices without the need to plan inspection points for each device, thereby improving the deployment efficiency of inspections and further saving labor costs.

[0032] Optionally, determining a plurality of first anchor points in the first map includes:

[0033] Based on a key point detection algorithm, a plurality of first anchor points are determined in the first map.

[0034] Optionally, the determining of a plurality of first anchor points in the first map includes:

[0035] Receive an anchor point configuration instruction from a user, where the anchor point configuration instruction is used to add at least one anchor point in a first map, delete at least one first anchor point from a plurality of first anchor points, or adjust the position of at least one first anchor point among a plurality of first anchor points; and adjust the plurality of first anchor points based on the anchor point configuration instruction.

[0036] Among them, the method of determining multiple second anchor points in the second map is the same as the method of determining multiple first anchor points in the first map, and will not be repeated here.

[0037] Optionally, the device for performing the inspection task is equipped with a camera, and the position and posture of the camera is adjustable. The method further includes:

[0038] An image captured by the camera at the second inspection point is obtained; based on the image, the position and posture of the camera corresponding to the second inspection point is adjusted to obtain the direction of the camera field of view of the second inspection point.

[0039] In the above method, on the basis of the coarse point position (that is, the second inspection point position) generated based on the anchor point, the coarse point position is fine-tuned based on the image taken by the equipment performing the inspection at the coarse point position to compensate for the error caused by the algorithm in the coordinate system transformation process based on the anchor point, so as to obtain a more accurate inspection point position, thereby ensuring that the equipment performing the inspection task can collect high-quality images during the inspection, thereby improving the accuracy of the inspection results.

[0040] Optionally, based on the image, adjusting the posture of the camera corresponding to the second inspection point to obtain the direction of the camera field of view of the second inspection point includes:

[0041] Based on the template matching algorithm, the object to be detected in the image is determined; based on the position of the center point of the object to be detected in the image, the position and posture of the camera are adjusted to control the center point of the camera's field of view to align with the center point of the object to be detected.

[0042] In a second aspect, a patrol inspection point planning device is provided, which is used to execute the patrol inspection point planning method. Specifically, the patrol inspection point planning device includes a functional module for executing the patrol inspection point planning method provided in the first aspect or any optional manner of the first aspect.

[0043] According to a third aspect, a computing device is provided, the computing device comprising a processor, wherein the processor is configured to execute program code so that the computing device executes to implement operations performed by the inspection point planning method described above.

[0044] In a fourth aspect, a computing device cluster is provided, which includes at least one computing device, each computing device including a processor and a memory, and the processor of at least one computing device is used to execute program code stored in the memory of at least one computing device, so that the computing device cluster executes to implement operations performed by the above-mentioned inspection point planning method.

[0045] In a fifth aspect, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is read by a processor of a computing device to enable the computing device to perform operations such as those performed by the above-mentioned inspection point planning method.

[0046] In a sixth aspect, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is read by a processor of a computing device in a computing device cluster to enable the computing device cluster to perform operations such as those performed by the above-mentioned inspection point planning method.

[0047] In the seventh aspect, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computing device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device executes the inspection point planning method provided in the above-mentioned one aspect or various optional implementations of the one aspect.

[0048] In an eighth aspect, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computing device in a computing device cluster reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device cluster executes the inspection point planning method provided in the above-mentioned one aspect or various optional implementations of one aspect.

[0049] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of an implementation environment of a patrol inspection point planning method provided in an embodiment of the present application;

[0051] Figure 2 This is a flow chart of a method for planning inspection points provided in an embodiment of the present application;

[0052] Figure 3 It is a flowchart of a method for planning inspection points provided in an embodiment of the present application;

[0053] Figure 4 This is a flow chart of a method for generating inspection points provided in an embodiment of the present application;

[0054] Figure 5 is a schematic diagram of a point adaptation module provided in an embodiment of the present application;

[0055] Figure 6 is a schematic diagram of a pan / tilt self-correction module provided in an embodiment of the present application;

[0056] Figure 7 It is a structural schematic diagram of a patrol point planning device provided in an embodiment of the present application;

[0057] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0058] Fig. 9 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0059] Fig.10 It is a schematic diagram of a possible implementation method of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0061] In order to facilitate understanding of the technical content of this application, several technical terms involved in this application are explained below.

[0062] Robot inspection: Inspection refers to the inspection and registration of the environment and equipment of the inspection site to promptly discover faults and hidden dangers. Inspection can realize standardized, standardized and process-based error detection, timely discover and solve problems to prevent them from happening. Robot inspection refers to the use of robots to perform inspection tasks to improve inspection efficiency, reduce inspection costs, and reduce the occurrence of casualties. Robot inspection can be applied to many fields, such as power inspection, oil pipeline inspection, highway inspection, fire scene inspection, etc.

[0063] Inspection point planning: Inspection point information includes the location of the inspection point, the direction of the inspection point, the position of the camera carried by the equipment performing the inspection, the camera magnification, and the camera focal length. During the deployment phase, it is necessary to determine the location and direction of the inspection points of the objects to be inspected in the inspection site, that is, to plan the inspection points to guide the execution of the inspection tasks.

[0064] Digital twin technology: Digital twin is an enabling technology that supports the digital transformation of the economy and society. It comprehensively uses information technologies such as perception, computing, and modeling to describe, diagnose, predict, and make decisions on physical space through software definition, thereby realizing the interactive mapping of physical space and cyberspace.

[0065] The implementation environment of the embodiments of the present application is introduced below.

[0066] Figure 1 is a schematic diagram of an implementation environment of a patrol inspection point planning method provided in an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a computing device 101 and a device for performing inspection tasks 102. The computing device 101 and the device for performing inspection tasks 102 are connected to each other in a communication manner via a wired network or a wireless network.

[0067] Among them, the computing device 101 can be at least one of a desktop computer, a laptop, a wireless terminal and a laptop portable computer, and the computing device 101 has a communication function and can access the Internet. The computing device 101 can refer to one of a plurality of computing devices. The embodiment of the present application is only illustrated by the computing device 101. Those skilled in the art can know that the number of the above computing devices can be more or less. The computing device 101 can also be an independent physical server, or a server cluster composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (content delivery networks, CDN), and big data and artificial intelligence platforms, etc., and the embodiment of the present application does not limit this. Exemplarily, the computing device 101 provides a patrol point planning function, which can determine the position and orientation of the object to be detected in the patrol site and the feasible area in the patrol site based on the 3D point cloud map of the patrol site, and then determine the position of the patrol point of the object to be detected in the feasible area. In some embodiments, the computing device 101 also provides a patrol point entry function, which can enter the position and orientation of each determined patrol point into the device that performs the patrol task, but this embodiment of the present application does not limit this.

[0068] In some embodiments, the computing device 101 is associated with a data acquisition device, and the computing device 101 can import the data of the site to be inspected collected by the data acquisition device into the computing device 101, and then perform digital twin modeling based on the imported data to generate a 3D point cloud map of the site to be inspected. Among them, the data acquisition device can be a backpack mapping device, which includes two 16-line laser radars and four fisheye cameras perpendicular to each other. The mapping device can collect 3D color point clouds of the area to be inspected, and the error does not exceed 1 centimeter (cm). The embodiment of the present application does not limit the type of data acquisition device.

[0069] Among them, the device 102 for performing the inspection task can be a robot or other device that can be used to perform the inspection task, and the device for performing the inspection task is equipped with a camera. Among them, the robot includes a robot body, a camera and a pan-tilt head. The robot body is used to control the movement and orientation of the robot; the camera is used to collect images; the pan-tilt head is used to control the orientation of the camera's field of view. Exemplarily, the device 102 for performing the inspection task can reach the position of the inspection point and adjust the orientation of the camera's field of view to the orientation of the inspection point based on the device's navigation map of the inspection site and the entered inspection points, and then collect the image corresponding to the inspection point to realize the detection of the object to be detected corresponding to the inspection point.

[0070] In some embodiments, the wireless network or wired network uses standard communication technology and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, peer-to-peer (P2P) communication is implemented between the computing device 101 and the device 102 performing the inspection task based on a remote procedure call protocol (RPC). In some embodiments, the computing device 101 and the device 102 performing the inspection task use technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. to represent data blocks exchanged through the network. In addition, conventional encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet protocol security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the above data communication technologies.

[0071] The embodiment of the present application provides a method for planning inspection points, wherein the position and orientation of the object to be inspected and the feasible area in the inspection site are determined based on the 3D point cloud map of the site to be inspected, and then the position of the inspection point of the object to be inspected is determined based on the feasible area and the position and orientation of the object to be inspected. In this method, the position of the inspection point can be automatically planned in the 3D point cloud map, without the need for manually remotely controlling the equipment that performs the inspection task to reach the vicinity of each object to be inspected and then determine the position of the inspection point. The inspection point planning takes a short time and is not affected by extreme environments, which can improve the deployment efficiency of the inspection and reduce labor costs.

[0072] The above inspection point planning method can be applied to substation inspection point planning scenarios, park inspection scenarios, and scenarios that require long-term monitoring of safety status, such as machine rooms, tunnels, and grain depots. The above application scenarios are introduced below.

[0073] (1) Planning scenario of substation inspection points: The safety of the power system plays an irreplaceable role in the development of the national economy. Any accident in any link may lead to a chain reaction, causing large-scale power outages, personal injury, damage to main equipment, and even catastrophic accidents such as the collapse of the entire network. As a key node connecting the backbone network and the distribution network, how to ensure its normal operation is directly related to the stability and safety of the entire power system. In order to ensure real-time monitoring of the operating status of major primary equipment such as main transformers, busbars, switches, etc. in the substation, it is necessary to inspect and maintain the substation. Traditional substation monitoring and inspection are mainly carried out manually, through human senses to make simple qualitative judgments on the equipment, mainly through methods such as seeing, touching, listening, and smelling. However, manual inspection has many shortcomings. The traditional manual inspection method has the shortcomings of high labor intensity, low work efficiency, scattered detection quality, and single means. The data of manual inspection cannot be accurately and timely connected to the management information system. In addition, with the promotion of unattended mode, the inspection workload is increasing, and the inspection rate and timeliness cannot be guaranteed. In addition, manual inspections still pose great safety risks in geographical conditions such as plateaus, hypoxia, cold weather, or in severe weather conditions, and there is a lack of effective inspection methods. Inspections cannot be carried out in a timely manner in severe weather conditions such as strong winds, fog, ice and snow, hail, and thunderstorms. Therefore, the traditional substation inspection method urgently needs a new "successor". The power inspection robot inspects outdoor high-voltage equipment in unmanned or undermanned substations in an autonomous or remotely controlled manner, and can promptly detect defects in power equipment, foreign objects hanging, and other abnormal phenomena, automatically alarm, or perform pre-set fault processing. It runs flexibly and freely, truly plays a role in reducing staff and increasing efficiency, and can promote the unmanned process of substations more quickly. Inspection point planning and point information recording are necessary deployment tasks for power inspection robots.

[0074] (2) Park inspection scenario: With the development of the economy, artificial intelligence has gradually come into people's sight. The construction of smart parks and intelligent communities has brought a series of conveniences to residents. Traditional patrol security work has moved towards "intelligent patrol" and "three-dimensional patrol" based on artificial intelligence technology, using patrol robots to replace manual patrol. Similarly, patrol point planning and point information recording are also necessary deployment tasks for park patrol robots.

[0075] (3) Scenarios that require long-term safety monitoring, such as computer rooms, tunnels, and grain warehouses: Inspection scenes such as computer rooms, tunnels, and grain warehouses all have objects to be inspected. Therefore, inspection point planning and point information recording are also necessary deployment tasks for inspection robots in these inspection scenes.

[0076] The following is an introduction to a method for planning inspection points provided in an embodiment of the present application. Figure 2 is a flow chart of a method for planning inspection points provided in an embodiment of the present application, such as Figure 2 As shown, taking the method executed by a computing device as an example, the method includes the following steps 201 to 205.

[0077] 201. The computing device performs digital twin modeling of the site to be inspected based on the data of the site to be inspected collected by the data collection device to obtain a 3D point cloud map of the site to be inspected.

[0078] The computing device is associated with a data acquisition device, which can import the data of the site to be inspected collected by the data acquisition device into the computing device, and then perform digital twin modeling based on the imported data to generate a 3D point cloud map of the site to be inspected. The data acquisition device can be a backpack mapping device, which includes two 16-line laser radars and four fisheye cameras perpendicular to each other. The mapping device can collect 3D color point clouds of the area to be inspected, and the error does not exceed 1 centimeter (cm). The embodiment of the present application does not limit the type of data acquisition device.

[0079] It should be noted that step 201 is an optional step. In some embodiments, step 201 may not be performed, that is, the computing device directly obtains the generated 3D point cloud map and executes subsequent steps 202 to 205 on the 3D point cloud map without the need for the computing device to perform digital twin modeling.

[0080] 202. The computing device determines, based on the 3D point cloud map of the site to be inspected, a position and orientation of a first object among a plurality of objects to be detected in the site to be inspected.

[0081] Among them, the objects to be inspected in the inspection site are also the inspection targets in the inspection site. Taking the substation inspection scenario as an example, the inspection site is the substation, and the objects to be inspected include meters, insulators, voltage transformers, current transformers, grounding switches, isolating switch pipes, bushings, lightning arrester rings, drainage wire clamps, electrical boxes, ladder locks, insulator connections, etc., which are not limited in the embodiments of the present application.

[0082] Among them, the first object is any object to be detected among multiple objects to be detected. The position of the first object is described by the coordinates of the midpoint of the first object in the 3D point cloud map. In some embodiments, the position of the first object is the coordinates of the midpoint of the key component on the first object in the 3D point cloud map. The position of the first object can be determined according to the rules corresponding to actual needs, and the embodiments of the present application do not limit this. The orientation of the first object is described by the normal direction of the plane where the front view of the first object is located.

[0083] The computing device automatically identifies the position and orientation of the object to be detected on the 3D point cloud map based on a 3D point cloud detection algorithm, such as BirdNet. In the above method, the computing device automatically identifies the position and orientation of the first object through a point cloud detection algorithm, without the need to manually remotely control the robot to go near the object to be detected and then determine the position and orientation of the object to be detected. This can improve the efficiency of inspection point planning and inspection deployment efficiency, reduce labor costs, and is not affected by extreme environments.

[0084] In some embodiments, the computing device uses a human-computer interaction method to determine the object selected in the 3D point cloud map of the site to be inspected as the object to be detected, and the position and orientation of the selected object are also the position and orientation of the object to be detected. In the above method, the position and orientation of the object to be detected are determined by using a human-computer interaction method, and what is selected is what is obtained, which can ensure the accuracy and convenience of determining the object to be detected.

[0085] In other embodiments, the computing device first automatically detects the positions and orientations of multiple objects to be detected in the 3D point cloud map, and then uses human-computer interaction to adjust the multiple objects to be detected, for example, deleting the positions and orientations of the objects to be detected that are detected incorrectly, adding the positions and orientations of the objects to be detected that are not detected, and correcting at least one of the positions or orientations of the objects to be detected that have been detected. In the above method, an algorithm is first used for automatic detection, and then the results of the automatic detection are adjusted through human-computer interaction. This can ensure the accuracy of the position and orientation of the objects to be detected while ensuring the efficiency of determining the position and orientation of the objects to be detected, thereby reducing the impact of errors caused by automatic detection.

[0086] In some embodiments, the computing device is configured with a display module. After the computing device determines the positions and orientations of multiple objects to be detected in the site to be inspected, the computing device displays the determined positions and orientations of the objects to be detected in the 3D point cloud map.

[0087] 203. The computing device determines a feasible area in the site to be inspected based on the 3D point cloud map, where the feasible area is an area that can be reached by the device performing the inspection task.

[0088] The computing device determines the convex hull range of the 3D point cloud map, and determines the feasible area in the site to be inspected by using the ground area corresponding to the convex hull range.

[0089] In some embodiments, the computing device automatically identifies feasible areas on the 3D point cloud map based on a 3D point cloud segmentation algorithm, for example, a segmentation algorithm based on point cloud convolution (PointNet). In the above method, the computing device identifies feasible areas through a point cloud segmentation algorithm, which can avoid the equipment performing the inspection task from sounding when driving in the identified feasible area, thereby ensuring the accuracy and availability of the subsequently determined inspection points.

[0090] In some embodiments, the computing device uses a human-computer interaction method to determine the area selected in the 3D point cloud map of the site to be inspected as a feasible area. In the above method, the feasible area is determined by a human-computer interaction method, and what is selected is what is obtained, which can ensure the accuracy and convenience of determining the feasible area.

[0091] In other embodiments, the computing device is configured with a display module, and the computing device first determines the convex hull range in the 3D point cloud map, displays the ground area corresponding to the determined convex hull range as a feasible area in the 3D point cloud map, and then uses human-computer interaction to adjust the displayed feasible area, for example, deleting the determined erroneous feasible area, adding the undetermined feasible area, and correcting at least one of the position or range of the determined feasible area. In the above method, the feasible area is first automatically segmented by an algorithm, and then the automatically segmented feasible area is adjusted through human-computer interaction. Under the premise of ensuring the efficiency of determining the feasible area, the accuracy of the determined feasible area can be guaranteed, and the impact of the error of the automatic segmentation algorithm can be reduced.

[0092] It should be noted that the above-mentioned execution of step 202 first and then step 203 is only exemplary. In some embodiments, step 203 may be executed first and then step 202, or step 202 and step 203 may be executed simultaneously. The embodiments of the present application do not limit this.

[0093] 204. The computing device determines the position of the inspection point of the first object based on the feasible area and the position and orientation of the first object.

[0094] The process of determining the position of the inspection point of the first object by the computing device includes: determining the intersection of the feasible area and the first plane, the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes multiple points; determining the position of the inspection point of the first object from the multiple points in the intersection. The position of the inspection point can be the position of the point farthest from the center of the first object among the multiple points, so as to expand the shooting range of the camera carried by the equipment performing the inspection task; the position of the inspection point can also be the position of the point closest to the center of the first object among the multiple points, so as to improve the clarity of the image captured by the camera on the equipment performing the inspection task. For example, the computing device is based on the midpoint x of the first object. d and orientation to determine plane A (ax+by+cz+d=0), which is also the first plane; the computing device intersects plane A with the feasible area to obtain a point equation ex+fy=0 (x0<=x<=x1), where x, y and z represent coordinate axes, a, b, c, d, e and f are constants, and x0 and x1 are the intersection points of plane A and the boundary of the feasible area; the computing device compares the midpoint x of the first object d With respect to the relationship between x0 and x1, if it is closer to x0, the position of the inspection point of the first object is x1, and if it is closer to x1, the position of the inspection point of the first object is x0.

[0095] In some embodiments, step 204 is implemented through a point planning algorithm, which is used to abstract the physical optimization conditions of point planning and optimization into constraint parameters of the algorithm to achieve optimized output of inspection points. The point planning algorithm includes the following conditions: inspection points can only be located in feasible areas; inspection points are as far away from the objects to be detected as possible; inspection points are located in the direction of the objects to be detected; if the distance between multiple inspection points is less than a threshold, the multiple inspection points are merged into a single point.

[0096] 205. The computing device determines a direction of the inspection point of the first object based on the position of the inspection point of the first object and the position of the first object.

[0097] The computing device uses the position of the inspection point of the first object and the direction of the line connecting the position of the first object as the orientation of the inspection point of the first object, and the orientation is used to indicate the orientation of the camera field of view carried by the device performing the inspection task.

[0098] The above steps 202 to 205 introduce the process of determining the inspection point of the first object among multiple objects to be detected in the inspection site, wherein the multiple objects to be detected include the first object and the second object, and if the distance between the inspection point of the first object and the inspection point of the second object is less than a threshold value, the computing device merges the inspection point of the first object and the inspection point of the second object to obtain a merged inspection point, and uses the merged inspection point as the inspection point of the first object and the second object. The process includes: the computing device determines the position of the inspection point of the second object based on the feasible area and the position and orientation of the second object; determines that the distance between the inspection point of the first object and the inspection point of the second object is less than a threshold value; determines the position of the merged inspection point based on the inspection point of the first object and the inspection point of the second object; uses the position of the merged inspection point as the position of the inspection point of the first object, and uses the position of the merged inspection point as the position of the inspection point of the second object. Wherein, determining the position of the merged inspection point based on the inspection point of the first object and the inspection point of the second object includes: determining the midpoint of the first inspection point and the second inspection point, and the midpoint is the position of the merged inspection point. In some embodiments, the computing device uses the direction of the line connecting the midpoint of the first object and the second object and the position of the merged inspection point as the orientation of the merged inspection point. It should be noted that if the distance between the merged inspection point and the inspection points of other objects in the object to be detected is less than a threshold value, the computing device merges the merged inspection point with the inspection points of other objects in the same manner as the above process until the merger cannot be performed. In the above method, multiple inspection points that can be merged are merged into one inspection point, so that the equipment performing the inspection task does not need to shoot different objects to be detected at multiple points that are close to each other, but only needs to shoot images of multiple objects to be detected at one point, which can improve the execution efficiency of the inspection task.

[0099] Below Figure 3 The process shown in the above steps 201 to 205 is described by taking the example of FIG. Figure 3 is a flow chart of a method for planning inspection points provided in an embodiment of the present application, such as Figure 3 As shown, a digital twin model is performed on the inspection site to obtain a 3D color laser point cloud map; the 3D color laser point cloud map is used to detect objects to be detected through a 3D point cloud detection algorithm, and the position and orientation of a first object among multiple objects to be detected in the inspection site are obtained; the 3D color laser point cloud map is segmented into feasible areas through a 3D point cloud segmentation algorithm, and the coordinate information of the feasible areas is obtained; based on the feasible area and the position and orientation of the first object, the position of the inspection point of the first object is determined.

[0100] In the above method, the positions of inspection points can be automatically planned in the 3D point cloud map of the site to be inspected, without the need for manual remote control of the equipment performing the inspection task to reach the vicinity of each object to be inspected and then determine the position of the inspection point. The inspection point planning takes a short time and is not affected by extreme environments, which can improve the deployment efficiency of robot inspections and reduce labor costs; further, multiple inspection points that can be merged are merged into one inspection point, so that the equipment performing the inspection task does not need to shoot different objects to be detected at multiple points at close distances, but only needs to shoot images of multiple objects to be detected at one point, which can improve the execution efficiency of the inspection task.

[0101] Above Figure 2 The illustrated embodiment introduces a process of planning inspection points in a site to be inspected based on a 3D point cloud map of the site to be inspected. Since there is an error between the navigation map used by the equipment (e.g., a robot) performing the inspection task when inspecting the site to be inspected and the 3D point cloud map, it is necessary to adapt the planned inspection points to the navigation map. The process is described below. Figure 4 This is a flowchart of a method for generating inspection points provided in an embodiment of the present application. Taking the process executed by a computing device as an example, the process includes the following steps 401 to 405.

[0102] 401. The computing device determines a plurality of first anchor points in a first map, and determines a plurality of second anchor points in a second map, wherein the first map is a grid map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected by a device performing an inspection task, and the plurality of first anchor points and the plurality of second anchor points are in one-to-one correspondence.

[0103] The computing device converts the 3D point cloud map of the site to be inspected into a 2D grid map (GM) by rasterization to obtain the first map. The computing device converts the 3D point cloud map into a 2D grid map using a 3D point cloud projection method, which can retain sufficient 3D information as much as possible on the basis of the original resolution, thereby facilitating the improvement of the accuracy of the 2D grid map. The navigation map of the site to be inspected by the device performing the inspection task is a 2D grid map.

[0104] The process of determining a plurality of first anchor points in the first map by the computing device includes any of the following methods.

[0105] The first method: based on a key point detection algorithm, multiple anchor points are determined in the first map, and the multiple anchor points are determined as multiple first anchor points.

[0106] In some embodiments, the computing device uses the Harris corner detection algorithm and uses gradient operators such as Sobel to calculate the gradient of each point in the first map in the x and y directions. and For each point in the first map, construct Find the corner point response function det(M)-α·trace(M) 2 , multiple points whose values ​​(response values) of the corner point response function are greater than a preset threshold are determined as multiple anchor points, and the multiple anchor points are determined as multiple first anchor points. Wherein, w is a weighting function, which can be a constant or a Gaussian weighting function; α is an empirical constant.

[0107] It should be noted that the above step 401 is described by taking the Harris corner detection algorithm as an example to determine the anchor point. In some embodiments, the anchor point is determined based on other methods, which is not limited in the embodiments of the present application.

[0108] The second method: using a human-computer interaction method, multiple anchor points are determined in the first map, and the multiple anchor points are determined as multiple first anchor points.

[0109] Among them, determining multiple anchor points in the first map by adopting a human-computer interaction method includes: receiving an anchor point configuration instruction from a user, the anchor point configuration instruction is used to determine multiple first anchor points in the first map; and determining multiple anchor points in the first map based on the anchor point configuration instruction.

[0110] In some embodiments, for a computing device configured with a display module, the computing device is capable of displaying a first map and displaying multiple anchor points in the first map. Therefore, a human-computer interaction method can be used to select multiple points in the first map displayed by the computing device. The selected multiple points are anchor points, and the computing device determines the multiple anchor points as multiple first anchor points.

[0111] The third method: based on the key point detection algorithm, multiple anchor points are determined in the first map, and at least one of the multiple anchor points is adjusted in a human-computer interaction manner to obtain multiple first anchor points.

[0112] Among them, the process of the computing device determining multiple anchor points in the first map based on the key point detection algorithm is the same as the first method mentioned above, and will not be repeated here. Among them, adjusting at least one of the multiple anchor points to obtain multiple first anchor points includes at least one of the following: adjusting the position of at least one of the multiple anchor points; deleting at least one of the multiple anchor points; adding at least one anchor point to the first map. In some embodiments, the computing device receives an anchor configuration instruction from the user, and the anchor configuration instruction is used to add at least one first anchor point to the first map, delete at least one first anchor point from multiple first anchor points, or adjust the position of at least one first anchor point from multiple first anchor points; the computing device adjusts the multiple first anchor points based on the anchor configuration instruction.

[0113] It should be noted that the above three methods are all described using the first map as an example. The process of the computing device determining multiple second anchor points in the second map is the same as the process of the computing device determining multiple second anchor points in the first map, and will not be repeated here.

[0114] 402. The computing device determines a coordinate transformation matrix based on the plurality of first anchor points and the plurality of second anchor points, where the coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map.

[0115] Among them, the coordinate transformation relationship between the coordinate system of the first map and the coordinate system of the second map can be described by the coordinate transformation relationship between the anchor points in the first map and the second map. The computing device matches multiple first anchor points and multiple second anchor points one by one to obtain multiple groups of anchor points, and determines the coordinate transformation relationship between the first anchor point and the second anchor point in each group of anchor points, that is, the coordinate transformation matrix. Among them, the computing device matches multiple first anchor points and multiple second anchor points one by one based on the feature matching algorithm. In some embodiments, the computing device also performs anchor point matching based on other methods, for example, dividing multiple first anchor points and multiple second anchor points into different groups, and determining the first anchor point and the second anchor point with the same group number and sequence number in the group as a group of matching anchor points. In other embodiments, multiple first anchor points and multiple second anchor points are matched one by one based on human-computer interaction, for example, the matching anchor points are marked as a group, and the embodiments of the present application are not limited to this.

[0116] The coordinate transformation matrix may be a homography matrix or an affine matrix, etc., which is not limited in the present application. Taking the coordinate transformation matrix as a homography matrix as an example, in any set of anchor points, the coordinate transformation relationship from the first anchor point (x1, y1) to the second anchor point (x2, y2) can be expressed as the following formula (1):

[0117]

[0118] In the above formula (1), H is the coordinate transformation matrix, and its value is

[0119] By expanding and arranging the above formula (1) in the form of a polynomial, the following coordinate system transformation formulas (2) and (3) are obtained.

[0120]

[0121]

[0122] For each set of anchor points determined, the coordinate transformation equations corresponding to the above formulas (2) and (3) can be listed. Based on multiple equations, the h in the coordinate transformation matrix can be solved. 00 ~h 22 , and then obtain the coordinate transformation matrix H.

[0123] It should be noted that the above description is made by taking the coordinate transformation matrix as a homography matrix as an example. The same is true when the coordinate transformation matrix is ​​an affine matrix or other matrices, and will not be repeated here.

[0124] 403. The computing device transforms at least one first inspection point of the equipment performing the inspection task into at least one corresponding second inspection point through a coordinate transformation matrix, wherein the at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

[0125] The computing device multiplies the coordinates of the first inspection point by the coordinate transformation matrix to obtain the corresponding second inspection point.

[0126] In the above steps 401 to 403, the planned first inspection point is automatically generated as the second inspection point used for actual inspection based on the anchor point, which can reduce the impact of errors between different maps on actual inspections and realize the reuse of planned inspection points on different devices performing inspection tasks, thereby eliminating the need to plan inspection points multiple times and saving labor costs.

[0127] In some embodiments, the process of generating the second inspection point based on the first inspection point includes two stages: generating a rough point and fine-tuning the rough point. The process of generating the rough point is used to determine the corresponding position of the first inspection point planned on the first map on the second map (that is, the second inspection point). The process is the same as the above steps 401 to 403 and will not be repeated. The process of fine-tuning the rough point is used to fine-tune the direction of the camera field of view of the second inspection point to compensate for the error in the process of generating the rough point and obtain an accurate inspection point. Taking the fine-tuning of any second inspection point as an example, the process includes the following steps 404 and 405.

[0128] 404. The computing device obtains an image captured at the second inspection point by a camera carried by the device performing the inspection task.

[0129] The position of the camera is adjustable. The second inspection point is the coarse point obtained in the coarse point generation stage. The device performing the inspection task reaches the position indicated by the second inspection point based on the second map, captures an image at the second inspection point, and sends the captured image to the computing device.

[0130] 405. The computing device adjusts the posture of the camera corresponding to the second inspection point based on the image to obtain the direction of the camera field of view of the second inspection point.

[0131] Among them, the center point of the object to be detected in the image taken by the camera carried by the device performing the inspection task in the adjusted posture is located at the center point of the image. In some embodiments, the device performing the inspection task carries the camera through a gimbal, the posture of the gimbal is adjustable, and the computing device adjusts the posture of the camera by adjusting the posture of the gimbal.

[0132] The process in which the computing device adjusts the posture of the camera corresponding to the second inspection point based on the image to obtain the direction of the camera field of view of the second inspection point includes the following steps 405A to 405B.

[0133] 405A. Determine the object to be detected in the image based on the template matching algorithm.

[0134] The template refers to an image of an object to be detected that is preset manually or at a site to be inspected. The computing device determines multiple object feature points in the image based on a key point detection algorithm, and matches the multiple object feature points with the template based on a template matching algorithm to determine the object to be detected in the image.

[0135] 405B. Based on the position of the center point of the object to be detected in the image, adjust the position and posture of the camera to control the center point of the camera's field of view to align with the center point of the object to be detected.

[0136] Among them, the center point of the camera's field of view is also the center point of the image taken by the camera. The computing device determines the relative position relationship between the center point of the object to be detected and the center point of the camera's field of view based on the position of the center point of the object to be detected in the image, and then adjusts the camera's posture based on the relative position relationship to control the center point of the camera's field of view to align with the center point of the object to be detected. In some embodiments, the computing device adjusts the camera's posture at least once, and after each adjustment, the camera retakes the image and sends it to the computing device. The computing device performs the next adjustment based on the retaken image until the center point of the camera's field of view is aligned with the center point of the object to be detected.

[0137] Among them, if the center point of the object to be detected is exactly in the center of the image, there is no need to adjust the camera's posture.

[0138] In some embodiments, based on the position of the center point of the object to be detected in the image, the position and posture of the camera is adjusted to control the field of view of the camera to include the object to be detected to the greatest extent.

[0139] It should be noted that the above steps 404 and 405 are described by taking one second inspection point as an example, and the other second inspection points are similar and will not be described in detail.

[0140] It should be noted that the process of fine-tuning the second inspection point shown in the above steps 404 and 405 can occur during the formal inspection process, that is, the device for performing the inspection task arrives at each second inspection point in turn based on the second map, and each time the device performing the inspection task arrives at a second inspection point, the computing device fine-tunes the second inspection point to obtain the fine-tuned second inspection point, and the device performing the inspection task completes the current inspection task based on the fine-tuned second inspection point, and then arrives at the next second inspection point. The fine-tuned second inspection point recorded in this process can be directly applied to the next inspection. The process shown in the above steps 404 and 405 can also occur in the pre-inspection process before the formal inspection, in which the computing device obtains the fine-tuned second inspection point and records the fine-tuned second inspection point, but the device performing the inspection task does not perform the inspection task at this point at this time, but arrives at the next second inspection point and records the next fine-tuned second inspection point until all the fine-tuned second inspection points are recorded. During the formal inspection, the pre-inspected equipment can perform inspections directly based on the recorded fine-tuned second inspection points, and the computing device can also enter the fine-tuned second inspection points into the device corresponding to the same second map as the pre-inspected equipment, so that the device can perform inspections directly based on the fine-tuned second inspection points without the need for temporary point fine-tuning.

[0141] Below Figure 5 and Figure 6 Taking the example of FIG. 4 as an example, the process shown in the above steps 401 to 405 is illustrated. Figure 5 is a schematic diagram of a point adaptation module provided in an embodiment of the present application. The point adaptation module can be deployed on a computing device or on a device such as a robot that performs inspection tasks. Figure 5 Corresponding to the above-mentioned stage of generating the coarse point position (that is, the above-mentioned steps 401 to 403), as Figure 5 shown, ground Figure 1 The first map may be a 2D grid map or a 3D point cloud map. Figure 2The second map may be a 2D grid map or a 3D point cloud map. Figure 2 A 3D point cloud map is formed by a map matching unit, an anchor point generation unit, a map matching unit and a point position adaptation unit. Figure 2 The 3D unit rasterizes the 3D point cloud map. If both the first map and the second map are 2D raster maps, no mode conversion is required. The anchor point generation unit determines multiple first anchor points in the first map and multiple second anchor points in the second map. The map matching unit matches the multiple first anchor points in the first map and the multiple second anchor points in the second map one by one to obtain multiple groups of anchor points. Based on the multiple groups of anchor points, the coordinate transformation matrix between the first map and the second map is determined. Based on the coordinate transformation matrix, the point adaptation unit transforms the first inspection point based on the first map into the second inspection point in the second map. The second inspection point is also the generated coarse point.

[0142] Figure 6 is a schematic diagram of a pan-tilt self-correction module provided in an embodiment of the present application. The pan-tilt self-correction module can be deployed on a computing device, or on a device such as a robot that performs inspection tasks. Figure 6 Corresponding to the stage of fine-tuning the coarse point position (that is, the above steps 404 to 405), as Figure 6 As shown, the pan-tilt self-correction module includes a template matching unit based on image features and a pan-tilt control unit. The template matching module based on image features processes the image captured by the device performing the inspection task at the coarse point position to determine the coordinates and confidence of the center point of the object to be detected in the image; the pan-tilt control unit controls the pan-tilt posture based on the relative position between the center point of the object to be detected and the center point of the image until the center of the object to be detected is moved to the center of the camera's field of view, and the obtained point position is recorded as the precise point position (that is, the second inspection point position after fine-tuning).

[0143] It should be noted that the above steps 404 and 405 are optional steps. In some embodiments, steps 404 and 405 may not be performed, and the embodiments of the present application do not limit this.

[0144] It should be noted that the above steps 401 to 405 are described by taking the computing device as the execution body as an example. In some embodiments, the above steps 401 to 405 are executed by a computing device cluster. In other embodiments, the above steps 401 to 405 are executed by robots and other devices that perform inspection tasks, that is, the inspection point generation function is built into the device that actually performs the inspection task. In some other embodiments, the above steps 401 to 403 are executed by a computing device, and steps 404 and 405 are executed by the device that performs the inspection task, that is, the computing device completes the generation of the rough point, and the device that performs the inspection task completes the fine adjustment of the rough point. The embodiments of the present application do not limit this.

[0145] In the above method, the computing device automatically adapts the planned inspection points and the inspection points used for actual inspection based on the anchor points, which can reduce the impact of errors between different maps on actual inspections without manually re-entering the inspection points, which can save labor costs. In addition, since the inspection points between different maps can be adapted, the inspection points obtained based on a single inspection point planning can be adapted to the navigation maps of multiple devices that perform inspection tasks, so that the inspection points can be reused on different devices without planning inspection points for each device, which can improve the deployment efficiency of the inspection and further save labor costs. Furthermore, on the basis of the coarse points generated based on the anchor points, the coarse points are fine-tuned based on the images taken by the device performing the inspection at the coarse points to compensate for the errors caused by the algorithm in the coordinate system transformation process based on the anchor points, so as to obtain more accurate inspection points, thereby ensuring that the equipment performing the inspection tasks collects high-quality images during the inspection, thereby improving the accuracy of the inspection results.

[0146] Figure 7 is a schematic diagram of the structure of a patrol point planning device provided in an embodiment of the present application, such as Figure 7 As shown, the device includes a first determination module 701, a second determination module 702 and a third determination module 703.

[0147] The first determination module 701 is used to determine the position and orientation of a first object among multiple objects to be detected in the site to be inspected based on the 3D point cloud map of the site to be inspected;

[0148] The second determination module is used to determine a feasible area in the site to be inspected based on the 3D point cloud map, where the feasible area is an area that can be reached by the equipment performing the inspection task;

[0149] The third determination module is used to determine the position of the inspection point of the first object based on the feasible area and the position and orientation of the first object.

[0150] Optionally, the device further comprises:

[0151] The fourth determination module is used to use the direction of the line connecting the position of the inspection point of the first object and the position of the first object as the orientation of the inspection point of the first object, which is used to indicate the orientation of the camera field of view carried by the equipment performing the inspection task.

[0152] Optionally, the third determining module 703 is used to:

[0153] Determine an intersection of the feasible area and a first plane, where the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes a plurality of points;

[0154] The position of the inspection point of the first object is determined from the multiple points in the intersection.

[0155] Optionally, the multiple objects to be detected include a first object and a second object, and the first determination module is further used to:

[0156] Determine the position of the inspection point of the second object based on the feasible area and the position and orientation of the second object;

[0157] The device also includes:

[0158] A distance determination module, used to determine that the distance between the inspection point of the first object and the inspection point of the second object is less than a threshold value;

[0159] A position determination module, used to determine the position of the combined inspection point based on the inspection point of the first object and the inspection point of the second object;

[0160] The third determining module 703 is further configured to: use the position of the combined inspection point as the position of the inspection point of the first object, and use the position of the combined inspection point as the position of the inspection point of the second object.

[0161] Optionally, the location determination module is used to:

[0162] Determine the midpoint between the first inspection point and the second inspection point, the midpoint being the position of the merged inspection point.

[0163] Optionally, the device further comprises:

[0164] The direction determination module is used to use the direction of the line connecting the midpoint of the first object and the second object and the position of the combined inspection point as the direction of the combined inspection point.

[0165] Optionally, the device further comprises:

[0166] The display module is used to display the locations of inspection points in the 3D point cloud map.

[0167] Optionally, the device further comprises:

[0168] The modeling module is used to perform digital twin modeling of the inspection site and obtain a 3D point cloud map of the inspection site.

[0169] Optionally, the device further comprises:

[0170] An anchor point determination module, used to determine multiple first anchor points in a first map, and multiple second anchor points in a second map, the first map is a grid map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected by the equipment performing the inspection task, and the multiple first anchor points and the multiple second anchor points are in one-to-one correspondence;

[0171] A matrix determination module, used to determine a coordinate transformation matrix based on the plurality of first anchor points and the plurality of second anchor points, wherein the coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map;

[0172] A coordinate transformation module is used to transform the coordinates of at least one first inspection point of a device performing an inspection task into the coordinates of at least one corresponding second inspection point through a coordinate transformation matrix, wherein at least one first inspection point is a point in the coordinate system of a first map, and at least one second inspection point is a point in the coordinate system of a second map.

[0173] Optionally, the anchor point determination module is used to:

[0174] Based on a key point detection algorithm, a plurality of first anchor points are determined in the first map.

[0175] Optionally, the anchor point determination module is used to:

[0176] receiving an anchor point configuration instruction from a user, where the anchor point configuration instruction is used to add at least one first anchor point in the first map, delete at least one first anchor point from a plurality of first anchor points, or adjust a position of at least one first anchor point among the plurality of first anchor points;

[0177] Based on the anchor point configuration instruction, multiple first anchor points are adjusted.

[0178] Optionally, the device for performing the inspection task is equipped with a camera, the position and posture of the camera is adjustable, and the device further includes:

[0179] An acquisition module is used to acquire an image taken by the camera at the second inspection point;

[0180] The adjustment module is used to adjust the posture of the camera corresponding to the second inspection point based on the image to obtain the direction of the camera field of view of the second inspection point.

[0181] Optionally, the adjustment module is used to:

[0182] Based on the template matching algorithm, determine the object to be detected in the image;

[0183] Based on the position of the center point of the object to be detected in the image, the position and posture of the camera are adjusted to control the center point of the camera's field of view to align with the center point of the object to be detected.

[0184] Among them, the first determination module 701, the second determination module 702 and the third determination module 703 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the first determination module 701 is introduced below by taking the first determination module 701 as an example. Similarly, the implementation of the second determination module 702 and the third determination module 703 can refer to the implementation of the first determination module 701.

[0185] As an example of a software functional unit, the first determination module 701 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first determination module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0186] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0187] As an example of a hardware functional unit, the first determination module 701 may include at least one computing device, such as a server, etc. Alternatively, the first determination module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0188] The multiple computing devices included in the first determination module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the first determination module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first determination module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0189] It should be noted that, in other embodiments, the steps that the above modules are responsible for implementing can be specified as needed, and the above modules can respectively implement different steps in the above inspection point planning method to realize all the functions of the above device. That is, the inspection point planning device provided in the above embodiment only uses the division of the above functional modules as an example when implementing the inspection point planning method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment belongs to the same concept as the corresponding method embodiment. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0190] The present application also provides a computing device 800 . Figure 8 is a schematic diagram of a computing device provided in an embodiment of the present application, such as Figure 8As shown, the computing device 800 includes: a bus 801, a processor 802, a memory 803 and a communication interface 804. The processor 802, the memory 803 and the communication interface 804 communicate with each other through the bus 801. The computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0191] The bus 801 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 801 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 801 may include a path for transmitting information between various components of the computing device 800 (for example, the memory 803, the processor 802, and the communication interface 804).

[0192] The processor 802 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0193] The memory 803 may include a volatile memory, such as a random access memory (RAM). The memory 803 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0194] The memory 803 stores executable program codes, and the processor 802 executes the executable program codes to respectively implement the functions of the first determination module 701, the second determination module 702, and the third determination module 703, thereby implementing the inspection point planning method. That is, the memory 803 stores program codes for executing the inspection point planning method.

[0195] The communication interface 804 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0196] The embodiment of the present application also provides a computing device cluster. Fig. 9 is a schematic diagram of a computing device cluster provided in an embodiment of the present application, such as Fig. 9 As shown, the computing device cluster includes at least one computing device 800. The memory 803 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the inspection point planning method.

[0197] In some possible implementations, the memory 803 of one or more computing devices 800 in the computing device cluster may also store part of the program code for executing the inspection point planning method. In other words, the combination of one or more computing devices 800 can jointly execute the program code for executing the inspection point planning method.

[0198] It should be noted that the memory 803 in different computing devices 800 in the computing device cluster can store different program codes, which are respectively used to execute part of the functions of the inspection point planning device. That is, the program codes stored in the memory 803 in different computing devices 800 can implement the functions of one or more modules in the first determination module 701, the second determination module 702 and the third determination module 703.

[0199] It should be understood that Fig. 9 The functionality of the computing device 800 shown in FIG. 8 may also be performed by multiple computing devices 800 .

[0200] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network, etc. Fig.10 A possible implementation is shown. Fig. 9 is a schematic diagram of a possible implementation of a computing device cluster provided in an embodiment of the present application, such as Fig.10 As shown, two computing devices 800 are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device.

[0201] The present application implements a computer program product including instructions, which may be software or a program product including program code and capable of running on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device implements the inspection point planning method provided in the above embodiment.

[0202] The embodiment of the present application provides a computer-readable storage medium, which can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes a program code, and when the program code is executed by a computing device, the computing device executes the inspection point planning method provided in the above embodiment; when the program code is executed by a computing device cluster, the computing device cluster executes the inspection point planning method provided in the above embodiment.

[0203] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the 3D point cloud maps involved in this application are all obtained with full authorization.

[0204] Those of ordinary skill in the art will appreciate that the various method steps and units described in the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0205] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0206] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0207] The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0208] In addition, each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or software unit.

[0209] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computing device (which can be a personal computer, a server, or a computing device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0210] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various examples, the first inspection point may be referred to as the second inspection point, and similarly, the second inspection point may be referred to as the first inspection point. Both the first inspection point and the second inspection point may be inspection points, and in some cases, may be separate and different inspection points.

[0211] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple inspection points means two or more inspection points. In this document, the terms "system" and "network" are often used interchangeably.

[0212] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0213] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0214] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the process or function in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0215] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state drive)), etc.

[0216] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0217] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for planning inspection points, characterized in that: For inspecting a plurality of objects to be inspected in an inspection site, the method comprises: Determine the position and orientation of a first object among the plurality of objects to be detected in the site to be inspected based on the 3D point cloud map of the site to be inspected; Based on the 3D point cloud map, determine a feasible area in the site to be inspected, where the feasible area is an area that can be reached by equipment performing the inspection task; Based on the feasible area and the position and orientation of the first object, the position of the inspection point of the first object is determined.

2. The method according to claim 1, characterized in that The method further comprises: The direction of the line connecting the position of the inspection point of the first object and the position of the first object is used as the orientation of the inspection point of the first object, and the orientation is used to indicate the orientation of the camera field of view carried by the equipment performing the inspection task.

3. The method according to claim 1 or 2, characterized in that: The determining the position of the inspection point of the first object includes: Determine an intersection of the feasible area and a first plane, where the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes a plurality of points; The position of the inspection point of the first object is determined from the multiple points in the intersection.

4. The method according to any one of claims 1 to 3, characterized in that: The plurality of objects to be detected include the first object and the second object, and the method further includes: Determine the position of the inspection point of the second object based on the feasible area and the position and orientation of the second object; Determining that a distance between an inspection point of the first object and an inspection point of the second object is less than a threshold; Determining a position of a merged inspection point based on the inspection point of the first object and the inspection point of the second object; The position of the combined inspection point is used as the position of the inspection point of the first object, and the position of the combined inspection point is used as the position of the inspection point of the second object.

5. The method according to claim 4, characterized in that The determining the position of the merged inspection point based on the inspection point of the first object and the inspection point of the second object includes: Determine a midpoint between the first inspection point and the second inspection point, the midpoint being the position of the combined inspection point.

6. The method according to claim 4, characterized in that The method further comprises: The direction of the line connecting the midpoint of the first object and the second object and the position of the combined inspection point is used as the orientation of the combined inspection point.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The locations of inspection points are displayed in the 3D point cloud map.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Digital twin modeling is performed on the site to be inspected to obtain a 3D point cloud map of the site to be inspected.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Determine a plurality of first anchor points in a first map, and determine a plurality of second anchor points in a second map, wherein the first map is a grid map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected by a device performing an inspection task, and the plurality of first anchor points and the plurality of second anchor points are in one-to-one correspondence; Based on the plurality of first anchor points and the plurality of second anchor points, determining a coordinate transformation matrix, wherein the coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map; Through the coordinate transformation matrix, the coordinates of at least one first inspection point of the equipment performing the inspection task are transformed into the coordinates of at least one corresponding second inspection point, the at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

10. The method according to claim 9, characterized in that The determining of a plurality of first anchor points in the first map comprises: Based on a key point detection algorithm, the plurality of first anchor points are determined in the first map.

11. The method according to claim 9 or 10, characterized in that: The determining of a plurality of first anchor points in the first map comprises: receiving an anchor point configuration instruction from a user, wherein the anchor point configuration instruction is used to add at least one first anchor point in the first map, delete at least one first anchor point from the plurality of first anchor points, or adjust the position of at least one first anchor point in the plurality of first anchor points; Based on the anchor point configuration instruction, the plurality of first anchor points are adjusted.

12. The method according to claim 9, characterized in that The device for performing the inspection task is equipped with a camera, and the position and posture of the camera is adjustable. The method further includes: Acquire an image captured by the camera at the second inspection point; Based on the image, the posture of the camera corresponding to the second inspection point is adjusted to obtain the direction of the camera field of view of the second inspection point.

13. The method according to claim 12, characterized in that The adjusting, based on the image, the posture of the camera corresponding to the second inspection point to obtain the direction of the camera field of view of the second inspection point includes: Determine the object to be detected in the image based on a template matching algorithm; Based on the position of the center point of the object to be detected in the image, the position and posture of the camera are adjusted to control the center point of the field of view of the camera to align with the center point of the object to be detected.

14. A patrol point planning device, characterized in that: Used for inspecting multiple objects to be inspected in an inspection site, the device comprises: A first determination module, configured to determine the position and orientation of a first object among the plurality of objects to be detected in the site to be inspected based on a 3D point cloud map of the site to be inspected; A second determination module is used to determine a feasible area in the site to be inspected based on the 3D point cloud map, where the feasible area is an area that can be reached by equipment performing the inspection task; The third determination module is used to determine the position of the inspection point of the first object based on the feasible area and the position and orientation of the first object.

15. The device according to claim 14, characterized in that The device also includes: The fourth determination module is used to use the direction of the line connecting the position of the inspection point of the first object and the position of the first object as the orientation of the inspection point of the first object, and the orientation is used to indicate the orientation of the camera field of view carried by the equipment performing the inspection task.

16. The device according to claim 14 or 15, characterized in that The third determining module is used to: Determine an intersection of the feasible area and a first plane, where the first plane is parallel to the orientation of the first object and passes through the center of the first object, and the intersection includes a plurality of points; The position of the inspection point of the first object is determined from the multiple points in the intersection.

17. The device according to any one of claims 14 to 16, characterized in that The device also includes: The display module is used to display the locations of inspection points in the 3D point cloud map.

18. The device according to claim 14, characterized in that The device also includes: An anchor point determination module, used to determine a plurality of first anchor points in a first map, and determine a plurality of second anchor points in a second map, wherein the first map is a grid map corresponding to a 3D point cloud map of the site to be inspected, and the second map is a navigation map of the site to be inspected by a device performing an inspection task, and the plurality of first anchor points and the plurality of second anchor points are in one-to-one correspondence; a matrix determination module, configured to determine a coordinate transformation matrix based on the plurality of first anchor points and the plurality of second anchor points, wherein the coordinate transformation matrix is ​​used to represent a coordinate transformation relationship between a coordinate system of the first map and a coordinate system of the second map; A coordinate transformation module is used to transform the coordinates of at least one first inspection point of the equipment performing the inspection task into the coordinates of at least one corresponding second inspection point through the coordinate transformation matrix, wherein the at least one first inspection point is a point in the coordinate system of the first map, and the at least one second inspection point is a point in the coordinate system of the second map.

19. The device according to claim 18, characterized in that The anchor point determination module is used to: Based on a key point detection algorithm, the plurality of first anchor points are determined in the first map.

20. The device according to claim 18 or 19, characterized in that The anchor point determination module is used to: receiving an anchor point configuration instruction from a user, wherein the anchor point configuration instruction is used to add at least one first anchor point in the first map, delete at least one first anchor point from the plurality of first anchor points, or adjust the position of at least one first anchor point in the plurality of first anchor points; Based on the anchor point configuration instruction, the plurality of first anchor points are adjusted.

21. A computing device, characterized in that The computing device comprises a processor, and the processor is used to execute program code, so that the computing device executes the inspection point planning method according to any one of claims 1 to 13.

22. A computing device cluster, characterized in that: comprising at least one computing device, each of the computing devices comprising a processor and a memory; The processor of the at least one computing device is used to execute the program code stored in the memory of the at least one computing device, so that the computing device cluster executes the inspection point planning method according to any one of claims 1 to 13.

23. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is read by a processor of a computing device to enable the computing device to execute the inspection point planning method according to any one of claims 1 to 13.

24. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is read by a processor of a computing device in a computing device cluster so that the computing device cluster executes the inspection point planning method according to any one of claims 1 to 13.

Images