Image processing method and image processing apparatus
Through cloud management platform and image processing technology, complete and accurate images taken by the robot in an outdoor environment are extracted, which solves the problem of incomplete images caused by low positioning accuracy of mobile robots and improves the effectiveness and accuracy of patrols.
Patent Information
- Application Number
- PCT/CN2023/140436
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-18
- Filing Date
- 2023-12-20
- Publication Date
- 2024-11-28
AI Technical Summary
When existing mobile robots inspect outdoor environments, the images captured are incomplete and inaccurate due to low positioning accuracy and environmental interference, which affects the maintenance effect of the inspection objects.
Using the image processing method, the cloud management platform refers to the position error between the desired checkpoint and the actual checkpoint, and extracts the target area image in the panoramic image captured by the robot to ensure that the image of the inspection object is complete and accurate. The method includes configuring the desired checkpoint location, acquiring the expected area, acquiring the robot patrol data, extracting the target area image in the second panoramic image based on the expected area and the first information, and improving positioning accuracy using the global navigation satellite system and dynamic post-processing technology. Spend.
Even if the robot itself is inaccurate in positioning, it can still capture complete and accurate images from actual checkpoints, improve the level of inspection and solve the problem of incomplete image information caused by positioning errors in the existing technology.
Smart Images

Figure CN2023140436_28112024_PF_FP_ABST
Abstract
Description
Image processing method and image processing device Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to an image processing method and an image processing device. Background Art
[0002] In order to maintain equipment in outdoor environments, it is necessary to conduct inspections on the equipment, such as inspections and maintenance of power plant equipment, power grids, and electric meters in power environments.
[0003] At present, inspection objects in outdoor environments are mainly inspected by mobile robots. However, due to the interference of factors such as the positioning accuracy of the mobile robots and the outdoor environment, the images taken by the current mobile robots of the inspection objects are not complete and accurate, and there is a problem of inadequate inspection of the inspection objects, which is not conducive to the maintenance of the inspection objects.
[0004] Summary of the Invention
[0005] The present application provides an image processing method and an image processing device, which can refer to the expected area of the inspection object related to the expected inspection point and the position of the actual inspection point to which the robot moves according to the expected inspection point, to extract the image of the area of interest from the second panoramic image related to the actual inspection point taken by the robot. Even if the robot's own positioning is inaccurate, it can refer to the accurate expected area to extract a complete and accurate image of the inspection object from the second panoramic image taken at the inaccurate positioning position, so as to conduct a more thorough inspection of the inspection object.
[0006] In one possible implementation, an embodiment of the present application provides an image processing method. The method includes: configuring the positions of multiple expected inspection points for the robot, the positions of the expected inspection points being expected positions for recording images of inspection objects when the robot moves along an inspection route, the robot being equipped with a camera for capturing panoramic images; acquiring an expected area associated with the expected inspection points, the expected area being an area of interest regarding the inspection object in a first panoramic image captured by the robot at the expected inspection points; acquiring inspection data of the robot, the inspection data including first information for determining the position of an actual inspection point and a second panoramic image associated with the actual inspection point captured by the robot, wherein the robot moves to the actual inspection point based on the position of the expected inspection point at a first time; and acquiring an image of a target area of the inspection object in the second panoramic image based on the expected area and the first information, the target area being an area of interest regarding the inspection object in the second panoramic image.
[0007] This method can be implemented by a cloud-based management platform (e.g., a cloud management platform), optionally in conjunction with a terminal device. For example, the cloud management platform can be deployed on a cloud server (e.g., a computing node). Alternatively, the method can be implemented by a cloud-based management platform (e.g., a server), optionally in conjunction with a terminal device. The following embodiments will be described using the cloud management platform as the execution entity.
[0008] Among them, the position of the expected checkpoint is the position to which the robot is expected to move, but because the robot's own positioning is not accurate enough, when the robot moves according to the position of the expected checkpoint, it moves to the position of the actual checkpoint that has a certain position deviation from the position of the expected checkpoint.
[0009] In addition, the robot takes a picture of the corresponding inspection object at the position of the desired inspection point, and the obtained image is the most complete and accurate. Therefore, it is expected that the robot can move to the desired position to take a picture of the inspection object.
[0010] The operation of configuring the desired checkpoint location for the robot can be performed by the cloud management platform or by the terminal device, and there is no restriction here.
[0011] The robot can be pre-positioned at the desired inspection point to capture a panoramic image and obtain a first panoramic image. In addition, the present application can pre-store the pixel position of the inspection object in the first panoramic image (also expressed as the above-mentioned expected area) in the cloud management platform; or pre-store the expected area on the side of a terminal device such as a robot, and the cloud management platform can obtain the expected area related to the desired inspection point from the side of a terminal device such as a robot.
[0012] When the robot inspects the inspection object near each expected inspection point according to the inspection route, the time it takes for the robot to move from the position of the expected inspection point to the actual inspection point is represented as the first time.
[0013] The second panoramic image captured by the robot and associated with the actual checkpoint and obtained by the cloud management platform may be a panoramic image captured by the robot at the first time, i.e., when the robot moves to the actual checkpoint. Alternatively, due to inaccurate positioning of the robot itself, there is a positional deviation between the actual checkpoint and the desired checkpoint. Therefore, the panoramic image captured by the robot at the actual checkpoint may not contain as clear, complete, and accurate images of the inspection object as the panoramic images captured by the robot before and / or after the first time as the robot approaches the desired checkpoint. In this case, the second panoramic image captured by the robot and associated with the actual checkpoint and obtained by the cloud management platform may also be a panoramic image captured by the robot when the robot moves to the vicinity of the actual checkpoint based on the desired checkpoint, or a fusion result of multiple panoramic images.
[0014] The cloud management platform can refer to the pixel positions of the inspection object (i.e., the expected area) in the first panoramic image taken by the robot at the expected inspection point, and the first information used to determine the position of the actual inspection point where the robot is located, to obtain an image of the target area (e.g., pixel positions) of the inspection object from the second panoramic image taken by the robot related to the actual inspection point. In this way, although the robot's own positioning is inaccurate, the cloud management platform can refer to the pixel positions of the inspection object obtained by the robot at the exact expected position to extract the image of the target area of the inspection object from the second panoramic image related to the actual inspection point (e.g., the panoramic image taken at the actual inspection point position), so that the image of the target area obtained is more complete, accurate, and precise, and the inspection of the inspection object is more thorough, which is beneficial to the maintenance of the inspection object.
[0015] In this way, even if the robot cannot accurately reach the desired inspection point due to the low accuracy of autonomous navigation, the cloud management platform of this application can still obtain a complete image of the inspection object (such as the image of the ROI) in the panoramic image captured at the actual inspection point when there is a certain error between the actual inspection point navigated by the robot and the desired inspection point, and conduct more thorough inspections of the inspection objects corresponding to each desired inspection point. This solves the problem in the prior art of incomplete image information of the inspection object (such as the inspected power equipment) due to the robot's inability to accurately reach the desired inspection point.
[0016] In addition, the robot performs intelligent inspections by using a camera that can capture panoramic images, and the cloud management platform selects images of the target area (a ROI) belonging to the inspection object from the panoramic images captured by the robot inspection. Since the panoramic image information is more comprehensive, the image of the target area obtained can contain more detailed information about the inspection object, which can improve the inspection accuracy of the inspection object.
[0017] In one possible embodiment, the position of the expected checkpoint and the position of the actual checkpoint are three-dimensional positions. Before obtaining the inspection data of the robot, the method further includes: obtaining a first distance related to the expected checkpoint, wherein the first distance is the distance between the camera and the inspection object when the robot is located at the expected checkpoint; obtaining an image of the target area of the inspection object in the second panoramic image based on the expected area and the first information, including: determining the target area of the inspection object in the second panoramic image based on the positional relationship between the position of the expected checkpoint and the position of the actual checkpoint, the expected area and the first distance, wherein the position of the actual checkpoint is determined based on the first information; and obtaining an image of the target area of the inspection object in the second panoramic image based on the target area.
[0018] In an embodiment of the present application, the position error between the desired inspection point and the actual inspection point, the ROI in the first panoramic image captured at the desired inspection point, and the corresponding first distance can be used to determine the pixel location of the inspection object (e.g., the target area) in a second panoramic image captured by the robot and related to the actual inspection point (e.g., a panoramic image captured at or near the actual inspection point). The image of the target area is then extracted from the second panoramic image to obtain a panoramic image of the corresponding inspection object.
[0019] In one possible embodiment, the robot is equipped with a receiver of a global navigation satellite system, and the receiver is used to receive satellite signals. The first information includes a first satellite signal received by the robot at the first time. The method also includes: obtaining a second satellite signal received by the base station at the first time; and processing the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
[0020] In this embodiment, the cloud management platform can utilize post-processing kinematic (PPK) technology to accurately locate the actual position of the robot 100 when it navigates to the desired checkpoint (e.g., the location of the actual checkpoint). The cloud management platform compensates and calibrates the satellite signals received by the robot, whose positioning position is constantly changing, by referencing the satellite signals of a base station whose positioning position remains unchanged. This accurately locates the actual position of the moving robot, thereby improving the accuracy of robot positioning. This allows the cloud management platform to extract an image of the ROI of the inspection object from the second panoramic image captured by the robot, which is associated with the actual checkpoint, based on the positional error between the desired checkpoint and the actual checkpoint.
[0021] In a possible implementation, acquiring the second satellite signal received by the base station at the first time includes: receiving multiple reference satellite signals and the second reception time of each of the reference satellite signals from the base station; and acquiring a reference satellite signal whose second reception time is the first time from the multiple reference satellite signals as the second satellite signal.
[0022] In this embodiment, the cloud management platform can select the reference satellite signal received at the first time from among multiple reference satellite signals received from the base station. This signal is used to calibrate and supplement the satellite signal received by the robot at the first time, thereby improving the positioning accuracy of the actual checkpoint at the robot's first time. By accurately identifying the robot's position, the position error between the actual checkpoint and the desired checkpoint can be used to extract the image of the ROI. In this way, the cloud management platform obtains the reference satellite signal used for robot positioning by directly interacting with the base station, reducing the frequency of link connections between the robot and the base station.
[0023] In other embodiments, the base station may also send multiple reference satellite signals and their second receiving times to the robot, and the robot may upload the multiple reference satellite signals and their second receiving times, or the second satellite signal whose second receiving time after screening is the first time, to the cloud management platform.
[0024] In a possible embodiment, obtaining the inspection data of the robot includes: obtaining the first time when the robot moves to the actual inspection point based on the position of the expected inspection point, multiple candidate panoramic images taken during the robot's inspection process and the shooting time of each candidate panoramic image, multiple candidate satellite signals received during the robot's inspection process and the first receiving time of each candidate satellite signal; obtaining at least one candidate panoramic image among the multiple candidate panoramic images whose shooting time matches the first time, and determining the second panoramic image related to the actual inspection point taken by the robot based on the at least one candidate panoramic image; obtaining the candidate satellite signal among the multiple candidate satellite signals whose first receiving time is the first time as the first information for determining the position of the actual inspection point.
[0025] The cloud management platform can use the first time obtained from the inspection data to screen out at least one candidate panoramic image from multiple candidate panoramic images in the inspection data whose shooting time matches the first time (for example, the shooting time is the first time, or the shooting time is within a certain time period before and / or after the first time); then, the cloud management platform can determine the second panoramic image taken by the robot related to the actual inspection point based on the at least one candidate panoramic image.
[0026] In example A, for example, if the at least one candidate panoramic image is a candidate panoramic image shot at a first time in the inspection data, the cloud management platform can use the candidate panoramic image shot at the first time as the second panoramic image.
[0027] In Example B, the at least one candidate panoramic image is a plurality of candidate panoramic images captured within a certain time period before and after the first time. For example, in Example B, the plurality of candidate panoramic images are multiple candidate panoramic images captured within a certain time period before and after the first time (e.g., 0.1 seconds before and after) when the robot patrols within a preset distance range (e.g., 0.5 meters) of the desired inspection point along the inspection route.
[0028] In Example B, when determining the second panoramic image, the cloud management platform may, for example, refer to a pre-stored desired area (a ROI) related to the desired inspection point to select a candidate panoramic image with the best image quality regarding the area of interest of the inspection object from the multiple candidate panoramic images shot within a certain period of time before and after the first time as the second panoramic image for ROI extraction.
[0029] Alternatively, in Example B, when determining the second panoramic image, the cloud management platform 200 may refer to the pre-stored expected area (a ROI) related to the expected inspection point, and perform image fusion on the multiple candidate panoramic images shot within a certain period of time before and after the first time, so as to obtain a panoramic image with a better image of the area of interest of the inspection object as the second panoramic image for ROI extraction.
[0030] In this way, the cloud management platform can use at least one candidate panoramic image related to the actual inspection point to obtain a second panoramic image. The image area of ROI in the second panoramic image is more complete, comprehensive, and of higher quality, which is conducive to the extraction of ROI, can improve the completeness of the image of the extracted target area, and reduce the acquisition error of the image of the inspection object.
[0031] In a possible embodiment, before obtaining the inspection data of the robot, the method also includes: when the robot inspects within a preset distance range of the location of the expected inspection point according to the inspection route, shooting the multiple candidate panoramic images through the camera, and recording the shooting time of each candidate panoramic image, and recording the first time when the robot moves to the actual inspection point based on the location of the expected inspection point.
[0032] When the robot patrols along the inspection route within a preset distance range of the desired inspection point, for example, if the robot determines through a lidar that its current location is within a certain distance range (e.g., within 0.5 meters) of the desired inspection point, the robot can control the camera to capture one or more candidate panoramic images and record the capture time of each panoramic image. In this way, the robot can capture multiple panoramic images within a preset distance range (e.g., within 0.5 meters, which can be flexibly configured based on needs) near the desired inspection point.
[0033] Since the robot's autonomous navigation technology itself has the problem of low accuracy, the current positioning of the robot determined by the laser radar is not accurate enough. Therefore, by starting to shoot panoramic images when the robot navigates to the position near the desired inspection point, a sufficient number of panoramic images shot when the robot's positioning position is close to the desired inspection point can be obtained, so as to extract images of the target area with higher image quality about the corresponding inspection object (such as the inspected power equipment) with a sufficient number of panoramic images, thereby improving the completeness and accuracy of the image of the inspected power equipment.
[0034] In a possible implementation, obtaining the inspection data of the robot includes: receiving the inspection data from the robot after the robot completes inspection along the inspection route.
[0035] In this embodiment, the cloud management platform can obtain the robot's inspection data after the robot completes the inspection task. In this way, the robot's uploading of inspection data does not affect the execution of the inspection task, nor does it interrupt the execution of the inspection task, thereby ensuring the continuity of the inspection task. Uploading inspection data during the inspection task does not occupy excessive network bandwidth, thereby affecting the quality of the network performing the inspection task. Furthermore, the cloud management platform can uniformly process (e.g., in parallel) the inspection data related to multiple desired checkpoints involved in this inspection task, thereby improving inspection efficiency.
[0036] In a possible implementation, the camera includes at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
[0037] In this embodiment, by installing at least two cameras with a total shooting angle of 360 degrees on the robot, the robot can capture a panoramic image without installing a pan / tilt platform.
[0038] Compared with the method of installing only a single or multiple cameras that do not cover a 360-degree viewing angle on the robot, and changing the shooting angle of the camera by rotating the pan-tilt head to obtain a panoramic image, the camera of the robot of the present application is more efficient in shooting panoramic images; and the mechanical stability of the pan-tilt head rotation is poor. The robot of the present application does not need a pan-tilt head, but uses multiple cameras with different shooting angles to shoot panoramic images, which can improve the shooting stability of the panoramic image; in addition, controlling the camera to shoot a panoramic image by rotating the pan-tilt head requires multiple shots at different times to obtain a panoramic image, while at least two cameras of the robot of the present application shoot simultaneously, thereby reducing the shooting delay of the panoramic image and helping to improve the inspection efficiency.
[0039] The robot's camera is capable of capturing omnidirectional images, providing a wider viewing angle than conventional cameras used in existing robots. Existing intelligent inspection robots generally use conventional cameras, resulting in a limited viewing angle and the potential for incomplete images of the inspection object. The camera in this application, capable of capturing panoramic images, expands the inspection field of view and improves inspection efficiency.
[0040] In a possible implementation, the robot obtains at least two first images by capturing them through the at least two cameras of the camera during an inspection along an inspection route; and the robot combines the at least two first images into a panoramic image.
[0041] In an embodiment of the present application, the robot can capture a panoramic image by synthesizing at least two first images captured by at least two cameras respectively to obtain a panoramic image.
[0042] In one possible embodiment, an embodiment of the present application provides an image processing device, which includes: a configuration module for configuring the positions of multiple expected checkpoints for a robot, where the positions of the expected checkpoints are expected positions for recording images of inspection objects when the robot moves along an inspection route, and the robot is equipped with a camera for capturing panoramic images; an expected area acquisition module for acquiring an expected area related to the expected checkpoint, where the expected area is an area of interest regarding the inspection object in a first panoramic image captured by the robot at the expected checkpoint; an inspection data acquisition module for acquiring inspection data of the robot, where the inspection data includes first information for determining the position of an actual checkpoint and a second panoramic image related to the actual checkpoint captured by the robot, wherein the robot moves to the actual checkpoint based on the position of the expected checkpoint at a first time; an interested image acquisition module for acquiring an image of a target area regarding the inspection object in the second panoramic image based on the expected area and the first information, where the target area is an area of interest regarding the inspection object in the second panoramic image.
[0043] In a possible embodiment, the device also includes: a distance acquisition module, used to acquire a first distance related to the expected inspection point, wherein the first distance is the distance between the camera and the inspection object when the robot is located at the expected inspection point; the image of interest acquisition module, specifically used to: determine the target area regarding the inspection object in the second panoramic image based on the positional relationship between the position of the expected inspection point and the position of the actual inspection point, the expected area and the first distance, wherein the position of the actual inspection point is determined based on the first information; based on the target area, obtain an image of the target area of the inspection object in the second panoramic image.
[0044] In one possible embodiment, the robot is equipped with a receiver of a global navigation satellite system, which is used to receive satellite signals. The first information includes a first satellite signal received by the robot at the first time. The device also includes: a signal acquisition module, which is used to acquire a second satellite signal received by the base station at the first time; and an image of interest acquisition module, which is specifically used to process the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
[0045] In a possible implementation, the signal acquisition module is specifically used to: receive multiple reference satellite signals and the second reception time of each reference satellite signal from a base station; and obtain a reference satellite signal whose second reception time is the first time among the multiple reference satellite signals as the second satellite signal.
[0046] In a possible embodiment, the inspection data acquisition module is specifically used to: obtain the first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, multiple candidate panoramic images taken during the robot's inspection process and the shooting time of each candidate panoramic image, multiple candidate satellite signals received during the robot's inspection process and the first receiving time of each candidate satellite signal; obtain at least one candidate panoramic image among the multiple candidate panoramic images whose shooting time matches the first time, and determine the second panoramic image related to the actual checkpoint taken by the robot based on the at least one candidate panoramic image; obtain the candidate satellite signal among the multiple candidate satellite signals whose first receiving time is the first time as the first information for determining the position of the actual checkpoint.
[0047] The effects of the image processing apparatuses of the above embodiments are similar to the effects of the image processing methods of the above embodiments, and are not described in detail here.
[0048] In a possible implementation, the present application provides an inspection system, which includes the image processing device and the robot described in any one of the above implementations.
[0049] In one possible embodiment, the robot is used to capture the multiple candidate panoramic images through the camera when patrolling within a preset distance range of the location of the expected checkpoint according to the inspection route, and record the shooting time of each candidate panoramic image, as well as the first time when the robot moves to the actual checkpoint based on the location of the expected checkpoint.
[0050] In a possible implementation, the robot is further configured to send inspection data to the image processing device after completing the inspection along the inspection route.
[0051] In a possible implementation, the camera of the robot has at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
[0052] In a possible embodiment, the robot is further configured to: obtain at least two first images by capturing at least two first images through the at least two cameras of the camera during an inspection process according to an inspection route; and synthesize the at least two first images into a panoramic image.
[0053] In one possible implementation, an embodiment of the present application provides a device cluster, including at least one processing device, each processing device including a processor and a memory; the processor of the at least one processing device is used to execute instructions stored in the memory of the at least one processing device, so that the device cluster performs the image processing method in any possible implementation described above.
[0054] The effect of the device cluster in this embodiment is similar to the effect of the image processing method in the above embodiments, and will not be described in detail here.
[0055] In one possible implementation, an embodiment of the present application provides a computer program product including instructions. When the instructions are executed by a device cluster, the device cluster executes the image processing method in any of the above possible implementations.
[0056] The effects of the computer program product of this embodiment are similar to the effects of the image processing methods in the above embodiments, and will not be described in detail here.
[0057] In a possible implementation, an embodiment of the present application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a device cluster, the device cluster executes the image processing method in any one of the above implementations.
[0058] The effects of the computer-readable storage medium of this embodiment are similar to the effects of the image processing methods in the above embodiments, and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] FIG1 is a schematic diagram of an inspection robot in the prior art;
[0060] FIG2 is a schematic diagram of an exemplary system architecture;
[0061] FIG3 a is a schematic diagram of an exemplary robot;
[0062] FIG3 b is a schematic diagram of an exemplary base station;
[0063] FIG4 a is a schematic diagram illustrating an exemplary processing process;
[0064] FIG4 b is a schematic diagram illustrating an exemplary processing process;
[0065] FIG5 is a schematic diagram showing an exemplary distance measurement;
[0066] FIG6 is a schematic diagram showing an exemplary image of a selected target area;
[0067] FIG7 is a schematic structural diagram of an exemplary image processing apparatus;
[0068] FIG8 is a schematic structural diagram of an exemplary inspection system;
[0069] FIG9 is a schematic diagram illustrating the structure of a computing device;
[0070] FIG10 is a schematic diagram illustrating the structure of a computing device;
[0071] FIG11 is a schematic diagram showing the structure of an exemplary computing device cluster. DETAILED DESCRIPTION
[0072] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0073] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0074] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.
[0075] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.
[0076] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.
[0077] Before describing the technical solutions of the embodiments of the present application, a brief introduction to the background technologies involved in the embodiments of the present application is first given:
[0078] A public cloud is a cloud platform provided by a third-party public cloud provider to individuals and businesses. In a public cloud, the hardware, software, and other infrastructure are all owned and managed by the third-party public cloud provider.
[0079] The cloud platform, which may also be referred to as a cloud management platform or cloud management platform in the embodiments of the present application, is a software system for cloud technology (also known as cloud computing technology) services provided by a cloud provider, which is used to provide an interface related to cloud services for tenants to remotely access cloud services. Tenants can log in to the cloud platform on the cloud service access page using a pre-registered account and password, and after a successful login, select and purchase the corresponding cloud service on the cloud service access page, such as object storage services, virtual machine services, container services, etc. Exemplarily, the cloud management platform can provide a unified entrance (also known as an operation and maintenance entrance), and users can access the cloud management platform through the unified entrance and manage the cloud resources (including hardware and software) of the corresponding cloud through the cloud management platform.
[0080] Cloud services include computing services, storage services, virtual machine services, network services, etc. Any device or function that a user device can access on the cloud platform can be considered a service provided by the cloud platform.
[0081] Cloud resources are hardware resources (also referred to as cloud infrastructure or infrastructure) and software resources (also referred to as public cloud services) used to provide services. The infrastructure includes multiple cloud data centers located in multiple regions, with at least one cloud data center located in each region. Cloud data centers may include computing nodes, storage nodes, and network nodes. Computing nodes may be computing servers, storage nodes may be storage servers, and network nodes may include network facilities such as switches. For example, the hardware resources corresponding to software resources such as computing services, storage services, or network services include computing resources, storage resources, or network resources. Computing resources may be deployed on computing nodes, storage resources may be deployed on storage nodes, and network resources may be deployed on network nodes.
[0082] Optionally, computing resources may include central processing unit (CPU) resources, memory resources, and / or hard disk resources. For example, a user (also referred to as a tenant) purchases a large number of virtual machine resources, and a large number of applications (also referred to as cloud applications) are deployed on the virtual machine resources. In this embodiment, the virtual machines and the applications in the virtual machines are cloud services (i.e., software resources), and the servers and other devices to which the virtual machines belong are the corresponding hardware resources.
[0083] As the scale of power systems continues to grow, higher requirements are placed on the safe operation of infrastructure within power environments, such as power grids and power plant equipment. To ensure the safe operation of infrastructure within power environments and to ensure power safety, inspections of these infrastructure are necessary. Manual inspections are gradually being phased out due to low operational efficiency, poor inspection quality, and numerous subjective factors. Currently, automated, autonomous digital inspections of facilities within power environments are primarily performed using robots (such as intelligent inspection robots on standby 24 hours a day). Robotic inspections not only improve operational and maintenance inspection efficiency but also help address issues inherent in manual inspections.
[0084] The robots in the existing technology are mainly intelligent inspection robots that rely on simultaneous localization and mapping (SLAM) technology for autonomous navigation. The intelligent inspection robot is composed of multiple sensors such as a robot motion controller, a laser sensor, a pan-tilt head, and a high-definition camera. The intelligent inspection robot uses SLAM technology to achieve autonomous trackless navigation in an electric power environment. During the navigation process, it can perceive the surrounding environment. When it senses an obstacle, it can avoid the obstacle and replan the route, thereby realizing autonomous patrol of the electric power environment.
[0085] However, for the above-mentioned intelligent inspection robot with trackless autonomous navigation outdoors, whether the robot can accurately reach the desired inspection point to inspect the inspection object (such as power plant equipment, power grid, electricity meter, etc.) is related to many factors, such as the robot's motion error in the outdoor environment, positioning accuracy, sensor fusion, and the robustness of the navigation algorithm, which will affect the robot's navigation accuracy.
[0086] Among the factors mentioned above that affect robot navigation accuracy, the most important one is the robot's positioning accuracy. In the prior art, intelligent inspection robots that rely on SLAM technology for positioning can infer the robot's current position through sensor observation of the environment. However, these sensors themselves are subject to certain observation errors, and the positioning accuracy of SLAM technology is also relatively low. As a result, these robots in the prior art suffer from inaccurate self-positioning estimates, resulting in the robot being unable to accurately reach the desired inspection point.
[0087] Among them, the SLAM technology used by robots in the prior art is a relative positioning technology, that is, it uses a technology that positions relative to an initial position. The disadvantage of relative positioning technology is that positioning errors will continue to accumulate over time. Therefore, when robots in the prior art are in long-term movement, their positioning errors will become increasingly larger as the movement duration increases, resulting in the robot's positioning accuracy decreasing as the inspection time increases. Therefore, in order to ensure positioning accuracy, robots in the prior art need to adjust the robot's pan / tilt or algorithm after a certain period of movement to reduce the robot's positioning error.
[0088] Furthermore, the robot's inspection environment can also affect its ability to accurately reach the desired inspection point. Actual outdoor inspection environments are often complex, with interference from dynamic objects (such as moving people or objects), random obstacles and obstructions along the inspection route, and uneven outdoor roads. These challenges the positioning accuracy of existing robot motion control systems and SLAM algorithms.
[0089] Figure 1 is a schematic diagram of an intelligent inspection robot in the prior art. It shows the robot's fixed-angle pan-tilt platform 10 and a conventional camera 11 mounted on the platform. The conventional camera 11 typically has a shooting angle of approximately 120 degrees. If the robot's positioning is inaccurate, it cannot accurately reach the desired inspection point. Consequently, the images captured by the conventional camera equipped with the fixed pan-tilt platform at the inaccurate inspection point are incomplete or contain errors, leading to incomplete inspections.
[0090] To address the aforementioned technical issues with existing intelligent inspection robots, this application provides an image processing method and a robotic inspection system. The robotic inspection system can execute the image processing method to inspect devices in an outdoor environment (using, but not limited to, an electrical environment as an example). The devices in the outdoor environment are fixed in position, such as an electric meter in an electrical environment.
[0091] The robot inspection system may include a robot 100, a base station 300, and a management platform, and may optionally further include terminal equipment, etc.
[0092] In this embodiment, the management platform is a cloud management platform 200 deployed on the cloud.
[0093] In other embodiments, the management platform may also be a server or server cluster under the cloud, and the execution principle of the image processing method is similar to the execution principle of the cloud management platform 200 deployed on the cloud, which will not be repeated here.
[0094] In this embodiment, the cloud management platform 200 can utilize post-processing kinematic (PPK) technology to accurately locate the actual position of the robot 100 when it navigates to a desired checkpoint (e.g., the position of the actual checkpoint), thereby improving the accuracy of the robot 100's positioning. Furthermore, the cloud management platform 200 can extract an image of the ROI of the inspection object from the panoramic image captured by the robot 100 at the actual checkpoint based on the position error between the desired checkpoint and the actual checkpoint. In this way, even if the robot 100 is unable to accurately reach the desired checkpoint due to low autonomous navigation accuracy, the cloud management platform 200 of the present application can still capture a complete image of the inspection object (e.g., the image of the ROI) in the panoramic image captured at the actual checkpoint, even if there is a certain error between the actual checkpoint to which the robot 100 navigates and the desired checkpoint, thereby ensuring a more thorough inspection of the inspection objects corresponding to each desired checkpoint. This solves the problem in the prior art of incomplete image information of the inspection object (e.g., the inspected electrical equipment) due to the robot 100's inability to accurately reach the desired checkpoint.
[0095] Among them, PPK technology is a satellite positioning technology. This application applies it to the positioning of robot 100. It can refer to the satellite signals of the base station whose positioning position remains unchanged to compensate and calibrate the satellite signals received by the robot 100 whose positioning position is constantly changing, so as to accurately locate the actual position of the moving robot 100, thereby achieving higher accuracy in the positioning of the robot 100 itself. Compared with the SLAM technology used by the robot 100 in the prior art, this PPK technology is an absolute positioning technology that can be used for long-distance positioning of the robot 100. It does not cause the problem of increasing positioning error due to the increase in the movement time of the robot 100, and can accurately locate the robot 100 during long-distance inspections.
[0096] Furthermore, in the method of the present application, the robot 100 only needs to autonomously navigate along the inspection route to collect data. For example, when navigating to a desired checkpoint (e.g., when moving to the actual checkpoint based on the location of the desired checkpoint), it can capture a panoramic image, receive satellite signals, and report the collected data to the cloud management platform 200. The cloud management platform 200 then processes the collected data from the robot 100 to obtain an image of the ROI. Therefore, during the robot 100 inspection process, each time the robot 100 navigates to a desired checkpoint, regardless of whether the robot 100 accurately reaches the desired checkpoint location, the robot 100 does not need to interact with the base station to make any adjustments. For example, each time the robot 100 navigates to the desired checkpoint, the robot 100 does not need to establish a radio communication data link with the base station. Therefore, the robot 100 does not need to use the data link to receive satellite signals from the base station (the satellite signals can be the satellite signals received by the base station when the robot 100 navigates to the desired inspection point) and use the satellite signals received from the base station and the satellite signals received by the robot itself when navigating to the desired inspection point to reduce the position error between the actual inspection point where the robot 100 is located and the desired inspection point. Instead, the robot 100 can continue to perform the inspection task. This can reduce the communication frequency between the robot 100 and the base station, avoid interference with the robot 100 caused by electromagnetic signals from the high-voltage power station, expand the operating range, make the operating method more flexible, simplify the subsequent maintenance of the robot 100, and make the robot 100 of the present application more anti-interference.
[0097] Before introducing the method of the present application in detail, a brief introduction to the system architecture of the present application is first given.
[0098] FIG2 is a schematic diagram of the system architecture of the present application, and FIG3a and FIG3b are schematic diagrams of the structures of the robot 100 and the base station 300 of the present application, respectively.
[0099] The system of the present application shown in FIG2 is introduced below in conjunction with FIG3a and FIG3b:
[0100] As shown in FIG. 2 , the system may include a robot 100 , a base station 300 , a cloud management platform 200 , and cloud resources 400 .
[0101] As shown in FIG3 a , the robot 100 may include but is not limited to: a panoramic camera 101 , a Global Navigation Satellite System (GNSS) receiver, here a GNSS receiving antenna 102 , a laser radar 103 , and the like.
[0102] As shown in FIG3 , the panoramic camera 101 may include a panoramic camera, which may include two cameras with different shooting angles, namely camera 1011 and camera 1012. The shooting angles of camera 1011 and camera 1012 both range from 0 to 180 degrees, and camera 1011 and camera 1012 are arranged relative to each other. Thus, the robot 100 may use camera 1011 and camera 1012 to capture two images with a viewing angle range of 0 to 180 degrees, respectively. The robot 100 may then combine the two images into a single panoramic image captured by the panoramic camera 101.
[0103] In other embodiments, the robot 100 may be equipped with more than two cameras for capturing 360-degree panoramic images, with the cameras having different viewing angles, and the sum of the viewing angles of the cameras being 360 degrees. In specific applications, the number and installation angles of the cameras for capturing panoramic images can be flexibly configured based on actual needs.
[0104] A panoramic camera is a camera that can capture images in all directions, providing a wider viewing angle than conventional cameras used in existing robots. Existing intelligent inspection robots generally use conventional cameras, resulting in a limited viewing angle and potentially incomplete images of the inspection object. This application utilizes a panoramic camera in the robot to expand the inspection field of view and improve inspection efficiency.
[0105] The panoramic image captured by the panoramic camera 101 of the present application is a photo presenting a 360° perspective, which is usually circular or annular. It is synthesized by taking multiple images and displayed in a planar manner. The initial synthesized image may have certain bending deformations, so the synthesized image can be processed in a certain way. The processing technology can be any related image processing technology developed in the existing technology or in the future. There is no limitation here. The panoramic image described in each embodiment of the present application can be obtained.
[0106] The robot 100 performs intelligent inspections by using a panoramic camera 101, and selects images of the target area (a ROI) belonging to the inspection object from the panoramic images taken during the inspection. Since the image information of the panoramic image is more comprehensive, the image of the target area obtained can contain more detailed information of the inspection object, which can improve the inspection accuracy of the inspection object, such as power equipment or power grids in an electric power environment.
[0107] In addition, in this embodiment, by installing two cameras with a shooting angle of view within 180 degrees on the robot 100, and the two cameras are arranged relative to each other to obtain a panoramic image, the robot 100 of the present application does not need to be installed with a pan-tilt head.
[0108] Compared with the method of only installing a single or multiple cameras that do not cover a 360-degree viewing angle on the robot 100, and changing the shooting angle of the camera by rotating the pan-tilt head to obtain a panoramic image, the panoramic camera 101 of the present application is more efficient in shooting panoramic images; and the mechanical stability of the pan-tilt head rotation is poor. The robot 100 of the present application does not need a pan-tilt head, but uses multiple cameras with different shooting angles to shoot panoramic images, which can improve the shooting stability of the panoramic image; in addition, controlling the camera to shoot a panoramic image by rotating the pan-tilt head requires multiple shots at different times to obtain a panoramic image, while at least two cameras of the panoramic camera 101 of the present application shoot at the same time, thereby reducing the shooting delay of the panoramic image and helping to improve inspection efficiency.
[0109] The GNSS receiving antenna 102 shown in FIG3 a can be used to receive satellite signals, which are observations of the carrier phase of the satellite. In addition, the robot 100 can use a laser radar 103 to perform autonomous navigation.
[0110] As shown in FIG3 b , a reference station receiver 301 may be installed on the base station 300 , and the base station 300 may use the reference station receiver 301 to receive a satellite signal, which is an observation of the carrier phase of the satellite.
[0111] Since the robot 100 and the base station 300 are located at different positions, their observations of the satellite's carrier phase may be different. The cloud management platform 200 shown in Figure 2 can use the respective observations of the satellite's carrier phase from the robot 100 and the base station 300 at the same time to determine the actual position of the robot 100.
[0112] The cloud management platform 200 can select an image of a target area of the inspection object of interest from a panoramic image taken by the robot 100 at the actual positioning position based on the position error between the actual positioning of the robot 100 and the positioning of the desired inspection point to which the robot is expected to move.
[0113] Exemplarily, when the cloud management platform 200 processes the data received from the base station 300 and the robot 100 to obtain the image of the target area, the cloud resource 400 shown in FIG. 2 may be used for computational processing.
[0114] 2 and FIG3 a, FIG3 b, the image processing method of the present application may include a configuration stage and a processing stage.
[0115] The configuration phase may include configuration operations on the robot 100, configuration operations on the cloud management platform 200, etc.
[0116] The processing stage may include but is not limited to: the cloud management platform 200 processes the inspection data obtained from the robot 100 performing the inspection task and the satellite signal from the base station 300 to obtain an image of the target area of the inspection object related to each desired inspection point.
[0117] As shown in Figure 4a, the configuration phase may include the following steps:
[0118] S100, pre-configuring the inspection task of the robot 100.
[0119] The inspection task may include but is not limited to: an inspection route (optional), first positions of multiple desired inspection points, etc.
[0120] The position of the expected inspection point is the expected position for recording the image of the inspection object when the robot 100 moves along the inspection route.
[0121] The position of the expected inspection point is the position where the robot is expected to reach. For example, empirical analysis has determined that when the robot 100 is located at the position of the expected inspection point to take a picture of the inspection object, the image taken of the inspection object is most consistent with expectations, for example, the image information about the inspection object is the most complete and accurate.
[0122] For example, to inspect an object, the robot 100 is expected to move to a desired location along an inspection route and photograph the object at the desired location to obtain an image of the object. However, due to the inaccuracy of the robot's autonomous navigation, when the robot 100 navigates based on the desired inspection point, the robot 100 determines that it has moved to the desired inspection point (determines that it is located at the desired inspection point). However, the actual location of the robot 100 (the actual inspection point location) at this time may deviate from the desired inspection point location by a certain amount, and the two locations may not be the same.
[0123] For example, the inspection objects corresponding to each desired inspection point are different. In this way, the robot 100 can be used to navigate to each desired inspection point to implement inspections of multiple inspection objects.
[0124] The inspection route may include, in sequence, an inspection starting point (optional), the first position of each desired inspection point, and an inspection end point (optional). For example, the inspection route is A->B1->B2->B3->C.
[0125] Point A is the inspection starting point, Points B1, B2, and B3 are the locations of the three desired inspection points, and Point B is the inspection endpoint. The inspection route starts at Point A and continues to Point B1, then from Point B1 to Point B2, then from Point B2 to Point B3, and finally, from Point B3 to Point C.
[0126] The robot 100 may be pre-configured at the positions of the points B1 , B2 , and B3 (an example of a first position).
[0127] Regarding the implementation method of pre-configuring the inspection task for the robot 100, the robot 100 can be configured through a terminal device.
[0128] In some embodiments, the cloud management platform 200 may also pre-configure the inspection task to the robot 100 by issuing the inspection task to the robot.
[0129] For example, information about the inspection task can be pre-entered into the cloud management platform 200 through the user interface, and then the robot 100 can capture the inspection task information (such as the coordinates of the desired inspection point, the inspection route, etc.) from the cloud management platform 200 to facilitate the execution of the inspection task.
[0130] Optionally, S200a, the cloud management platform 200 pre-stores the desired area related to the desired checkpoint.
[0131] The expected area is a region of interest (ROI) of the inspection object in the first panoramic image pre-photographed by the robot 100 at the expected inspection point, such as the position of a pixel point.
[0132] Optionally, at S200b, the cloud management platform 200 pre-stores a first distance associated with the desired inspection point, wherein the first distance is the distance between a panoramic camera of the robot 100 (e.g., the panoramic camera 101 shown in FIG. 3a ) and the inspection object when the robot 100 is located at the desired inspection point.
[0133] 5 , in a specific implementation, the robot 100 can be manually remotely controlled to be placed at the first position of the desired inspection point, or the robot 100 can be directly placed at the first position of the desired inspection point, where the position of the desired inspection point is taken as point B1 as an example; then, the first distance d between the panoramic camera 101 (for example, the center point 1013 of the panoramic camera 101, not specifically limited) of the robot 100 located at the position of the desired inspection point (point B3) is measured manually or by using the laser radar 103 (or the distance sensor of the robot 100) as shown in FIG3a and the corresponding inspection object P is measured.
[0134] It should be understood that the multiple expected inspection points each correspond to a different inspection object, so that different inspection objects are inspected separately at the multiple inspection points.
[0135] Then, the robot 100 can be manually remotely controlled to use the panoramic camera 101 to capture a panoramic image at the desired inspection point B1;
[0136] Next, the image region of the inspection object can be extracted from the panoramic image by controlling the robot 100 or through post-processing, and the pixel locations of the image region in the panoramic image can be determined to obtain the desired region of the inspection object in the panoramic image taken at the desired inspection point. It should be understood that the image of the desired region is also a panoramic image, but the panoramic image only includes the panoramic image of the inspection object of interest and does not include image content of other facilities or the on-site environment.
[0137] In this way, when there are multiple expected inspection points, an expected area and a first distance related to the corresponding inspection object can be obtained for each expected inspection point.
[0138] In some embodiments, the robot 100 may also capture multiple panoramic images while at the first position of a desired inspection point, thereby processing the image to obtain a desired region of the inspection object associated with the desired inspection point. This prevents the accuracy of the obtained desired region from being affected by the poor image quality of a single panoramic image. By pre-acquiring a highly accurate desired region of the inspection object associated with the desired inspection point, which can be used to extract an image of the target region of the inspection object from the panoramic image captured at the actual inspection point, the recognition accuracy and image quality of the target region image can be improved.
[0139] In one possible embodiment, the cloud management platform 200 may be configured with a user interface, through which the user may transmit at least one of the above-mentioned expected area and first distance related to the expected checkpoint to the cloud management platform 200. In this way, the cloud management platform 200 may pre-acquire at least one of the expected area and corresponding first distance related to the expected checkpoint regarding the corresponding inspection object through the user interface.
[0140] For example, the cloud management platform 200 may store at least one of the expected area and the first distance related to the expected checkpoint obtained in advance in a storage node in the cloud data center.
[0141] In another possible implementation, at least one of the aforementioned desired area and the first distance related to the desired inspection point may be pre-stored in the robot 100 before the robot 100 performs the inspection task.
[0142] Then, before performing the inspection task, the robot 100 may upload at least one of the pre-stored desired area and the first distance to the cloud management platform 200 so that the cloud management platform 200 pre-stores at least one of the desired area and the first distance.
[0143] In other embodiments, the robot 100 may also upload at least one of the pre-stored expected area and the first distance to the cloud management platform 200 during the execution of the inspection task or after the inspection task is completed, so that the cloud management platform 200 can obtain at least one of the expected area and the first distance related to the expected checkpoint in the above processing stage.
[0144] That is, the cloud management platform 200 may acquire the desired area and first distance associated with the desired inspection point during the configuration phase and / or the processing phase, depending on the application scenario and actual needs, as well as the network quality of the mobile network (e.g., 5G network) of the robot 100. The cloud management platform 200 may acquire all data regarding the desired area and first distance during the configuration phase or the processing phase; or the cloud management platform 200 may acquire a portion of the data regarding the desired area and first distance during the configuration phase and another portion during the processing phase. This allows the cloud management platform 200 to receive inspection data obtained by the robot 100 during the processing phase, and then utilize the acquired desired area and first distance to obtain an image of the target area of the corresponding inspection object.
[0145] This application does not restrict the execution order of S200a and S200b. For example, S200a and S200b may be executed serially or in parallel. Furthermore, this application does not restrict the execution order of S100 executed by the robot 100 and S200a and S200b executed by the cloud management platform 200.
[0146] After the configuration stage shown in FIG4a , as shown in FIG4b , the processing stage may include but is not limited to the following steps:
[0147] The robot 100 starts to perform the inspection task.
[0148] For example, the inspection task may include, but is not limited to: inspection time, inspection route, the first position of each expected inspection point, etc.
[0149] For example, the inspection route may include the inspection starting position (optional), the first position of each desired inspection point, and the inspection route A->B1->B2->B3.
[0150] Point A is the inspection starting point, Points B1, B2, and B3 are the locations of the three desired inspection points, and Point B is the inspection endpoint. The inspection route starts at Point A, inspects Point B1, then inspects Point B1 to Point B2, and then inspects Point B2 to Point B3, concluding the inspection.
[0151] When performing an inspection task, the robot 100 may begin autonomous navigation along the inspection route in an electric power environment at the inspection time (e.g., 10:00 PM every Sunday evening), navigating from point A to point B1. Upon reaching point B1, the robot captures a panoramic image. Similarly, the robot then autonomously navigates from point B1 to point B2, capturing a panoramic image upon reaching point B2. Similarly, the robot then autonomously navigates from point B2 to point B3, capturing a panoramic image upon reaching point B3, and the inspection is complete. However, due to the positioning inaccuracy inherent in the autonomous navigation of the robot 100, when the robot 100 determines that it has navigated to point B1, there is still a certain distance (also known as position error) between the actual inspection point (e.g., point D1) and the desired inspection point (point B1). Therefore, the panoramic image captured by the robot 100 upon navigating to point B1 is the panoramic image captured by the robot 100 at point D1.
[0152] When the robot 100 autonomously navigates to each desired inspection point, the operations performed are similar. Therefore, the following takes the robot 100 navigating to point B1 as an example to illustrate the operations of its inspection process. The operations when the robot autonomously navigates to other desired inspection points can be referred to the operations of navigating to point B1, which will not be repeated here.
[0153] The following describes in detail the steps S101a to S105 performed by the robot 100 during the inspection task.
[0154] S101a, the robot 100 captures a panoramic image and records the panoramic image and its capturing time.
[0155] In this embodiment, the robot 100 can use autonomous navigation technology to sequentially move to the first position of each desired inspection point along the inspection route. When the robot 100 inspects within a preset distance range of the first position of the desired inspection point along the inspection route, for example, if the robot 100, using the laser radar 103 shown in FIG3a , determines that the robot 100 is currently located within a certain distance range of the desired inspection point (point B1) (e.g., within 0.5 meters of point B1), the robot 100 can control the panoramic camera 101 to capture one or more panoramic images (also referred to as candidate panoramic images) and record the capture time of each panoramic image. In this way, the robot 100 can capture multiple panoramic images within a preset distance range (e.g., within 0.5 meters, which can be flexibly configured based on needs) near the desired inspection point.
[0156] Since the technology of autonomous navigation of the robot 100 itself has the problem of low accuracy, the current positioning of the robot 100 determined by the laser radar 103 is not accurate enough. Therefore, by starting to shoot panoramic images when the robot 100 navigates to the position near the desired inspection point, a sufficient number of panoramic images shot when the positioning position of the robot 100 is close to the desired inspection point can be obtained, so as to facilitate the extraction of images of the target area with higher image quality of the corresponding inspection object (such as the inspected power equipment) with a sufficient number of panoramic images, thereby improving the completeness and accuracy of the image of the inspected power equipment.
[0157] In other embodiments, the robot 100 may also execute S101a to capture a panoramic image when it is determined in S102 that the robot has navigated to a desired checkpoint.
[0158] That is, the robot 100 can also use autonomous navigation technology to determine the first time it will navigate to a desired checkpoint, such as the first time the robot 100 moves to the corresponding actual checkpoint based on the desired checkpoint. At this first time, the robot 100 can use the panoramic camera 101 to capture a panoramic image and associate the panoramic image with the first time. In this way, the robot 100 does not need to capture multiple panoramic images near the desired checkpoint during the execution of the inspection task. It only needs to capture the panoramic image at the first time it navigates to the desired checkpoint. This can reduce the proportion of the panoramic image cache space on the robot 100, save the robot 100's cache space, and also reduce the performance loss of the robot 100 due to continuous image capture.
[0159] S101b, the robot 100 receives a first satellite signal and records a first receiving time of the first satellite signal.
[0160] Among them, at the beginning of the inspection task, the robot 100 can start the switch of the GNSS receiving antenna 102 as shown in Figure 3, and during the execution of the inspection task by the robot 100, before the inspection task ends, it can receive satellite signals through the GNSS receiving antenna 102 and record the reception time of each satellite signal.
[0161] The first satellite signal received by the robot 100 may be an observation value of the carrier phase of the satellite (for example, represented by observation value 1).
[0162] S101a and S101b are parallel steps in the process of the robot 100 performing the inspection task.
[0163] S102 , the robot 100 determines whether it has navigated to the desired checkpoint.
[0164] As shown in S100 of FIG4a , the robot 100 has pre-configured the first position of each desired checkpoint. The robot 100 can navigate to each first position according to the inspection route through autonomous navigation technology. When navigating to a first position, the robot 100 can determine that it has currently navigated to the desired checkpoint (for example, moving to the actual checkpoint based on the position of the desired checkpoint), and thus can execute S103.
[0165] S103, the robot 100 records the first time.
[0166] Among them, the robot 100 can record the time of navigation to the desired checkpoint. From the perspective of the robot 100, at the first time, the robot 100 is already at the desired checkpoint (for example, point B1). However, in fact, because the accuracy of the robot 100's self-positioning in autonomous navigation is not high enough, the robot 100 moves to the position of the actual checkpoint at the first time according to the position of the desired checkpoint, also known as the position of the actual checkpoint, which still has a certain position error compared with the position of the desired checkpoint.
[0167] After S103 , in S104 , the robot 100 determines whether the inspection task is completed.
[0168] For example, the inspection route is A->B1->B2->B3.
[0169] For example, the robot 100 moves to each desired inspection point, namely point B1, point B2, and point B3, according to the inspection route within a specified time period (optional), and takes panoramic images at the three desired inspection points (or near the three desired inspection points). Then, the inspection mission of the robot 100 is completed.
[0170] For another example, if the robot 100 does not move to all the expected checkpoints configured for the robot 100 in FIG. 4 a according to the inspection route, or if corresponding panoramic images are not captured at all the expected checkpoints, it is determined that the inspection task is not completed.
[0171] After S104 , the robot 100 determines that the inspection task is not completed, and then proceeds to S102 .
[0172] Among them, when the robot 100 determines that the inspection task has not been completed, it means that the robot 100 has not inspected all the expected inspection points, and then continues to navigate according to the autonomous navigation technology, according to the inspection route and the first position of the expected inspection point configured in S100 in Figure 4a. When navigating to the next expected inspection point in the inspection route (for example, point B2), it continues to execute S103 and S104.
[0173] After S104 , the robot 100 determines that the inspection task has ended and then executes S105 .
[0174] When the robot 100 determines that the inspection task has been completed, it means that the robot 100 has moved to all the expected inspection points and captured a panoramic image when navigating to the first position of the corresponding expected inspection point (or near the first position, for example, within 0.5 meters of the first position). Then the robot 100 can execute S105 to upload the initial inspection data of the inspection task to the cloud management platform 200.
[0175] S105 , the robot 100 uploads the initial inspection data to the cloud management platform 200 .
[0176] The initial inspection data may include each panoramic image (also referred to as a candidate panoramic image) captured by the robot 100 when it moves to each actual inspection point (e.g., within 0.5 meters of the desired inspection point) during the inspection mission, and its capture time (obtained in S101a), as well as each first satellite signal received during the inspection mission and its first reception time (obtained in S101b), and the first time each time the robot 100 navigates to the desired inspection point, also referred to as the first time the robot 100 moves to the actual inspection point based on the position of the desired inspection point (obtained in S103). At this first time, the robot 100 is determined to have moved to the desired inspection point, but there is still a certain position error between the actual position of the robot 100 and the first position of the desired inspection point.
[0177] In other embodiments, when the expected area and the first distance are not pre-stored in the cloud management platform 200 through S200a and S200b shown in Figure 4a during the configuration stage, the robot 100 may upload the expected area and the first distance of the inspection object related to each expected inspection point described in S200a and S200b as part of the initial inspection data or separately to the cloud management platform 200 when executing S105.
[0178] In other embodiments, the panoramic image and its capture time in the inspection data uploaded by the robot 100 to the cloud management platform 200 may also be at least one candidate panoramic image and its capture time, obtained by the robot 100 through screening the data obtained in S101a, whose capture time matches the first time. The cloud management platform 200 can then determine the second panoramic image captured by the robot 100 at the actual inspection point based on the at least one candidate panoramic image (see the detailed example of S201a below for the specific implementation process).
[0179] Alternatively, in some embodiments, the panoramic image and its shooting time in the inspection data uploaded by the robot 100 to the cloud management platform 200 may also be obtained by filtering the data obtained in S101a by the robot 100, and the second panoramic image shot by the robot 100 at the actual inspection point, wherein the process of the robot 100 screening and processing the candidate panoramic images in the initial inspection data to determine the second panoramic image is similar to the process performed by the cloud management platform 200 in the following S201a, and will not be repeated here.
[0180] In this way, if the robot 100 has high computing power, it can pre-screen the panoramic images in the inspection data before uploading them. Thus, from multiple candidate panoramic images captured by the robot 100 within 0.5 meters of the desired inspection point during the inspection process, it can select a panoramic image captured at the first time the robot 100 navigated to the desired inspection point, or select one or more candidate panoramic images captured around the first time (e.g., within a certain time period before or after the first time), or select a second panoramic image from the multiple candidate panoramic images captured above, and upload them to the cloud management platform 200. This reduces the amount of data transmitted, thereby reducing network bandwidth usage, increasing data transmission speed, and accelerating the processing of inspection data by the cloud management platform 200. For example, the certain time period can be 0.1 seconds, which can be set according to needs and is not limited here.
[0181] In this embodiment, after the inspection task is completed, the robot 100 can upload initial inspection data. This initial inspection data includes data related to each desired inspection point involved in the inspection task, such as a candidate panoramic image captured at or near the first time, and a first satellite signal received at the first time. In this way, the robot 100's uploading of inspection data does not affect or interrupt the execution of the inspection task, ensuring the continuity of the inspection task. Uploading inspection data during the inspection task also avoids excessive network bandwidth usage and impacting the quality of the network used for the inspection task.
[0182] In some embodiments, when the robot 100 has a relatively good 5G network quality (e.g., a relatively strong signal strength), the robot 100 may also, during the execution of an inspection mission, upload at least one candidate panoramic image that matches the first time (or a second panoramic image determined based on the at least one candidate panoramic image) and the first satellite signal received at the first time to the cloud management platform 200 whenever it navigates to a desired inspection point and obtains one or more corresponding candidate panoramic images and first satellite signals. This allows the cloud management platform 200 to promptly obtain inspection data obtained by the robot 100 during inspections near the desired inspection point, for use in obtaining an image of the target area of the inspection object corresponding to the desired inspection point.
[0183] In the method of the present application, the robot 100 only needs to autonomously navigate along the inspection route to collect data. For example, it captures panoramic images and receives satellite signals when moving from the desired inspection point to the actual inspection point, and reports the collected data to the cloud management platform 200. The cloud management platform 200 then processes the collected data from the robot 100 to obtain an image of the ROI. Therefore, during the robot 100's inspection process, each time the robot 100 navigates to the desired inspection point, regardless of whether the robot 100 accurately reaches the desired inspection point, the robot 100 does not need to interact with the base station to make any adjustments. For example, each time the robot 100 navigates to the desired inspection point, the robot 100 does not need to establish a radio communication data link with the base station. Instead, the robot 100 uses this data link to receive satellite signals from the base station 300 and uses the satellite signals received from the base station and its own satellite signals to reduce the position error between the actual inspection point where the robot 100 is located and the desired inspection point, and continues to perform the inspection task. This can reduce the communication frequency between the robot 100 and the base station 300, avoid interference of the high-voltage power station electromagnetic signal on the robot 100, make the operation range wider, the operation method more flexible, and the subsequent maintenance of the robot 100 simpler, making the robot 100 of this application more anti-interference capable.
[0184] While the robot 100 performs the inspection task, as shown in FIG4 b , the base station 300 may execute S300 .
[0185] S300: The base station 300 receives a second satellite signal and records a second receiving time of the second satellite signal.
[0186] The base station 300 may receive the second satellite signals through the reference station receiver 301 installed on the base station 300 as shown in FIG3 b , and record the reception time of each second satellite signal.
[0187] The second satellite signal received by the base station 300 may be an observation value of the carrier phase of the satellite (for example, represented by observation value 2).
[0188] S301 , the base station 300 may synchronously upload the second satellite signal and its second receiving time to the cloud management platform 200 .
[0189] After S105 and S301 , the cloud management platform 200 may execute S201 a to S203 .
[0190] S201a, the cloud management platform 200 obtains a second panoramic image related to the actual checkpoint taken by the robot 100.
[0191] For example, the second panoramic image may be a candidate panoramic image taken when the robot 100 is located at the actual inspection point (eg, the shooting time is the first time).
[0192] Alternatively, the second panoramic image may be a candidate panoramic image taken by the robot 100 near the actual checkpoint, or a panoramic image obtained by the cloud management platform 200 fusing multiple candidate panoramic images taken by the robot 100 near the actual checkpoint with reference to the desired area.
[0193] Specifically, the initial inspection data may include each panoramic image (also referred to as a candidate panoramic image) taken by the robot 100 during the inspection task and its shooting time (obtained through S101a), as well as the first time each time the robot 100 navigates to the desired inspection point (obtained through S103), wherein, at the first time, the robot 100 is determined to have moved to the desired inspection point, but there is still a certain position error between the actual position of the robot 100 and the first position of the desired inspection point.
[0194] The cloud management platform 200 can then use the first time obtained from the initial inspection data to filter out at least one candidate panoramic image in the initial inspection data whose shooting time matches the first time (for example, the shooting time is the first time, or within a certain time period before and after the first time); then, the cloud management platform 200 can determine the second panoramic image related to the actual inspection point shot by the robot 100 based on the at least one candidate panoramic image.
[0195] In example A, for example, if the at least one candidate panoramic image mentioned above is a candidate panoramic image with a shooting time of the first time in the initial inspection data, then the cloud management platform 200 can use the candidate panoramic image with a shooting time of the first time as the second panoramic image.
[0196] In Example B, the at least one candidate panoramic image is a plurality of candidate panoramic images captured within a certain time period before and after the first time. In Example B, the plurality of candidate panoramic images are multiple candidate panoramic images captured within a certain time period before and after the first time (e.g., 0.1 seconds before and after) when the robot 100 is patrolling within a preset distance range (e.g., 0.5 meters) of the desired inspection point along the inspection route.
[0197] In Example B, when determining the second panoramic image, the cloud management platform 200 may, for example, refer to a pre-stored desired region (a type of ROI) associated with the desired inspection point to select a candidate panoramic image with the best image quality regarding the inspection object's region of interest from the multiple candidate panoramic images as the second panoramic image for use in performing the ROI extraction step S203 described below. Alternatively, the cloud management platform 200 may refer to a pre-stored desired region (a type of ROI) associated with the desired inspection point to perform image fusion on the multiple candidate panoramic images to obtain a panoramic image with a relatively good image quality regarding the inspection object's region of interest as the second panoramic image for use in performing the ROI extraction step S203 described below.
[0198] In this way, the cloud management platform 200 uses multiple candidate panoramic images related to the actual inspection point to obtain a second panoramic image for ROI extraction, which can improve the completeness of the image of the extracted target area and reduce the acquisition error of the image of the inspection object. The target area is the area of interest (such as pixel position) of the inspection object in the second panoramic image.
[0199] S201b, the cloud management platform 200 obtains a first satellite signal whose first receiving time is a first time.
[0200] The initial inspection data also includes each first satellite signal received during the robot 100 performing the inspection task and its first receiving time (obtained through S101b).
[0201] The cloud management platform 200 can then filter out the first satellite signal with the first receiving time being the first time from the received initial inspection data as the first information for determining the location of the actual inspection point. For example, the first information is the first satellite signal received by the robot 100 at the first time.
[0202] S201c, the cloud management platform 200 obtains a second satellite signal whose second receiving time is the first time.
[0203] The cloud management platform 200 may filter the second satellite signal whose second receiving time is the first time from the data uploaded by the base station 300 in S301 .
[0204] The present application does not limit the execution order between S201a and S201c, and they can be executed serially or in parallel.
[0205] S202, the cloud management platform 200 processes the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the second position of the robot 100 at the first time (also expressed as the position of the actual checkpoint).
[0206] Among them, dynamic post-processing technology (PPK technology) is a GNSS positioning technology that uses post-differentiation of carrier phase. In this case, the positioning of the base station 300 is fixed, while the positioning of the robot 100 is constantly changing. The cloud management platform can use PPK technology to compensate and calibrate the first time measurement 1 of the satellite carrier phase observed by the GNSS receiving antenna 102 of the robot 100 using the first time measurement 2 of the satellite carrier phase observed by the base station receiver 301 as shown in Figure 3b. This method obtains the actual position of the checkpoint at the first time (i.e., the second position) of the robot 100 while the robot 100 is moving. This second position is the three-dimensional coordinate of the robot 100. The pre-configured first position of the desired checkpoint is also a three-dimensional coordinate. In this way, after the robot 100 inspects the desired checkpoint (e.g., point B1), the cloud management platform 200 can obtain the second position of the actual checkpoint at which the robot 100 inspected the desired checkpoint (e.g., point B1).
[0207] The cloud management platform 200 can use PPK to locate the robot 100. It can use the satellite signals from the base station, whose positioning position remains unchanged, to compensate and calibrate the satellite signals received by the robot 100, whose positioning position is constantly changing. This allows the robot 100 to accurately locate its actual position while in motion, resulting in a higher degree of accuracy in its positioning. Compared to the SLAM technology used by the robot 100 in the prior art, the PPK technology is an absolute positioning technology that can be used for long-distance positioning of the robot 100. It avoids the problem of increasing positioning errors due to the increased movement time of the robot 100, and can accurately locate the robot 100 during long-distance inspections.
[0208] In some embodiments, PPK technology is used to position the robot, which can achieve centimeter-level positioning accuracy, thereby improving the positioning precision and accuracy of the robot.
[0209] Optionally, after S202, the cloud management platform 200 may process the initial inspection data into data of a first time that is interrelated, a second position of the robot at the actual inspection point, and a second panoramic image taken by the robot 100 and related to the actual inspection point, so that the cloud management platform 200 can use the interrelated data to determine the image of the target area of the inspection object in the second panoramic image. In this way, the image of the target area of the inspection object corresponding to each expected inspection point can be determined more quickly, thereby improving the efficiency of acquiring the image of the target area of the inspection object involved in this inspection task.
[0210] Among them, after S201b and S201c, the above S202 can be executed.
[0211] S203 , the cloud management platform 200 obtains an image of the target area of the inspection object in the second panoramic image according to the positional relationship between the first position and the second position, and the pre-stored desired area and the first distance.
[0212] The target area is a region of interest (eg, pixel position) of the inspection object in the second panoramic image.
[0213] Among them, after S201a and S202, the above S203 can be executed.
[0214] As described in S200a and S200b above, each desired inspection point may be associated with a desired area related to the corresponding inspection object and a first distance between the desired inspection point and the inspection object.
[0215] In this embodiment, the expected area and the first distance associated with the expected inspection point are data pre-stored in the cloud management platform 200 before the robot 100 performs the inspection task.
[0216] In other embodiments, the expected area and the first distance associated with the expected inspection point may also be uploaded by the robot 100 to the cloud management platform 200 through the initial inspection data in S105 above, or may not be merged into the initial inspection data but uploaded separately to the cloud management platform 200 by the robot 100 during the execution of the inspection task or after the inspection task is completed. There is no limitation here.
[0217] In this embodiment, the cloud management platform 200 pre-stores a first position of a desired checkpoint (e.g., point B1) and determines the actual second position of the robot 100 when navigating to the first position (e.g., actual checkpoint D1). The cloud management platform 200 can then use the first position and the second position to determine the positional relationship (also known as position error, or three-dimensional coordinate transformation relationship) between them. In three-dimensional space, this position error can include rotational error and translational error.
[0218] Then, the cloud management platform can use the transformation relationship of the three-dimensional coordinates between the first position and the second position, as well as the desired area (pixel position) of the inspection object obtained when the robot 100 takes a panoramic image of the inspection object at the first position located at the desired inspection point, and the first distance between the panoramic camera 101 and the inspection object when the robot 100 is located at the desired inspection point, to determine the pixel position (for example, the target area) of the inspection object in the second panoramic image related to the position of the actual inspection point (the second position) taken by the robot 100, and then based on the target area, the image of the target area of the inspection object (for example, the image composed of pixels at the pixel positions expressed by the target area) can be selected in the second panoramic image taken by the robot 100 at the actual inspection point.
[0219] FIG6 is a schematic diagram showing an example of selecting an image of a target area.
[0220] In FIG6 , P represents the inspection object that the robot 100 inspects at the desired inspection point B1; D1 represents the actual position (also called the second position or actual inspection point) of the robot 100 when navigating to the desired inspection point B1; {C} represents the three-dimensional coordinates of the desired inspection point B1; {C} represents the three-dimensional coordinates of the desired inspection point B1; ′} represents the three-dimensional coordinates of the actual inspection point D1; the vector u represents the pixel position of the inspection object P associated with the expected inspection point B1 in the corresponding first panoramic image (also called the expected area, or expected ROI), that is, the position of the pixel point of the inspection object P in the first panoramic image taken when the robot 100 is located at the expected inspection point B1.
[0221] As shown in Figure 6, with the position of the expected checkpoint B1 as the center (also known as the origin) of the spherical model, in the spherical coordinate system of the panoramic camera 101 established, the coordinates P of the inspection object P c are represented by Formula 1, where P c can be measured as:
[0222] where represents a three-dimensional real vector.
[0223] The vector u is represented by Formula 2:
[0224] where represents a two-dimensional real vector, and [u, v] represents the pixel position of the expected ROI in the corresponding first panoramic image.
[0225] In Formula 3, the spherical model projected from the above spherical coordinate system of the panoramic camera 101 of the robot 100 is represented by m, and [H, W] represents the pixel position of the panoramic image captured by the panoramic camera 101. m = [H, W] T , Formula 3;
[0226] In Expression 1, lon and lat respectively represent the longitude and latitude in the spherical coordinate system of the panoramic camera 101: -π < lon < π, Expression 1; -π / 2 < lat < π / 2, Expression 2;
[0227] In the following Formulas 4 to 7, the projection function that maps the three-dimensional space in the spherical coordinate system of the panoramic camera 100 to the two-dimensional image space of the point P c is represented by π(P c ), and K is the intrinsic parameter of the panoramic camera 101:
[0228] where Formula 4 represents the mapping between the two-dimensional image space of the image, the longitude and latitude of the three-dimensional space, and the three-dimensional coordinates.
[0229] where Formula 6 represents the mapping between the three-dimensional coordinates of the three-dimensional space and the longitude and latitude.[[ID=D44]]
[0230] In Formula 8, represents the reciprocal of the first distance (such as the first distance d measured in Figure 5) between the point P c and the center of the spherical model m (i.e., the above {C}):
[0231] Given T ijis the transformation matrix between the three-dimensional coordinates of the desired checkpoint B1 and the actual checkpoint D1, as shown in Formula 9. ij Divided into rotation transformation matrix R ij and the translation transformation matrix t ij .
[0232] Combined with the above formulas 1 to 9, formula 10 can be used to obtain the vector u' (an example of the target area) formed by the pixel point position of the inspection object in the second panoramic image (for example, the second panoramic image taken at the actual inspection point D1) taken by the robot 100 and related to the actual inspection point D1. Thus, in the second panoramic image taken by the robot 100 at the actual inspection point D1, the vector u' is used to select the pixel located at the pixel point position, and the image of the target area of the inspection object P in the second panoramic image is extracted.
[0233] The image u in formula 10 is the pixel position of the inspection object P in the first panoramic image pre-taken at the desired inspection point B1, also known as the desired area. It is the reciprocal of the first distance between the panoramic camera 101 and the inspection object P when the panoramic camera 101 is located at the desired inspection point B1, which is obtained in advance by the robot 100 through the distance sensor.
[0234] In an embodiment of the present application, the cloud management platform 200 can refer to the desired area of the inspection object photographed at the first position of the desired inspection point and the first distance of the photographing based on the posture change of the robot 100 (for example, the position change between the first position and the second position), and select an image of the target area of the inspection object in the second panoramic image photographed by the robot 100 at the second position of the actual inspection point. In this way, even if the robot's autonomous navigation is not accurate, so that the robot 100 does not actually reach the desired inspection point when navigating to the desired inspection point, the cloud management platform 200 of the present application can combine the position error between the desired inspection point and the actual inspection point where the robot 100 is located to obtain an image of the target area of the inspection object with complete information in the second panoramic image photographed at the actual inspection point, thereby making the inspection more comprehensive and reducing the error of the image collected of the inspection object.
[0235] In other embodiments, each time the robot 100 navigates to a desired checkpoint, for example, after executing S103 and before S104 shown in FIG4b , the robot 100 may upload inspection data related to the desired checkpoint to the cloud management platform 200. The inspection data uploaded in a single time may include, but is not limited to: a second panoramic image captured and related to the actual checkpoint (also referred to as related to the first time), a first satellite signal received at the first time, and the first time.
[0236] In this way, the robot 100 can upload the inspection data related to the expected inspection point to the cloud management platform 200 every time it reaches a desired inspection point (the robot 100 believes that it has reached the expected inspection point, but there is actually a certain position error). In this way, the cloud management platform 200 can execute S202 and S203 shown in Figure 4b above every time it receives inspection data, so as to obtain a panoramic image (represented by an image of the target area) obtained by inspecting the corresponding inspection object at the expected inspection point, thereby obtaining panoramic images of the inspection of the respective inspection objects at the three expected inspection points B1, B2, and B3 in turn.
[0237] An embodiment of the present application provides an image processing device 800. FIG7 is a schematic diagram of the structure of the image processing device 800. Referring to FIG7, the image processing device 800 includes: a configuration module 801 for configuring the positions of a plurality of desired inspection points for a robot, wherein the positions of the desired inspection points are desired positions for recording images of an inspection object when the robot moves along an inspection route, wherein the robot is equipped with a camera for capturing panoramic images; a desired region acquisition module 802 for acquiring a desired region associated with the desired inspection point, wherein the desired region is a region of interest of the inspection object in a first panoramic image captured by the robot at the desired inspection point; an inspection data acquisition module 803 for acquiring inspection data of the robot, wherein the inspection data includes first information for determining the position of an actual inspection point and a second panoramic image captured by the robot and associated with the actual inspection point, wherein the robot first moves to the actual inspection point based on the position of the desired inspection point; and an image of interest acquisition module 804 for acquiring an image of a target region of the inspection object in the second panoramic image based on the desired region and the first information, wherein the target region is a region of interest of the inspection object in the second panoramic image.
[0238] In a possible embodiment, the device 800 also includes: a distance acquisition module 805, used to acquire a first distance related to the expected checkpoint, wherein the first distance is the distance between the camera and the inspection object when the robot is located at the expected checkpoint; the image of interest acquisition module 804, specifically used to: determine the target area regarding the inspection object in the second panoramic image based on the positional relationship between the position of the expected checkpoint and the position of the actual checkpoint, the expected area and the first distance, wherein the position of the actual checkpoint is determined based on the first information; based on the target area, obtain an image of the target area of the inspection object in the second panoramic image.
[0239] In one possible embodiment, the robot is equipped with a receiver of a global navigation satellite system, and the receiver is used to receive satellite signals. The first information includes a first satellite signal received by the robot at the first time. The device 800 also includes: a signal acquisition module 806, used to obtain a second satellite signal received by the base station at the first time; the image of interest acquisition module 804 is specifically used to process the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
[0240] In one possible implementation, the signal acquisition module 806 is specifically used to: receive multiple reference satellite signals and the second reception time of each reference satellite signal from a base station; and obtain a reference satellite signal whose second reception time is the first time among the multiple reference satellite signals as the second satellite signal.
[0241] In a possible embodiment, the inspection data acquisition module 803 is specifically used to: obtain the first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, the multiple candidate panoramic images taken during the robot's inspection process and the shooting time of each candidate panoramic image, the multiple candidate satellite signals received during the robot's inspection process and the first receiving time of each candidate satellite signal; obtain at least one candidate panoramic image among the multiple candidate panoramic images whose shooting time matches the first time, and determine the second panoramic image related to the actual checkpoint taken by the robot based on the at least one candidate panoramic image; obtain the candidate satellite signal among the multiple candidate satellite signals whose first receiving time is the first time as the first information for determining the position of the actual checkpoint.
[0242] In a possible implementation, the image processing device 800 is deployed on a cloud management platform, which is used to manage the infrastructure for providing cloud services. The infrastructure includes multiple cloud data centers located in different regions, with at least one cloud data center located in each region.
[0243] The effects of the image processing apparatus 800 in the above-mentioned embodiments are similar to the effects of the image processing methods in the above-mentioned embodiments, and are not described in detail here.
[0244] In one possible implementation, the present application provides an inspection system 700. FIG8 is a schematic structural diagram of the inspection system 700. Referring to FIG8, the system 700 includes the image processing device 800 and the robot 100 described in any one of the above implementations.
[0245] In a possible implementation, the inspection system 700 further includes a base station 300 .
[0246] In one possible embodiment, the robot 100 is used to capture the multiple candidate panoramic images through the camera when patrolling within a preset distance range of the location of the expected checkpoint according to the inspection route, and record the shooting time of each candidate panoramic image, as well as the first time when the robot moves to the actual checkpoint based on the location of the expected checkpoint.
[0247] In a possible implementation, the robot 100 is further configured to send inspection data to the image processing device 800 after completing the inspection along the inspection route.
[0248] In a possible implementation, the camera of the robot 100 includes at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
[0249] In a possible embodiment, the robot 100 is further used to: obtain at least two first images by capturing at least two first images through the at least two cameras of the camera during the inspection process according to the inspection route; and synthesize the at least two first images into a panoramic image.
[0250] Among them, the above modules can all be implemented by software, or can be implemented by hardware. Among them, the module is an example of a software functional unit, and the above module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the image of interest acquisition module 804 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Among them, usually a region can include multiple AZs.
[0251] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.
[0252] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, the module may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0253] The multiple computing devices included in the image processing apparatus 800 can be distributed in the same region or in different regions. The multiple computing devices included in the image processing apparatus 800 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the image processing apparatus 800 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.
[0254] It should be noted that, in other embodiments, the above modules may be used to execute corresponding steps in the image processing method to realize all functions of the image processing device 800 .
[0255] This application also provides a computing device 900. As shown in Figure 9, computing device 900 includes a bus 902, a processor 904, a memory 906, and a communication interface 909. Processor 904, memory 906, and communication interface 909 communicate with each other via bus 902. Computing device 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 900.
[0256] Bus 902 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG9 shows only one s-line, but this does not imply a single bus or type of bus. Bus 902 may include a path for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, and communication interface 909).
[0257] The processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0258] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0259] Memory 906 stores executable program code. Processor 904 executes this executable program code to implement the functions of the aforementioned configuration module 801, desired area acquisition module 802, inspection data acquisition module 803, image of interest acquisition module 804, distance acquisition module 805, and signal acquisition module 806, thereby implementing the image processing method. In other words, memory 906 stores instructions for executing the image processing method.
[0260] The communication interface 909 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.
[0261] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0262] As shown in Figure 10, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the image processing method.
[0263] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store some instructions for executing the image processing method. In other words, the combination of one or more computing devices 1000 can jointly execute the instructions for executing the image processing method.
[0264] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions, each used to execute part of the functions of the image processing apparatus 800. That is, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more of the configuration module 801, the desired area acquisition module 802, the inspection data acquisition module 803, the image of interest acquisition module 804, the distance acquisition module 805, and the signal acquisition module 806.
[0265] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG11 shows a possible implementation. As shown in FIG11 , two computing devices 1100A and 1100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1106 in the computing device 1100A stores instructions for executing the functions of the configuration module 801 and the desired area acquisition module 802. At the same time, the memory 1106 in the computing device 1100B stores instructions for executing the functions of the inspection data acquisition module 803, the image of interest acquisition module 804, the distance acquisition module 805, and the signal acquisition module 806.
[0266] It should be understood that the functionality of the computing device 1100A shown in FIG11 may also be implemented by multiple computing devices 1100. Similarly, the functionality of the computing device 1100B may also be implemented by multiple computing devices 1100.
[0267] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection methods of the computing device clusters described in Figures 9 and 11. However, the memory 1106 in one or more computing devices 1100 in this computing device cluster can store the same instructions for executing the image processing method.
[0268] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store some instructions for executing the image processing method. In other words, the combination of one or more computing devices 1100 can jointly execute the instructions for executing the image processing method.
[0269] It should be noted that the memory 1106 in different computing devices 1100 in the computing device cluster may store different instructions for executing partial functions of the image processing apparatus 800. In other words, the instructions stored in the memory 1106 in different computing devices 1100 may implement the functions of one or more devices in the image processing apparatus 800.
[0270] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the image processing method described in the above embodiments.
[0271] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the image processing method in the above embodiment.
[0272] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.
Claims
1. An image processing method, It is characterized in that The method comprises: Configuring the positions of a plurality of desired inspection points for the robot, wherein the positions of the desired inspection points are desired positions for recording images of inspection objects when the robot moves, and the robot is equipped with a camera for capturing panoramic images; Acquire an expected area related to the expected inspection point, where the expected area is an area of interest about the inspection object in a first panoramic image taken by the robot at the expected inspection point; Acquire inspection data of the robot, the inspection data comprising first information for determining a position of an actual inspection point and a second panoramic image taken by the robot and related to the actual inspection point, wherein the robot moves to the actual inspection point based on the position of the expected inspection point; Based on the expected area and the first information, an image of a target area of the inspection object in the second panoramic image is acquired, where the target area is a region of interest of the inspection object in the second panoramic image.
2. The method according to claim 1, It is characterized in that The position of the expected inspection point and the position of the actual inspection point are three-dimensional positions. Before obtaining the inspection data of the robot, the method further includes: Acquire a first distance associated with the desired inspection point, wherein the first distance is a distance between the camera and the inspection object when the robot is located at the desired inspection point; The acquiring, based on the expected area and the first information, an image of the target area of the inspection object in the second panoramic image comprises: determining a target area with respect to the inspection object in the second panoramic image based on a positional relationship between the position of the expected inspection point and the position of the actual inspection point, the expected area, and the first distance, wherein the position of the actual inspection point is determined based on the first information; Based on the target area, an image of the target area of the inspection object is obtained in the second panoramic image.
3. The method according to claim 1 or 2, It is characterized in that The robot moves to the actual checkpoint based on the position of the expected checkpoint at a first time, the robot is equipped with a receiver of a global navigation satellite system, the receiver is used to receive satellite signals, the first information includes a first satellite signal received by the robot at the first time, and the method further includes: Acquire a second satellite signal received by the base station at the first time; The first satellite signal and the second satellite signal are processed by a dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
4. The method according to claim 3, It is characterized in that The acquiring the second satellite signal received by the base station at the first time includes: receiving a plurality of reference satellite signals and a second receiving time of each of the reference satellite signals from a base station; A reference satellite signal whose second receiving time is the first time among the multiple reference satellite signals is obtained as the second satellite signal.
5. The method according to claim 3 or 4, It is characterized in that The obtaining of the inspection data of the robot comprises: Acquire a first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, a plurality of candidate panoramic images captured by the robot during the inspection process and the shooting time of each candidate panoramic image, a plurality of candidate satellite signals received by the robot during the inspection process and the first receiving time of each candidate satellite signal; Acquire at least one candidate panoramic image of the plurality of candidate panoramic images whose shooting time matches the first time, and determine a second panoramic image taken by the robot and related to the actual checkpoint based on the at least one candidate panoramic image; A candidate satellite signal whose first receiving time is the first time among the multiple candidate satellite signals is obtained as first information for determining the position of the actual checkpoint.
6. The method according to claim 5, It is characterized in that Before obtaining the inspection data of the robot, the method further includes: When the robot inspects within a preset distance range of the position of the desired inspection point, the robot captures the plurality of candidate panoramic images through the camera, and records the shooting time of each candidate panoramic image, as well as the time when the robot moves to the desired inspection point based on the position of the desired inspection point. The actual checkpoint for the first time.
7. The method according to any one of claims 1 to 6, It is characterized in that The obtaining of the inspection data of the robot comprises: After the robot completes the inspection, inspection data is received from the robot.
8. The method according to any one of claims 1 to 7, It is characterized in that The camera has at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
9. The method according to claim 8, It is characterized in that The method further comprises: During the inspection process, the robot obtains at least two first images by photographing the at least two cameras of the camera; The robot combines the at least two first images into a panoramic image.
10. An image processing device, It is characterized in that The device comprises: a configuration module, configured to configure the positions of a plurality of desired inspection points for the robot, wherein the positions of the desired inspection points are desired positions for recording images of inspection objects when the robot moves, and the robot is equipped with a camera for capturing panoramic images; An expected area acquisition module, used to acquire an expected area related to the expected inspection point, wherein the expected area is an area of interest about the inspection object in the first panoramic image taken by the robot at the expected inspection point; an inspection data acquisition module, configured to acquire inspection data of the robot, the inspection data comprising first information for determining a position of an actual inspection point and a second panoramic image taken by the robot and related to the actual inspection point, wherein the robot moves to the actual inspection point based on the position of the desired inspection point; An interested image acquisition module is used to acquire an image of a target area of the inspection object in the second panoramic image based on the expected area and the first information, wherein the target area is an interested area of the inspection object in the second panoramic image.
11. The device according to claim 10, It is characterized in that The device also includes: a distance acquisition module, configured to acquire a first distance associated with the desired inspection point, wherein the first distance is a distance between the camera and the inspection object when the robot is located at the desired inspection point; The image of interest acquisition module is specifically used for: determining a target area with respect to the inspection object in the second panoramic image based on a positional relationship between the position of the expected inspection point and the position of the actual inspection point, the expected area, and the first distance, wherein the position of the actual inspection point is determined based on the first information; Based on the target area, an image of the target area of the inspection object is obtained in the second panoramic image.
12. The device according to claim 10 or 11, It is characterized in that The robot moves to the actual checkpoint based on the position of the expected checkpoint at a first time, the robot is equipped with a receiver of a global navigation satellite system, the receiver is used to receive satellite signals, the first information includes a first satellite signal received by the robot at the first time, and the device further includes: a signal acquisition module, configured to acquire a second satellite signal received by the base station at the first time; The image of interest acquisition module is specifically used to process the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
13. The device according to claim 12, It is characterized in that The signal acquisition module is specifically used for: receiving a plurality of reference satellite signals and a second receiving time of each of the reference satellite signals from a base station; A reference satellite signal whose second receiving time is the first time among the multiple reference satellite signals is obtained as the second satellite signal.
14. The device according to claim 12 or 13, It is characterized in that The inspection data acquisition module is specifically used for: Acquire a first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, a plurality of candidate panoramic images captured by the robot during the inspection process and the shooting time of each candidate panoramic image, a plurality of candidate satellite signals received by the robot during the inspection process and the first receiving time of each candidate satellite signal; Acquire at least one candidate panoramic image whose shooting time matches the first time among the plurality of candidate panoramic images, based on the at least one candidate panoramic image determines a second panoramic image taken by the robot and associated with the actual checkpoint; A candidate satellite signal whose first receiving time is the first time among the multiple candidate satellite signals is obtained as first information for determining the position of the actual checkpoint.
15. A patrol inspection system, It is characterized in that The system comprises the image processing device according to any one of claims 10 to 14 and the robot.
16. The system according to claim 15, It is characterized in that The robot is used to capture the multiple candidate panoramic images through the camera when patrolling within a preset distance range of the location of the expected checkpoint, and record the shooting time of each candidate panoramic image, as well as the first time when the robot moves to the actual checkpoint based on the location of the expected checkpoint.
17. The system according to claim 15 or 16, It is characterized in that The robot is also used to send inspection data to the image processing device after the inspection is completed.
18. A system according to any one of claims 15 to 17, It is characterized in that The camera of the robot comprises at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
19. The system according to claim 17, It is characterized in that The robot is also used to: During the inspection process, at least two first images are obtained by photographing by the at least two cameras of the camera; The at least two first images are combined into a panoramic image.
20. A device cluster, It is characterized in that comprising at least one processing device, each processing device comprising a processor and a memory; The processor of the at least one processing device is configured to execute instructions stored in the memory of the at least one processing device, so that the device cluster executes the method according to any one of claims 1 to 9.
21. A computer program product comprising instructions, It is characterized in that When the instructions are executed by a device cluster, the device cluster executes the method according to any one of claims 1 to 9.
22. A computer-readable storage medium, It is characterized in that The method comprises computer program instructions. When the computer program instructions are executed by a device cluster, the device cluster performs the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Robot inspection method and device, computer readable storage medium and robot
CN110850872A
Inspection control method of movable platform and movable platform
CN111316185A
Unmanned aerial vehicle inspection method, device and equipment and storage medium
CN111967299A
Image processing method and device, and storage medium
CN114026595A
Detection area recovery method and device, equipment and storage medium
CN115393433A
