Image processing method and image processing device
By configuring expected checkpoints in mobile robots and using cloud management platforms to process images, combining satellite signals and dynamic post-processing technology, the incomplete image caused by inaccurate positioning of robots is solved, and complete and accurate shooting at actual checkpoints is achieved Images improve the in-placement and accuracy of inspections.
Patent Information
- Application Number
- CN202311044779.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2023-08-18
- Publication Date
- 2024-11-26
AI Technical Summary
When existing mobile robots inspect outdoor environments, the images captured are incomplete and inaccurate due to low positioning accuracy and environmental interference, which affects the maintenance effect of the inspection objects.
Provide an image processing method and device to configure the desired checkpoint, use a cloud management platform or terminal device to obtain panoramic images related to the expected checkpoint, extract the area of interest, combine satellite signals and dynamic post-processing technology to correct the position error of the robot , ensure complete and accurate images are taken at actual checkpoints.
Even if the robot itself is inaccurate, it can still take pictures of the complete and accurate images of the inspection object from the actual checkpoint, improve the placement and accuracy of the inspection, and solve the problem of incomplete image information in the prior art.
Smart Images

Figure CN119031095A_ABST
Abstract
Description
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 23, 2023, with application number 202310586842.9 and application name “A method, device and system for image correction”, all contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of data processing technology, and in particular to an image processing method and an image processing device. Background Art
[0003] In order to maintain equipment in outdoor environments, it is necessary to inspect the equipment, such as inspecting and maintaining power plant equipment, power grids, and electric meters in an electric power environment.
[0004] At present, mobile robots are mainly used to inspect inspection objects in outdoor environments. However, due to the interference of factors such as the positioning accuracy of the mobile robots and the outdoor environment, the images taken by the current mobile robots of the inspection objects are not complete and accurate, and there is a problem of inadequate inspection of the inspection objects, which is not conducive to the maintenance of the inspection objects. Summary of the invention
[0005] The present application provides an image processing method and an image processing device, which can refer to the expected area of the inspection object related to the expected inspection point and the position of the actual inspection point to which the robot moves according to the expected inspection point, to extract the image of the area of interest from the second panoramic image related to the actual inspection point taken by the robot. Even if the robot itself is positioned inaccurately, it can refer to the accurate expected area to extract a complete and accurate image of the inspection object from the second panoramic image taken at the inaccurate positioning position, so as to conduct a more thorough inspection of the inspection object.
[0006] In a possible implementation, an embodiment of the present application provides an image processing method. The method includes: configuring the positions of multiple expected inspection points for the robot, the positions of the expected inspection points are expected positions for recording images of inspection objects when the robot moves along the inspection route, the robot is equipped with a camera, and the camera is used to take panoramic images; obtaining an expected area related to the expected inspection point, the expected area is an area of interest about the inspection object in the first panoramic image taken by the robot at the expected inspection point; obtaining the inspection data of the robot, the inspection data includes first information for determining the position of the actual inspection point and a second panoramic image related to the actual inspection point taken by the robot, wherein the robot moves to the actual inspection point based on the position of the expected inspection point at the first time; based on the expected area and the first information, obtaining an image of a target area of the inspection object in the second panoramic image, the target area being an area of interest about the inspection object in the second panoramic image.
[0007] The method can be implemented by a management platform on the cloud (e.g., a cloud management platform), optionally in combination with a terminal device. For example, the cloud management platform can be deployed on a cloud server (e.g., a computing node). Alternatively, the method can be implemented by a management platform under the cloud (e.g., a server), optionally in combination with a terminal device. The subsequent implementation methods are described by taking the cloud management platform as the execution subject as an example.
[0008] Among them, the position of the expected checkpoint is the position to which the robot is expected to move, but because the robot's own positioning is not accurate enough, when the robot moves according to the position of the expected checkpoint, it moves to the position of the actual checkpoint that has a certain position deviation from the position of the expected checkpoint.
[0009] In addition, the robot takes a picture of the corresponding inspection object at the position of the desired inspection point, and the obtained image is the most complete and accurate. Therefore, it is expected that the robot can move to the desired position to take a picture of the inspection object.
[0010] The operation of configuring the location of the desired checkpoint for the robot can be performed by the cloud management platform or by the terminal device, and there is no restriction here.
[0011] The robot can be pre-positioned at the desired inspection point to capture a panoramic image and obtain a first panoramic image. In addition, the present application can pre-store the pixel locations of the inspection object in the first panoramic image (also expressed as the above-mentioned expected area) in the cloud management platform; or pre-store the expected area on the robot and other terminal devices, and the cloud management platform can obtain the expected area related to the desired inspection point from the robot and other terminal devices.
[0012] When the robot inspects the inspection object near each expected inspection point according to the inspection route, the time when the robot moves from the position of the expected inspection point to the actual inspection point is represented as the first time.
[0013] The second panoramic image related to the actual checkpoint taken by the robot and obtained by the cloud management platform may be a panoramic image taken by the robot at the first time, that is, when it moves to the actual checkpoint. Alternatively, due to the inaccurate positioning of the robot itself, there is a positional deviation between the position of the actual checkpoint and the position of the expected checkpoint. Therefore, the panoramic image taken by the robot at the actual checkpoint may not be as clear, complete, and accurate as the image area of the inspection object in the panoramic image taken by the robot before and / or after the first time when the robot approaches the expected checkpoint. Then, the second panoramic image related to the actual checkpoint taken by the robot and obtained by the cloud management platform may also be a panoramic image taken by the robot when it moves to the vicinity of the actual checkpoint based on the expected checkpoint, or a fusion result of multiple panoramic images.
[0014] The cloud management platform can refer to the pixel position of the inspection object (i.e., the expected area) in the first panoramic image taken by the robot at the expected inspection point, and the first information for determining the position of the actual inspection point where the robot is located, to obtain an image of the target area (e.g., pixel position) of the inspection object from the second panoramic image taken by the robot related to the actual inspection point. In this way, although the robot's own positioning is inaccurate, the cloud management platform can refer to the pixel position of the inspection object obtained by the robot at the accurate expected position to extract the image of the target area of the inspection object from the second panoramic image related to the actual inspection point (e.g., the panoramic image taken at the actual inspection point position), so that the image of the target area obtained is more complete, accurate, and precise, and the inspection of the inspection object is more in place, which is conducive to the maintenance of the inspection object.
[0015] In this way, even if the robot cannot accurately reach the desired inspection point due to the low accuracy of autonomous navigation, the cloud management platform of the present application can obtain a complete image of the inspection object (such as the image of ROI) in the panoramic image taken at the actual inspection point when there is a certain error between the actual inspection point to which the robot navigates and the desired inspection point, and the inspection of the inspection object corresponding to each desired inspection point is more thorough. This solves the problem in the prior art that the image information of the inspection object (such as the inspected power equipment) is incomplete due to the robot's inability to accurately reach the desired inspection point.
[0016] In addition, the robot performs intelligent inspections by using a camera that can take panoramic images, and the cloud management platform selects images of the target area (a type of ROI) belonging to the inspection object from the panoramic images taken by the robot inspection. Since the panoramic image information is more comprehensive, the image of the target area obtained can contain more detailed information about the inspection object, which can improve the inspection accuracy of the inspection object.
[0017] In a possible embodiment, the position of the expected checkpoint and the position of the actual checkpoint are three-dimensional positions. Before obtaining the inspection data of the robot, the method also includes: obtaining a first distance related to the expected checkpoint, wherein the first distance is the distance between the camera and the inspection object when the robot is located at the expected checkpoint; obtaining an image of a target area of the inspection object in the second panoramic image based on the expected area and the first information, including: determining the target area of the inspection object in the second panoramic image based on the positional relationship between the position of the expected checkpoint and the actual checkpoint, the expected area and the first distance, wherein the position of the actual checkpoint is determined based on the first information; and obtaining an image of the target area of the inspection object in the second panoramic image based on the target area.
[0018] In the embodiment of the present application, the position error between the expected inspection point and the actual inspection point, the ROI in the first panoramic image taken at the expected inspection point, and the corresponding first distance can be used to determine the pixel position (e.g., target area) of the inspection object in the second panoramic image (e.g., a panoramic image taken at or near the actual inspection point) taken by the robot. Thus, the image of the target area is extracted from the second panoramic image to obtain a panoramic image of the corresponding inspection object.
[0019] In one possible implementation, the robot is equipped with a receiver of a global navigation satellite system, and the receiver is used to receive satellite signals. The first information includes a first satellite signal received by the robot at the first time. The method also includes: acquiring a second satellite signal received by the base station at the first time; and processing the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
[0020] In this embodiment, the cloud management platform can use the post-processing kinematic (PPK) technology to accurately locate the actual position of the robot 100 when navigating to the desired checkpoint (e.g., the position of the actual checkpoint), wherein the cloud management platform compensates and calibrates the satellite signals received by the robot whose positioning position is constantly changing by referring to the satellite signals of the base station whose positioning position remains unchanged, so as to accurately locate the actual position of the moving robot, so as to improve the accuracy of positioning the robot. In this way, the cloud management platform can extract the image of the ROI of the inspection object from the second panoramic image related to the actual checkpoint taken by the robot based on the position error between the desired checkpoint and the actual checkpoint.
[0021] In a possible implementation, acquiring a second satellite signal received by the base station at the first time includes: receiving multiple reference satellite signals and the second receiving time of each of the reference satellite signals from the base station; acquiring a reference satellite signal whose second receiving time is the first time among the multiple reference satellite signals as the second satellite signal.
[0022] In this embodiment, the cloud management platform can select the reference satellite signal with the first reception time from the multiple reference satellite signals received from the base station, so as to calibrate and supplement the satellite signal received by the robot at the first time, so as to improve the positioning accuracy of the actual checkpoint where the robot is located at the first time, and to extract the image of the ROI by accurately identifying the position of the robot, using the position error between the actual checkpoint and the expected checkpoint. In this way, the cloud management platform can reduce the link connection frequency between the robot and the base station by directly interacting with the base station to obtain the reference satellite signal for positioning the robot.
[0023] In other embodiments, the base station may also send multiple reference satellite signals and their second reception times to the robot, and the robot may upload the multiple reference satellite signals and their second reception times, or the second satellite signal whose screened second reception time is the first time, to the cloud management platform.
[0024] In a possible embodiment, obtaining the inspection data of the robot includes: obtaining the first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, multiple candidate panoramic images taken during the robot's inspection process and the shooting time of each candidate panoramic image, multiple candidate satellite signals received during the robot's inspection process and the first receiving time of each candidate satellite signal; obtaining at least one candidate panoramic image among the multiple candidate panoramic images whose shooting time matches the first time, and determining a second panoramic image related to the actual checkpoint taken by the robot based on the at least one candidate panoramic image; obtaining a candidate satellite signal among the multiple candidate satellite signals whose first receiving time is the first time as the first information for determining the position of the actual checkpoint.
[0025] The cloud management platform can use the first time obtained from the inspection data to screen out at least one candidate panoramic image from multiple candidate panoramic images in the inspection data, the shooting time of which matches the first time (for example, the shooting time is the first time, or the shooting time is within a certain period of time before and / or after the first time); then, the cloud management platform can determine the second panoramic image taken by the robot and related to the actual inspection point based on the at least one candidate panoramic image.
[0026] In an example A, for example, if the at least one candidate panoramic image is a candidate panoramic image shot at a first time in the inspection data, the cloud management platform may use the candidate panoramic image shot at the first time as the second panoramic image.
[0027] In an example B, the at least one candidate panoramic image is a plurality of candidate panoramic images captured within a certain time period before and after the first time. For example, in this example B, the plurality of candidate panoramic images are a plurality of candidate panoramic images captured within a certain time period (for example, 0.1 seconds before and after) of the first time when the robot inspects within a preset distance range (for example, 0.5 meters) of the position of the expected inspection point according to the inspection route.
[0028] In Example B, when determining the second panoramic image, the cloud management platform may, for example, refer to a pre-stored expected area (an ROI) associated with the expected inspection point to select a candidate panoramic image with the best image quality regarding the area of interest of the inspection object from among the multiple candidate panoramic images whose shooting time is within a certain period of time before and after the first time, as the second panoramic image for extracting the ROI.
[0029] Alternatively, in Example B, when determining the second panoramic image, the cloud management platform 200 may refer to a pre-stored expected area (an ROI) associated with the expected inspection point, and perform image fusion on the multiple candidate panoramic images shot within a certain period of time before and after the first time, so as to obtain a panoramic image with a better image of the area of interest of the inspection object as the second panoramic image, for extraction of the ROI.
[0030] In this way, the cloud management platform can use at least one candidate panoramic image related to the actual inspection point to obtain a second panoramic image. The image area about the ROI in the second panoramic image is more complete, comprehensive, and of higher quality, which is conducive to the extraction of the ROI, can improve the completeness of the image of the extracted target area, and reduce the acquisition error of the image of the inspection object.
[0031] In a possible embodiment, before obtaining the inspection data of the robot, the method also includes: when the robot inspects within a preset distance range of the location of the expected inspection point according to the inspection route, shooting the multiple candidate panoramic images by the camera, and recording the shooting time of each of the candidate panoramic images, and recording the first time when the robot moves to the actual inspection point based on the location of the expected inspection point.
[0032] When the robot can inspect within a preset distance range of the location of the desired inspection point according to the inspection route, for example, when the robot can determine through the laser radar that the current location of the robot is within a certain distance range of the desired inspection point (for example, within 0.5 meters), the robot can control the camera to take one or more candidate panoramic images and record the shooting time of each panoramic image. In this way, the robot can take multiple panoramic images within a preset distance range (for example, within 0.5 meters, which can be flexibly configured according to needs) close to the desired inspection point.
[0033] Since the robot's autonomous navigation technology itself has the problem of low accuracy, the robot's current positioning determined by the laser radar is not accurate enough. Therefore, by starting to shoot panoramic images when the robot navigates to the position near the desired inspection point, a sufficient number of panoramic images shot when the robot's positioning position is close to the desired inspection point can be obtained. This is conducive to extracting images of the target area with higher image quality about the corresponding inspection object (for example, the inspected power equipment) with a sufficient number of panoramic images, thereby improving the completeness and accuracy of the image of the inspected power equipment.
[0034] In a possible implementation, acquiring the inspection data of the robot includes: receiving the inspection data from the robot after the robot completes inspection along the inspection route.
[0035] In this embodiment, the cloud management platform can obtain the robot's inspection data after the robot completes the inspection task. In this way, the robot uploading the inspection data does not affect the execution of the inspection task, does not interrupt the execution of the inspection task, and can ensure the continuity of the inspection task execution. It will not occupy too much network bandwidth due to the inspection data upload during the inspection task, affecting the quality of the network for executing the inspection task. In addition, the cloud management platform can uniformly process the inspection data related to multiple expected inspection points involved in this inspection task (for example, parallel processing), so as to improve the inspection efficiency.
[0036] In a possible implementation, the camera has at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
[0037] In this embodiment, by installing at least two cameras whose total shooting angle of view is 360 degrees on the robot, the robot can capture a panoramic image without installing a pan / tilt platform.
[0038] Compared with the method of installing only a single or multiple cameras that do not cover a 360-degree viewing angle on the robot, and changing the camera's shooting angle of view by rotating the gimbal to obtain a panoramic image, the camera of the robot of the present application is more efficient in shooting panoramic images; and the mechanical stability of the gimbal rotation is poor. The robot of the present application does not need a gimbal, but uses multiple cameras with different shooting angles to shoot panoramic images, which can improve the shooting stability of panoramic images; in addition, controlling the camera to shoot panoramic images by rotating the gimbal requires multiple shots at different times to obtain a panoramic image, while at least two cameras of the robot of the present application shoot simultaneously, thereby reducing the shooting delay of the panoramic image and helping to improve inspection efficiency.
[0039] The camera of the robot of the present application is a camera capable of capturing omnidirectional images, which can provide a wider shooting angle than the ordinary camera used by the robot in the prior art. The intelligent inspection robots in the prior art generally use ordinary cameras, so their shooting angle range is limited, and the inspection object may not be fully imaged. The camera of the present application has a panoramic camera capable of capturing panoramic images, which can increase the inspection field of view and improve the inspection efficiency.
[0040] In a possible implementation, the robot obtains at least two first images by capturing them through the at least two cameras of the camera during the inspection process along the inspection route; and the robot combines the at least two first images into a panoramic image.
[0041] In an embodiment of the present application, the robot can capture a panoramic image by synthesizing at least two first images captured by at least two cameras respectively to obtain a panoramic image.
[0042] In one possible implementation, an embodiment of the present application provides an image processing device, the device comprising: a configuration module, used to configure the positions of multiple expected checkpoints for a robot, the positions of the expected checkpoints being expected positions for recording images of inspection objects when the robot moves along an inspection route, the robot being equipped with a camera, the camera being used to capture panoramic images; an expected area acquisition module, used to acquire an expected area associated with the expected checkpoint, the expected area being an area of interest regarding the inspection object in a first panoramic image captured by the robot at the expected checkpoint; an inspection data acquisition module, used to acquire inspection data of the robot, the inspection data comprising first information for determining the position of an actual checkpoint and a second panoramic image associated with the actual checkpoint captured by the robot, wherein the robot moves to the actual checkpoint based on the position of the expected checkpoint at a first time; an interested image acquisition module, used to acquire an image of a target area regarding the inspection object in the second panoramic image based on the expected area and the first information, the target area being an area of interest regarding the inspection object in the second panoramic image.
[0043] In a possible embodiment, the device also includes: a distance acquisition module, used to acquire a first distance related to the expected checkpoint, wherein the first distance is the distance between the camera and the inspection object when the robot is located at the expected checkpoint; the image of interest acquisition module, specifically used to: determine a target area regarding the inspection object in the second panoramic image based on a positional relationship between the position of the expected checkpoint and the position of the actual checkpoint, the expected area and the first distance, wherein the position of the actual checkpoint is determined based on the first information; based on the target area, obtain an image of the target area of the inspection object in the second panoramic image.
[0044] In a possible implementation, the robot is equipped with a receiver of a global navigation satellite system, and the receiver is used to receive satellite signals. The first information includes a first satellite signal received by the robot at the first time. The device also includes: a signal acquisition module, used to acquire a second satellite signal received by the base station at the first time; and an image of interest acquisition module, specifically used to process the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
[0045] In a possible implementation, the signal acquisition module is specifically used to: receive multiple reference satellite signals and the second receiving time of each of the reference satellite signals from a base station; and acquire a reference satellite signal whose second receiving time is the first time among the multiple reference satellite signals as the second satellite signal.
[0046] In a possible embodiment, the inspection data acquisition module is specifically used to: obtain the first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, multiple candidate panoramic images taken during the robot's inspection process and the shooting time of each candidate panoramic image, multiple candidate satellite signals received during the robot's inspection process and the first receiving time of each candidate satellite signal; obtain at least one candidate panoramic image among the multiple candidate panoramic images whose shooting time matches the first time, and determine a second panoramic image related to the actual checkpoint taken by the robot based on the at least one candidate panoramic image; obtain a candidate satellite signal among the multiple candidate satellite signals whose first receiving time is the first time as the first information for determining the position of the actual checkpoint.
[0047] The effects of the image processing apparatus of the above-mentioned embodiments are similar to the effects of the image processing methods of the above-mentioned embodiments, and will not be described in detail here.
[0048] In a possible implementation, the present application provides an inspection system, which includes the image processing device and the robot described in any one of the above implementations.
[0049] In one possible embodiment, the robot is used to capture the multiple candidate panoramic images through the camera when patrolling within a preset distance range of the location of the expected checkpoint along the inspection route, and record the shooting time of each of the candidate panoramic images, and record the first time when the robot moves to the actual checkpoint based on the location of the expected checkpoint.
[0050] In a possible implementation, the robot is further configured to send inspection data to the image processing device after completing the inspection along the inspection route.
[0051] In a possible implementation, the camera of the robot has at least two cameras with different shooting angle ranges, and the sum of the shooting angle ranges of the at least two cameras is 360 degrees.
[0052] In a possible implementation, the robot is further used to: during an inspection process according to an inspection route, obtain at least two first images by capturing them through the at least two cameras of the camera; and combine the at least two first images into a panoramic image.
[0053] In one possible implementation, an embodiment of the present application provides a device cluster, including at least one processing device, each processing device including a processor and a memory; the processor of the at least one processing device is used to execute instructions stored in the memory of the at least one processing device, so that the device cluster executes the image processing method in any possible implementation described above.
[0054] The effect of the device cluster in this embodiment is similar to the effect of the image processing method in the above embodiments, and will not be described in detail here.
[0055] In a possible implementation, an embodiment of the present application provides a computer program product including instructions, and when the instructions are executed by a device cluster, the device cluster executes the image processing method in any possible implementation described above.
[0056] The effects of the computer program product of this embodiment are similar to the effects of the image processing methods in the above embodiments, and will not be described in detail here.
[0057] In a possible implementation, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a device cluster, the device cluster executes the image processing method in any one of the above implementations.
[0058] The effect of the computer-readable storage medium of this embodiment is similar to the effect of the image processing method in the above-mentioned embodiments, and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 is a schematic diagram of an inspection robot in the prior art shown as an example;
[0060] Figure 2 is a schematic diagram of a system architecture shown as an example;
[0061] Figure 3a is a schematic diagram of a robot shown as an example;
[0062] Figure 3b is a schematic diagram of a base station shown as an example;
[0063] Figure 4a A schematic diagram for exemplarily illustrating the processing process;
[0064] Figure 4b A schematic diagram for exemplarily illustrating the processing process;
[0065] Figure 5 is a schematic diagram of measuring distance for exemplary purposes;
[0066] Figure 6 is a schematic diagram of an image of a selected target area shown as an example;
[0067] Figure 7 is a schematic structural diagram of an image processing device shown as an example;
[0068] Figure 8 is a schematic diagram of the structure of an inspection system shown as an example;
[0069] Fig. 9 A schematic diagram of the structure of a computing device is shown as an example;
[0070] Fig.10 A schematic diagram of the structure of a computing device is shown as an example;
[0071] Fig.11 The figure is a schematic diagram of the structure of a computing device cluster shown as an example. DETAILED DESCRIPTION
[0072] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0073] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0074] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.
[0075] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0076] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.
[0077] Before describing the technical solution of the embodiment of the present application, the background technology involved in the embodiment of the present application is briefly introduced:
[0078] Public cloud is a cloud platform provided by third-party public cloud providers for individuals or enterprises. In the public cloud, the hardware, software and other structures are owned and managed by the third-party public cloud providers.
[0079] The cloud platform, which may also be referred to as a cloud management platform or cloud management platform in the embodiments of the present application, is a software system for cloud technology (also known as cloud computing technology) services provided by a cloud provider, and is used to provide interfaces related to cloud services for tenants to remotely access cloud services. Tenants can log in to the cloud platform on the cloud service access page using a pre-registered account and password, and after a successful login, select and purchase the corresponding cloud service on the cloud service access page, such as object storage services, virtual machine services, container services, etc. Exemplarily, the cloud management platform can provide a unified entrance (also referred to as an operation and maintenance entrance), and users can access the cloud management platform through the unified entrance, and manage the cloud resources (including hardware and software) of the corresponding cloud through the cloud management platform.
[0080] Cloud services include computing services, storage services, virtual machine services, network services, etc. Any device or function that a user device can access on a cloud platform can be considered a service provided by the cloud platform.
[0081] Cloud resources are hardware resources (also referred to as cloud infrastructure or infrastructure) and software resources (also referred to as public cloud services) used to provide services. Among them, the infrastructure includes multiple cloud data centers set up in multiple regions, and each region is equipped with at least one cloud data center. Cloud data centers may include computing nodes, storage nodes, and network nodes. Among them, computing nodes may be computing servers, storage nodes may be storage servers, and network nodes may include network facilities such as switches. For example, the hardware resources corresponding to software resources such as computing services, storage services, or network services include computing resources, storage resources, or network resources. Among them, computing resources can be deployed on computing nodes, storage resources can be deployed on storage nodes, and network resources can be deployed on network nodes.
[0082] Optionally, computing resources may include central processing unit (CPU) resources, memory resources and / or hard disk resources, etc. For example, a user (also referred to as a tenant) purchases a large number of virtual machine resources, and a large number of applications (also referred to as cloud applications) are deployed on the virtual machine resources. In this embodiment, the virtual machine and the applications in the virtual machine are both cloud services (i.e., software resources), and the server and other devices to which the virtual machine belongs are the corresponding hardware resources.
[0083] As the scale of the power system continues to grow, higher requirements are placed on the safe operation of infrastructure such as power grids and power plant equipment in the power environment. In order to ensure the safe operation of infrastructure in the power environment and to ensure power safety, it is necessary to inspect the infrastructure in the power environment. Manual inspections are gradually being eliminated due to low operation and maintenance efficiency, poor detection quality, and many subjective factors. Currently, robots (such as 24-hour standby intelligent inspection robots) are mainly used to conduct automatic and autonomous digital inspections of facilities in the power environment. Robot inspections can not only improve the efficiency of operation and maintenance inspections, but also help solve the problems existing in manual inspections.
[0084] The robots in the prior art are mainly intelligent inspection robots that rely on simultaneous localization and mapping (SLAM) technology for autonomous navigation. The intelligent inspection robot is composed of multiple sensors such as a robot motion controller, a laser sensor, a pan-tilt head, and a high-definition camera. The intelligent inspection robot uses SLAM technology to achieve autonomous trackless navigation in an electric power environment. During the navigation process, it can sense the surrounding environment, avoid obstacles and re-plan the route when obstacles are sensed, thereby achieving autonomous patrols of the electric power environment.
[0085] However, for the above-mentioned intelligent inspection robot with outdoor trackless autonomous navigation, whether the robot can accurately reach the desired inspection point to inspect the inspection object (such as power plant equipment, power grid, electric meter, etc.) is related to many factors, such as the robot's motion error in the outdoor environment, positioning accuracy, sensor fusion, and the robustness of the navigation algorithm. These factors will affect the robot's navigation accuracy.
[0086] The most important factor among the factors affecting the robot's navigation accuracy is the robot's positioning accuracy. In the prior art, the above-mentioned intelligent inspection robot that relies on SLAM technology for positioning can infer the robot's current positioning through the sensor's observation of the environment. Among them, the sensor itself will have a certain observation error. In addition, the positioning accuracy of SLAM technology is also low, which leads to the problem of inaccurate estimation of the robot's own positioning in the prior art, resulting in the robot being unable to accurately reach the desired inspection point.
[0087] Among them, the SLAM technology used by the robot in the prior art is a relative positioning technology, that is, it uses a technology for positioning relative to the initial position. The disadvantage of relative positioning technology is that the positioning error will continue to accumulate with the growth of time. Therefore, when the robot in the prior art is moving for a long time, its positioning error will become larger and larger as the movement time increases, resulting in the robot's positioning accuracy becoming lower and lower as the inspection time increases. Therefore, in order to ensure positioning accuracy, the robot in the prior art needs to adjust the robot's pan / tilt or algorithm after a certain period of movement to reduce the robot's positioning error.
[0088] In addition, the robot's inspection environment will also affect the robot's accurate arrival at the desired inspection point. Since the actual outdoor inspection environment is often more complex, such as interference from dynamic objects (such as the movement of people or objects), random obstacles and shielding on the inspection route, and rugged outdoor roads, etc., all of these will bring challenges to the positioning accuracy of the robot's motion control system and SLAM technology algorithm in the existing technology.
[0089] Figure 1 is a schematic diagram of an intelligent inspection robot in the prior art, Figure 1 The robot's fixed-angle pan-tilt 10 and a common camera 11 mounted on the pan-tilt 10 are shown, wherein the shooting angle of the common camera 11 is generally about 120 degrees. If the robot is not positioned accurately, it cannot accurately reach the desired inspection point, resulting in incomplete or erroneous images of the inspection object captured by the common camera equipped with the fixed pan-tilt at the inaccurate inspection point, and the problem of inadequate inspection occurs.
[0090] In view of the above technical problems existing in the intelligent inspection robot in the prior art, the present application provides an image processing method and a robot inspection system. The robot inspection system can execute the image processing method to inspect the inspected equipment in the outdoor environment (the power environment is taken as an example in this paper, but is not limited to the above power environment), wherein the inspected equipment in the outdoor environment is fixed in position, for example, an electric meter in the power environment can be the inspected equipment.
[0091] The robot inspection system may include a robot 100, a base station 300, and a management platform, and may optionally also include terminal equipment, etc.
[0092] In this embodiment, the management platform is a cloud management platform 200 deployed on the cloud.
[0093] In other embodiments, the management platform may also be a server or server cluster under the cloud, and the execution principle of the image processing method is similar to the execution principle of the cloud management platform 200 deployed on the cloud, which will not be repeated here.
[0094] In this embodiment, the cloud management platform 200 can use the post-processing kinematic (PPK) technology to accurately locate the actual position of the robot 100 when it navigates to the desired checkpoint (e.g., the position of the actual checkpoint) to improve the accuracy of positioning the robot 100. And the cloud management platform 200 can extract the image of the ROI of the inspection object from the panoramic image taken by the robot 100 at the actual checkpoint based on the position error between the desired checkpoint and the actual checkpoint. In this way, even if the robot 100 cannot accurately reach the desired checkpoint due to the low accuracy of autonomous navigation, the cloud management platform 200 of the present application can also obtain the complete image of the inspection object (e.g., the image of the ROI) in the panoramic image taken at the actual checkpoint when there is a certain error between the actual checkpoint to which the robot 100 navigates and the desired checkpoint, and the inspection of the inspection object corresponding to each desired checkpoint is more in place. The problem of incomplete image information of the inspection object (e.g., the inspected power equipment) collected due to the inability of the robot 100 to accurately reach the desired checkpoint in the prior art is solved.
[0095] Among them, the PPK technology is a satellite positioning technology. The present application applies it to the positioning of the robot 100. It can refer to the satellite signal of the base station whose positioning position remains unchanged to compensate and calibrate the satellite signal received by the robot 100 whose positioning position is constantly changing, so as to accurately locate the actual position of the moving robot 100, so that the accuracy of the positioning of the robot 100 itself is higher. Compared with the SLAM technology used by the robot 100 in the prior art, the PPK technology is an absolute positioning technology, which can be used for long-distance positioning of the robot 100. It will not have the problem of increasing positioning error due to the increase in the movement time of the robot 100, and can accurately locate the robot 100 for long-distance inspection.
[0096] Furthermore, in the method of the present application, the robot 100 only needs to perform inspections according to the inspection route through autonomous navigation to collect data, such as taking panoramic images when navigating to the desired inspection point (for example, when moving to the actual inspection point based on the position of the desired inspection point), receiving satellite signals, etc., and reporting the collected data to the cloud management platform 200. The cloud management platform 200 processes the collected data of the robot 100 to obtain an image of the ROI. Therefore, during the inspection process of the robot 100, each time the robot 100 navigates to the desired inspection point, regardless of whether the robot 100 accurately reaches the position of the desired inspection point, the robot 100 does not need to interact with the base station to make any adjustment operations. For example, each time the robot 100 navigates to the desired inspection point, the robot 100 does not need to establish a radio communication data link with the base station. Therefore, the robot 100 does not need to use the data link to receive satellite signals from the base station (the satellite signal can be the satellite signal received by the base station when the robot 100 navigates to the desired checkpoint), and use the satellite signal received from the base station and the satellite signal received by itself when navigating to the desired checkpoint to reduce the position error between the actual checkpoint where the robot 100 is located and the desired checkpoint, but continues to perform the inspection task. This can reduce the communication frequency between the robot 100 and the base station, avoid the interference of the electromagnetic signal of the high-voltage power station to the robot 100, make the operation range wider, the operation method more flexible, and the later maintenance of the robot 100 simpler, so that the robot 100 of the present application has a stronger anti-interference ability.
[0097] Before introducing the method of the present application in detail, a brief introduction to the system architecture of the present application is first given.
[0098] Figure 2 The following is a schematic diagram of the system architecture of the present application, Figure 3a and Figure 3b The structural schematic diagrams of the robot 100 and the base station 300 of the present application are respectively exemplified.
[0099] Combine the following Figure 3a , Figure 3b Come to Figure 2 The system of the present application is introduced as shown:
[0100] like Figure 2 As shown, the system may include a robot 100 , a base station 300 , a cloud management platform 200 and cloud resources 400 .
[0101] like Figure 3aAs shown, the robot 100 may include but is not limited to: a panoramic camera 101, a Global Navigation Satellite System (GNSS) receiver, here a GNSS receiving antenna 102, a laser radar 103, etc.
[0102] As shown in FIG3 , the panoramic camera 101 may include a panoramic camera, and the panoramic camera may include two cameras with different shooting angles, namely, camera 1011 and camera 1012. The shooting angles of camera 1011 and camera 1012 are both 0 to 180 degrees, and camera 1011 and camera 1012 are arranged relative to each other, so that the robot 100 can use camera 1011 and camera 1012 to respectively capture two images with a viewing angle range of 0 to 180 degrees, and the robot 100 can synthesize the two images into a panoramic image captured by the panoramic camera 101.
[0103] In other embodiments, the number of cameras installed on the robot 100 for shooting 360-degree panoramic images may be more than two, the shooting angles of the multiple cameras are different, and the sum of the shooting angles of the multiple cameras is 360 degrees. In specific applications, the number and installation angles of the cameras for shooting panoramic images can be flexibly configured according to needs.
[0104] A panoramic camera is a camera that can capture all-round images. It can provide a wider shooting angle than the ordinary camera used by the robot in the prior art. The intelligent inspection robots in the prior art generally use ordinary cameras, so their shooting angle range is limited, and the inspection object may not be fully imaged. The present application uses a panoramic camera for the robot to increase the inspection field of view and improve the inspection efficiency.
[0105] The panoramic image taken by the panoramic camera 101 of the present application is a photo presenting a 360° viewing angle, which is usually circular or annular. It is synthesized by taking multiple images and displayed in a plane. The initial synthesized image may have certain bending deformation, so the synthesized image can be processed in a certain way. The processing technology can be any related image processing technology developed in the existing technology or in the future. There is no limitation here, and the panoramic image described in each embodiment of the present application can be obtained.
[0106] The robot 100 performs intelligent inspections by using the panoramic camera 101, and selects images of the target area (a ROI) belonging to the inspection object from the panoramic images taken during the inspection. Since the panoramic image information is more comprehensive, the image of the target area obtained may contain more detailed information of the inspection object, thereby improving the inspection accuracy of the inspection object, such as power equipment or power grids in an electric power environment.
[0107] In addition, in this embodiment, by installing two cameras with a shooting angle of view within 180 degrees on the robot 100, and the two cameras are arranged relative to each other to obtain a panoramic image, the robot 100 of the present application does not need to be installed with a pan / tilt head.
[0108] Compared to the method of only installing a single or multiple cameras that do not cover a 360-degree viewing angle on the robot 100, and changing the shooting angle of the camera by rotating the gimbal to obtain a panoramic image, the panoramic camera 101 of the present application is more efficient in shooting panoramic images; and the mechanical stability of the gimbal rotation is poor. The robot 100 of the present application does not need a gimbal, but uses multiple cameras with different shooting angles to shoot panoramic images, which can improve the shooting stability of the panoramic image; in addition, controlling the camera to shoot a panoramic image by rotating the gimbal requires multiple shots at different times to obtain a panoramic image, while at least two cameras of the panoramic camera 101 of the present application shoot simultaneously, thereby reducing the shooting delay of the panoramic image and helping to improve the inspection efficiency.
[0109] like Figure 3a The GNSS receiving antenna 102 shown can be used to receive satellite signals, which are observations of the carrier phase of the satellite. In addition, the robot 100 can use a laser radar 103 to perform autonomous navigation.
[0110] like Figure 3b As shown, the base station 300 may be installed with a reference station receiver 301, and the base station 300 may use the reference station receiver 301 to receive a satellite signal, which is an observed value of the carrier phase of the satellite.
[0111] Since the robot 100 and the base station 300 are located at different positions, their observations of the satellite's carrier phase may differ, such as Figure 2 The cloud management platform 200 shown can determine the actual positioning of the robot 100 by using the respective observations of the carrier phase of the satellite from the robot 100 and the base station 300 at the same time.
[0112] And the cloud management platform 200 can select an image of a target area of the inspection object of interest from a panoramic image taken by the robot 100 at the actual positioning position based on the position error between the actual positioning of the robot 100 and the positioning of the expected inspection point to which the robot is expected to move.
[0113] Exemplarily, when the cloud management platform 200 processes the data received from the base station 300 and the robot 100 to obtain the image of the target area, it can use Figure 2 The cloud resources 400 shown are used to perform computing processing.
[0114] Combination Figure 2 and Figure 3a , Figure 3b , the image processing method of the present application may include a configuration stage and a processing stage.
[0115] The configuration phase may include configuration operations on the robot 100, configuration operations on the cloud management platform 200, etc.
[0116] The processing stage may include but is not limited to: the cloud management platform 200 processes the inspection data obtained from the robot 100 performing the inspection task and the satellite signal from the base station 300 to obtain the image of the target area of the inspection object related to each desired inspection point.
[0117] like Figure 4a As shown, the configuration phase may include the following steps:
[0118] S100, pre-configuring the inspection task of the robot 100.
[0119] The inspection tasks may include but are not limited to: inspection routes (optional), first positions of multiple desired inspection points, etc.
[0120] The position of the expected inspection point is the expected position for recording the image of the inspection object when the robot 100 moves along the inspection route.
[0121] The position of the expected inspection point is the position that the robot is expected to reach. For example, empirical analysis has determined that when the robot 100 is located at the expected inspection point to take a picture of the inspection object, the image taken of the inspection object is most consistent with expectations, for example, the image information about the inspection object is the most complete and accurate.
[0122] For example, in order to implement the inspection of the inspection object, the robot 100 is expected to move to the expected position according to the inspection route, and take a picture of the inspection object at the expected position to obtain an image of the inspection object. However, because the autonomous navigation of the robot 100 is not accurate enough, when the robot 100 navigates and moves based on the position of the expected inspection point, when the robot 100 determines that it has moved to the expected inspection point (determines that it is located at the position of the expected inspection point), the actual position of the robot 100 at this time (called the actual inspection point position) will have a certain position deviation from the position of the expected inspection point, and the two are not the same position.
[0123] For example, the inspection objects corresponding to the various desired inspection points are different. Thus, the robot 100 can be used to navigate to the various desired inspection points to implement inspections of multiple inspection objects.
[0124] The inspection route may include the inspection starting point position (optional), the first position of each desired inspection point, and the inspection end point position (optional) in sequence. For example, the inspection route is A->B1->B2->B3->C.
[0125] Among them, point A is the inspection starting point, point B1, point B2, and point B3 are the locations of the three expected inspection points, and point B is the inspection end point. The inspection route is to inspect from point A to point B1, then from point B1 to point B2, then from point B2 to point B3, and finally, move from point B3 to point C, and the inspection ends.
[0126] The positions of the points B1, B2, and B3 (an example of a first position) may be configured in advance for the robot 100.
[0127] Regarding the implementation method of pre-configuring the inspection tasks for the robot 100, the robot 100 may be configured through a terminal device.
[0128] In some embodiments, the cloud management platform 200 may also pre-configure the inspection task to the robot 100 by issuing the inspection task to the robot.
[0129] Exemplarily, information about the inspection task can be pre-entered into the cloud management platform 200 through the user interface, and then the robot 100 can capture the inspection task information (such as the coordinates of the desired inspection points, the inspection route, etc.) from the cloud management platform 200 to facilitate the execution of the inspection task.
[0130] Optionally, S200a, the cloud management platform 200 pre-stores the expected area related to the expected checkpoint.
[0131] The expected area is a region of interest (ROI) of the inspection object in the first panoramic image pre-photographed by the robot 100 at the expected inspection point, such as the position of a pixel point.
[0132] Optionally, S200b, the cloud management platform 200 pre-stores a first distance associated with the desired checkpoint. The first distance is the distance between the robot 100 and the panoramic camera (e.g. Figure 3a The distance between the panoramic camera 101) shown in FIG. 1 and the inspection object.
[0133] Reference Figure 5 In a specific implementation, the robot 100 can be manually remotely controlled to be placed at the first position of the desired inspection point, or the robot 100 can be directly placed at the first position of the desired inspection point, where the position of the desired inspection point is point B1 as an example; then, manual measurement or by Figure 3a The laser radar 103 (or the distance sensor of the robot 100) shown is used to measure the first distance d between the panoramic camera 101 of the robot 100 (for example, the center point 1013 of the panoramic camera 101, not specifically limited) located at the position of the desired inspection point (point B3) and the corresponding inspection object P.
[0134] It should be understood that the multiple expected inspection points each correspond to a different inspection object, so different inspection objects are inspected separately at the multiple inspection points.
[0135] Then, the robot 100 can be manually remotely controlled to use the panoramic camera 101 to capture a panoramic image at the desired inspection point B1;
[0136] Then, the image area to which the inspection object belongs can be extracted from the panoramic image by controlling the robot 100 or by post-processing, and the pixel position of the image area in the panoramic image can be determined to obtain the desired area of the inspection object in the panoramic image taken at the desired inspection point. It should be understood that the image of the desired area is also a panoramic image, but the panoramic image only includes the panoramic image of the inspection object of interest, and does not include image content of other facilities or on-site environment.
[0137] In this way, when there are multiple expected inspection points, an expected area and a first distance related to the corresponding inspection object can be obtained for each expected inspection point.
[0138] In some embodiments, the robot 100 can also take multiple panoramic images when it is located at the first position of the desired inspection point, so as to process and obtain a desired area of the inspection object related to the desired inspection point, thereby avoiding affecting the accuracy of the acquired desired area due to the poor image quality of a panoramic image taken. By pre-acquiring a high-precision desired area of the inspection object related to the desired inspection point, and using it to extract the image of the target area of the inspection object from the panoramic image taken at the actual inspection point, the recognition accuracy of the image of the target area and the image quality of the image of the target area can be improved.
[0139] In one possible implementation, the cloud management platform 200 may be configured with a user interface, through which the user may transmit at least one of the above-mentioned expected area and the first distance related to the expected checkpoint to the cloud management platform 200. In this way, the cloud management platform 200 may pre-acquire at least one of the expected area and the corresponding first distance related to the expected checkpoint regarding the corresponding inspection object through the user interface.
[0140] For example, the cloud management platform 200 may store at least one of the expected area and the first distance related to the expected checkpoint acquired in advance in a storage node in the cloud data center.
[0141] In another possible implementation, at least one of the expected area and the first distance related to the expected inspection point may be pre-stored in the robot 100 before the robot 100 performs the inspection task.
[0142] Then, before performing the inspection task, the robot 100 may upload at least one of the pre-stored expected area and the first distance to the cloud management platform 200, so that the cloud management platform 200 pre-stores at least one of the expected area and the first distance.
[0143] In other embodiments, the robot 100 may also upload at least one of the pre-stored expected area and the first distance to the cloud management platform 200 during the execution of the inspection task or after the execution of the inspection task, so that the cloud management platform 200 can obtain at least one of the expected area and the first distance related to the expected checkpoint in the above processing stage.
[0144] That is to say, the cloud management platform 200 can obtain the expected area and the first distance related to the expected inspection point in the configuration stage and / or the processing stage according to the application scenario and actual needs and the network quality of the mobile network (such as the 5G network) of the robot 100. Among them, all the data in the expected area and the first distance can be fully obtained by the cloud management platform 200 in the configuration stage or the processing stage; or, part of the data in the expected area and the first distance is obtained by the cloud management platform 200 in the configuration stage, and the other part of the data is obtained by the cloud management platform 200 in the processing stage, so that the cloud management platform 200 can use the obtained expected area and the first distance to obtain the image of the target area of the corresponding inspection object after receiving the inspection data obtained by the robot 100 performing the inspection task in the processing stage.
[0145] The present application does not limit the execution order between the above S200a and S200b. For example, S200a and S200b can be executed in series or in parallel. Moreover, the present application does not limit the execution order between S100 executed by the robot 100 and S200a and S200b executed by the cloud management platform 200.
[0146] exist Figure 4a After the configuration phase shown, Figure 4b As shown, this processing stage may include but is not limited to the following steps:
[0147] The robot 100 starts to perform the inspection task.
[0148] For example, the inspection task may include, but is not limited to: inspection time, inspection route, the first position of each expected inspection point, etc.
[0149] For example, the inspection route may include the inspection starting position (optionally), the first position of each desired inspection point in sequence. For example, the inspection route is A->B1->B2->B3.
[0150] Among them, point A is the inspection starting point, point B1, point B2, and point B3 are the locations of the three expected inspection points, and point B is the inspection end point. The inspection route is to inspect from point A to point B1, then from point B1 to point B2, and then from point B2 to point B3, and the inspection ends.
[0151] When the robot 100 performs the inspection task, it can start autonomous navigation in the power environment according to the inspection route when the inspection time arrives (for example, 22:00 every Sunday night), so as to navigate from point A to point B1, and take a panoramic image when navigating to point B1; similarly, the robot autonomously navigates from point B1 to point B2, and takes a panoramic image when reaching point B2; similarly, the robot autonomously navigates from point B2 to point B3, and takes a panoramic image when reaching point B3, and the inspection is completed. Among them, since the autonomous navigation of the robot 100 has the problem of inaccurate positioning, when the robot 100 determines that it has navigated to point B1, there is actually a certain distance (also called position error) between the actual inspection point (for example, point D1) where the robot 100 is located and the expected inspection point (point B1), so the panoramic image taken by the robot 100 when navigating to point B1 is the panoramic image taken by the robot 100 at point D1.
[0152] When the robot 100 autonomously navigates to each desired checkpoint, the operations performed are similar. Therefore, the following takes the robot 100 navigating to point B1 as an example to illustrate the operations of its inspection process. The operations when the robot autonomously navigates to other desired checkpoints can be referred to the operations of navigating to point B1, which will not be repeated here.
[0153] The following describes in detail the steps S101a to S105 performed by the robot 100 during the inspection task:
[0154] S101a, the robot 100 captures a panoramic image, and records the panoramic image and its capturing time.
[0155] In this embodiment, the robot 100 can use autonomous navigation technology to navigate the inspection route and move to the first position of each desired inspection point in sequence. When the robot 100 inspects within a preset distance range of the first position of the desired inspection point according to the inspection route, for example, the robot 100 can pass through the inspection route. Figure 3a When the laser radar 103 shown determines that the current position of the robot 100 is within a certain distance range of the desired inspection point (point B1) (e.g., within 0.5 meters of point B1), the robot 100 can control the panoramic camera 101 to take one or more panoramic images (also referred to as candidate panoramic images) and record the shooting time of each panoramic image. In this way, the robot 100 can take multiple panoramic images within a preset distance range (e.g., within 0.5 meters, which can be flexibly configured according to needs) close to the desired inspection point.
[0156] Since the technology of autonomous navigation of robot 100 itself has the problem of low accuracy, the current positioning of robot 100 determined by laser radar 103 is not accurate enough. Therefore, by starting to capture panoramic images when robot 100 navigates to the vicinity of the desired inspection point, a sufficient number of panoramic images captured when the positioning position of robot 100 is close to the desired inspection point can be obtained, so as to extract images of target areas with higher image quality of corresponding inspection objects (such as the inspected power equipment) with a sufficient number of panoramic images, thereby improving the completeness and accuracy of the images of the inspected power equipment.
[0157] In other embodiments, the robot 100 may also execute S101a to capture a panoramic image when it is determined in S102 that the robot has navigated to a desired checkpoint.
[0158] That is to say, the robot 100 can also use the autonomous navigation technology to determine the first time to navigate to the expected checkpoint, such as the first time when the robot 100 moves to the corresponding actual checkpoint based on the expected checkpoint. The robot 100 can use the above-mentioned panoramic camera 101 to capture a panoramic image at the first time, and associate the panoramic image with the first time. In this way, the robot 100 does not need to capture multiple panoramic images at a position close to the expected checkpoint during the inspection task. It only needs to capture the panoramic image at the first time of navigating to the expected checkpoint, which can reduce the proportion of the cache space occupied by the panoramic image on the robot 100 side, save the cache space of the robot 100, and reduce the loss of the performance of the robot 100 due to continuous image capture.
[0159] S101b, the robot 100 receives a first satellite signal, and records a first receiving time of the first satellite signal.
[0160] Among them, when the inspection task begins, the robot 100 can start the switch of the GNSS receiving antenna 102 as shown in Figure 3, and during the robot 100 performing the inspection task, before the inspection task ends, the GNSS receiving antenna 102 can receive satellite signals and record the reception time of each satellite signal.
[0161] The first satellite signal received by the robot 100 may be an observation value of the carrier phase of the satellite (for example, represented by observation value 1).
[0162] Among them, S101a and S101b are parallel steps in the process of the robot 100 performing the inspection task.
[0163] S102, the robot 100 determines whether to navigate to the desired inspection point.
[0164] Among them, Figure 4a In S100, the robot 100 has pre-configured the first position of each expected checkpoint. The robot 100 can navigate to each first position according to the inspection route through autonomous navigation technology. When navigating to a first position, the robot 100 can determine that it has currently navigated to the expected checkpoint (for example, moving to the actual checkpoint based on the position of the expected checkpoint), and thus can execute S103.
[0165] S103, the robot 100 records the first time.
[0166] Among them, the robot 100 can record the time of navigation to the expected checkpoint. From the perspective of the robot 100, at the first time, the robot 100 is already at the expected checkpoint (for example, point B1). In fact, because the accuracy of the robot 100's self-positioning in autonomous navigation is not high enough, the robot 100 moves to the position of the actual checkpoint at the first time according to the position of the expected checkpoint, also called the position of the actual checkpoint, which still has a certain position error with the position of the expected checkpoint.
[0167] After S103 , in S104 , the robot 100 determines whether the inspection task is completed.
[0168] For example, the inspection route is A->B1->B2->B3.
[0169] For example, if the robot 100 moves to each desired inspection point, namely point B1, point B2, and point B3, according to the inspection route within a specified time period (optionally), and takes panoramic images at the three desired inspection points (or near the three desired inspection points), the inspection mission of the robot 100 is completed.
[0170] For example, the robot 100 follows the inspection route and does not move to the above Figure 4a If the positions of all expected inspection points configured for the robot 100 are not obtained, or if corresponding panoramic images are not captured at the positions of all expected inspection points, it is determined that the inspection task is not completed.
[0171] After S104 , the robot 100 determines that the inspection task is not completed, and then proceeds to S102 .
[0172] If the robot 100 determines that the inspection task is not completed, it means that the robot 100 has not inspected all the expected inspection points, and then continues to use the autonomous navigation technology to follow the inspection route and Figure 4a The first position of the desired inspection point configured in S100 is used for navigation. When navigating to the next desired inspection point (such as point B2) in the inspection route, S103 and S104 are continued.
[0173] After S104 , the robot 100 determines that the inspection task has been completed, and then executes S105 .
[0174] When the robot 100 determines that the inspection task has been completed, it means that the robot 100 has moved to all the expected inspection points and has captured a panoramic image when navigating to the first position of the corresponding expected inspection point (or near the first position, for example, within 0.5 meters of the first position). Then the robot 100 can execute S105 to upload the initial inspection data of the inspection task to the cloud management platform 200.
[0175] S105 , the robot 100 uploads initial inspection data to the cloud management platform 200 .
[0176] The initial inspection data may include each panoramic image (also referred to as a candidate panoramic image) and its shooting time (obtained by S101a) taken when the robot 100 moves to each actual inspection point (e.g., within 0.5 meters of the corresponding expected inspection point) during the inspection task, as well as each first satellite signal received during the inspection task and its first reception time (obtained by S101b), and the first time each time the robot 100 navigates to the expected inspection point, also referred to as the first time based on the position of the expected inspection point to move to the actual inspection point (obtained by S103). At the first time, the robot 100 is determined to have moved to the expected inspection point, but there is still a certain position error between the actual position of the robot 100 and the first position of the expected inspection point.
[0177] In other embodiments, the configuration phase is not completed. Figure 4a When S200a and S200b are used to pre-store the expected area and the first distance to the cloud management platform 200, the robot 100 can upload the expected area and the first distance of the inspection object related to each expected inspection point described in S200a and S200b to the cloud management platform 200 as part of the initial inspection data or separately when executing S105.
[0178] In addition, in other embodiments, the panoramic image and its shooting time in the inspection data uploaded by the robot 100 to the cloud management platform 200 may also be at least one candidate panoramic image and its shooting time obtained by filtering the data obtained in S101a by the robot 100, whose shooting time matches the first time. Then, the cloud management platform 200 may determine the second panoramic image shot by the robot 100 at the actual inspection point based on the at least one candidate panoramic image (the specific implementation process may refer to the detailed example of S201a below).
[0179] Alternatively, in some embodiments, the panoramic image and its shooting time in the inspection data uploaded by the robot 100 to the cloud management platform 200 may also be obtained by screening the data obtained in S101a by the robot 100, and the second panoramic image photographed by the robot 100 at the actual inspection point, wherein the process of the robot 100 screening and processing the candidate panoramic images in the initial inspection data to determine the second panoramic image is similar to the process performed by the cloud management platform 200 in S201a described below, and will not be repeated here.
[0180] In this way, when the robot 100 has a high computing power, it can perform data screening in advance when uploading panoramic images in the inspection data, so as to select a panoramic image with a shooting time of the first time when the robot 100 navigates to the desired inspection point from among the multiple candidate panoramic images taken within 0.5 meters of the desired inspection point during the inspection process of the robot 100, or select one or more candidate panoramic images with a shooting time near the first time (for example, within a certain time before and after the first time), or determine a second panoramic image from the multiple candidate panoramic images taken above, and upload it to the cloud management platform 200, so as to reduce the amount of data transmission, thereby reducing the network bandwidth occupation, improving the data transmission speed, and speeding up the processing speed of the inspection data by the cloud management platform 200. For example, the certain time length can be 0.1s, which can be set according to the needs and is not limited here.
[0181] In this embodiment, the robot 100 can upload initial inspection data after the inspection task is completed. The initial inspection data includes data related to each desired inspection point involved in this inspection task, such as a candidate panoramic image captured at or near the first time, and a first satellite signal received at the first time. In this way, the robot 100 uploading inspection data does not affect the execution of the inspection task, does not interrupt the execution of the inspection task, can ensure the continuity of the inspection task execution, and does not occupy too much network bandwidth due to the inspection data upload during the execution of the inspection task, affecting the quality of the network for executing the inspection task.
[0182] In some embodiments, when the 5G network quality of the robot 100 is good (for example, the signal strength is strong), the robot 100 may also upload at least one candidate panoramic image matching the first time (or the second panoramic image determined based on the at least one candidate panoramic image) and the first satellite signal received at the first time to the cloud management platform 200 each time it navigates to a desired inspection point and obtains one or more corresponding candidate panoramic images and first satellite signals during the inspection task. This allows the cloud management platform 200 to obtain the inspection data obtained by the robot 100 during the inspection near the desired inspection point in a timely manner, so as to obtain the image of the target area of the inspection object corresponding to the desired inspection point.
[0183] In the method of the present application, the robot 100 only needs to perform inspections according to the inspection route through autonomous navigation to collect data, such as taking panoramic images when moving from the expected inspection point to the actual inspection point, receiving satellite signals, etc., and reporting the collected data to the cloud management platform 200. The cloud management platform 200 processes the collected data of the robot 100 to obtain the image of the ROI. Therefore, during the inspection process of the robot 100, each time the robot 100 navigates to the expected inspection point, regardless of whether the robot 100 accurately reaches the position of the expected inspection point, the robot 100 side does not need to interact with the base station to make any adjustment operations. For example, each time the robot 100 navigates to the expected inspection point, the robot 100 does not need to establish a radio communication data link with the base station, so that the robot 100 uses the data link to receive satellite signals from the base station 300, and uses the satellite signals received from the base station and the satellite signals received by itself to reduce the position error between the actual inspection point where the robot 100 is located and the expected inspection point, but continues to perform the inspection task. This can reduce the communication frequency between the robot 100 and the base station 300, avoid interference of the electromagnetic signal of the high-voltage power station on the robot 100, make the operation range wider, the operation method more flexible, and the subsequent maintenance of the robot 100 simpler, making the robot 100 of the present application more anti-interference.
[0184] During the inspection task performed by the robot 100, Figure 4b As shown, the base station 300 may execute S300.
[0185] S300, the base station 300 receives a second satellite signal and records a second receiving time of the second satellite signal.
[0186] The base station 300 can be Figure 3b The reference station receiver 301 installed on the base station 300 is shown to receive the second satellite signals and record the reception time of each second satellite signal.
[0187] The second satellite signal received by the base station 300 may be an observation value of the carrier phase of the satellite (for example, represented by observation value 2).
[0188] S301 , the base station 300 may synchronously upload the second satellite signal and its second receiving time to the cloud management platform 200 .
[0189] After S105 and S301 , the cloud management platform 200 may execute S201 a to S203 .
[0190] S201a, the cloud management platform 200 obtains a second panoramic image related to the actual checkpoint taken by the robot 100.
[0191] For example, the second panoramic image may be a candidate panoramic image taken when the robot 100 is located at the actual inspection point (eg, the shooting time is the first time).
[0192] Alternatively, the second panoramic image may be a candidate panoramic image taken by the robot 100 near the actual checkpoint, or a panoramic image obtained by the cloud management platform 200 fusing multiple candidate panoramic images taken by the robot 100 near the actual checkpoint with reference to the desired area.
[0193] Specifically, the initial inspection data may include each panoramic image (also referred to as a candidate panoramic image) taken by the robot 100 during the inspection task and its shooting time (obtained through S101a), as well as the first time each time the robot 100 navigates to the desired checkpoint (obtained through S103), wherein, at the first time, the robot 100 is deemed to have moved to the desired checkpoint, but there is still a certain position error between the actual position of the robot 100 and the first position of the desired checkpoint.
[0194] The cloud management platform 200 can then use the first time obtained from the initial inspection data to screen out at least one candidate panoramic image in the initial inspection data whose shooting time matches the first time (for example, the shooting time is the first time, or within a certain time period before and after the first time); then, the cloud management platform 200 can determine the second panoramic image taken by the robot 100 and related to the actual inspection point based on the at least one candidate panoramic image.
[0195] In an example A, for example, if the at least one candidate panoramic image is a candidate panoramic image shot at the first time in the initial inspection data, the cloud management platform 200 can use the candidate panoramic image shot at the first time as the second panoramic image.
[0196] In an example B, the at least one candidate panoramic image is a plurality of candidate panoramic images captured within a certain time period before and after the first time. In this example B, the plurality of candidate panoramic images are a plurality of candidate panoramic images captured within a certain time period (e.g., 0.1 seconds before and after) of the first time when the robot 100 inspects within a preset distance range (e.g., 0.5 meters) of the position of the desired inspection point according to the inspection route.
[0197] In Example B, when determining the second panoramic image, the cloud management platform 200 may, for example, refer to the pre-stored expected region (a ROI) associated with the expected inspection point to select a candidate panoramic image with the best image quality regarding the region of interest of the inspection object from the multiple candidate panoramic images as the second panoramic image, for performing the following S203 for ROI extraction. Alternatively, the cloud management platform 200 may refer to the pre-stored expected region (a ROI) associated with the expected inspection point to perform image fusion on the multiple candidate panoramic images to obtain a panoramic image with a better image quality regarding the region of interest of the inspection object as the second panoramic image, for performing the following S203 for ROI extraction.
[0198] In this way, the cloud management platform 200 uses multiple candidate panoramic images related to the actual inspection points to obtain a second panoramic image for ROI extraction, which can improve the completeness of the image of the extracted target area and reduce the acquisition error of the image of the inspection object. The target area is the area of interest (such as pixel position) regarding the inspection object in the second panoramic image.
[0199] S201b, the cloud management platform 200 obtains a first satellite signal whose first receiving time is a first time.
[0200] The initial inspection data also includes each first satellite signal received during the robot 100 performing the inspection task and its first receiving time (obtained through S101b).
[0201] The cloud management platform 200 can then filter out the first satellite signal with a first reception time as the first time in the received initial inspection data as the first information for determining the location of the actual checkpoint. For example, the first information is the first satellite signal received by the robot 100 at the first time.
[0202] S201c, the cloud management platform 200 obtains a second satellite signal whose second receiving time is the first time.
[0203] The cloud management platform 200 may filter the second satellite signal whose second receiving time is the first time from the data uploaded by the base station 300 in S301.
[0204] The present application does not limit the execution order between S201a to S201c, and they can be executed serially or in parallel.
[0205] S202, the cloud management platform 200 processes the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the second position (also expressed as the position of the actual checkpoint) of the robot 100 at the first time.
[0206] Among them, the dynamic post-processing technology (PPK technology) is a GNSS positioning technology that uses carrier phase for post-differentiation. Among them, the positioning of the base station 300 is fixed, while the positioning of the robot 100 is constantly changing. The cloud management platform can use the PPK technology to Figure 3b The base station receiver 301 shown in the figure uses the observation 2 of the carrier phase of the satellite at the first time to compensate and calibrate the observation 1 of the carrier phase of the satellite at the first time of the GNSS receiving antenna 102 of the robot 100, so as to obtain the position of the actual checkpoint at the first time of the robot 100 that keeps moving (i.e., the second position), which is the three-dimensional coordinate of the robot 100, and the first position of the above-mentioned pre-configured desired checkpoint is also a three-dimensional coordinate. In this way, after the robot 100 inspects the desired checkpoint (for example, point B1), the cloud management platform 200 can obtain the second position of the actual checkpoint where the robot 100 is when inspecting the desired checkpoint (point B1).
[0207] The cloud management platform 200 can use PPK to locate the robot 100, and can refer to the satellite signal of the base station whose positioning position remains unchanged to compensate and calibrate the satellite signal received by the robot 100 whose positioning position is constantly changing, so as to accurately locate the actual position of the moving robot 100, so that the accuracy of the positioning of the robot 100 itself is higher. Compared with the SLAM technology used by the robot 100 in the prior art, the PPK technology is an absolute positioning technology, which can be used for long-distance positioning of the robot 100, and will not have the problem of increasing positioning error due to the increase in the movement time of the robot 100, and can accurately locate the robot 100 for long-distance inspection.
[0208] In some embodiments, the PPK technology is used to position the robot, which can achieve centimeter-level positioning accuracy, thereby improving the positioning accuracy and positioning precision of the robot.
[0209] Optionally, after S202, the cloud management platform 200 may process the initial inspection data into data of an interrelated first time, a second position of the robot at the actual inspection point, and a second panoramic image taken by the robot 100 and related to the actual inspection point, so that the cloud management platform 200 can use the interrelated data to determine the image of the target area of the inspection object in the second panoramic image. In this way, the image of the target area of the inspection object corresponding to each expected inspection point can be determined more quickly, thereby improving the efficiency of acquiring the image of the target area of the inspection object involved in this inspection task.
[0210] Among them, after S201b and S201c, the above S202 can be executed.
[0211] S203, the cloud management platform 200 obtains an image of the target area of the inspection object in the second panoramic image according to the positional relationship between the first position and the second position, and the pre-stored desired area and the first distance.
[0212] The target area is a region of interest (eg, pixel position) regarding the inspection object in the second panoramic image.
[0213] Among them, after S201a and S202, the above S203 can be executed.
[0214] As described in S200a and S200b above, each expected inspection point may be associated with an expected area related to a corresponding inspection object and a first distance between the expected inspection point and the inspection object.
[0215] In this embodiment, the expected area and the first distance associated with the expected inspection point are data pre-stored in the cloud management platform 200 before the robot 100 performs the inspection task.
[0216] In other embodiments, the expected area and the first distance associated with the expected checkpoint may also be uploaded by the robot 100 to the cloud management platform 200 through the initial inspection data in S105 above, or may not be merged into the initial inspection data but uploaded separately to the cloud management platform 200 by the robot 100 during the execution of the inspection task or after the inspection task is completed. There is no limitation here.
[0217] In this embodiment, the cloud management platform 200 pre-stores the first position of the desired checkpoint (e.g., point B1), and determines the second position (e.g., actual checkpoint D1) where the robot 100 actually is when navigating to the first position. Then, the cloud management platform 200 can use the first position and the second position to determine the positional relationship (also called position error, three-dimensional coordinate transformation relationship) between them. In the three-dimensional space, the position error may include rotation error and translation error.
[0218] Then, the cloud management platform can use the transformation relationship of the three-dimensional coordinates between the first position and the second position, as well as the expected area (pixel position) of the inspection object obtained when the robot 100 takes a panoramic image of the inspection object at the first position located at the expected inspection point, and the first distance between the panoramic camera 101 and the inspection object when the robot 100 is located at the expected inspection point, to determine the pixel position (for example, the target area) of the inspection object in the second panoramic image taken by the robot 100 and related to the position of the actual inspection point (the second position), and then based on the target area, an image of the target area of the inspection object (for example, an image composed of pixels at the pixel positions expressed by the target area) can be selected in the second panoramic image taken by the robot 100 at the actual inspection point.
[0219] Figure 6 The figure is a schematic diagram of an image of an exemplary selected target area.
[0220] exist Figure 6 In the figure, P represents the inspection object that the robot 100 inspects at the expected inspection point B1; D1 represents the actual position of the robot 100 when it navigates to the expected inspection point B1 (also called the second position or the actual inspection point); {C} represents the three-dimensional coordinates of the expected inspection point B1; {C′} represents the three-dimensional coordinates of the actual inspection point D1; the vector u represents the pixel position of the inspection object P associated with the expected inspection point B1 in the corresponding first panoramic image (also called the expected region, or expected ROI), that is, the pixel position of the inspection object P in the first panoramic image taken when the robot 100 is located at the expected inspection point B1.
[0221] like Figure 6 As shown, the position of the desired inspection point B1 is taken as the center (also called the origin) of the spherical model, and the coordinates P of the inspection object P are established in the spherical coordinate system of the panoramic camera 101. c It is expressed as formula 1, where P c Measurable results:
[0222]
[0223] in, Represents a three-dimensional real vector.
[0224] The vector u is expressed as formula 2:
[0225]
[0226] in, represents a two-dimensional real vector, and [u,,v] represents the pixel position of the desired ROI in the corresponding first panoramic image.
[0227] In Equation 3, the spherical model obtained by projecting the omnidirectional camera 101 of the robot 100 in the above spherical coordinate system is denoted by m, and [H, W] represents the pixel position of the omnidirectional image captured by the omnidirectional camera 101.
[0228] m = [H, W] T , Equation 3;
[0229] In Equation 1, lon and lat respectively represent the longitude and latitude in the spherical coordinate system of the omnidirectional camera 101:
[0230] -π < lon < π, Equation 1;
[0231] -π / 2 < lat < π / 2, Equation 2;
[0232] In the following Equations 4 to 7, the projection function that maps the three-dimensional space in the spherical coordinate system of the omnidirectional camera 100 to the two-dimensional image space of point P c is denoted by π(P c ), and K is the intrinsic parameter of the omnidirectional camera 101:
[0233]
[0234] Among them, Equation 4 represents the mapping between the two-dimensional image space of the image and the longitude and latitude, as well as the three-dimensional coordinates, of the three-dimensional space.
[0235]
[0236]
[0237] Among them, Equation 6 represents the mapping between the three-dimensional coordinates and the longitude and latitude of the three-dimensional space.
[0238]
[0239] In Equation 8, represents the reciprocal of the first distance (e.g., c the first distance d measured in Figure 5 ) between point P
[0240]
[0241] It is known that T ij is the transformation matrix between the three-dimensional coordinates of the expected inspection point B1 and the actual inspection point D1 respectively, as shown in Equation 9. This transformation matrix T ij is divided into a rotation transformation matrix R ij and a translation transformation matrix t ij .
[0242]
[0243] Combined with Formulas 1 to 9 above, Formula 10 can be used to obtain the vector u' (an example of the target area) formed by the pixel point position of the inspection object in the second panoramic image (for example, the second panoramic image taken at the actual inspection point D1) taken by the robot 100 and related to the actual inspection point D1, thereby using the vector u' to select the pixel located at the pixel point position in the second panoramic image taken by the robot 100 at the actual inspection point D1, and extract the image of the target area of the inspection object P in the second panoramic image.
[0244]
[0245] The image u in Formula 10 is the pixel position of the inspection object P in the first panoramic image pre-photographed at the desired inspection point B1, also called the desired area. It is the reciprocal of the first distance between the panoramic camera 101 and the inspection object P when the panoramic camera 101 is located at the desired inspection point B1, which is obtained in advance by the robot 100 through the distance sensor.
[0246] In the embodiment of the present application, the cloud management platform 200 can refer to the desired area of the inspection object photographed at the first position of the desired inspection point and the first shooting distance according to the posture change of the robot 100 (for example, the position change between the first position and the second position), and select the image of the target area of the inspection object in the second panoramic image taken by the robot 100 at the second position of the actual inspection point. In this way, even if the robot's autonomous navigation is not accurate, so that the robot 100 does not actually reach the desired inspection point when navigating to the desired inspection point, the cloud management platform 200 of the present application can also combine the position error between the desired inspection point and the actual inspection point where the robot 100 is located to obtain the image of the target area of the inspection object with complete information in the second panoramic image taken at the actual inspection point, so that the inspection is more comprehensive and the error of the image collected for the inspection object is reduced.
[0247] In other embodiments, the robot 100 may also navigate to a desired inspection point each time, for example, when performing Figure 4b After S103 and before S104, the inspection data related to the desired inspection point navigated to this time is uploaded to the cloud management platform 200. The inspection data uploaded once may include but is not limited to: a second panoramic image captured related to the actual inspection point (also expressed as related to the first time) and a first satellite signal received at the first time, the first time, etc.
[0248] In this way, the robot 100 can upload the inspection data related to the expected inspection point to the cloud management platform 200 every time it reaches an expected inspection point (the robot 100 thinks that it has reached the expected inspection point, but there is still a certain position error in reality). In this way, the cloud management platform 200 can execute the above-mentioned operation once every time it receives the inspection data. Figure 4b S202 and S203 are shown to obtain a panoramic image (represented by an image of the target area) obtained by inspecting the corresponding inspection object at the desired inspection point, thereby obtaining panoramic images of the inspections of the respective inspection objects at the three desired inspection points of point B1, point B2, and point B3 in turn.
[0249] An embodiment of the present application provides an image processing device 800 . Figure 7 FIG. 8 is a schematic diagram of an exemplary structure of an image processing device 800. Figure 7 The image processing device 800 includes: a configuration module 801, which is used to configure the positions of multiple expected inspection points for the robot, the positions of the expected inspection points are expected positions for recording images of inspection objects when the robot moves along the inspection route, and the robot is equipped with a camera, and the camera is used to capture panoramic images; an expected area acquisition module 802, which is used to acquire an expected area related to the expected inspection point, and the expected area is an area of interest about the inspection object in a first panoramic image captured by the robot at the expected inspection point; an inspection data acquisition module 803, which is used to acquire inspection data of the robot, and the inspection data includes first information for determining the position of the actual inspection point and a second panoramic image related to the actual inspection point captured by the robot, wherein the robot moves to the actual inspection point based on the position of the expected inspection point at a first time; an interested image acquisition module 804, which is used to acquire an image of a target area about the inspection object in the second panoramic image based on the expected area and the first information, and the target area is an area of interest about the inspection object in the second panoramic image.
[0250] In a possible embodiment, the device 800 also includes: a distance acquisition module 805, used to acquire a first distance related to the expected checkpoint, wherein the first distance is the distance between the camera and the inspection object when the robot is located at the expected checkpoint; the image of interest acquisition module 804, specifically used to: determine a target area regarding the inspection object in the second panoramic image based on a positional relationship between the position of the expected checkpoint and the position of the actual checkpoint, the expected area and the first distance, wherein the position of the actual checkpoint is determined based on the first information; based on the target area, obtain an image of the target area of the inspection object in the second panoramic image.
[0251] In a possible implementation, the robot is equipped with a receiver of a global navigation satellite system, and the receiver is used to receive satellite signals. The first information includes a first satellite signal received by the robot at the first time. The device 800 also includes: a signal acquisition module 806, used to acquire a second satellite signal received by the base station at the first time; the image of interest acquisition module 804 is specifically used to process the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
[0252] In a possible implementation, the signal acquisition module 806 is specifically used to: receive multiple reference satellite signals from a base station and a second receiving time of each of the reference satellite signals; and obtain a reference satellite signal whose second receiving time is the first time among the multiple reference satellite signals as the second satellite signal.
[0253] In a possible implementation, the inspection data acquisition module 803 is specifically used to: obtain the first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, multiple candidate panoramic images taken during the robot's inspection process and the shooting time of each candidate panoramic image, multiple candidate satellite signals received during the robot's inspection process and the first receiving time of each candidate satellite signal; obtain at least one candidate panoramic image among the multiple candidate panoramic images whose shooting time matches the first time, and determine a second panoramic image related to the actual checkpoint taken by the robot based on the at least one candidate panoramic image; obtain a candidate satellite signal among the multiple candidate satellite signals whose first receiving time is the first time as the first information for determining the position of the actual checkpoint.
[0254] In a possible implementation, the image processing device 800 is deployed on a cloud management platform, which is used to manage infrastructure for providing cloud services. The infrastructure includes multiple cloud data centers located in different regions, and each region is provided with at least one cloud data center.
[0255] The effects of the image processing device 800 in the above-mentioned embodiments are similar to the effects of the image processing methods in the above-mentioned embodiments, and will not be described in detail here.
[0256] In a possible implementation, the embodiment of the present application provides an inspection system 700 . Figure 8 The schematic diagram of the structure of the inspection system 700 is shown as an example. Figure 8 The system 700 includes the image processing device 800 and the robot 100 described in any one of the above embodiments.
[0257] In a possible implementation manner, the inspection system 700 further includes a base station 300 .
[0258] In one possible embodiment, the robot 100 is used to capture the multiple candidate panoramic images through the camera when patrolling within a preset distance range of the location of the expected checkpoint along the inspection route, and record the shooting time of each candidate panoramic image, as well as the first time when the robot moves to the actual checkpoint based on the location of the expected checkpoint.
[0259] In a possible implementation manner, the robot 100 is further configured to send inspection data to the image processing device 800 after completing the inspection along the inspection route.
[0260] In a possible implementation, the camera of the robot 100 has at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
[0261] In a possible implementation, the robot 100 is further used to: during an inspection process according to an inspection route, obtain at least two first images by capturing them through the at least two cameras of the camera; and combine the at least two first images into a panoramic image.
[0262] Among them, the above modules can all be implemented by software, or can be implemented by hardware. Among them, the module is an example of a software functional unit, and the above module may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the image acquisition module 804 of interest may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.
[0263] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0264] As an example of a hardware functional unit, the module may include at least one computing device, such as a server, etc. Alternatively, the module may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0265] The multiple computing devices included in the above-mentioned image processing apparatus 800 can be distributed in the same region or in different regions. The multiple computing devices included in the above-mentioned image processing apparatus 800 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the above-mentioned image processing apparatus 800 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0266] It should be noted that, in other embodiments, the above modules can be used to execute corresponding steps in the image processing method to realize all functions of the image processing device 800.
[0267] The present application also provides a computing device 900. Fig. 9 As shown, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 909. The processor 904, the memory 906, and the communication interface 909 communicate with each other through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.
[0268] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 902 may include a path for transmitting information between various components of the computing device 900 (eg, the memory 906, the processor 904, and the communication interface 909).
[0269] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0270] The memory 906 may include a volatile memory, such as a random access memory (RAM). The processor 904 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0271] The memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned configuration module 801, the desired area acquisition module 802, the inspection data acquisition module 803, the image of interest acquisition module 804, the distance acquisition module 805, and the signal acquisition module 806, thereby implementing the image processing method. That is, the memory 906 stores instructions for executing the image processing method.
[0272] The communication interface 909 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.
[0273] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.
[0274] like Fig.10 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the image processing method.
[0275] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, the combination of one or more computing devices 1000 may jointly execute instructions for executing the image processing method.
[0276] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the image processing apparatus 800. That is, the instructions stored in the memory 1006 in different computing devices 1000 may implement the functions of one or more modules among the configuration module 801, the desired area acquisition module 802, the inspection data acquisition module 803, the image of interest acquisition module 804, the distance acquisition module 805, and the signal acquisition module 806.
[0277] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11 As shown, two computing devices 1100A and 1100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1106 in the computing device 1100A stores instructions for executing the functions of the configuration module 801 and the desired area acquisition module 802. At the same time, the memory 1106 in the computing device 1100B stores instructions for executing the functions of the inspection data acquisition module 803, the image of interest acquisition module 804, the distance acquisition module 805, and the signal acquisition module 806.
[0278] It should be understood that Fig.11 The functions of the computing device 1100A shown in FIG. 1100A may also be completed by multiple computing devices 1100. Similarly, the functions of the computing device 1100B may also be completed by multiple computing devices 1100.
[0279] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.11 The connection mode of the computing device cluster is different in that the memory 1106 in one or more computing devices 1100 in the computing device cluster may store the same instructions for executing the image processing method.
[0280] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, the combination of one or more computing devices 1100 may jointly execute instructions for executing the image processing method.
[0281] It should be noted that the memory 1106 in different computing devices 1100 in the computing device cluster may store different instructions for executing partial functions of the image processing apparatus 800. That is, the instructions stored in the memory 1106 in different computing devices 1100 may implement the functions of one or more devices in the image processing apparatus 800.
[0282] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the image processing method in the above embodiment.
[0283] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the image processing method in the above embodiment.
[0284] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image processing method, It is characterized in that The method comprises: Configuring the positions of a plurality of desired inspection points for the robot, wherein the positions of the desired inspection points are desired positions for recording images of inspection objects when the robot moves, and the robot is equipped with a camera for capturing panoramic images; Acquire an expected area related to the expected inspection point, where the expected area is an area of interest about the inspection object in a first panoramic image taken by the robot at the expected inspection point; Acquire inspection data of the robot, the inspection data comprising first information for determining a position of an actual inspection point and a second panoramic image taken by the robot and related to the actual inspection point, wherein the robot moves to the actual inspection point based on the position of the expected inspection point; Based on the expected area and the first information, an image of a target area of the inspection object in the second panoramic image is acquired, where the target area is a region of interest of the inspection object in the second panoramic image.
2. The method according to claim 1, It is characterized in that The position of the expected inspection point and the position of the actual inspection point are three-dimensional positions. Before obtaining the inspection data of the robot, the method further includes: Acquire a first distance associated with the desired inspection point, wherein the first distance is a distance between the camera and the inspection object when the robot is located at the desired inspection point; The acquiring, based on the expected area and the first information, an image of the target area of the inspection object in the second panoramic image comprises: determining a target area with respect to the inspection object in the second panoramic image based on a positional relationship between the position of the expected inspection point and the position of the actual inspection point, the expected area, and the first distance, wherein the position of the actual inspection point is determined based on the first information; Based on the target area, an image of the target area of the inspection object is obtained in the second panoramic image.
3. The method according to claim 1 or 2, It is characterized in that The robot moves to the actual checkpoint based on the position of the expected checkpoint at a first time, the robot is equipped with a receiver of a global navigation satellite system, the receiver is used to receive satellite signals, the first information includes a first satellite signal received by the robot at the first time, and the method further includes: Acquire a second satellite signal received by the base station at the first time; The first satellite signal and the second satellite signal are processed by a dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
4. The method according to claim 3, It is characterized in that The acquiring the second satellite signal received by the base station at the first time includes: receiving a plurality of reference satellite signals and a second receiving time of each of the reference satellite signals from a base station; A reference satellite signal whose second receiving time is the first time among the multiple reference satellite signals is obtained as the second satellite signal.
5. The method according to claim 3 or 4, It is characterized in that The obtaining of the inspection data of the robot comprises: Acquire a first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, a plurality of candidate panoramic images captured by the robot during the inspection process and the shooting time of each candidate panoramic image, a plurality of candidate satellite signals received by the robot during the inspection process and the first receiving time of each candidate satellite signal; Acquire at least one candidate panoramic image of the plurality of candidate panoramic images whose shooting time matches the first time, and determine a second panoramic image taken by the robot and related to the actual checkpoint based on the at least one candidate panoramic image; A candidate satellite signal whose first receiving time is the first time among the multiple candidate satellite signals is obtained as first information for determining the position of the actual checkpoint.
6. The method according to claim 5, It is characterized in that Before obtaining the inspection data of the robot, the method further includes: When the robot patrols within a preset distance range of the location of the expected checkpoint, the robot captures the multiple candidate panoramic images through the camera, records the shooting time of each candidate panoramic image, and records the first time when the robot moves to the actual checkpoint based on the location of the expected checkpoint.
7. The method according to any one of claims 1 to 6, It is characterized in that The obtaining of the inspection data of the robot comprises: After the robot completes the inspection, inspection data is received from the robot.
8. The method according to any one of claims 1 to 7, It is characterized in that The camera has at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
9. The method according to claim 8, It is characterized in that The method further comprises: During the inspection process, the robot obtains at least two first images by photographing the at least two cameras of the camera; The robot combines the at least two first images into a panoramic image.
10. An image processing device, It is characterized in that The device comprises: a configuration module, configured to configure the positions of a plurality of desired inspection points for the robot, wherein the positions of the desired inspection points are desired positions for recording images of inspection objects when the robot moves, and the robot is equipped with a camera for capturing panoramic images; An expected area acquisition module, used to acquire an expected area related to the expected inspection point, wherein the expected area is an area of interest about the inspection object in the first panoramic image taken by the robot at the expected inspection point; an inspection data acquisition module, configured to acquire inspection data of the robot, the inspection data comprising first information for determining a position of an actual inspection point and a second panoramic image taken by the robot and related to the actual inspection point, wherein the robot moves to the actual inspection point based on the position of the desired inspection point; An interested image acquisition module is used to acquire an image of a target area of the inspection object in the second panoramic image based on the expected area and the first information, wherein the target area is an interested area of the inspection object in the second panoramic image.
11. The device according to claim 10, It is characterized in that The device also includes: a distance acquisition module, configured to acquire a first distance associated with the desired inspection point, wherein the first distance is a distance between the camera and the inspection object when the robot is located at the desired inspection point; The image of interest acquisition module is specifically used for: determining a target area with respect to the inspection object in the second panoramic image based on a positional relationship between the position of the expected inspection point and the position of the actual inspection point, the expected area, and the first distance, wherein the position of the actual inspection point is determined based on the first information; Based on the target area, an image of the target area of the inspection object is obtained in the second panoramic image.
12. The device according to claim 10 or 11, It is characterized in that The robot moves to the actual checkpoint based on the position of the expected checkpoint at a first time, the robot is equipped with a receiver of a global navigation satellite system, the receiver is used to receive satellite signals, the first information includes a first satellite signal received by the robot at the first time, and the device further includes: a signal acquisition module, configured to acquire a second satellite signal received by the base station at the first time; The image of interest acquisition module is specifically used to process the first satellite signal and the second satellite signal through dynamic post-processing technology to determine the position of the actual checkpoint where the robot is located at the first time.
13. The device according to claim 12, It is characterized in that The signal acquisition module is specifically used for: receiving a plurality of reference satellite signals and a second receiving time of each of the reference satellite signals from a base station; A reference satellite signal whose second receiving time is the first time among the multiple reference satellite signals is obtained as the second satellite signal.
14. The device according to claim 12 or 13, It is characterized in that The inspection data acquisition module is specifically used for: Acquire a first time when the robot moves to the actual checkpoint based on the position of the expected checkpoint, a plurality of candidate panoramic images captured by the robot during the inspection process and the shooting time of each candidate panoramic image, a plurality of candidate satellite signals received by the robot during the inspection process and the first receiving time of each candidate satellite signal; Acquire at least one candidate panoramic image of the plurality of candidate panoramic images whose shooting time matches the first time, and determine a second panoramic image taken by the robot and related to the actual checkpoint based on the at least one candidate panoramic image; A candidate satellite signal whose first receiving time is the first time among the multiple candidate satellite signals is obtained as first information for determining the position of the actual checkpoint.
15. A patrol inspection system, It is characterized in that The system comprises the image processing device according to any one of claims 10 to 14 and the robot.
16. The system according to claim 15, It is characterized in that The robot is used to capture the multiple candidate panoramic images through the camera when patrolling within a preset distance range of the location of the expected checkpoint, and record the shooting time of each candidate panoramic image, as well as the first time when the robot moves to the actual checkpoint based on the location of the expected checkpoint.
17. The system according to claim 15 or 16, It is characterized in that The robot is also used to send inspection data to the image processing device after the inspection is completed.
18. A system according to any one of claims 15 to 17, It is characterized in that The camera of the robot comprises at least two cameras with different shooting angles of view, and the sum of the shooting angles of view of the at least two cameras is 360 degrees.
19. The system according to claim 17, It is characterized in that The robot is also used to: During the inspection process, at least two first images are obtained by photographing by the at least two cameras of the camera; The at least two first images are combined into a panoramic image.
20. A device cluster, It is characterized in that comprising at least one processing device, each processing device comprising a processor and a memory; The processor of the at least one processing device is configured to execute instructions stored in the memory of the at least one processing device, so that the device cluster executes the method according to any one of claims 1 to 9.
21. A computer program product comprising instructions, It is characterized in that When the instructions are executed by a device cluster, the device cluster executes the method according to any one of claims 1 to 9.
22. A computer-readable storage medium, It is characterized in that The method comprises computer program instructions. When the computer program instructions are executed by a device cluster, the device cluster performs the method according to any one of claims 1 to 9.
