Field inspection method and device based on quadruped robot and quadruped robot

Abstract

The present application relates to a field inspection method, device and quadruped robot based on a quadruped robot. The method includes: determining the position information and posture information of key inspection points; obtaining the field environment image collected by the quadruped robot during the field inspection based on the position information and posture information of the key inspection points; segmenting the inspection path area image from the field environment image, and determining the obstacle point information from the inspection path area image; performing local path planning based on the obstacle point information in combination with the dynamic window method; local path planning is used to guide the quadruped robot to inspect along the center line of the inspection path; based on the path of local path planning, the field inspection site is inspected in combination with the position information and posture information of the key inspection points. The present application can correct deviations in real time, thereby improving the inspection navigation accuracy and ensuring that the quadruped robot can inspect along the ideal route.

Description

Technical Field

[0001] The present application relates to the fields of agricultural automation and robotics, artificial intelligence fields such as deep learning, and in particular to a quadruped robot for autonomous field inspections, and more particularly to a field inspection robot that integrates GPS and visual information to achieve real-time local path planning. Background Art

[0002] In recent years, the application of robot technology in the agricultural field has gradually increased. Most of the robots used for inspection in the existing agricultural industry are wheeled or tracked. Compared with other types of robots, wheeled or tracked robots can operate in a wider range of environments. However, wheeled robots have problems such as poor climbing ability, obstacle crossing and ditch crossing ability. Tracked robots have greater sliding and steering resistance, greater motion loss, and higher cost. As can be seen from the above, wheeled or tracked robots have relatively weak generalization ability and scalability.

[0003] In addition, existing agricultural robots have the following problems in field inspections: the navigation methods used are mostly relatively simple and are easily affected by the environment, such as light, rain and snow; the accuracy of positioning and navigation technology is related to the atmospheric environment, buildings, and the equipment's own performance configuration. Summary of the invention

[0004] The present application provides a field inspection method and device based on a quadruped robot, and a quadruped robot.

[0005] According to a first aspect of an embodiment of the present application, a field inspection method based on a quadruped robot is provided, comprising the following steps:

[0006] Determine the position information and posture information of the key inspection points, wherein the position information and posture information of the key inspection points are obtained based on the fusion of multi-sensor data collected at the field inspection site, wherein the multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data;

[0007] During the field inspection by the quadruped robot based on the position information and posture information of the inspection key points, obtaining a field environment image collected by the quadruped robot;

[0008] Segmenting an inspection path area image from the field environment image, and determining obstacle point information from the inspection path area image;

[0009] According to the obstacle point information, local path planning is performed in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path;

[0010] Based on the path planned by the local path, the field inspection site is inspected in combination with the position information and posture information of the key inspection points.

[0011] According to a second aspect of an embodiment of the present application, a field inspection device based on a quadruped robot is provided, comprising:

[0012] A first determination module is used to determine the position information and posture information of the inspection key points, where the position information and posture information of the inspection key points are obtained based on the fusion of multi-sensor data collected at the field inspection site, where the multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data;

[0013] An acquisition module, used for acquiring a field environment image collected by the quadruped robot during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points;

[0014] A second determination module is used to segment an inspection path area image from the field environment image, and determine obstacle point information from the inspection path area image;

[0015] A path planning module, used to perform local path planning based on the obstacle point information in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path;

[0016] The inspection module is used to inspect the field inspection site based on the path planned by the local path and in combination with the position information and posture information of the inspection key points.

[0017] According to a third aspect of an embodiment of the present application, there is provided a quadruped robot, comprising:

[0018] at least one processor;

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0021] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed on a quadruped robot, the quadruped robot executes the method described in the first aspect above.

[0022] According to a fifth aspect of an embodiment of the present application, a program product is provided, comprising at least one of a program or an instruction, wherein at least one of the program or the instruction, when executed by a quadruped robot, implements the steps of the method described in the first aspect.

[0023] The technical solution provided by the embodiments of the present application may include the following beneficial effects: navigation based on GPS and visual fusion, in the stage of GPS key point information collection, multi-sensor data is integrated to improve the accuracy of GPS key points; in the visual navigation stage, based on deep learning to segment the inspection path area, and then combined with DWA for local path planning, it can be helpful to avoid the deviation of the inspection direction of the quadruped robot, improve the inspection navigation accuracy, and ensure that the quadruped robot can inspect along the ideal route. In addition, the present application uses a bionic quadruped robot for research and development. The robot's appearance adopts a dog-like structure, and the movement is carried out by four dog-like legs. Its joints can be adjusted at any time, and it can achieve functions such as crossing obstacles and ditches during the inspection process; in addition, the quadruped robot can make its operation more stable during the inspection process, can adapt to different agricultural inspection environments more quickly, has a strong generalization ability, and its functions also have good expansibility.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] Figure 1 The present invention is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment.

[0027] Figure 2 The present invention is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment.

[0028] Figure 3 The present invention is a flow chart of a field navigation method based on GPS and vision fusion according to an exemplary embodiment.

[0029] Figure 4 It is a block diagram of a field inspection device based on a quadruped robot according to an exemplary embodiment.

[0030] Figure 5 is a block diagram of a quadruped robot 500 for field inspection according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0033] It is worth noting that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0034] First, the terms used in this article are explained accordingly.

[0035] 1. FCN (Fully Convolution Networks)

[0036] FCN is a framework for image semantic segmentation. Replacing the fully connected layer of the traditional CNN (Convolutional Neural Networks) structure with a convolutional layer and using upsampling to restore the image size can effectively avoid the problem of image size reduction caused by convolution and pooling.

[0037] 2. DWA (Dynamic Window Approach)

[0038] DWA is a local path planning method and is the main method used in ROS (Robot Operating System). It mainly uses multiple groups of speeds in the speed space, simulates the movement trajectories of these speeds within a certain period of time, and then scores these trajectories through an evaluation function. The optimal trajectory and its corresponding speed will be output to the robot platform. Based on the actual research and development situation of this application, the quadruped robot involved in this application has a constant movement speed during the inspection process. The DWA evaluation function part is improved, the evaluation indicators and weights are adjusted, and local path planning is performed for the constant speed of this application.

[0039] 3. DiffusionEdge

[0040] DiffusionEdge is an edge detection method based on the probability diffusion model, designed for accurate edge detection. DiffusionEdge has led a new trend in image processing through its innovative technical means. The core of DiffusionEdge is the probability diffusion model, which uses incremental noise introduction and subsequent back-propagation process to learn complex image features, especially in edge detection. Compared with traditional edge detection algorithms, DiffusionEdge can handle edge blurring more delicately and strengthen edges while retaining details. DiffusionEdge trains a diffusion model with a decoupled structure in the latent space through technical structure designs such as adaptive frequency filtering and uncertainty distillation. Frequency analysis is performed with an adaptive filter to retain multiple pixel-level uncertainty information and reduce the requirements for computing resources; cross entropy loss is used in a distillation manner to optimize the latent space. Without post-processing, edge maps that meet both accuracy and clarity can be generated.

[0041] Most of the robots currently used for inspection in the agricultural industry are wheeled or tracked. Compared with other types of robots, wheeled or tracked robots can operate in a wider range of environments. However, wheeled robots have problems such as poor climbing ability, obstacle crossing and ditch crossing ability. Tracked robots have greater sliding and steering resistance, greater motion loss, and higher cost. As can be seen from the above, wheeled or tracked robots have relatively weak generalization and scalability.

[0042] The main navigation technologies currently used include visual navigation, SLAM (Simultaneous Localization and Mapping) navigation and other technologies. Most of the navigation methods used are relatively simple and are easily affected by the environment, such as light, rain and snow; the accuracy of positioning and navigation technology is related to the atmospheric environment, buildings, and the performance configuration of the equipment itself. As can be seen from the above, the accuracy of the navigation technology currently used is greatly affected by the outside world. In an environment that may be dynamically changing, it is difficult to achieve accurate navigation and real-time path planning.

[0043] Based on this, the present application proposes a field inspection method based on a quadruped robot, which is developed by selecting a quadruped robot with a bionic form. The robot's appearance adopts a dog-like structure, and the movement is carried out by the four legs of the dog-like form. The robot's four-legged structure, its joints can be adjusted at any time, and can achieve functions such as crossing obstacles and ditches during the inspection process. In addition, due to the low center of gravity of the quadruped robot, it makes its operation more stable during the inspection process, can adapt to different agricultural inspection environments more quickly, has a strong generalization ability, and its functions also have good expansibility. In addition, the present application develops a quadruped robot navigation system by integrating GPS positioning navigation and visual navigation technology. Combined with multi-sensor data, the key point positioning of the inspection is realized; during the inspection process, the visual sensor is enabled, the inspection section is segmented based on deep learning and the center position of the inspection section is extracted, the obstacle site is sampled, and then combined with the key points of GPS positioning for real-time fusion, and the deviation is corrected in real time, so as to improve the inspection navigation accuracy and ensure that the quadruped robot inspects along the ideal route.

[0044] Figure 1 FIG. 1 is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment. Figure 1 As shown, the field inspection method based on the quadruped robot may include but is not limited to the following steps.

[0045] In step 101, the position information and posture information of the inspection key points are determined.

[0046] In some embodiments, the location information and posture information of the above-mentioned key inspection points can be obtained based on the fusion of multi-sensor data collected in the field inspection site. The multi-sensor data may include but is not limited to GPS (Global Positioning System) data, IMU (Inertial Measurement Unit) data and odometer (Odom) data, etc.

[0047] In some embodiments, before the quadruped robot inspects, key point data information can be collected in advance at the field inspection site. In some embodiments, the quadruped robot can be equipped with RTK (Real-Time Kinematic, real-time dynamic carrier phase difference technology)-GPS equipment, inertial sensors, and odometers to collect GPS, IMU, and Odom data. The robot_localization (a series of robot state estimation node geometry) package based on extended Kalman filtering is used to fuse GPS data, IMU data, and Odom data to establish the location information and posture information of the key inspection points, so that the quadruped robot can inspect along the key points.

[0048] In some embodiments, the optional implementation of the inspection key point positioning method can be as follows: first, conduct a GPS precision calibration experiment. For example, conduct RTK-GPS error calibration experiments in three groups of 3 meters, 10 meters, and 30 meters, optimize the error calculation method, and control the GPS error within 5 centimeters; carry out key point related data collection along the target inspection route of the field inspection site with RTK-GPS and multi-sensor equipment, and record the key point trajectory information with RTK-GPS equipment, obtain IMU data with inertial sensors, and obtain Odom data with odometers; integrate GPS data, IMU data, and Odom data of key points based on the robot state estimation node (EKF_localization_node) and navigation satellite conversion node (navsat_transform_node) in robot_localization to make the key point location information more accurate.

[0049] Exemplarily, configure RTK-GPS equipment to collect GPS data and record the information of the quadruped robot at key points, including longitude and latitude information. The positioning accuracy of the GPS positioning system is not high enough when used alone, and the accuracy is also easily affected by the weather. Therefore, the quadruped robot's own inertial sensor and odometer are integrated to assist in positioning, improve the reliability and positioning accuracy of inspections using key points, and synchronously record the position information of the quadruped robot when recording the GPS information of key points. The key point related information collected above is transmitted to the Raspberry Pi module in real time, and the Raspberry Pi then sends the received key point information to the cloud server for data processing and storage. Taking into account the WIFI coverage, this application can use a mobile data (such as 4G, 5G, etc.) communication module as a networking tool for Raspberry Pi to transmit data.

[0050] Since GPS information will be affected by clouds and the environment, resulting in positioning drift, and Odom data will produce cumulative errors, in order to ensure that the quadruped robot walks accurately on the right road, data fusion is required to reduce the hardware or other factor errors caused by a single sensor to improve the accuracy of key point location information. Before fusing the above collected location information data, the coordinate data format needs to be unified. In one possible implementation method, the GPS data can be converted from the spherical coordinate system to the Universal Transverse Mercator Grid System UTM coordinate system through navsat_transform_node; based on the translation and rotation of the coordinate system, the GPS data in the UTM coordinate system is converted to the coordinate system of the quadruped robot's working environment; the GPS data, IMU data and odometer data in the coordinate system of the quadruped robot's working environment are fused through EKF_localization_node to obtain the location information and posture information of the key inspection points.

[0051] Exemplarily, in this application, GPS obtains two-dimensional geographic location coordinates including longitude and latitude. The coordinate system data format is inconsistent with the coordinate system in the working environment of the quadruped robot, and the GPS data needs to be formatted. The Cartesian coordinate system is used in the working environment of the quadruped robot. When performing navigation and path planning, the spherical coordinates based on longitude and latitude need to be converted into a plane coordinate system. Considering the computational complexity, navigation accuracy and compatibility, the UTM (Universal Transverse Mercator Grid System) coordinate system is in meters, which can measure distance more directly. In addition, the UTM coordinate system uses the transverse Mercator projection method, which can effectively reduce the degree of distortion of regional conversion, and its coordinates are more compatible with maps and navigation systems. Therefore, in the navsat_transform_node, this application receives GPS data, converts the GPS data into UTM coordinates, and then considers the translation and rotation of the coordinate system and converts it into the coordinate system in the working environment of the quadruped robot.

[0052] The inertial sensor records the angular velocity and acceleration of the quadruped robot along the axis at the key point, and calculates the specific posture of the quadruped robot in space. The odometer can measure the posture data information of the quadruped robot more accurately in a short time. The angular velocity accuracy of the quadruped robot motion information obtained by the odometer is low, so the IMU and Odom data are fused to reduce the positioning error generated by each sensor. Considering that the motion model and measurement model of the quadruped robot are nonlinear during the actual inspection process, this application uses EKF (Extended Kalman Filter) to process multi-sensor data fusion, that is, after completing the GPS coordinate system conversion in navsat_transform_node, the GPS data, IMU data and Odom data are fused through EKF_localization_node, thereby outputting the position information and posture information of the key inspection points.

[0053] In step 102, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, a field environment image collected by the quadruped robot is obtained.

[0054] For example, when the quadruped robot conducts field inspection based on the position information and posture information of key inspection points, the visual sensor on the quadruped robot can be started synchronously to conduct inspection navigation together with the GPS key point information. When the quadruped robot conducts field inspection based on the position information and posture information of key inspection points, the visual sensor can collect field environment images in real time.

[0055] In step 103, an inspection path area image is segmented from the field environment image, and obstacle point information is determined from the inspection path area image.

[0056] In some embodiments, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, the visual sensor on the quadruped robot can be synchronously started, the field environment image collected by the visual sensor can be segmented based on FCN, and the inspection path area image can be segmented from the field environment image.

[0057] It should be noted that, after obtaining the segmented inspection path area image, in order to further improve the accuracy of path edge segmentation and extract edge feature points, edge detection operations can be performed on the inspection path area image segmented based on FCN. In some embodiments, edge detection operations can be performed on the inspection path area image based on an edge detection model. In one possible implementation, edge detection can be performed on the inspection path area image based on a pre-trained edge detection module DiffusionEdge to obtain an edge image of the inspection path area; feature points are extracted from the edge image of the inspection path area, and obstacle point information is determined based on the extracted feature points.

[0058] In step 104, local path planning is performed based on the obstacle point information in combination with a dynamic window method. The local path planning can be used to guide the quadruped robot to perform inspections along the center line of the inspection path.

[0059] In some embodiments, the dynamic window method (DWA) can form a local search space according to the speed space, form multiple motion trajectories according to the obstacle point information, and evaluate the multiple motion trajectories according to the scoring function to obtain the optimal trajectory. In the embodiment of the present application, the change in the forward speed of the quadruped robot is not considered, the quadruped robot performs inspection at the same speed, and the DWA controls the steering of the quadruped robot.

[0060] In an embodiment of the present application, local path planning is performed by improving the DWA evaluation index and weight, so that the local path planning can guide the quadruped robot to patrol according to the center point of the patrol path, thereby ensuring that the quadruped robot navigates and patrols on the center line of the path area.

[0061] In step 105, based on the path planned by the local path, the field inspection site is inspected in combination with the position information and posture information of the key inspection points.

[0062] Exemplarily, when a quadruped robot conducts field inspections, the inspection is carried out by integrating GPS key points and visual navigation methods. Taking rice fields as an example, this application pre-collects GPS key point information on the inspection ridges (including GPS data, IMU data, and Odom data), turns on the visual sensor during the inspection, samples the feature points on the edge of the ridges as obstacle points, and performs local path planning based on the obstacle points combined with DWA. Based on the path planned based on the local path, combined with the position information and posture information of the inspection key points, the quadruped robot is controlled to inspect the field inspection site, thereby realizing field navigation based on the fusion of GPS and vision.

[0063] In the above embodiments, the present application is developed by selecting a quadruped robot with a bionic form. The robot's exterior adopts a dog-like structure, and the movement is carried out by four legs that imitate the dog's form. The robot's quadruped structure has joints that can be adjusted at any time, and can achieve functions such as crossing obstacles and ditches during the inspection process. In addition, due to the low center of gravity of the quadruped robot, it runs more stably during the inspection process, can adapt to different agricultural inspection environments more quickly, has a strong generalization ability, and its functions also have good expansibility. In addition, the present application develops a quadruped robot navigation system by integrating GPS positioning navigation and visual navigation technology. Combined with multi-sensor data, the key point positioning of the inspection is achieved; during the inspection process, the visual sensor is enabled, the inspection section is segmented based on deep learning, and the center position of the inspection section is extracted, the obstacle site is sampled, and then combined with the key points of GPS positioning for real-time fusion, and the deviation is corrected in real time, thereby improving the inspection navigation accuracy and ensuring that the quadruped robot inspects along the ideal route.

[0064] Figure 2 The present invention is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment. Figure 3 FIG. 1 is a flow chart of a field navigation method based on GPS and vision fusion according to an exemplary embodiment. Figure 2 and Figure 3 As shown, the method may include but is not limited to the following steps.

[0065] In step 201, the position information and posture information of the inspection key points are determined.

[0066] In some embodiments, the position information and posture information of the above-mentioned key inspection points are obtained based on the fusion of multi-sensor data collected at the field inspection site, and the multi-sensor data may include GPS data, IMU data and odometer data.

[0067] In the embodiments of the present application, step 201 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0068] In step 202, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, a field environment image collected by the quadruped robot is obtained.

[0069] In the embodiments of the present application, step 202 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0070] In step 203, an inspection path area image is segmented from the field environment image.

[0071] In some embodiments, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, the visual sensor on the quadruped robot can be synchronously started, the field environment image collected by the visual sensor can be segmented based on FCN, and the inspection path area image can be segmented from the field environment image.

[0072] Exemplarily, the FCN network is an improvement on the traditional CNN, which is divided into two parts: full convolution and deconvolution. The full convolution part is some classic CNN network structure, which performs feature extraction; the deconvolution part obtains the semantic segmentation image of the original size through upsampling method. The full convolution part in the FCN network extracts image feature information by continuously downsampling the field environment image, i.e., convolution and pooling operations; the deconvolution part in the FCN network performs upsampling, restores the extracted feature image to the original size, obtains the category corresponding to each pixel, and outputs the segmentation result. In some embodiments, in order to prevent the loss of image details due to the size of the feature map during the upsampling process, a skip structure can be added to the FCN network to fuse features rich in global information and rich in local information, thereby improving the segmentation accuracy of the model.

[0073] In some embodiments, the FCN network can be pre-trained. For example, image data of the field environment can be collected first, and data enhancement methods such as noise enhancement, rotation, and contrast enhancement can be performed on the data to expand the data set used in the model training phase. The training set and the test set are divided into a certain ratio (such as an 8:2 ratio), and learning and training are performed based on the manually annotated paths and the other two types of label graphs. The test set data is used to evaluate the model performance and adjust the model parameters.

[0074] In step 204, edge detection is performed on the inspection path area image based on the pre-trained edge detection module DiffusionEdge to obtain an edge image of the inspection path area.

[0075] In some embodiments, after obtaining the segmented inspection path area image, an edge detection operation may be performed on the inspection path area image. Exemplarily, edge detection may be performed on the inspection path area image based on the pre-trained DiffusionEdge to obtain an edge image of the inspection path area. DiffusionEdge may be an edge detection method based on a probability diffusion model. DiffusionEdge can generate a clear and accurate edge map with limited computing resources without any post-processing.

[0076] In some embodiments, the network structure of DiffusionEdge mainly includes the following parts: Condition Encoder, Adaptive FFT Filter, Latent Space, Uncertainty Distillation, and Decoder. Among them, the Swin Transformer structure can be used as the backbone in the conditional encoder structure, and the input image (i.e., the inspection path area image) can be encoded into a latent feature representation. The use of Swin Transformer can improve the efficiency of the model while maintaining the feature extraction capability, and extract the path edge features more efficiently and accurately. The adaptive FFT filter can filter the features output by the conditional encoder. The features are converted from the spatial domain to the frequency domain by fast Fourier transform, and then the frequency domain features are adaptively filtered using learnable weights, and finally the features are converted back to the spatial domain by inverse Fourier transform. The features processed by the conditional encoder and the adaptive FFT filter are used as the input of the diffusion model in the latent space. The diffusion model denoises the noise features through step-by-step iteration to obtain edge prediction. Uncertainty distillation can avoid the calculation of gradient backpropagation through redundant autoencoders by directly optimizing the pixel-level cross entropy loss into the latent space, while retaining the edge uncertainty information provided by multiple annotators. The decoder can decode the edge prediction obtained by the diffusion model in the latent space back to the original image size to obtain the final edge image.

[0077] For example, to solve the problem that too many sampling steps of the diffusion model lead to too long model inference time, DiffusionEdge can use a decoupled diffusion model structure to accelerate the sampling inference process, where the decoupled forward diffusion process is controlled by a combination of explicit transition probability and standard Wiener process, which can be described as the following Markov chain:

[0078]

[0079] in, is the initial edge, is the noise edge, is the transition function of the reverse edge gradient, t is the time step, I is the identity matrix, () indicates that it obeys the normal distribution. To train the decoupled diffusion model, both the data and the noise components need to be supervised. Therefore, the training objective can be parameterized as follows:

[0080]

[0081] in, θ are the denoising network parameters, n~N ( 0 , I )express The mean is 0, and the covariance matrix is The noise component of the standard normal distribution, It is parameterized as follows: is the identity matrix; is the time step; is a hyperparameter, The training objective is to minimize and Considering the computational cost of the diffusion model, the training process can be transferred to the latent space with a 4-fold downsampled space size.

[0082] For example, in the DiffusionEdge network structure, an autoencoder is first trained, which consists of an encoder for compressing edge reality into latent code and a decoder for recovering edge reality from latent code. In the U-Net stage of training denoising, the network weights can be fixed, and the denoising process can be trained in the latent space to maintain network performance while reducing computing resource consumption. The above process can be expressed by the following formula:

[0083]

[0084]

[0085] in represents the denoising network, is the compressed latent code in the autoencoder, t is the time step.

[0086] In the decoupled structure of DiffusionEdge, an adaptive fast Fourier transform filter (AdaptiveFFT-filter) is introduced to filter different frequency feature modules. The filter is integrated into the denoising network to adaptively filter and separate the edge map and noise components in the frequency domain. Given the encoder features, a two-dimensional fast Fourier transform is performed along the spatial dimension, and then the Fourier transform filter is integrated into the denoising network. Then a learnable weight map is constructed and mapped to the features after Fourier transform. After adaptive filtering, the features are projected from the frequency domain back to the spatial domain through the inverse fast Fourier transform. Finally, the residual connection of the encoder features is used to avoid filtering out useful information. The above process can be expressed as follows:

[0087]

[0088] Where F is the encoder feature, is the feature after Fourier transformation, W is the learnable weight map, is the output feature, Represents the two-dimensional inverse Fourier transform.

[0089] Furthermore, in order to solve the problem of highly unbalanced number of edge pixels and non-edge pixels and the problem of latent space gradient transfer, uncertainty distillation is introduced in DiffusionEdge to directly optimize the gradient of the latent space. The gradient of the uncertainty-aware binary cross entropy loss is directly calculated based on the chain rule. The gradient transfer process skips the autoencoder gradient, optimizes the gradient space, reduces the computational cost, and allows the use of uncertainty-aware loss functions to be directly optimized on the latent variables. The final training objective is as follows:

[0090]

[0091] in, The latent code reconstructed by the decoder The decoded edge, is the uncertainty perception loss, is the time-varying loss weight, is the objective function.

[0092] Finally, considering the size of the training data set, the data of the already labeled inspection path area images is expanded. This application refers to the DiffusionEdge algorithm to randomly flip and scale the data samples for path edge detection training. Based on the trained edge detection model DiffusionEdge, the edge detection result map is output in real time during the inspection process to complete the real-time inspection path area segmentation, thereby obtaining the edge image of the inspection path area.

[0093] In step 205, feature points are extracted from the edge image of the inspection path area, and obstacle point information is determined based on the extracted feature points.

[0094] It should be noted that in order to enable the quadruped robot to walk in the center of the path as much as possible during the inspection process and the trajectory does not deviate towards the direction of the crops, DWA can be used for local path planning. Based on the DWA principle, feature points on the edge of the inspection path area need to be extracted as obstacle point information for the DWA algorithm.

[0095] In some embodiments, the edge image of the inspection path area can be processed by equally spaced horizontal sampling, and the intersection of the equally spaced sampling line and the edge of the inspection path area is regarded as the feature point to be extracted; the coordinate information of the feature point is determined based on the mean method, and the obstacle point information is determined based on the coordinate information of the feature point.

[0096] In a possible implementation, a Cartesian coordinate system can be established based on the edge image of the inspection path area, wherein the lower left corner of the edge image is set as the origin, and the coordinates from bottom to top are Axis positive direction, from left to right The positive direction of the axis; based on the mean method, the mean of the horizontal coordinates of the intersections of the equally spaced sampling lines and the edge of the inspection path area is determined as the horizontal coordinate of the feature point; based on the interval pixel distance between the sampling lines and the extraction order, the vertical coordinate of the feature point is determined.

[0097] In a possible implementation, the calculation formulas for the horizontal and vertical coordinates of the above feature points are expressed as follows:

[0098]

[0099] in, is the horizontal coordinate of the feature point, is the edge of the inspection path area and the sampling line i The horizontal coordinates of the intersection points, is the total number of intersections between the inspection path area edge and the sampling line; is the pixel distance between sampling lines; To extract the order, is the pixel of the edge image. In the image processing and analysis experiment, according to the experimental results, this application takes The value is 10.

[0100] In step 206, local path planning is performed based on the obstacle point information in combination with a dynamic window method. The local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path.

[0101] In some embodiments, the obstacle point information can be integrated into the working environment of the quadruped robot to establish a local environment map; based on the current angular velocity, an angular velocity control is selected for sampling; wherein the velocity space of the dynamic window method is , is a constant linear speed, is the angular velocity; the sampled angular velocity space trajectory is evaluated, and local path planning is performed by adjusting the evaluation indicators and weights of the dynamic window method.

[0102] For example, after obtaining the obstacle point information, the obstacle point information based on the original image (i.e., the coordinates of the obstacle point) can be converted to the working environment of the quadruped robot through the rotation and translation operator to establish a local environment map. Based on the current angular velocity, an angular velocity control is selected for sampling. Here, the velocity space of DWA is , is a constant linear speed, is the angular velocity. Considering the kinematics and obstacle distance constraints, the angular velocity space sampling is performed; the sampled angular velocity space trajectory is evaluated, the DWA evaluation index and weight are improved, and the local path planning is performed to ensure that the quadruped robot performs navigation inspection on the center line of the path area. After obtaining the optimal trajectory through the DWA algorithm, the sport_client interface in the SDK (Software Development Kit) can be called to send the angular velocity command to the quadruped robot platform to realize DWA control of the quadruped robot's motion steering.

[0103] In step 207, the field inspection site is inspected based on the path planned by the local path and combined with the position information and posture information of the key inspection points.

[0104] In the embodiments of the present application, step 207 may be implemented in any of the embodiments of the present application, and the embodiments of the present application do not limit this and will not be described in detail.

[0105] like Figure 3As shown, before the quadruped robot inspects, the present application needs to collect key point data information in advance at the inspection site. In the GPS key point collection stage, key point information can be collected through multiple sensors, the collected GPS data can be converted into the coordinate system of the quadruped robot's working environment, and the GPS data and other sensor data (such as IMU data, Odom data) can be fused to obtain the position information and posture information of the inspection key points. In the process of navigation inspection based on the position information and posture information of the inspection key points of the quadruped robot, due to the error of the GPS key points recorded in advance, its trajectory may not be the ideal path center line. In order to prevent the robot from deviating in the direction of crop planting during the inspection process, the visual sensor is started synchronously during the inspection process, and the field environment image is segmented based on the FCN through the real-time field environment image of the visual sensor, and then the segmented inspection path area image is detected by edge detection. The edge feature points are extracted using the mean method as obstacle points in the DWA algorithm; local real-time path planning is performed based on the DWA algorithm to improve navigation accuracy and ensure that the quadruped robot inspects along the path center line.

[0106] As shown in Table 1 below, the specific configuration of the software and hardware of the training and testing platform of the entire model of the embodiment of the present application is shown. Through the specific configuration of the software and hardware shown in Table 1 below, the technical solution provided by the embodiment of the present application is applied to the actual rice test field operation environment for testing. From the experimental results, it can be seen that the automatic inspection route of the quadruped robot on the rice field ridges achieved by the present application has a small deviation from the ideal path center line, thereby improving the navigation accuracy.

[0107] Table 1 Experimental configuration information

[0108]

[0109] In summary, this application is based on GPS and visual fusion for navigation. In the GPS key point information collection stage, the fusion of multi-sensor data can improve the accuracy of GPS key points; in the visual navigation stage, the inspection path area is segmented based on deep learning, and then combined with DWA for local path planning. The combination of the above technologies can help avoid the deviation of the inspection direction of the quadruped robot, thereby improving the navigation accuracy.

[0110] Figure 4 FIG. 1 is a block diagram of a field inspection device based on a quadruped robot according to an exemplary embodiment. Figure 4 The field inspection device based on the quadruped robot includes a first determination module 401, an acquisition module 402, a second determination module 403, a path planning module 404 and an inspection module 405.

[0111] Among them, the first determination module 401 is used to determine the position information and posture information of the key inspection points. The position information and posture information of the key inspection points are obtained based on the fusion of multi-sensor data collected in the field inspection site. The multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data.

[0112] In some embodiments, the first determination module 401 is used to: convert the GPS data from the spherical coordinate system to the Universal Transverse Mercator Grid UTM coordinate system through the navigation satellite conversion node navsat_transform_node; based on the translation and rotation of the coordinate system, convert the GPS data in the UTM coordinate system into the coordinate system of the quadruped robot working environment; through the robot state estimation node EKF_localization_node, fuse the GPS data, IMU data and odometer data in the coordinate system of the quadruped robot working environment to obtain the position information and posture information of the key inspection points.

[0113] The acquisition module 402 is used to acquire the field environment image collected by the quadruped robot during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points.

[0114] The second determination module 403 is used to segment the inspection path area image from the field environment image, and determine the obstacle point information from the inspection path area image.

[0115] In some embodiments, the second determination module 403 is used to: perform edge detection on the inspection path area image based on a pre-trained edge detection module DiffusionEdge to obtain an edge image of the inspection path area; wherein DiffusionEdge is an edge detection method based on a probability diffusion model; perform feature point extraction on the edge image of the inspection path area, and determine obstacle point information based on the extracted feature points.

[0116] In some embodiments, the second determination module 403 is used to: process the edge image of the inspection path area using equally spaced horizontal sampling, and regard the intersection of the equally spaced sampling line and the edge of the inspection path area as the feature points to be extracted; determine the coordinate information of the feature points based on the mean method, and determine the obstacle point information based on the coordinate information of the feature points.

[0117] In a possible implementation, the second determination module 403 is used to: establish a Cartesian coordinate system based on the edge image of the inspection path area, wherein the lower left corner of the edge image is set as the origin, and the coordinates from bottom to top are Axis positive direction, from left to right The positive direction of the axis; based on the mean method, the mean of the horizontal coordinates of the intersections of the equally spaced sampling lines and the edge of the inspection path area is determined as the horizontal coordinate of the feature point; based on the interval pixel distance between the sampling lines and the extraction order, the vertical coordinate of the feature point is determined.

[0118] In an optional implementation, the calculation formula of the horizontal and vertical coordinates of the feature point is expressed as follows:

[0119]

[0120] in, is the horizontal coordinate of the feature point, is the edge of the inspection path area and the sampling line i The horizontal coordinates of the intersection points, is the total number of intersections between the inspection path area edge and the sampling line; is the pixel distance between sampling lines; To extract the order, is the pixel of the edge image.

[0121] The path planning module 404 is used to perform local path planning based on obstacle point information in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path.

[0122] In some embodiments, the path planning module 404 is used to: integrate the obstacle point information into the working environment of the quadruped robot and establish a local environment map; based on the current angular velocity, select an angular velocity control for sampling; wherein the velocity space of the dynamic window method is , is a constant linear speed, is the angular velocity; the sampled angular velocity space trajectory is evaluated, and local path planning is performed by adjusting the evaluation indicators and weights of the dynamic window method.

[0123] The inspection module 405 is used to inspect the field inspection site based on the path planned by the local path and the position information and posture information of the key inspection points.

[0124] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0125] Figure 5This is a block diagram of a quadruped robot 500 for field inspection according to an exemplary embodiment. For example, the quadruped robot 500 may have a dog-like structure, and move by means of four dog-like legs. The joints of the robot's quadruped structure can be adjusted at any time, and can achieve functions such as crossing obstacles and ditches during the inspection process. In addition, since the center of gravity of the quadruped robot is low, it can run more stably during the inspection process, can adapt to different agricultural inspection environments more quickly, has strong generalization ability, and its functions also have good expansibility.

[0126] Reference Figure 5 The quadruped robot 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0127] The processing component 502 generally controls the overall operation of the quadruped robot 500, such as operations associated with display, data communication, camera operation, and recording operation. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0128] The memory 504 is configured to store various types of data to support the operation of the quadruped robot 500. Examples of such data include instructions for any application or method operating on the quadruped robot 500, GPS data, IMU data, odometer data, location information and posture information of key inspection points, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0129] The power supply assembly 506 provides power to various components of the quadruped robot 500. The power supply assembly 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the quadruped robot 500.

[0130] In some embodiments, the multimedia component 508 may include a visual sensor, and for example, the visual sensor may include a camera. The quadruped robot 500 may use the camera to photograph the field environment.

[0131] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the quadruped robot 500 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0132] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0133] The sensor assembly 514 includes one or more sensors for providing various aspects of status evaluation for the quadruped robot 500. For example, the sensor assembly 514 may include an inertial sensor, an odometer, a GPS positioning system, etc. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0134] The communication component 516 is configured to facilitate wired or wireless communication between the quadruped robot 500 and other devices. The quadruped robot 500 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0135] In an exemplary embodiment, the quadruped robot 500 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned methods.

[0136] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by the processor 520 of the quadruped robot 500 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0137] In an exemplary embodiment, a program product is also provided, including at least one of a program and an instruction, and the at least one of the program and the instruction is executed by the processor 520 of the quadruped robot 500 to complete the above method.

[0138] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0139] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A field inspection method based on a quadruped robot, characterized in that: The following steps are involved: Determine the position information and posture information of the key inspection points, wherein the position information and posture information of the key inspection points are obtained based on the fusion of multi-sensor data collected at the field inspection site, and the multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data; During the field inspection by the quadruped robot based on the position information and posture information of the inspection key points, obtaining a field environment image collected by the quadruped robot; Based on the fully convolutional network FCN, the inspection path area image is segmented from the field environment image, and the inspection path area image is edge detected based on the pre-trained edge detection module DiffusionEdge to obtain the edge image of the inspection path area, and feature points are extracted from the edge image of the inspection path area, and obstacle point information is determined based on the extracted feature points; wherein the FCN adds a skip structure to fuse features rich in global information and rich in local information, and the DiffusionEdge is an edge detection method based on a probability diffusion model; According to the obstacle point information, local path planning is performed in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path; Based on the path planned by the local path, the field inspection site is inspected in combination with the position information and posture information of the inspection key points; Wherein, performing local path planning based on the obstacle point information and in combination with a dynamic window method includes: After integrating the obstacle point information into the working environment of the quadruped robot, a local environment map is established; Based on the current angular velocity, an angular velocity control is selected for sampling; wherein the velocity space of the dynamic window method is , For a constant linear speed, is the angular velocity; The sampled angular velocity space trajectory is evaluated, and local path planning is performed by adjusting the evaluation index and weight of the dynamic window method.

2. The method according to claim 1, characterized in that The extracting feature points from the edge image of the inspection path area and determining the obstacle point information based on the extracted feature points includes: The edge image of the inspection path area is processed by using equally spaced horizontal sampling, and the intersection points of the equally spaced distance sampling lines and the edge of the inspection path area are regarded as feature points to be extracted; The coordinate information of the feature point is determined based on the mean value method, and the obstacle point information is determined based on the coordinate information of the feature point.

3. The method according to claim 2, characterized in that The determining the coordinate information of the feature point based on the mean value method includes: A Cartesian coordinate system is established based on the edge image of the inspection path area, wherein the lower left corner of the edge image is set as the origin, the positive direction of the y-axis is from bottom to top, and the positive direction of the x-axis is from left to right; Based on the mean method, the mean of the horizontal coordinates of the intersections of the equally spaced sampling lines and the edge of the inspection path area is determined as the horizontal coordinate of the feature point; The ordinate of the feature point is determined based on the interval pixel distance between the sampling lines and the extraction order.

4. The method according to claim 3, characterized in that The calculation formula of the horizontal and vertical coordinates of the feature point is as follows: in, is the horizontal coordinate of the feature point, is the first i The horizontal coordinates of the intersection points, is the total number of intersections between the edge of the inspection path area and the sampling line; is the distance in pixels between the sampling lines; is the extraction order, is the pixel of the edge image.

5. The method according to claim 1, characterized in that The location information and posture information of the inspection key points are obtained based on the fusion of multi-sensor data collected at the field inspection site, including: The GPS data is converted from the spherical coordinate system to the Universal Transverse Mercator Grid (UTM) coordinate system through the navigation satellite conversion node navsat_transform_node; Based on the translation and rotation of the coordinate system, the GPS data in the UTM coordinate system is converted into the coordinate system in the working environment of the quadruped robot; The GPS data, the IMU data and the odometer data in the coordinate system of the quadruped robot working environment are fused and processed through the robot state estimation node EKF_localization_node to obtain the position information and posture information of the inspection key points.

6. A field inspection device based on a quadruped robot, characterized in that: include: A first determination module is used to determine the position information and posture information of the inspection key points, where the position information and posture information of the inspection key points are obtained based on the fusion of multi-sensor data collected at the field inspection site, where the multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data; An acquisition module, used for acquiring a field environment image collected by the quadruped robot during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points; The second determination module is used to segment the inspection path area image from the field environment image based on the fully convolutional network FCN, and perform edge detection on the inspection path area image based on the pre-trained edge detection module DiffusionEdge to obtain the edge image of the inspection path area, and extract feature points from the edge image of the inspection path area, and determine the obstacle point information based on the extracted feature points; wherein the FCN adds a skip structure to fuse features rich in global information and rich in local information, and the DiffusionEdge is an edge detection method based on a probability diffusion model; A path planning module, used to perform local path planning based on the obstacle point information in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path; An inspection module, used to inspect the field inspection site based on the path planned by the local path and in combination with the position information and posture information of the inspection key points; The path planning module is used to: integrate the obstacle point information into the working environment of the quadruped robot to establish a local environment map; Based on the current angular velocity, an angular velocity control is selected for sampling; wherein the velocity space of the dynamic window method is , For a constant linear speed, is the angular velocity; The sampled angular velocity space trajectory is evaluated, and local path planning is performed by adjusting the evaluation index and weight of the dynamic window method.

7. A quadruped robot, characterized in that: include: at least one processor; a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A storage medium storing instructions, characterized in that: When the instructions are executed on a quadruped robot, the quadruped robot is caused to execute the method as claimed in any one of claims 1 to 5.

9. A program product, comprising at least one of a program and an instruction, characterized in that: When at least one of the programs and instructions is executed by a quadruped robot, the steps of the method described in any one of claims 1 to 5 are implemented.

Images