Field inspection method and device based on quadruped robot and quadruped robot

Abstract

The invention relates to a field inspection method and device based on a quadruped robot and the quadruped robot. The method comprises the following steps: determining position information and pose information of an inspection key point; in the field inspection process of the quadruped robot based on the position information and the pose information of the inspection key point, acquiring a field environment image acquired by the quadruped robot; segmenting an inspection path area image from the field environment image, and determining obstacle point information from the inspection path area image; performing local path planning by combining a dynamic window method according to the obstacle point information; the local path planning is used for guiding the quadruped robot to inspect according to the inspection path center line; and based on the path of the local path planning, performing routing inspection on the field routing inspection site in combination with the position information and the pose information of the routing inspection key points. According to the invention, the deviation can be corrected in real time, so that the inspection navigation precision is improved, and the quadruped robot can inspect according to an ideal route.

Description

Technical Field

[0001] The present application relates to the fields of agricultural automation and robotics, deep learning and other artificial intelligence fields, and in particular to a quadruped robot for autonomous field inspections, and more particularly to a field inspection robot that integrates GPS and visual information to achieve real-time local path planning. Background Art

[0002] In recent years, the application of robotics in agriculture has steadily increased. Most existing inspection robots in the agricultural sector are wheeled or tracked. Compared to other types of robots, wheeled or tracked robots can operate in a wider range of environments. However, wheeled robots have limited climbing ability and the ability to navigate obstacles and ditches. Tracked robots, on the other hand, have greater sliding and steering resistance, higher motion losses, and higher costs. As can be seen from the above, wheeled or tracked robots suffer from relatively weak generalization and scalability.

[0003] In addition, existing agricultural robots have the following problems in field inspections: the navigation methods used are mostly relatively simple and are easily affected by the environment, such as light, rain and snow; the accuracy of positioning and navigation technology is related to the atmospheric environment, buildings, and the equipment's own performance configuration. Summary of the Invention

[0004] The present application provides a field inspection method and device based on a quadruped robot, and a quadruped robot.

[0005] According to a first aspect of an embodiment of the present application, a field inspection method based on a quadruped robot is provided, comprising the following steps: Determining the position and posture information of key inspection points, where the position and posture information of the key inspection points are obtained based on the fusion of multi-sensor data collected at the field inspection site, where the multi-sensor data includes global positioning system (GPS) data, inertial measurement unit (IMU) data, and odometer data; During the field inspection process performed by the quadruped robot based on the position information and posture information of the inspection key points, obtaining a field environment image collected by the quadruped robot; Segmenting an inspection path area image from the field environment image, and determining obstacle point information from the inspection path area image; Based on the obstacle point information, a local path planning is performed in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path; Based on the path planned by the local path, the field inspection site is inspected in combination with the position information and posture information of the inspection key points.

[0006] According to a second aspect of an embodiment of the present application, a field inspection device based on a quadruped robot is provided, comprising: A first determination module is used to determine the position information and posture information of the key inspection points, where the position information and posture information of the key inspection points are obtained based on the fusion of multi-sensor data collected at the field inspection site, where the multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data, and odometer data; an acquisition module, configured to acquire a field environment image collected by the quadruped robot during a field inspection performed by the quadruped robot based on the position information and posture information of the inspection key points; a second determining module, configured to segment an inspection path area image from the field environment image, and determine obstacle point information from the inspection path area image; A path planning module is used to perform local path planning based on the obstacle point information in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path; An inspection module is used to inspect the field inspection site based on the path planned by the local path and in combination with the position information and posture information of the inspection key points.

[0007] According to a third aspect of the embodiments of the present application, there is provided a quadruped robot, comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0008] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, wherein the storage medium stores instructions. When the instructions are executed on a quadruped robot, the quadruped robot executes the method described in the first aspect above.

[0009] According to a fifth aspect of an embodiment of the present application, a program product is provided, comprising at least one of a program or an instruction, wherein when the at least one of the program or the instruction is executed by a quadruped robot, the steps of the method described in the first aspect are implemented.

[0010] The technical solutions provided by the embodiments of the present application may include the following beneficial effects: navigation based on GPS and visual fusion, in the GPS key point information collection stage, multi-sensor data is integrated to improve the accuracy of GPS key points; in the visual navigation stage, the inspection path area is segmented based on deep learning, and then local path planning is performed in combination with DWA, which can help avoid the deviation of the quadruped robot's inspection direction, improve the inspection navigation accuracy, and ensure that the quadruped robot can inspect along the ideal route. In addition, the present application uses a bionic quadruped robot for research and development. The robot's appearance adopts a dog-like structure, and its movement is carried out by its four dog-like legs. Its joints can be adjusted at any time, and it can achieve functions such as crossing obstacles and ditches during the inspection process. In addition, the quadruped robot can make its operation more stable during the inspection process, can quickly adapt to different agricultural inspection environments, has strong generalization capabilities, and its functions are also well expandable.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 The figure is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment.

[0014] Figure 2 The figure is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment.

[0015] Figure 3 The figure is a flow chart of a field navigation method based on GPS and vision fusion according to an exemplary embodiment.

[0016] Figure 4 The figure is a block diagram of a field inspection device based on a quadruped robot according to an exemplary embodiment.

[0017] Figure 5 is a block diagram of a quadruped robot 500 for field inspection according to an exemplary embodiment. DETAILED DESCRIPTION

[0018] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0019] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0020] It is worth noting that in the embodiments of this application, certain software, components, models, etc. that already exist in the industry may be mentioned. They should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0021] First, the terms involved in this article are explained accordingly.

[0022] 1. FCN (Fully Convolution Networks) FCN is a framework for image semantic segmentation. It replaces the fully connected layers of the traditional CNN (Convolutional Neural Network) architecture with convolutional layers and uses upsampling to restore the image size, effectively avoiding the problem of image size reduction caused by convolution and pooling.

[0023] 2. DWA (Dynamic Window Approach) DWA is a local path planning method primarily used in ROS (Robot Operating System). It primarily uses multiple speed groups in velocity space and simulates the trajectories of these speeds over a specified timeframe. These trajectories are then scored using an evaluation function, and the optimal trajectory and its corresponding speed are output to the robot platform. Based on the actual development of this application, the quadruped robot involved in this application maintains a constant speed during inspections. The DWA evaluation function was improved, with adjustments to the evaluation indicators and weights, to perform local path planning for this application's constant speed scenario.

[0024] 3. DiffusionEdge DiffusionEdge is an edge detection method based on a probability diffusion model, designed for precise edge detection. DiffusionEdge is leading a new trend in image processing through its innovative technical means. The core of DiffusionEdge is the probability diffusion model, which uses the incremental introduction of noise and the subsequent backpropagation process to learn complex image features, especially in edge detection. Compared with traditional edge detection algorithms, DiffusionEdge can handle edge blur issues more delicately and enhance edges while preserving details. DiffusionEdge trains a diffusion model with a decoupled structure in the latent space through technical structural designs such as adaptive frequency filtering and uncertainty distillation. Frequency analysis is performed using adaptive filters to retain multiple pixel-level uncertainty information, reducing the requirements for computing resources; cross-entropy loss is used in a distillation manner to optimize the latent space. Without the need for post-processing, an edge map that satisfies both accuracy and clarity can be generated.

[0025] Most existing inspection robots in the agricultural industry are wheeled or tracked. Compared to other types of robots, wheeled or tracked robots can operate in a wider range of environments. However, wheeled robots have limited climbing ability and the ability to navigate obstacles and ditches. Tracked robots, on the other hand, have greater sliding and steering resistance, high motion losses, and higher costs. As can be seen from the above, wheeled or tracked robots suffer from relatively weak generalization and scalability.

[0026] The main navigation technologies currently in use include visual navigation and SLAM (Simultaneous Localization and Mapping). These navigation methods are often relatively simple and easily affected by environmental factors, such as lighting, rain, and snow. The accuracy of positioning and navigation technology is dependent on the atmospheric environment, buildings, and the device's performance and configuration. As can be seen from the above, the accuracy of currently used navigation technologies is significantly affected by external factors, making it difficult to achieve accurate navigation and real-time path planning in dynamically changing environments.

[0027] Based on this, this application proposes a field inspection method based on a quadruped robot. This method utilizes a bionic quadruped robot. The robot's exterior features a dog-like structure, and its movement is achieved through its four legs. The robot's legs feature adjustable joints, enabling it to navigate obstacles and ditches during inspections. Furthermore, the quadruped robot's low center of gravity makes it more stable during inspections, allowing it to quickly adapt to diverse agricultural inspection environments. It exhibits strong generalization capabilities and offers excellent scalability. Furthermore, this application develops a quadruped robot navigation system by integrating GPS positioning and navigation with visual navigation technologies. This system integrates multi-sensor data to locate key inspection points. During the inspection process, visual sensors are activated, and deep learning is used to segment and extract the center of the inspection route. Obstacle locations are sampled and then integrated with GPS-based key points for real-time fusion and real-time correction. This improves inspection navigation accuracy and ensures the quadruped robot follows the ideal inspection route.

[0028] Figure 1 FIG is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment. Figure 1 As shown, the field inspection method based on the quadruped robot may include but is not limited to the following steps.

[0029] In step 101, the position information and posture information of the inspection key points are determined.

[0030] In some embodiments, the location information and posture information of the above-mentioned inspection key points can be obtained based on the fusion of multi-sensor data collected in the field inspection site. The multi-sensor data may include but is not limited to GPS (Global Positioning System) data, IMU (Inertial Measurement Unit) data and odometer (Odom) data, etc.

[0031] In some embodiments, prior to a quadruped robot's inspection, key point data can be collected at the field inspection site. In some embodiments, the quadruped robot can be equipped with an RTK (Real-Time Kinematic) GPS device, inertial sensors, and an odometer to collect GPS, IMU, and Odom data. A robot_localization package (a series of robot state estimation node geometry) based on extended Kalman filtering is used to fuse GPS, IMU, and Odom data to establish the position and pose information of key inspection points, enabling the quadruped robot to perform inspections along these key points.

[0032] In some embodiments, an optional implementation of the inspection key point positioning method can be as follows: First, conduct a GPS accuracy calibration experiment. For example, three RTK-GPS error calibration tests are conducted at 3, 10, and 30 meter intervals to optimize the error calculation method and control the GPS error to within 5 centimeters. Equipped with RTK-GPS and multi-sensor equipment, key point-related data is collected along the target inspection route in the field inspection area. The RTK-GPS equipment records key point trajectory information, the inertial sensor obtains IMU data, and the odometer obtains Odom data. The robot state estimation node (EKF_localization_node) and the navigation satellite transformation node (navsat_transform_node) in the robot_localization function are used to fuse the GPS data, IMU data, and Odom data of the key points to achieve more accurate key point location information.

[0033] For example, an RTK-GPS device is configured to collect GPS data and record the information of the quadruped robot at key points, including latitude and longitude information. The positioning accuracy of the GPS positioning system is not high enough when used alone, and the accuracy is also easily affected by weather. Therefore, the quadruped robot's own inertial sensor and odometer are integrated to assist in positioning, thereby improving the reliability and positioning accuracy of inspections using key points. When recording the GPS information of key points, the quadruped robot's posture information is recorded synchronously. The key point related information collected above is transmitted to the Raspberry Pi module in real time, and the Raspberry Pi then sends the received key point information to the cloud server for data processing and storage. Taking into account the WIFI coverage, this application can use a mobile data (such as 4G, 5G, etc.) communication module as a networking tool for Raspberry Pi to transmit data.

[0034] Because GPS information can be affected by clouds and the environment, causing positioning drift, and Odom data can produce cumulative errors, data fusion is required to ensure that the quadruped robot accurately walks on the correct path. This reduces hardware or other errors caused by a single sensor, thereby improving the accuracy of key point location information. Before fusing the above collected location information data, the coordinate data format must be unified. In one possible implementation, the GPS data can be converted from a spherical coordinate system to a Universal Transverse Mercator Grid (UTM) coordinate system using the navsat_transform_node. Based on the translation and rotation of the coordinate system, the GPS data in the UTM coordinate system is converted to the coordinate system of the quadruped robot's working environment. The GPS data, IMU data, and odometer data in the coordinate system of the quadruped robot's working environment are fused using the EKF_localization_node to obtain the location and posture information of key inspection points.

[0035] For example, in this application, the GPS obtains two-dimensional geographic location coordinates including longitude and latitude. The coordinate system data format is inconsistent with the coordinate system in the working environment of the quadruped robot, and the GPS data needs to be formatted. The Cartesian coordinate system is used in the working environment of the quadruped robot. When performing navigation and path planning, the spherical coordinates based on longitude and latitude need to be converted into a plane coordinate system. Taking into account the computational complexity, navigation accuracy and compatibility, the UTM (Universal Transverse Mercator Grid System) coordinate system is in meters, which can measure distances more directly. In addition, the UTM coordinate system uses the transverse Mercator projection method, which can effectively reduce the degree of distortion of regional conversion, and its coordinates are more compatible with maps and navigation systems. Therefore, in the navsat_transform_node, this application receives GPS data, converts the GPS data into UTM coordinates, and then considers the translation and rotation of the coordinate system and converts it into the coordinate system in the working environment of the quadruped robot.

[0036] Inertial sensors record the angular velocity and acceleration of the quadruped robot along its axis at key points, calculating the robot's specific posture in space. The odometer can accurately measure the robot's position and posture data in a short period of time. The angular velocity accuracy of the quadruped robot's motion information obtained by the odometer is relatively low, so the IMU and Odom data are fused to reduce the positioning errors generated by each sensor. Considering that the quadruped robot's motion model and measurement model are both nonlinear during actual inspections, this application uses an EKF (Extended Kalman Filter) to process multi-sensor data fusion. Specifically, after completing the GPS coordinate system conversion in the navsat_transform_node, the GPS data, IMU data, and Odom data are fused through the EKF_localization_node to output the position and posture information of key inspection points.

[0037] In step 102, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, a field environment image collected by the quadruped robot is obtained.

[0038] For example, while a quadruped robot is conducting a field inspection based on the position and posture information of key inspection points, the visual sensor on the quadruped robot can be activated synchronously to perform inspection navigation in conjunction with GPS key point information. During the field inspection based on the position and posture information of key inspection points, the visual sensor can capture real-time images of the field environment.

[0039] In step 103, an inspection path area image is segmented from the field environment image, and obstacle point information is determined from the inspection path area image.

[0040] In some embodiments, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, the visual sensor on the quadruped robot can be synchronously started, and the field environment image collected by the visual sensor can be segmented based on FCN, and the inspection path area image can be segmented from the field environment image.

[0041] It should be noted that after obtaining the segmented inspection path area image, in order to further improve the accuracy of path edge segmentation and extract edge feature points, an edge detection operation can be performed on the inspection path area image segmented based on FCN. In some embodiments, an edge detection operation can be performed on the inspection path area image based on an edge detection model. In one possible implementation, edge detection can be performed on the inspection path area image based on a pre-trained edge detection module DiffusionEdge to obtain an edge image of the inspection path area; feature points are extracted from the edge image of the inspection path area, and obstacle point information is determined based on the extracted feature points.

[0042] In step 104, local path planning is performed based on the obstacle point information in combination with a dynamic window method. The local path planning can be used to guide the quadruped robot to perform inspections along the center line of the inspection path.

[0043] In some embodiments, the dynamic window algorithm (DWA) can form a local search space based on the velocity space, generate multiple motion trajectories based on obstacle point information, and evaluate these trajectories using a scoring function to determine the optimal trajectory. In this embodiment, the quadruped robot's forward speed is not considered; the quadruped robot performs inspections at a constant speed, and DWA controls the robot's steering.

[0044] In an embodiment of the present application, local path planning is performed by improving the DWA evaluation index and weight, so that the local path planning can guide the quadruped robot to patrol according to the center point of the patrol path, thereby ensuring that the quadruped robot navigates and patrols on the center line of the path area.

[0045] In step 105, the field inspection site is inspected based on the path planned by the local path and combined with the position information and posture information of the key inspection points.

[0046] For example, when a quadruped robot conducts on-site inspections in the field, the inspection is carried out by integrating GPS key points and visual navigation methods. Taking rice fields as an example, this application pre-collects GPS key point information on the inspection ridges (including GPS data, IMU data and Odom data), turns on the visual sensor during the inspection, samples the edge feature points of the ridges as obstacle points, and performs local path planning based on the obstacle points combined with DWA. Based on the path planned locally, combined with the position information and posture information of the inspection key points, the quadruped robot is controlled to inspect the field inspection site, thereby realizing field navigation based on the fusion of GPS and vision.

[0047] In the above-mentioned embodiments, the present application is developed by selecting a bionic quadruped robot. The robot's exterior adopts a dog-like structure, and its movement is achieved by its four legs that resemble a dog. The joints of the robot's legs can be adjusted at any time, enabling it to perform functions such as crossing obstacles and ditches during inspections. In addition, due to the quadruped robot's low center of gravity, it operates more stably during inspections, can adapt more quickly to different agricultural inspection environments, has strong generalization capabilities, and its functions are also well expandable. In addition, the present application develops a quadruped robot navigation system by integrating GPS positioning navigation and visual navigation technology. Combining multi-sensor data, it achieves the positioning of key points for inspections; during the inspection process, the visual sensor is activated, and the inspection section is segmented based on deep learning, and the center position of the inspection section is extracted. Obstacle locations are sampled, and then combined with the key points of GPS positioning for real-time fusion and real-time correction of deviations, thereby improving the inspection navigation accuracy and ensuring that the quadruped robot inspects along the ideal route.

[0048] Figure 2 The figure is a flow chart of a field inspection method based on a quadruped robot according to an exemplary embodiment. Figure 3 FIG is a flow chart of a field navigation method based on GPS and vision fusion according to an exemplary embodiment. Figure 2 and Figure 3 As shown, the method may include but is not limited to the following steps.

[0049] In step 201, the position information and posture information of the inspection key points are determined.

[0050] In some embodiments, the position information and posture information of the above-mentioned inspection key points are obtained based on the fusion of multi-sensor data collected in the field inspection site, and the multi-sensor data may include GPS data, IMU data and odometer data.

[0051] In the embodiments of the present application, step 201 can be implemented in any of the ways in the embodiments of the present application. The embodiments of the present application do not limit this and will not be described in detail.

[0052] In step 202, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, a field environment image collected by the quadruped robot is obtained.

[0053] In the embodiment of the present application, step 202 can be implemented in any of the ways in the embodiments of the present application. The embodiment of the present application does not limit this and will not be described in detail.

[0054] In step 203, an inspection path area image is segmented from the field environment image.

[0055] In some embodiments, during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points, the visual sensor on the quadruped robot can be synchronously started, and the field environment image collected by the visual sensor can be segmented based on FCN, and the inspection path area image can be segmented from the field environment image.

[0056] For example, the FCN network is an improvement on the traditional CNN and is divided into two parts: full convolution and deconvolution. The full convolution part is some classic CNN network structures, which performs feature extraction; the deconvolution part obtains the original-size semantic segmentation image through upsampling methods. The full convolution part in the FCN network extracts image feature information by continuously downsampling the field environment image, i.e., convolution and pooling operations; the deconvolution part in the FCN network performs upsampling, restores the extracted feature image to the original size, obtains the category corresponding to each pixel, and outputs the segmentation result. In some embodiments, in order to prevent the loss of image details due to the size of the feature map during the upsampling process, a skip structure can be added to the FCN network to fuse features rich in global information and rich in local information, thereby improving the segmentation accuracy of the model.

[0057] In some embodiments, the FCN network can be pre-trained. For example, image data of a field environment can be collected and augmented with methods such as noise reduction, rotation, and contrast enhancement to expand the dataset used for model training. The training and test sets are divided into a specific ratio (e.g., 8:2). Learning and training are performed based on manually annotated paths and two other types of labeled images. The test set data is used to evaluate model performance and adjust model parameters.

[0058] In step 204, edge detection is performed on the inspection path area image based on the pre-trained edge detection module DiffusionEdge to obtain an edge image of the inspection path area.

[0059] In some embodiments, after obtaining the segmented inspection path area image, an edge detection operation can be performed on the inspection path area image. For example, edge detection can be performed on the inspection path area image based on a pre-trained DiffusionEdge algorithm to obtain an edge image of the inspection path area. DiffusionEdge can be an edge detection method based on a probabilistic diffusion model. DiffusionEdge can generate clear and accurate edge maps with limited computing resources, without the need for any post-processing.

[0060] In some embodiments, the DiffusionEdge network architecture primarily includes the following components: a conditional encoder, an adaptive FFT filter, a latent space, uncertainty distillation, and a decoder. The conditional encoder architecture can use a Swin Transformer as its backbone, encoding the input image (i.e., an image of the inspection path area) into a latent feature representation. The Swin Transformer improves model efficiency while maintaining feature extraction capabilities, enabling more efficient and accurate extraction of path edge features. The adaptive FFT filter filters the features output by the conditional encoder. Features are converted from the spatial domain to the frequency domain using a fast Fourier transform. The frequency domain features are then adaptively filtered using learnable weights, and finally converted back to the spatial domain using an inverse Fourier transform. The features processed by the conditional encoder and adaptive FFT filter in the latent space serve as input to the diffusion model. The diffusion model iteratively denoises noisy features to obtain edge predictions. Uncertainty distillation can directly optimize the pixel-level cross entropy loss in the latent space, avoiding the calculation of redundant autoencoder back-propagation gradients while retaining the edge uncertainty information provided by multiple annotators. The decoder can decode the edge predictions obtained by the diffusion model in the latent space back to the original image size to obtain the final edge image.

[0061] For example, to address the problem of excessive number of sampling steps in the diffusion model leading to long model inference time, DiffusionEdge can use a decoupled diffusion model structure to accelerate the sampling inference process. The decoupled forward diffusion process is controlled by a combination of explicit transition probabilities and a standard Wiener process, which can be described as the following Markov chain:

[0062] in, is the initial edge, is the noise edge, is the transition function of the reverse edge gradient, t is the time step, I is the identity matrix, () indicates that it obeys the normal distribution. To train the decoupled diffusion model, it is necessary to supervise both the data and the noise components. Therefore, the training objective can be parameterized as follows:

[0063] in, θ are the denoising network parameters, n~N ( 0 , I )express To obey the mean of 0, the covariance matrix is The noise component of the standard normal distribution, Parameterize it, where is the identity matrix; is the time step; is a hyperparameter, The training objective is to minimize and Considering the computational cost of the diffusion model, the training process can be transferred to the latent space with a 4-fold downsampled space size.

[0064] For example, in the DiffusionEdge network structure, an autoencoder is first trained, which consists of an encoder that compresses edge real-time information into a latent code and a decoder that recovers edge real-time information from the latent code. During the denoising U-Net training phase, the network weights can be fixed and the denoising process can be trained in the latent space, reducing computational resource consumption while maintaining network performance. The above process can be expressed as follows:

[0065]

[0066] in represents the denoising network, is the compressed latent code in the autoencoder, t is the time step.

[0067] In the decoupled structure of DiffusionEdge, an adaptive fast Fourier transform filter (AdaptiveFFT-filter) is introduced to filter different frequency feature modules. The filter is integrated into the denoising network to adaptively filter and separate the edge map and noise components in the frequency domain. Given the encoder features, a two-dimensional fast Fourier transform is performed along the spatial dimension, and the Fourier transform filter is integrated into the denoising network. Then, a learnable weight map is constructed and mapped to the Fourier transformed features. After adaptive filtering, the features are projected from the frequency domain back to the spatial domain through the inverse fast Fourier transform. Finally, the residual connection of the encoder features is used to avoid filtering out useful information. The above process can be expressed as follows:

[0068] Where F is the encoder feature, is the feature after Fourier transformation, W is the learnable weight map, is the output feature, Represents the two-dimensional inverse Fourier transform.

[0069] Furthermore, to address the highly unbalanced number of edge and non-edge pixels and the problem of latent space gradient transfer, uncertainty distillation is introduced in DiffusionEdge to directly optimize the gradient of the latent space. The gradient of the uncertainty-aware binary cross entropy loss is directly calculated based on the chain rule. The gradient transfer process skips the autoencoder gradient, optimizing the gradient space, reducing computational cost, and allowing the use of uncertainty-aware loss functions to directly optimize the latent variables. The final training objective is as follows:

[0070] in, The latent code reconstructed by the decoder The decoded edge, is the uncertainty perception loss, is the time-varying loss weight, is the objective function.

[0071] Finally, considering the size of the training dataset, we augmented the existing labeled inspection path images. Using the DiffusionEdge algorithm, we randomly flipped and scaled the data samples to train path edge detection. Based on the trained DiffusionEdge edge detection model, we output edge detection results in real time during the inspection process, completing real-time inspection path segmentation and ultimately obtaining edge images of the inspection path.

[0072] In step 205 , feature points are extracted from the edge image of the inspection path area, and obstacle point information is determined based on the extracted feature points.

[0073] It should be noted that in order to ensure that the quadruped robot stays as close to the center of the path as possible during inspection and does not deviate toward the crops, DWA can be used for local path planning. Based on the DWA principle, feature points on the edge of the inspection path need to be extracted as obstacle point information for the DWA algorithm.

[0074] In some embodiments, the edge image of the inspection path area can be processed using equally spaced horizontal sampling, and the intersection of the equally spaced sampling line and the edge of the inspection path area is regarded as the feature point to be extracted; the coordinate information of the feature point is determined based on the mean method, and the obstacle point information is determined based on the coordinate information of the feature point.

[0075] In a possible implementation, a Cartesian coordinate system can be established based on the edge image of the inspection path area, wherein the lower left corner of the edge image is set as the origin, and the coordinates from bottom to top are Axis positive direction, from left to right The positive direction of the axis; based on the mean method, the mean of the horizontal coordinates of the intersection of the equally spaced sampling lines and the edge of the inspection path area is determined as the horizontal coordinate of the feature point; based on the interval pixel distance between the sampling lines and the extraction order, the vertical coordinate of the feature point is determined.

[0076] In a possible implementation, the calculation formulas for the horizontal and vertical coordinates of the above feature points are expressed as follows:

[0077] in, is the horizontal coordinate of the feature point, The edge of the inspection path area and the sampling line i The horizontal coordinates of the intersection points, is the total number of intersections between the inspection path area edge and the sampling line; is the pixel distance between sampling lines; To extract the order, is the pixel of the edge image. In the image processing and analysis experiment, according to the experimental results, this application takes The value is 10.

[0078] In step 206, local path planning is performed based on the obstacle point information in combination with a dynamic window method. The local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path.

[0079] In some embodiments, the obstacle point information can be integrated into the working environment of the quadruped robot to establish a local environment map; based on the current angular velocity, an angular velocity control is selected for sampling; wherein the velocity space of the dynamic window method is , For a constant linear velocity, is the angular velocity; the sampled angular velocity space trajectory is evaluated, and local path planning is performed by adjusting the evaluation indicators and weights of the dynamic window method.

[0080] For example, after obtaining the obstacle point information, the obstacle point information based on the original image (i.e., the coordinates of the obstacle point) can be converted to the quadruped robot working environment through the rotation and translation operators to establish a local environment map. Based on the current angular velocity, an angular velocity control is selected for sampling. Here, the velocity space of DWA is , For a constant linear velocity, is the angular velocity. Angular velocity space sampling is performed, taking into account kinematic and obstacle distance constraints. The sampled angular velocity space trajectory is evaluated, and the DWA evaluation metrics and weights are refined to perform local path planning, ensuring that the quadruped robot navigates and inspects along the centerline of the path area. After obtaining the optimal trajectory through the DWA algorithm, the sport_client interface in the SDK (Software Development Kit) is called to send angular velocity commands to the quadruped robot platform, enabling DWA-controlled motion and steering of the quadruped robot.

[0081] In step 207, the field inspection site is inspected based on the path planned by the local path and in combination with the position information and posture information of the key inspection points.

[0082] In the embodiment of the present application, step 207 can be implemented in any of the ways in the embodiments of the present application. The embodiment of the present application does not limit this and will not be described in detail.

[0083] like Figure 3As shown, before the quadruped robot conducts an inspection, the present application requires collecting key point data information at the inspection site. During the GPS key point collection phase, key point information can be collected using multiple sensors. The collected GPS data is converted to the coordinate system of the quadruped robot's working environment. The GPS data and other sensor data (such as IMU data and Odom data) are then fused to obtain the position and posture information of the inspection key points. During the navigation inspection process of the quadruped robot based on the position and posture information of the inspection key points, due to errors in the pre-recorded GPS key points, its trajectory may not be the ideal path centerline. To prevent the robot from deviating toward the crop planting direction during the inspection process, the visual sensor is synchronously activated during the inspection process. The field environment image is segmented based on the FCN, and the segmented inspection path area image is then subjected to edge detection. The mean method is used to extract edge feature points as obstacle points in the DWA algorithm. Local real-time path planning is performed based on the DWA algorithm to improve navigation accuracy and ensure that the quadruped robot conducts inspections along the path centerline.

[0084] Table 1 below shows the specific hardware and software configurations of the training and testing platform for the entire model of this embodiment. Using the hardware and software configurations shown in Table 1, the technical solutions provided by this embodiment were applied to an actual rice field operating environment. The experimental results demonstrate that the quadruped robot's automated inspection route along rice paddies exhibited minimal deviation from the ideal path centerline, improving navigation accuracy.

[0085] Table 1 Experimental configuration information

[0086] In summary, this application uses GPS and vision fusion for navigation. During the GPS keypoint information collection phase, multi-sensor data is integrated to improve GPS keypoint accuracy. During the visual navigation phase, deep learning is used to segment inspection path areas, and then DWA is combined for local path planning. This combination of technologies can help prevent deviations in the quadruped robot's inspection direction, thereby improving navigation accuracy.

[0087] Figure 4 FIG is a block diagram of a field inspection device based on a quadruped robot according to an exemplary embodiment. Figure 4 The field inspection device based on the quadruped robot includes a first determination module 401 , an acquisition module 402 , a second determination module 403 , a path planning module 404 and an inspection module 405 .

[0088] Among them, the first determination module 401 is used to determine the position information and posture information of the key inspection points. The position information and posture information of the key inspection points are obtained based on the fusion of multi-sensor data collected in the field inspection site. The multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data.

[0089] In some embodiments, the first determination module 401 is used to: convert the GPS data from the spherical coordinate system to the Universal Transverse Mercator Grid UTM coordinate system through the navigation satellite conversion node navsat_transform_node; based on the translation and rotation of the coordinate system, convert the GPS data in the UTM coordinate system into the coordinate system of the quadruped robot working environment; through the robot state estimation node EKF_localization_node, fuse the GPS data, IMU data and odometer data in the coordinate system of the quadruped robot working environment to obtain the position information and posture information of the key inspection points.

[0090] The acquisition module 402 is used to acquire the field environment image collected by the quadruped robot during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points.

[0091] The second determining module 403 is configured to segment an inspection path area image from the field environment image and determine obstacle point information from the inspection path area image.

[0092] In some embodiments, the second determination module 403 is used to: perform edge detection on the inspection path area image based on the pre-trained edge detection module DiffusionEdge to obtain an edge image of the inspection path area; wherein DiffusionEdge is an edge detection method based on a probability diffusion model; perform feature point extraction on the edge image of the inspection path area, and determine obstacle point information based on the extracted feature points.

[0093] In some embodiments, the second determination module 403 is used to: process the edge image of the inspection path area using equally spaced horizontal sampling, and regard the intersection of the equally spaced sampling line and the edge of the inspection path area as the feature points to be extracted; determine the coordinate information of the feature points based on the mean method, and determine the obstacle point information based on the coordinate information of the feature points.

[0094] In a possible implementation, the second determining module 403 is configured to: establish a Cartesian coordinate system based on the edge image of the inspection path area, wherein the lower left corner of the edge image is set as the origin, and the coordinates from bottom to top are Axis positive direction, from left to right The positive direction of the axis; based on the mean method, the mean of the horizontal coordinates of the intersection of the equally spaced sampling lines and the edge of the inspection path area is determined as the horizontal coordinate of the feature point; based on the interval pixel distance between the sampling lines and the extraction order, the vertical coordinate of the feature point is determined.

[0095] In an optional implementation, the calculation formulas for the horizontal and vertical coordinates of the feature points are expressed as follows:

[0096] in, is the horizontal coordinate of the feature point, The edge of the inspection path area and the sampling line i The horizontal coordinates of the intersection points, is the total number of intersections between the inspection path area edge and the sampling line; is the pixel distance between sampling lines; To extract the order, is the pixel of the edge image.

[0097] The path planning module 404 is used to perform local path planning based on obstacle point information in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path.

[0098] In some embodiments, the path planning module 404 is used to: integrate the obstacle point information into the quadruped robot working environment and establish a local environment map; based on the current angular velocity, select an angular velocity control for sampling; wherein the velocity space of the dynamic window method is , For a constant linear velocity, is the angular velocity; the sampled angular velocity space trajectory is evaluated, and local path planning is performed by adjusting the evaluation indicators and weights of the dynamic window method.

[0099] The inspection module 405 is used to inspect the field inspection site based on the path planned by the local path and the position information and posture information of the inspection key points.

[0100] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0101] Figure 5This is a block diagram of a quadruped robot 500 for field inspections, according to an exemplary embodiment. For example, the robot can have a dog-like appearance, with its four legs acting as a locomotive for movement. The robot's legs feature adjustable joints, enabling it to navigate obstacles and ditches during inspections. Furthermore, the robot's low center of gravity makes it more stable during inspections, allowing it to quickly adapt to diverse agricultural inspection environments. This provides strong generalization capabilities and broad functionality.

[0102] Reference Figure 5 The quadruped robot 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0103] The processing component 502 generally controls the overall operation of the quadruped robot 500, such as operations associated with display, data communication, camera operation, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 502 may include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0104] The memory 504 is configured to store various types of data to support the operation of the quadruped robot 500. Examples of such data include instructions for any application or method operating on the quadruped robot 500, GPS data, IMU data, odometer data, position information and pose information of key inspection points, images, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0105] The power supply assembly 506 provides power to the various components of the quadruped robot 500. The power supply assembly 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the quadruped robot 500.

[0106] In some embodiments, the multimedia component 508 may include a visual sensor, for example, the visual sensor may include a camera, and the quadruped robot 500 may use the camera to photograph the field environment.

[0107] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) configured to receive external audio signals when the quadruped robot 500 is in an operating mode, such as a call mode, a recording mode, or a speech recognition mode. The received audio signals may be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0108] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0109] The sensor assembly 514 includes one or more sensors for providing various state assessments for the quadruped robot 500. For example, the sensor assembly 514 may include an inertial sensor, an odometer, a GPS positioning system, etc. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0110] The communication component 516 is configured to facilitate wired or wireless communication between the quadruped robot 500 and other devices. The quadruped robot 500 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0111] In an exemplary embodiment, the quadruped robot 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.

[0112] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions. The instructions can be executed by the processor 520 of the quadruped robot 500 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0113] In an exemplary embodiment, a program product is also provided, including at least one of a program and instructions, which is executed by the processor 520 of the quadruped robot 500 to implement the above method.

[0114] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0115] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A field inspection method based on a quadruped robot, characterized in that: The following steps are involved: Determine the position information and posture information of the key inspection points, wherein the position information and posture information of the key inspection points are obtained based on the fusion of multi-sensor data collected at the field inspection site, and the multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data; During the field inspection by the quadruped robot based on the position information and posture information of the inspection key points, obtaining a field environment image collected by the quadruped robot; Segmenting an inspection path area image from the field environment image, and determining obstacle point information from the inspection path area image; According to the obstacle point information, local path planning is performed in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path; Based on the path planned by the local path, the field inspection site is inspected in combination with the position information and posture information of the key inspection points.

2. The method according to claim 1, characterized in that The determining of obstacle point information from the inspection path area image includes: Performing edge detection on the inspection path area image based on a pre-trained edge detection module DiffusionEdge to obtain an edge image of the inspection path area; wherein DiffusionEdge is an edge detection method based on a probability diffusion model; Feature points are extracted from the edge image of the inspection path area, and the obstacle point information is determined based on the extracted feature points.

3. The method according to claim 2, characterized in that The extracting feature points from the edge image of the inspection path area and determining the obstacle point information based on the extracted feature points includes: The edge image of the inspection path area is processed by using equally spaced horizontal sampling, and the intersection points of the equally spaced distance sampling lines and the edge of the inspection path area are regarded as feature points to be extracted; The coordinate information of the feature point is determined based on the mean value method, and the obstacle point information is determined based on the coordinate information of the feature point.

4. The method according to claim 3, characterized in that The determining the coordinate information of the feature point based on the mean value method includes: A Cartesian coordinate system is established based on the edge image of the inspection path area, wherein the lower left corner of the edge image is set as the origin, the positive direction of the y-axis is from bottom to top, and the positive direction of the x-axis is from left to right; Based on the mean method, the mean of the horizontal coordinates of the intersections of the equally spaced sampling lines and the edge of the inspection path area is determined as the horizontal coordinate of the feature point; The ordinate of the feature point is determined based on the interval pixel distance between the sampling lines and the extraction order.

5. The method according to claim 4, characterized in that The calculation formula of the horizontal and vertical coordinates of the feature point is as follows: in, is the horizontal coordinate of the feature point, is the first i The horizontal coordinates of the intersection points, is the total number of intersections between the edge of the inspection path area and the sampling line; is the distance in pixels between the sampling lines; is the extraction order, is the pixel of the edge image.

6. The method according to claim 1, characterized in that The local path planning is performed according to the obstacle point information in combination with a dynamic window method, including: After integrating the obstacle point information into the working environment of the quadruped robot, a local environment map is established; Based on the current angular velocity, an angular velocity control is selected for sampling; wherein the velocity space of the dynamic window method is , is a constant linear speed, is the angular velocity; The sampled angular velocity space trajectory is evaluated, and local path planning is performed by adjusting the evaluation index and weight of the dynamic window method.

7. The method according to claim 1, characterized in that The location information and posture information of the inspection key points are obtained based on the fusion of multi-sensor data collected at the field inspection site, including: The GPS data is converted from the spherical coordinate system to the Universal Transverse Mercator Grid (UTM) coordinate system through the navigation satellite conversion node navsat_transform_node; Based on the translation and rotation of the coordinate system, the GPS data in the UTM coordinate system is converted into the coordinate system in the working environment of the quadruped robot; The GPS data, the IMU data and the odometer data in the coordinate system of the quadruped robot working environment are fused and processed through the robot state estimation node EKF_localization_node to obtain the position information and posture information of the inspection key points.

8. A field inspection device based on a quadruped robot, characterized in that: include: A first determination module is used to determine the position information and posture information of the inspection key points, where the position information and posture information of the inspection key points are obtained based on the fusion of multi-sensor data collected at the field inspection site, where the multi-sensor data includes global positioning system GPS data, inertial measurement unit IMU data and odometer data; An acquisition module, used for acquiring a field environment image collected by the quadruped robot during the field inspection process of the quadruped robot based on the position information and posture information of the inspection key points; A second determination module is used to segment an inspection path area image from the field environment image, and determine obstacle point information from the inspection path area image; A path planning module, used to perform local path planning based on the obstacle point information in combination with a dynamic window method; the local path planning is used to guide the quadruped robot to perform inspections along the center line of the inspection path; The inspection module is used to inspect the field inspection site based on the path planned by the local path and in combination with the position information and posture information of the inspection key points.

9. A quadruped robot, characterized in that: include: at least one processor; a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A storage medium storing instructions, characterized in that: When the instructions are executed on a quadruped robot, the quadruped robot is caused to execute the method as claimed in any one of claims 1 to 7.

11. A program product, comprising at least one of a program and an instruction, characterized in that: When at least one of the programs and instructions is executed by a quadruped robot, the steps of the method described in any one of claims 1 to 7 are implemented.

Images