A mine-used inspection robot vision servo control method based on FPGA

Abstract

The application discloses a kind of based on FPGA's mine inspection robot vision servo control method, it is related to artificial intelligence vision technical field, including, pre-processing image data is input to the inference after training depth neural network model deployed on FPGA, generates target recognition result;With space-time attention method, target recognition result is fused with environmental sensor data in real time, dynamically adjusts the influence of space and time, generates after fusion multi-source sensor data;To the sensor data after fusion is carried out path planning and motion control, combines vision, laser radar and inertial measurement data, path optimization is carried out by Dijkstra algorithm, generates motion control instruction;According to smooth planning path, robot is driven by motion control instruction and moves, and real-time monitoring and acquisition robot position, speed and acceleration, generates execution feedback data.The application improves the accuracy and robustness of path planning and obstacle avoidance decision.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence vision technology, and in particular to a vision servo control method for a mining inspection robot based on FPGA. Background Technology

[0002] In recent years, with the innovation of Field Programmable Gate Arrays (FPGAs) and computer vision algorithms, the functions of mine inspection robots have gradually evolved from simple mechanical inspection to intelligent operation. Early solutions mainly relied on general-purpose CPUs or GPUs, combining image filtering, feature extraction, and PID closed-loop control to identify pipe cracks and panel indicator changes. However, in the low-light, dusty, and electromagnetically interference environment of mines, the processing of high-frame-rate image streams often suffers significant delays due to computational resource bottlenecks, while simultaneously struggling to balance energy consumption and real-time response.

[0003] Furthermore, the matching and fusion of sensor information from radar ranging, inertial measurement, and temperature and humidity acquisition with visual data is often completed across multiple chips or processors, resulting in significant data alignment and transmission delays. This hinders rapid response in path replanning and closed-loop operation. In actual inspections, if pipe leaks or water seepage occur, failure to complete target identification, data synchronization, and trajectory updates within milliseconds may cause the optimal time for hazard avoidance to be missed, leading to safety risks. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a vision servo control method for mining inspection robots based on FPGA, which solves the problem of insufficient real-time performance and accuracy caused by excessive delay in deep inference and multi-source fusion.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a visual servo control method for a mining inspection robot based on FPGA. The method includes: acquiring and preprocessing images of the mining environment to generate preprocessed image data; inputting the preprocessed image data into a trained deep neural network model deployed on the FPGA for inference to generate target recognition results; using a spatiotemporal attention method to fuse the target recognition results with environmental sensor data in real time, dynamically adjusting the influence of space and time to generate fused multi-source sensor data; performing path planning and motion control on the fused sensor data, combining visual, lidar, and inertial measurement data, and optimizing the path using the Dijkstra algorithm to generate motion control commands; driving the robot to move along a smoothly planned path using the motion control commands, and monitoring and acquiring the robot's position, velocity, and acceleration in real time to generate execution feedback data; and processing the execution feedback data in parallel on the FPGA to dynamically optimize motion control parameters and generate adjusted motion control commands.

[0007] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the mining environment images include pipeline appearance images, instrument panel images, roadway wall crack images, water and silt accumulation area images, safety obstacle images, and lighting blind spot images.

[0008] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the specific steps for generating preprocessed image data are as follows: Noise suppression, dark detail enhancement, and dust and haze removal are performed on mining area environmental images to generate dehazed and enhanced images; The dehazing enhanced image is subjected to frequency domain bandstop filtering to generate a stripe suppression image, and multi-scale smoothing and edge enhancement are performed in the spatial domain to generate a detail enhancement image. Extract pipe appearance, instrument panel, cracks, and water accumulation areas from the detailed enhanced image and integrate them to generate preprocessed image data.

[0009] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the trained deep neural network model is trained by supervised learning using mining environment image data, and after training, the deep neural network model structure is pruned and quantized for optimization and then deployed on an FPGA.

[0010] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the specific steps for generating target recognition results are as follows: The preprocessed image data is transmitted to the FPGA via the on-chip data bus, and the trained deep neural network model is used to perform pixel normalization and quantization to generate a quantized pixel matrix. The quantized pixel matrix is ​​subjected to depthwise separable convolution and pooling downsampling to generate a downsampled feature map. The downsampled feature map is input into the fully connected layer of the trained deep neural network model, and the class probability is calculated by approximating Softmax to generate the target recognition result.

[0011] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the environmental sensor data is obtained by real-time acquisition of spatial information, climate data, and obstacle distance data of the mining area through lidar, temperature and humidity sensors, and ultrasonic sensors.

[0012] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the specific steps for generating fused multi-source sensor data are as follows: The target recognition results are synchronized with environmental sensor data in time to generate a synchronized data stream. When processing synchronous data streams using the spatiotemporal attention method, a dynamic weighting strategy combining space and time is used to transform the synchronous data streams into spatial coordinates and map them to the robot base coordinate system to generate standardized sensor data. The spatial weighting is dynamically adjusted based on the relative position of the sensor and the robot base, while the temporal weighting is dynamically attenuated based on the newness of the synchronous data stream. Noise filtering and data smoothing are performed on synchronous data streams and standardized sensor data to generate denoised multi-source data. The denoised multi-source data is fused to generate preliminary fused data. The credibility and influence of the preliminary fused data are evaluated by a trained deep neural network model to generate fused multi-source sensor data.

[0013] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the specific steps for generating motion control commands are as follows: Based on the fused sensor data, and taking into account obstacle positions, environmental changes, and target positions, the optimal driving path is calculated, and preliminary path planning results are generated. The preliminary path planning results are smoothed to generate a smoothed planned path. Based on the smooth planning path and with reference to the robot's current position, velocity, and acceleration information, motion control parameters are calculated, and motion control commands are generated.

[0014] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the specific steps for generating execution feedback data are as follows: The motion control commands are converted into motor speed and torque targets, which drive the pulse width modulation controller to generate corresponding voltage pulses, and simultaneously acquire encoder and inertial measurement data to form raw motion measurement data. The raw motion measurement data is time-stamped and filtered at multiple levels to generate smooth motion state data. The smooth motion state data is compared with the smooth planned path to calculate the driving deviation and generate execution feedback data.

[0015] As a preferred embodiment of the FPGA-based visual servo control method for mining inspection robots described in this invention, the specific steps for generating the adjusted motion control commands are as follows: The execution feedback data is normalized and mapped to a state vector on the FPGA; Based on the state vector, the gain correction amount is calculated using an adaptive neural-fuzzy inference algorithm to generate an updated gain vector; The updated gain vector is fused with the motion control parameters to generate fused motion control parameters. Then, the fused motion control parameters are optimized and adjusted through gain scheduling logic to generate new motion control parameters. The new motion control parameters are encoded into a motion control instruction format to generate the adjusted motion control instructions.

[0016] The beneficial effects of this invention are as follows: by performing time alignment, coordinate transformation, and noise removal on the target recognition results, LiDAR, inertial measurement, and environmental sensor data collected by temperature and humidity, the synchronous fusion of visual and sensor data on the FPGA is achieved. This enables the robot to perceive the distance of minute cracks, water and silt, and obstacles in front in real time in complex mining environments, improving the accuracy and robustness of path planning and obstacle avoidance decisions, and achieving the effect of millisecond-level multi-source data fusion. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a vision servo control method for a mining inspection robot based on FPGA.

[0019] Figure 2 A flowchart for generating preprocessed image data.

[0020] Figure 3 This is a flowchart for generating multi-source sensor data after fusion.

[0021] Figure 4 This is a flowchart for generating the adjusted motion control commands. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a vision servo control method for a mining inspection robot based on FPGA, including the following steps: S1. Collect environmental images of the mining area and perform preprocessing to generate preprocessed image data.

[0026] S1.1 The mining area environmental images include images of pipeline appearance, instrument panel images, roadway wall crack images, water and silt accumulation areas, safety obstacles images, and lighting blind spots images.

[0027] Specifically, pipeline exterior images are captured using high-resolution cameras to photograph the outer surface of the pipeline, ensuring that details and cracks are visible; instrument panel images are captured using cameras to photograph the instrument panels of mine machinery, capturing changes in instrument readings; tunnel wall crack images are captured using LiDAR or high-resolution cameras to photograph the inner walls of the tunnels, extracting cracks and damaged areas; water and silt area images are captured using cameras or thermal imaging to collect images of water and silt areas within the mine, ensuring image quality in low-light environments; safety obstacle images are captured using cameras to photograph potential obstacles within the mining area, ensuring accurate identification of obstacles; and lighting blind spot images are captured using infrared cameras or low-light cameras in low-light areas of the mining area to obtain information on lighting blind spots. The pipeline exterior images, instrument panel images, tunnel wall crack images, water and silt area images, safety obstacle images, and lighting blind spot images are then integrated to form a mining environment image.

[0028] S1.2. Perform noise suppression, enhance dark details, and eliminate dust and haze on the mining area environmental image to generate a dehazed and enhanced image.

[0029] Specifically, an adaptive filtering method is used to smooth the mining area environment image, removing random noise and ensuring that details are not affected. A local contrast-limited adaptive histogram equalization method is applied to enhance the details in the dark areas of the image, making the details in low-light areas clearer, especially improving the visibility of the image in low-light environments. A dehazing algorithm based on dark channel priors is used to eliminate the image blurring caused by dust and haze in the mining area. By adjusting the brightness and contrast of the mining area environment image, the clarity and perspective of the mining area environment image are enhanced, generating a dehazed and enhanced image.

[0030] It should also be noted that the adaptive filtering method is a method of adjusting filter parameters according to the local characteristics of the mining area environment image. It is used to remove noise in the mining area environment image while preserving edges and details as much as possible. By calculating the mean and standard deviation of the local area of ​​the mining area environment image, the strength of the filter is dynamically adjusted to avoid excessive smoothing of the details of the mining area environment image. Local contrast-limited adaptive histogram equalization (CLAHE) is a method for enhancing dark details in images. It limits the enhancement magnitude by dividing the image into blocks and performing histogram equalization on each small region, thus avoiding over-enhancement that could lead to distortion of mining environment images. It is particularly suitable for enhancing mining environment images under low-light conditions. The dehazing algorithm based on dark channel priors utilizes the color features in the mining area environmental image, extracts the channel information of the darkest pixel to estimate the degree of haze, and restores the transmittance and atmospheric light of the mining area environmental image to complete the dehazing process.

[0031] S1.3 Perform frequency domain bandstop filtering on the dehazing enhanced image to generate a stripe suppression image, and perform multi-scale smoothing and edge enhancement in the spatial domain to generate a detail enhancement image.

[0032] Specifically, the dehazed and enhanced image is converted to the frequency domain, and a band-stop filter is applied to selectively remove low-frequency components and periodic interference. The dehazed and enhanced image is then converted back to the spatial domain to obtain a stripe-suppressed image. In the spatial domain, the stripe-suppressed image is subjected to multi-scale smoothing. A Gaussian smoothing filter is used to blur the stripe-suppressed image at different scales to eliminate noise and fine textures while preserving the main structure. Finally, the Laplacian operator is applied to enhance the edges of the stripe-suppressed image, highlighting the edges and details and making the outline of the stripe-suppressed image clearer, resulting in a detail-enhanced image.

[0033] It should also be noted that the working principle of the band-stop filter is to convert the dehazing and enhanced image into the frequency domain, identify and delete high-frequency or low-frequency regions through Fourier transform, and then convert it back to the spatial domain through inverse Fourier transform, thereby obtaining an image with interference removed. Gaussian smoothing filters use a weight matrix generated by a Gaussian function to convolve the stripe suppression image, thereby performing a weighted average of pixels, removing noise while preserving the main structure of the stripe suppression image.

[0034] S1.4 Extract pipe appearance, instrument panel, cracks and water accumulation areas from the detail-enhanced image, and integrate them to generate preprocessed image data.

[0035] Specifically, the process involves extracting the pipe appearance, instrument panel, cracks, and water accumulation areas from the enhanced detail image. Image segmentation methods are used to process the enhanced detail image, and threshold segmentation is employed to identify the pipe and instrument areas. Canny edge detection is used to extract the contours of the pipe appearance and instrument panel, and morphological operations are used to further refine the region extraction. For crack areas, image segmentation methods are used for region growing or edge-based crack identification to extract and enhance the crack areas. Water accumulation areas are segmented using color space conversion (e.g., to HSV space) to extract water or silt areas. Image stitching is used to align the pipe appearance, instrument panel, cracks, and water accumulation areas according to the same coordinate system, followed by edge smoothing to avoid obvious seams. Pixel-level fusion is used to fuse the pipe appearance, instrument panel, cracks, and water accumulation areas, ensuring seamless integration of details in each area. Specific operations include: extracting each area through image segmentation; using a weighted average method to fuse the pixel values ​​between areas to ensure a natural transition; smoothing the fused image to remove artifacts and seams, eliminating inconsistencies between areas, and generating preprocessed image data.

[0036] It should also be noted that image segmentation is an image processing method that aims to divide a detail-enhanced image into multiple regions or objects so that each region can be analyzed independently. The goal of detail-enhanced image segmentation is to divide the detail-enhanced image into several regions based on the similarity of pixels, such as pipe appearance region, instrument panel region, crack region and water and silt region, so that the pixels in the same region are as similar as possible in some feature (such as color, intensity or texture). Thresholding is a common image segmentation method. By setting a fixed threshold, the pixel values ​​of the detail-enhancing image are compared with the fixed threshold to divide the detail-enhancing image into foreground and background. For example, in a grayscale image, all pixels with pixel values ​​greater than the fixed threshold can be classified as foreground, while pixels with values ​​less than the fixed threshold are classified as background. The fixed threshold is usually determined by manual setting, global histogram analysis, or adaptive thresholding methods, and the range is usually, for example, between 0 and 255. Canny edge detection is a classic edge detection algorithm used to extract edge information in detail-enhanced images. It removes noise through Gaussian filtering, calculates the gradient value of the detail-enhanced image, identifies edge regions, refines edges and removes non-edge pixels through non-maximum suppression, and determines strong and weak edges through dual threshold detection to extract the final edges in the detail-enhanced image. Canny edge detection can effectively detect object contours in detail-enhanced images and is often used to extract structural information from them.

[0037] S2. Input the preprocessed image data into the trained deep neural network model deployed on the FPGA for inference and generate target recognition results.

[0038] S2.1 The trained deep neural network model is trained by supervised learning using mining area environmental image data. After training, the deep neural network model structure is pruned and quantized to optimize it before being deployed on an FPGA.

[0039] Specifically, when training deep neural network models, a convolutional neural network (CNN) architecture is chosen. A CNN typically consists of multiple convolutional layers (usually 3×3 or 5×5 kernels) to extract low-level features from mining environment images, such as edges and textures. This is followed by pooling layers (e.g., 2×2 max-pooling layers) to reduce spatial dimensionality and preserve feature information. The CNN structure often stacks multiple convolutional layers to extract more complex high-level features, and then outputs the classification results through fully connected layers. To optimize CNN training, the Adam optimizer is used, which dynamically adjusts the learning rate, accelerating training and improving stability. The loss function is typically the cross-entropy loss function, suitable for multi-class image classification tasks, and capable of measuring the difference between the CNN's predictions and the actual labels. The training process involves several steps: first, optimizing the model based on the actual features and targets of the mining environment images; second, expanding the training dataset using data augmentation methods such as rotation, scaling, and translation to increase the robustness of the deep neural network model; third, employing transfer learning strategies to fine-tune the model on mining environment image data, enabling rapid adaptation to new tasks; fourth, pruning the deep neural network model after training to remove redundant neurons and connections, reducing computational complexity; fifth, converting network parameters from floating precision to fixed precision through quantization optimization to reduce storage requirements and improve inference speed; and finally, deploying the pruned and quantized deep neural network model onto an FPGA to accelerate the inference process using the FPGA's parallel computing capabilities, ensuring efficient target recognition and real-time processing in the mining environment.

[0040] S2.2 The preprocessed image data is transmitted to the FPGA via the on-chip data bus, and the trained deep neural network model is used to perform pixel normalization and quantization to generate a quantized pixel matrix.

[0041] Specifically, the preprocessed image data is transmitted to the FPGA via the on-chip data bus, and each pixel is stored in the FPGA memory in a row-column order. The trained deep neural network model is used to normalize the transmitted preprocessed image data. Specifically, based on the grayscale range of the image, the value of each pixel is mapped to a standardized range to generate normalized pixel values. Then, pixel quantization is performed, converting each normalized pixel value into a fixed integer value. The normalized pixel values ​​are mapped using quantization levels to generate a quantized pixel matrix. The quantized pixel matrix will be used for subsequent deep neural network inference calculations within the FPGA.

[0042] It should also be noted that the grayscale value range refers to the range of grayscale values ​​that pixels in an image can take. For example, in a common 8-bit grayscale image, the grayscale value range is from 0 to 255, where 0 represents black and 255 represents white. Values ​​between 0 and 255 represent different grayscale levels. Standardization range is the process of converting grayscale values ​​in an image into a uniform standard range, typically mapped to the interval between 0 and 1, to facilitate subsequent processing. The basis of standardization is to ensure that the range of each pixel value is at a uniform scale through normalization operations, so that it can be processed by deep learning models.

[0043] S2.3 Perform depthwise separable convolution and pooling downsampling on the quantized pixel matrix to generate a downsampled feature map.

[0044] Specifically, each quantized pixel matrix is ​​convolved pixel-by-pixel with its corresponding convolution kernel to generate an output feature map, expressed as: ; in, Indicates the first Output feature map of the channel Indicates the first Quantized pixel matrix of the channel This represents the convolution operation. Indicates the first The convolution kernel corresponding to the channel, An index representing the number of channels; Perform a pointwise convolution operation on the output feature map, and use the pointwise convolution kernel to extract higher-level features. The expression is as follows: ; in, Indicates the output feature map at the th row and number Column pixel values, Indicates the first The output feature map of the channel is in the first... row and number Column pixel values, Indicates the first Channel-wise convolution kernel, Indicates the row index. Indicates the index of the column; Using the max pooling method, the size of the pooling window is defined based on the size of the output feature map, the resolution of the output feature map, and the computational requirements (e.g., ...). or The sliding pooling window iterates through each small region on the output feature map, selects the maximum value in each small region, and generates a downsampled feature map.

[0045] S2.4 Input the downsampled feature map into the fully connected layer of the trained deep neural network model, and calculate the class probability through approximate Softmax to generate the target recognition result.

[0046] Specifically, the downsampled feature map is input into the fully connected layer of the trained deep neural network model. The downsampled feature map is unfolded into a one-dimensional vector, which serves as the input to the fully connected layer. Each neuron in the fully connected layer is connected to all elements in the input vector. After weighted summation and bias processing, activation values ​​for each category are generated. The activation values ​​are normalized using an approximate softmax function, and the probability of each category is calculated. The approximate softmax function normalizes the activation values ​​of each category by calculating the ratio of the activation value of each category to the activation values ​​of all categories using an exponential function method. The ratio is then normalized to ensure that the sum of the probabilities of all categories is 1, thus obtaining the relative probability value of each category. The category with the highest relative probability value is selected as the target recognition result.

[0047] It should also be noted that the approximate Softmax function is a simplified version of the standard Softmax function, mainly used to reduce computational complexity and improve inference speed, especially when computational resources are limited; the standard Softmax function transforms the output score of a neural network into a probability distribution, while the approximate Softmax achieves a similar function by optimizing or simplifying the computation process.

[0048] S3. Using the spatiotemporal attention method, the target recognition results are fused with environmental sensor data in real time, and the influence of space and time is dynamically adjusted to generate fused multi-source sensor data.

[0049] S3.1 Environmental sensor data is obtained by collecting spatial information, climate data and obstacle distance data of the mining area in real time through lidar, temperature and humidity sensors and ultrasonic sensors.

[0050] Specifically, the system uses lidar to scan the mining area environment in real time, generating point cloud data. This point cloud data contains the three-dimensional position and shape data of objects within the mining area. Temperature and humidity sensors collect climate data in the mining area in real time, recording temperature and humidity changes. The output of the temperature and humidity sensors is temperature and humidity values, reflecting the climate changes in the mining area. Ultrasonic sensors emit sound waves and receive reflected waves to measure the distance to obstacles in the mining area, generating obstacle distance data. Each time the ultrasonic sensor emits a sound wave and receives the echo, it calculates and records the distance to the obstacle. The spatial information, climate data, and obstacle distance data collected by lidar, temperature and humidity sensors, and ultrasonic sensors are integrated to generate environmental sensor data.

[0051] S3.2 When processing synchronous data streams using the spatiotemporal attention method, a dynamic weighting strategy combining space and time is used to transform the synchronous data streams into spatial coordinates and map them to the robot base coordinate system to generate standardized sensor data. The spatial weighting is dynamically adjusted based on the relative position of the sensor and the robot base, while the temporal weighting is dynamically attenuated based on the newness of the synchronous data stream.

[0052] Specifically, each item in the target recognition results and environmental sensor data is time-stamped to ensure they share the same time reference. By comparing the timestamps of the target recognition results and environmental sensor data, target recognition results and environmental sensor data with acceptable time differences are identified and matched. For target recognition results and environmental sensor data with different timestamps, the time of the target recognition results and environmental sensor data is adjusted using a time alignment method to ensure that the data streams are synchronized in time. The time-synchronized target recognition results and environmental sensor data are integrated to generate a synchronized data stream, where each data point contains a corresponding timestamp and the corresponding target recognition result and environmental data. Spatial weighting is dynamically adjusted based on the relative position between the sensor and the robot base to determine the weighting coefficient of each sensor data in the spatial coordinate system. Temporal weighting is dynamically attenuated based on the difference in the newness of the synchronized data stream to ensure that newer data has a higher weight in the time dimension.

[0053] It should also be noted that the acceptable range refers to the maximum allowable time difference range during time synchronization. Within the acceptable range, the target recognition result and environmental sensor data can be considered synchronized, sufficient for effective fusion and subsequent processing. The specific acceptable range usually depends on the real-time requirements and the frequency of data acquisition. For example, for rapidly changing environments, the time difference may be required to be no more than 10 milliseconds, while for slower-changing environments, the acceptable range of time difference can be larger, such as 50 milliseconds or 100 milliseconds. The dynamic weighting strategy of space and time is a method that assigns different importance to different data sources based on the characteristics of real-time data. When processing multi-source sensor data, spatial weighting dynamically adjusts the spatial weights by considering the relative positions of each sensor in physical space, thereby increasing the influence of sensor data closer to the robot on decision-making. For example, sensor data closer to the robot may be more accurate and reliable, and is given a higher spatial weight, while sensor data farther away has a relatively lower importance. Temporal weighting, on the other hand, dynamically adjusts the temporal weights according to the freshness of the data, ensuring that the most recently collected data contributes more to the decision, while outdated data has a smaller impact. For example, recently collected data reflects the current environmental state and has a stronger influence on the decision, while older data may be affected by environmental changes and is therefore given a lower weight.

[0054] S3.3. Using the spatiotemporal attention method, the synchronous data stream is transformed into spatial coordinates and mapped to the robot base coordinate system. Spatial and temporal information is dynamically weighted to generate standardized sensor data.

[0055] Specifically, based on the original coordinate system of the target recognition results and environmental sensor data, the spatial position of each data point is recorded. A spatial coordinate transformation method is used to convert the environmental sensor data (e.g., LiDAR, temperature and humidity sensor, and ultrasonic sensor data) into a unified coordinate system. This involves mapping the spatial coordinates of each data point from the original coordinate system to the robot's base coordinate system. During this process, a spatiotemporal attention method is used to dynamically weight time and spatial information. In the spatial dimension, spatial coordinates are weighted based on the local spatial features of the sensor data to highlight important areas. In the temporal dimension, data points are weighted based on the relationship between timestamps to enhance their influence on the target recognition results. The transformed data points are then scaled to a standardized range to generate standardized sensor data, where the spatial position and environmental information of each data point are unified to the same coordinate system and standardized scale.

[0056] S3.4. Perform noise filtering and data smoothing on synchronous data streams and standardized sensor data to generate denoised multi-source data.

[0057] Specifically, the synchronous data stream and standardized sensor data are input into a noise filtering algorithm to remove noise from the synchronous data stream and standardized sensor data. Gaussian filtering is used to smooth the neighborhood around each data point, removing abrupt changes and outliers to ensure the smoothness of the synchronous data stream and standardized sensor data. Low-pass filtering is applied to further process the synchronous data stream and standardized sensor data to reduce high-frequency fluctuations and irregular changes, generating denoised multi-source data.

[0058] S3.5. The denoised multi-source data is fused to generate preliminary fused data. The credibility and influence of the preliminary fused data are evaluated by a trained deep neural network model to generate fused multi-source sensor data.

[0059] Specifically, the denoised multi-source data is fused by combining synchronous data streams with standardized sensor data through data stitching to generate preliminary fused data. This preliminary fused data is then input into a trained deep neural network model, which processes the data and evaluates the credibility and influence of each data source during the fusion process. During processing, the trained deep neural network model analyzes the characteristics of each data source (e.g., sensor data accuracy, noise level, and time consistency) to determine the reliability of each data source. The characteristics of each data source (e.g., data stability, resolution, noise level, and time alignment accuracy) are extracted through the hidden layers of the trained deep neural network model to obtain a credibility value. Based on the credibility evaluation results of each data source, the trained deep neural network model further optimizes the preliminary fused data to generate fused multi-source sensor data.

[0060] It should be noted that the visual recognition results are spatiotemporally aligned, coordinate transformed, and noise removed on the same FPGA with multi-source environmental data such as LiDAR, inertial measurement, and temperature and humidity. After training a deep neural network model, the credibility and influence of the preliminary fused data are evaluated, which enables the robot to perceive multi-dimensional information such as cracks, water accumulation, and obstacle distances with high precision and low latency.

[0061] S4. Perform path planning and motion control on the fused sensor data. Combine vision, lidar and inertial measurement data, optimize the path using the Dijkstra algorithm, and generate motion control commands.

[0062] S4.1 Based on the fused sensor data, and with reference to obstacle positions, environmental changes and target positions, calculate the optimal driving path and generate preliminary path planning results.

[0063] Specifically, based on the fused sensor data, the robot's current position, obstacle positions, environmental change information, and target position data are obtained. Using the Dijkstra algorithm, the optimal travel path from the current robot position to the target position is calculated by combining the obstacle position, environmental change, and target position data. Specifically, a map model is built, the location of obstacles is marked, the cost of feasible paths is calculated, and the travel path is dynamically adjusted according to environmental change data to ensure that the path planning can avoid new obstacles or changed areas, generating preliminary path planning results.

[0064] It should also be noted that Dijkstra's algorithm is an algorithm for calculating the shortest path from the source node to all other nodes in a graph. The basic idea is to start from the source node and gradually expand to other nodes in the graph, updating the shortest path by continuously selecting the node closest to the source node among the currently unvisited nodes. Specifically, the process is as follows: set the initial distance to the source node to 0, and set the distances to other nodes to infinity; select the node closest to the source node; check the neighbor nodes of the node closest to the source node and update their distances; if the path to a neighbor node through the current node is shorter, update the distance of the neighbor node; repeat this process until the shortest path to all nodes has been calculated. Dijkstra's algorithm is chosen for calculating the optimal driving path, efficiently handling the shortest path problem in non-negative weighted graphs, adapting to obstacles and environmental changes, and ensuring that the robot finds the shortest path from the current position to the target position in complex environments. Map modeling refers to the process of transforming environmental information into a processable mathematical representation structure, typically in graphical or grid form, during path planning or robot navigation. A map model abstracts and represents elements in the physical environment such as obstacles, passable areas, and target locations, constructing a data structure that reflects the spatial layout of the real world. Common map models include grid maps and topological maps. In a grid map, the environment is divided into a series of small grids, each representing a free area or obstacle. A topological map, on the other hand, represents the environment through the connections between grid nodes and edges; grid nodes represent key locations, and edges represent feasible paths between locations. By building map models, robots can perform path planning, navigation, and obstacle avoidance operations based on these models. Obstacle location refers to any physical obstacle in the environment, such as walls and buildings, which is usually acquired and its coordinates marked in real time by sensors (such as lidar and ultrasonic sensors). Environmental change refers to the dynamic changes in the environment caused by various factors (such as the movement of objects and climate change), which may affect path planning and navigation, and usually needs to be reflected by continuous sensor data acquisition and updates. Target location refers to the final destination that the robot needs to reach when performing a task, which is usually determined by task requirements or user instructions. By combining obstacle location, environmental change, and target location, the robot can dynamically adjust its path planning, avoid obstacles, and cope with environmental changes, ultimately reaching the target location safely.

[0065] S4.2 Smooth the preliminary path planning results to generate a smoothed planning path.

[0066] Specifically, the preliminary path planning results are represented as a series of path points (e.g., coordinate positions). The path points are processed by median filtering, which replaces the original pixel value with the median of the neighboring pixels, effectively eliminating salt-and-pepper noise. For each path point, a smooth transition with adjacent points is calculated to reduce sharp turns or irregular changes in the path. The connection between path points is smoothed by linear interpolation, which connects two data points and performs linear estimation within this interval. This method is suitable for cases where data changes are gradual, ensuring that the path curve is smoother and more natural, generating a smooth planning path where the transition between each path point and its adjacent path points is smoother, avoiding abrupt changes or excessive curvature that may exist in the original path.

[0067] S4.3. Based on the smooth planning path and with reference to the robot's current position, speed, and acceleration information, calculate the motion control parameters and generate motion control commands.

[0068] Specifically, the coordinates of each path point and information of adjacent path points are extracted from the smoothed planned path. Combined with the robot's current position, velocity, and acceleration information, the error between the current robot state and the target path point is calculated. Using a PID control algorithm (proportional-integral-derivative control), the robot's direction, velocity, and acceleration are adjusted based on the error between the current robot state and the target path point. Specifically, Euclidean distance and angular errors between the current position and the target path point are calculated. The robot's velocity (forward speed and turning angular velocity) is adjusted based on the magnitude of the distance and angular errors. Using acceleration information, considering the robot's current motion state and expected trajectory, the acceleration control parameters are adjusted to ensure smooth motion and conformity to the path planning. Based on the calculated distance error, angular error, and acceleration adjustment parameters, motion control parameters are synthesized to generate motion control commands, instructing the robot to execute corresponding speed and acceleration adjustments to travel along the smoothed planned path.

[0069] It should also be noted that the PID control algorithm (proportional-integral-derivative control) is a widely used feedback control algorithm used to precisely regulate the output. PID control calculates the deviation (the difference between the set value and the actual value) and adjusts the control signal according to the combination of the proportional, integral and derivative components to achieve the desired target. The robot's current position refers to its specific coordinates in three-dimensional space, typically measured in real time by position sensors (such as GPS and LiDAR), representing the robot's position relative to a reference coordinate system. Robot velocity refers to the rate of motion of the robot at its current position, usually including forward and turning speeds, and can be obtained through velocity sensors (such as encoders and inertial measurement units). Robot acceleration describes the change in robot velocity, representing the rate of change of robot velocity over time, and can be obtained through acceleration sensors (such as gyroscopes and accelerometers). The robot's current position, velocity, and acceleration information together reflect its motion state, aiding in path planning, motion control, and stability control.

[0070] S5. Drive the robot to move along a smoothly planned path through motion control commands, and monitor and collect the robot's position, speed and acceleration in real time to generate execution feedback data.

[0071] S5.1. Convert motion control commands into motor speed and torque targets, drive the pulse width modulation controller to generate corresponding voltage pulses, and simultaneously acquire encoder and inertial measurement data to form raw motion measurement data.

[0072] Specifically, the forward speed and turning angular velocity in the motion control commands are converted into motor speed and torque targets. Control signals are calculated based on the error between the current position and the target position. A PID algorithm is used to adjust the position error, speed error, and acceleration error respectively, generating corresponding control outputs. The PID control outputs are then converted into motor speed and torque values ​​to ensure the robot travels precisely along the smoothly planned path. A pulse width modulation controller generates corresponding voltage pulses to control the motor's speed and torque. The duty cycle of the voltage pulses determines the motor's speed and torque, thus executing the motion control commands. An encoder collects real-time data on motor speed and position changes, recording the robot's actual motion state. An inertial measurement unit (IMU) collects information such as acceleration and angular velocity to reflect the robot's dynamic behavior. The raw data collected by the encoder and IMU are integrated to form raw motion measurement data.

[0073] S5.2. Perform timestamp correction and multi-level filtering on the raw motion measurement data to generate smooth motion state data.

[0074] Specifically, the raw motion measurement data undergoes timestamp correction. By comparing the timestamps of data from different sensors, interpolation methods are used to align data from different sources to the same time base. Multi-stage filtering methods are then used to process the raw data to remove high-frequency noise. Specifically, low-pass filters, Kalman filters, or moving average filters are employed, smoothing each data point layer by layer through progressive filtering to gradually reduce noise and abrupt changes. The low-pass filter smooths the motion data by removing frequency components above a set cutoff frequency. The Kalman filter is optimized based on the difference between prediction and measurement to accurately estimate the data. The moving average filter smooths the data by calculating the average value of data points within a window. The data after timestamp correction and multi-stage filtering generates smoothed motion state data.

[0075] It should also be noted that the cutoff frequency is usually set based on the characteristics of the data and the range of noise frequencies that need to be removed. Generally, the cutoff frequency should be selected at the upper limit of the noise components in the data to ensure that unwanted high-frequency noise is removed during the filtering process while retaining the effective information in the signal. For dynamic motion data, the selection of the cutoff frequency should also take into account the robot's speed and response time. For example, if the robot runs at a high speed, the filter's cutoff frequency should be set higher to avoid filtering out useful motion information. The cutoff frequency is set above the frequency of the expected noise in the signal, usually between several hertz and several hundred hertz. The specific range should be adjusted according to experimental data and noise characteristics to ensure that the filtering effect does not affect the accuracy of the actual motion data.

[0076] S5.3 Compare the smooth motion state data with the smooth planned path, calculate the driving deviation, and generate execution feedback data.

[0077] Specifically, the current position in the smooth motion state data is compared with the target path point in the smooth planned path. The Euclidean distance calculation method is used to calculate the distance between the robot's current position and the target path point. By extracting the coordinates of the current position and the target path point, the straight-line distance between the current position and the target path point is calculated to obtain the position error. The direction angle calculation method is used to calculate the angular difference between the robot's orientation and the target path to obtain the angle error. Combining the position error and the angle error, the travel deviation is obtained. The PID control algorithm is used to adjust the robot's motion parameters (such as speed and acceleration) according to the travel deviation (including position error and angle error) to ensure that the robot can travel along the planned path. The PID control algorithm adjusts the robot's motion direction and speed by calculating the proportional, integral, and derivative terms of the deviation, and considers acceleration information to optimize control, generating execution feedback data, including the robot's current error state, motion parameter adjustment values, and deviation from the target path.

[0078] S6. The execution feedback data is processed in parallel on the FPGA to dynamically optimize the motion control parameters and generate adjusted motion control commands.

[0079] S6.1 Normalize the execution feedback data on the FPGA and map it into a state vector.

[0080] Specifically, the position error, angle error, velocity, and acceleration information in the execution feedback data are standardized. The maximum and minimum values ​​of each data point are calculated by traversing the dataset, and the execution feedback data is normalized so that the value of each data point is mapped to a uniform range. The normalized execution feedback data is then mapped into a state vector according to a predetermined rule. The state vector includes information such as the robot's current position error, velocity error, and acceleration error.

[0081] It should also be noted that the dataset traversal method refers to the process of accessing and processing each element in the execution feedback data one by one. All data points are extracted from the execution feedback data, and each data point is processed sequentially. An initial value is set as the maximum and minimum value (for example, the initial maximum value can be set to negative infinity, and the minimum value to positive infinity). Each element in the dataset is traversed, and each element is compared with the current maximum and minimum value. If a data point is greater than the current maximum value, the maximum value is updated; if a data point is less than the current minimum value, the minimum value is updated. Predefined rules are typically based on task requirements, data characteristics, or algorithm requirements, setting standards and scope for data processing. The purpose is to ensure that the data processing or transformation process can achieve the expected results. For example, when mapping execution feedback data to a state vector, the predefined rules may include data sorting, priority allocation, normalization methods, or requirements for the mapped data structure.

[0082] S6.2. Based on the state vector, calculate the gain correction amount and generate the updated gain vector using the adaptive neural-fuzzy inference algorithm.

[0083] Specifically, based on the position error, velocity error, and acceleration error in the state vector, an adaptive neural-fuzzy inference algorithm is input to fuzzify the input state vector, transforming continuous error data into a fuzzy set. An appropriate membership function is defined to represent the membership degree of each error. The algorithm searches for matching rules in a fuzzy rule base based on the fuzzy set, typically using an "if...then..." format. Inference is then performed using rules matching the input values ​​to generate fuzzy inference results. These results are then fused using the maximum value method to obtain a fuzzy output. The fuzzy output is converted into a precise gain correction using the centroid method. Leveraging the learning ability of the neural network, feedback from the error and gain correction is used to adjust and update the weights and parameters in the neural network to adapt to the current motion state and control requirements. Finally, an updated gain vector is generated based on the calculated gain correction.

[0084] It should also be noted that the adaptive neural-fuzzy inference algorithm is an intelligent algorithm that combines neural networks and fuzzy logic control, designed to handle uncertain and fuzzy data. Its adaptive characteristic is reflected in the algorithm's ability to dynamically adjust parameters based on changes in input data, achieving a more efficient learning and optimization process. Specifically, by training the data through a neural network and continuously adjusting the network's weights and biases, it can adaptively perform inference and optimization based on the characteristics of the input data. During the adaptive process, the fuzzy inference part adjusts the fuzzy rules based on real-time input data to cope with environmental changes or data uncertainty. Specifically, it monitors fluctuations and changes in input data, adjusts the fuzzy sets in the fuzzy rule base and the membership function in the inference process accordingly, and adjusts the inference strategy in real time for the current task and environment to achieve optimized control. The fuzzy rule base is the core component of fuzzy logic control, used to describe the relationships and behaviors between input variables. In the fuzzy inference process, the fuzzy rule base consists of a set of fuzzy rules. Each rule is usually expressed in the form of "if-then", such as "if the temperature is high, then the wind speed is high". The rules are set according to the needs of the actual problem or obtained through data training. Each rule fuzzifies the input variables and uses a membership function to map the input to a fuzzy set, thereby deriving the fuzzy inference result.

[0085] S6.3. The updated gain vector is fused with the motion control parameters to generate fused motion control parameters. The fused motion control parameters are then optimized and adjusted through gain scheduling logic to generate new motion control parameters.

[0086] Specifically, the correction value in the updated gain vector is linearly superimposed with the motion control parameters (such as forward speed, turning angular velocity, and acceleration) to obtain the fused motion control parameters. Using gain scheduling logic, the fused motion control parameters are optimized and adjusted according to the current robot motion state and path error. The gain scheduling logic adjusts the gain of the motion control parameters in real time according to preset scheduling rules to ensure that the robot can perform tasks smoothly and accurately under different working conditions, generating new motion control parameters that will be used to guide the robot's actual movement and ensure that the corresponding actions are performed according to the adjusted control signals, such as adjusting speed, acceleration, and steering.

[0087] It should also be noted that gain scheduling logic is a method for dynamically adjusting control gain, aiming to optimize control parameters in real time based on the current state or environmental conditions. It automatically selects and adjusts different gain values ​​according to preset scheduling rules to adapt to different operating scenarios. Specifically, gain scheduling logic selects an appropriate gain value based on the robot's motion state (e.g., speed, acceleration, and error magnitude), and the gain value determines the robot's response to control inputs. For example, when the robot approaches a target, gain scheduling logic may decrease the gain for a smooth transition; while when the robot deviates from its trajectory, gain scheduling logic may increase the gain to strengthen the correction. Scheduling rules typically set a series of predefined conditions based on input variables (such as robot speed, acceleration, and error magnitude), and determine the gain selection based on the current state. Scheduling rules can be based on experience or optimized through data-driven methods. For example, when the path deviation is large, the scheduling rule may choose a higher gain to quickly correct the robot's deviation from the trajectory; while when the path is close to the target, the scheduling rule may choose a lower gain to avoid over-adjustment and maintain smooth driving.

[0088] S6.4. Encode the new motion control parameters into a motion control instruction format and generate the adjusted motion control instructions.

[0089] Specifically, new motion control parameters are converted into corresponding digital signals using linear quantization. The specific steps are as follows: The value range of each motion control parameter is determined based on the quantization level (e.g., 8-bit and 16-bit); the values ​​of the motion control parameters are mapped to this range, and continuous analog signals are converted into discrete digital signals according to a proportional relationship; during the conversion process, the motion control parameters may be rounded or truncated to ensure compatibility with digital signal processing; the value of each new motion control parameter is mapped to a discrete range, for example, by setting a maximum and minimum value and dividing it into multiple discrete levels for controller compatibility; for example, speed and acceleration values ​​are converted into duty cycles or voltage values ​​recognizable by the controller using a proportional mapping method, while steering angular velocity is discretized into corresponding motor control signals. According to a predetermined encoding format, the converted new motion control parameters are organized into motion control commands in the order of forward speed, steering angular velocity, and acceleration, generating adjusted motion control commands that include forward speed, steering angular velocity, and acceleration.

[0090] It should also be noted that the setting of the discrete range needs to take into account the maximum and minimum values ​​that the controller can handle, as well as the required accuracy. For example, if the input signal of the controller is an 8-bit digital signal, then the discrete range of the control parameters will be divided into 255 levels between 0 and 255. The predetermined encoding format is usually based on the controller's input requirements and data transmission protocol. The controller's hardware specifications and supported signal types will affect the choice of encoding format, such as whether it supports analog or digital signals, the signal bit width (e.g., 8-bit and 16-bit), and the signal resolution. The predetermined encoding format needs to take into account the accuracy of the data and the transmission efficiency to ensure that the control signals can be transmitted accurately, while avoiding data loss or excessive redundancy.

[0091] In summary, this invention achieves synchronous fusion of visual and sensor data on an FPGA by performing time alignment, coordinate transformation, and noise removal on target recognition results, LiDAR, inertial measurement, and environmental sensor data collected by temperature and humidity. This enables the robot to perceive the distance to minute cracks, water and silt, and obstacles in front in real time in complex mining environments, improving the accuracy and robustness of path planning and obstacle avoidance decisions, and achieving millisecond-level multi-source data fusion.

[0092] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual servo control method for a mine inspection robot based on FPGA, characterized in that: include, Collect environmental images of the mining area and preprocess them to generate preprocessed image data; The preprocessed image data is input into a trained deep neural network model deployed on the FPGA for inference, generating target recognition results; Using the spatiotemporal attention method, the target recognition results are fused with environmental sensor data in real time, and the influence of space and time is dynamically adjusted to generate fused multi-source sensor data. The fused sensor data is used for path planning and motion control. By combining visual, lidar and inertial measurement data, the path is optimized using the Dijkstra algorithm to generate motion control commands. The robot is driven to move along a smoothly planned path by motion control commands, and its position, speed and acceleration are monitored and collected in real time to generate execution feedback data. The execution feedback data is processed in parallel on the FPGA to dynamically optimize motion control parameters and generate adjusted motion control commands.

2. The FPGA-based visual servo control method for mine inspection robots as described in claim 1, characterized in that: The mining area environmental images include images of pipeline appearance, instrument panel, roadway wall cracks, water and silt accumulation areas, safety obstacles, and lighting blind spots.

3. The FPGA-based visual servo control method for mine inspection robots as described in claim 1, characterized in that: The specific steps for generating preprocessed image data are as follows. Noise suppression, dark detail enhancement, and dust and haze removal are performed on mining area environmental images to generate dehazed and enhanced images; The dehazing enhanced image is subjected to frequency domain bandstop filtering to generate a stripe suppression image, and multi-scale smoothing and edge enhancement are performed in the spatial domain to generate a detail enhancement image. Extract pipe appearance, instrument panel, cracks, and water accumulation areas from the detailed enhanced image and integrate them to generate preprocessed image data.

4. The FPGA-based visual servo control method for a mining inspection robot as described in claim 1, characterized in that: The trained deep neural network model is trained by supervised learning using mining area environmental image data. After training, the deep neural network model structure is pruned and quantized to optimize it before being deployed on an FPGA. This is achieved through data augmentation, transfer learning, and online fine-tuning strategies.

5. The FPGA-based visual servo control method for a mining inspection robot as described in claim 1, characterized in that: The specific steps for generating the target recognition result are as follows: The preprocessed image data is transmitted to the FPGA via the on-chip data bus, and the trained deep neural network model is used to perform pixel normalization and quantization to generate a quantized pixel matrix. The quantized pixel matrix is ​​subjected to depthwise separable convolution and pooling downsampling to generate a downsampled feature map. The downsampled feature map is input into the fully connected layer of the trained deep neural network model, and the class probability is calculated by approximating Softmax to generate the target recognition result.

6. The FPGA-based visual servo control method for a mining inspection robot as described in claim 1, characterized in that: The environmental sensor data is obtained by collecting spatial information, climate data, and obstacle distance data of the mining area in real time through lidar, temperature and humidity sensors, and ultrasonic sensors.

7. The FPGA-based visual servo control method for a mining inspection robot as described in claim 1, characterized in that: The specific steps for generating the fused multi-source sensor data are as follows: The target recognition results are synchronized with environmental sensor data in time to generate a synchronized data stream. When processing synchronous data streams using the spatiotemporal attention method, a dynamic weighting strategy combining space and time is used to transform the synchronous data streams into spatial coordinates and map them to the robot base coordinate system to generate standardized sensor data. The spatial weighting is dynamically adjusted based on the relative position of the sensor and the robot base, while the temporal weighting is dynamically attenuated based on the newness of the synchronous data stream. Noise filtering and data smoothing are performed on synchronous data streams and standardized sensor data to generate denoised multi-source data. The denoised multi-source data is fused to generate preliminary fused data. The credibility and influence of the preliminary fused data are evaluated by a trained deep neural network model to generate fused multi-source sensor data.

8. The FPGA-based visual servo control method for a mining inspection robot as described in claim 1, characterized in that: The specific steps for generating motion control commands are as follows: Based on the fused sensor data, and taking into account obstacle positions, environmental changes, and target positions, the optimal driving path is calculated, and preliminary path planning results are generated. The preliminary path planning results are smoothed to generate a smoothed planned path. Based on the smooth planning path and with reference to the robot's current position, velocity, and acceleration information, motion control parameters are calculated, and motion control commands are generated.

9. The FPGA-based visual servo control method for a mining inspection robot as described in claim 1, characterized in that: The specific steps for generating execution feedback data are as follows: The motion control commands are converted into motor speed and torque targets, which drive the pulse width modulation controller to generate corresponding voltage pulses, and simultaneously acquire encoder and inertial measurement data to form raw motion measurement data. The raw motion measurement data is time-stamped and filtered at multiple levels to generate smooth motion state data. The smooth motion state data is compared with the smooth planned path to calculate the driving deviation and generate execution feedback data.

10. The FPGA-based visual servo control method for a mining inspection robot as described in claim 1, characterized in that: The specific steps for generating the adjusted motion control commands are as follows: The execution feedback data is normalized and mapped to a state vector on the FPGA; Based on the state vector, the gain correction amount is calculated using an adaptive neural-fuzzy inference algorithm to generate an updated gain vector; The updated gain vector is fused with the motion control parameters to generate fused motion control parameters. Then, the fused motion control parameters are optimized and adjusted through gain scheduling logic to generate new motion control parameters. The new motion control parameters are encoded into a motion control instruction format to generate the adjusted motion control instructions.

Images