Transfer robot obstacle avoidance strategy generation method and system based on deep learning

Abstract

The invention provides a transfer robot obstacle avoidance strategy generation method and system based on deep learning, and relates to the technical field of robot obstacle avoidance, and the method comprises the steps: firstly obtaining a transfer robot surrounding environment perception data set which comprises obstacle form, motion state and path topographic data; carrying out feature association processing on the data based on a pre-trained deep learning obstacle avoidance model, generating environment obstacle association features, obtaining current motion state data of the robot, constructing obstacle avoidance decision constraint conditions in combination with the environment obstacle association features, and carrying out obstacle avoidance; environment obstacle correlation characteristics and obstacle avoidance decision constraint conditions are input into a model decision output layer to generate a preliminary obstacle avoidance path scheme, and then a final obstacle avoidance strategy is obtained through adjustment according to the constraint conditions and comprises advancing speed adjustment parameters, steering operation instructions and path change node information; therefore, the environment can be sensed comprehensively, the efficient obstacle avoidance strategy is generated dynamically, and the obstacle avoidance capability and operation efficiency of the robot are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot obstacle avoidance, in particular to a carrying robot obstacle avoidance strategy generation method and system based on deep learning. BACKGROUND

[0002] In industrial production and logistics transportation scenarios, carrying robots play a crucial role, as they can efficiently and accurately complete the task of carrying goods, improving production efficiency and reducing labor costs. However, carrying robots inevitably encounter various obstacles during operation, and how to achieve safe and effective obstacle avoidance becomes a key issue in the application of carrying robots.

[0003] Currently, traditional carrying robot obstacle avoidance methods mainly rely on simple sensor detection and preset rules. For example, an ultrasonic sensor or an infrared sensor is used to detect the distance of an obstacle, and when the distance of an obstacle is detected to be less than a set threshold, the robot performs a deceleration, stop or turning operation according to a preset rule. However, the above method has obvious limitations. On the one hand, the environmental information obtained by traditional sensors is relatively single, and only the approximate position and distance of the obstacle can be obtained, making it difficult to fully perceive the complex situation of the surrounding environment, such as the shape, motion state and path terrain of the obstacle. On the other hand, the obstacle avoidance strategy based on preset rules lacks flexibility and adaptability, and cannot dynamically adjust the obstacle avoidance scheme according to the real-time changes in the environment. In the face of complex and variable obstacle scenarios, the obstacle avoidance may fail or be inefficient, seriously affecting the running efficiency and safety of the carrying robot. SUMMARY

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present application, the embodiments of the present application provide a carrying robot obstacle avoidance strategy generation method based on deep learning, which comprises:

[0005] Obtain a set of environmental perception data of the carrying robot, which includes obstacle shape data, obstacle motion state data and path terrain data within the robot's travel path range;

[0006] Perform feature correlation processing on the set of environmental perception data based on a pre-trained deep learning obstacle avoidance model to generate environmental obstacle correlation features, which reflect the spatial correlation relationship between obstacles and between obstacles and path terrain;

[0007] Obtain the current motion state data of the carrying robot, and construct obstacle avoidance decision constraints in combination with the environmental obstacle correlation features, which include robot motion speed limits, turning angle limits and path deviation range limits;

[0008] input the environmental obstacle correlation features and the obstacle avoidance decision constraint conditions into a decision output layer of the deep learning obstacle avoidance model to generate a preliminary obstacle avoidance path scheme;

[0009] adjust the preliminary obstacle avoidance path scheme according to the obstacle avoidance decision constraint conditions to obtain a final obstacle avoidance strategy, the final obstacle avoidance strategy including a robot travel speed adjustment parameter, a steering operation instruction, and path change node information.

[0010] In still another aspect, an embodiment of the present application also provides a deep learning-based obstacle avoidance strategy generation system for a carrying robot, including a processor and a machine-readable storage medium, the machine-readable storage medium being connected to the processor, the machine-readable storage medium being used to store programs, instructions or codes, and the processor being used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0011] Based on the above aspects, an embodiment of the present application obtains a multi-dimensional environmental perception data set including obstacle shapes, motion states and path terrains around a carrying robot, performs feature correlation processing on the environmental perception data set based on a pre-trained deep learning obstacle avoidance model to generate environmental obstacle correlation features reflecting spatial correlation relationships between obstacles and between obstacles and path terrains, constructs obstacle avoidance decision constraint conditions in combination with current motion state data of the carrying robot and the environmental obstacle correlation features, and covers multiple aspects of limitations such as robot motion speed, steering angle and path deviation range, so that the generation of the obstacle avoidance strategy is more in line with the actual motion ability and safety requirements of the robot. The environmental obstacle correlation features and the obstacle avoidance decision constraint conditions are input into a decision output layer of the deep learning obstacle avoidance model to generate a preliminary obstacle avoidance path scheme, and the preliminary scheme is adjusted according to the obstacle avoidance decision constraint conditions to finally obtain a final obstacle avoidance strategy including a robot travel speed adjustment parameter, a steering operation instruction and path change node information. The final obstacle avoidance strategy can dynamically generate an efficient and safe obstacle avoidance strategy according to real-time changing environmental information, greatly improves the obstacle avoidance ability and operation efficiency of the carrying robot in a complex environment, and ensures the smooth completion of the carrying task. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is an execution flow diagram of the deep learning-based obstacle avoidance strategy generation method for a carrying robot provided by an embodiment of the present application.

[0013] Figure 2 is a schematic diagram of exemplary hardware and software components of the deep learning-based obstacle avoidance strategy generation system for a carrying robot provided by an embodiment of the present application. DETAILED DESCRIPTION

[0014] The present application will be described in detail below with reference to the accompanying drawings, Figure 1is a flowchart of a deep learning-based obstacle avoidance strategy generation method for a carrying robot provided by an embodiment of the present application. The deep learning-based obstacle avoidance strategy generation method will be described in detail below.

[0015] Step S110: Obtain a set of environment perception data around the carrying robot, which includes obstacle shape data, obstacle motion state data and path terrain data within the robot's travel path range.

[0016] In this embodiment, a carrying robot running in an intelligent warehouse center is taken as an example. The robot needs to perform cargo carrying operations in the passageway between shelves, and the surrounding environment may have temporary stacked cargo, other running robots, workers and other obstacles, and the path terrain may include flat cement ground, laid rubber mat area and other different situations. In order to achieve safe obstacle avoidance, the robot's surrounding environment perception data needs to be comprehensively obtained first.

[0017] Step S111: Start the multi-type environment perception device carried by the carrying robot, collect initial environment data within the robot's travel path range, and the initial environment data includes image data collected by the visual sensor, point cloud data collected by the laser radar and distance data collected by the ultrasonic sensor.

[0018] In the scenario of an intelligent warehouse center, a carrying robot usually carries multiple environment perception devices. Among them, the visual sensor can be a high-definition camera installed at the front, rear and sides of the robot to cover the visual field in different directions, and the image data collected by the visual sensor can reflect the appearance information such as color, shape and texture of objects in the surrounding environment; the laser radar can be a multi-line laser radar installed on the top of the robot, which can collect three-dimensional point cloud data of the surrounding environment at a high frequency and high accuracy, thereby obtaining the spatial position and contour information of the object; the ultrasonic sensor is distributed around the robot to detect obstacles at close range, making up for the possible blind spots of the visual sensor and laser radar in close-range detection. When the robot starts and begins to perform the carrying task, these multi-type environment perception devices are started simultaneously, and continuously collect initial environment data within the robot's travel path range according to the preset sampling frequency.

[0019] Step S112: Perform obstacle contour extraction processing on the image data collected by the visual sensor to obtain obstacle contour information in the image data, and convert the obstacle contour information into structured shape description data as the first component of the obstacle shape data.

[0020] For the image data collected by the visual sensor, since it contains a large amount of pixel information, it is relatively complex to be directly used for obstacle shape analysis, and therefore needs to be subjected to obstacle contour extraction processing. First, the image data is preprocessed, including image denoising, contrast enhancement and the like, to improve the image quality and reduce the interference of subsequent processing. Then, an edge detection algorithm such as the Canny edge detection algorithm is used to perform edge detection on the preprocessed image to identify the edge pixel points of the objects in the image. Next, the edge pixel points are connected and closed to form complete object contours. Then, a contour tracking algorithm such as the contour tracking algorithm based on eight-neighborhood search is used to obtain the pixel coordinate sequence of each object contour. The pixel coordinate sequence of the contour is converted into structured shape description data, such as the perimeter, area, length and width of the circumscribed rectangle and the like of the contour, which together constitute the first part of the obstacle shape data.

[0021] Step S1121: converting the image data collected by the visual sensor into gray-scale image data, removing noise from the gray-scale image data, and performing edge enhancement processing on the denoised gray-scale image data to obtain edge-enhanced gray-scale image data.

[0022] In the intelligent warehouse center scenario, the image data collected by the visual sensor is usually a color image containing information of three channels of red, green and blue. In order to reduce the data processing amount and highlight the contour features of the objects, the color image is first converted into gray-scale image data, and the gray-scale value is obtained by calculating the weighted average value of the three channels of each pixel point. Due to factors such as light flickering and equipment vibration in the warehouse environment, noise may be introduced into the gray-scale image data, and therefore noise removal processing is needed. A Gaussian filter algorithm can be used to filter the gray-scale image data, and by setting appropriate filter kernel size and standard deviation, the noise in the image can be effectively suppressed while the edge information of the image is preserved. After the denoising processing is completed, in order to make the edges of the objects clearer and facilitate subsequent contour extraction, edge enhancement processing is performed on the denoised gray-scale image data. A Laplacian operator can be used to perform convolution operation on the image to enhance the regions with large gray-scale changes in the image, i.e. the edge regions of the objects, thereby obtaining edge-enhanced gray-scale image data.

[0023] Step S1122: performing binarization processing on the edge-enhanced gray-scale image data to obtain binarized image data, and performing connected region analysis processing on the binarized image data to identify all connected white regions in the binarized image data, each connected white region corresponding to a potential obstacle, to obtain a connected region set.

[0024] In the edge-enhanced grayscale image data, the gray values of the object's edge and the background region still have differences. In order to separate the object from the background, binarization processing is needed. A suitable threshold value is selected, and the pixel points with a gray value greater than the threshold value in the grayscale image are set to white (pixel value is 255), representing the possible object region; the pixel points with a gray value less than or equal to the threshold value are set to black (pixel value is 0), representing the background region, thereby obtaining the binarized image data. Since there may be some isolated white pixel points or small white regions in the binarized image, these may be noise or irrelevant small objects, and therefore a connected region analysis process is needed. A connected region analysis algorithm based on region growing is adopted, starting from the top left corner of the binarized image, and scanning each pixel point in turn. When a white pixel point is encountered and has not been marked, the pixel point is taken as a seed point, and all adjacent white pixel points are merged into a connected region with the seed point, and are marked. In the above manner, all connected white regions in the binarized image data are identified, and each connected white region corresponds to a potential obstacle, thereby obtaining a connected region set.

[0025] Step S1123: performing region morphological processing on each connected region in the connected region set, filling small cavities inside the connected region with a dilation algorithm, and eliminating small protrusions on the edge of the connected region with an erosion algorithm, to obtain a morphologically optimized connected region.

[0026] In the connected region set, each connected region may have small cavities inside or small protrusions on the edge, which will affect the accuracy of subsequent contour extraction. Therefore, region morphological processing is needed for each connected region. First, a dilation algorithm is used to process the connected region. The dilation algorithm can expand the boundary of the connected region outward, thereby filling the small cavities inside the connected region. A suitable size and shape of a structural element, such as a 3x3 square structural element, is selected, and multiple dilation operations are performed on the connected region until the small cavities inside are completely filled. Next, an erosion algorithm is used to process the dilated connected region. The erosion algorithm can contract the boundary of the connected region inward, thereby eliminating the small protrusions on the edge of the connected region. Similarly, a suitable structural element is selected, and multiple erosion operations are performed on the dilated connected region to restore the original shape of the connected region while removing the small protrusions on the edge, finally obtaining a morphologically optimized connected region.

[0027] Step S1124: performing contour tracking processing on each morphologically optimized connected region, recording the coordinate sequence of the edge pixel points of the connected region with a chain code tracking algorithm, and the coordinate sequence constitutes the contour information of the obstacle.

[0028] The connected region after morphological optimization has a relatively regular shape, and contour tracking processing can be performed at this time. A chain code tracking algorithm is used to start from an arbitrary pixel point on the boundary of the connected region, and the pixel points on the boundary are sequentially tracked in a predetermined direction order (such as clockwise or counterclockwise direction), and the coordinates of each pixel point are recorded. The chain code tracking algorithm represents the shape of the contour by recording the direction change between adjacent pixel points, for example, using the eight numbers 0-7 to represent the eight directions of up, upper right, right, lower right, down, lower left, left, and upper left. In the tracking process, when returning to the starting pixel point, it indicates that a complete contour has been tracked. Each connected region corresponds to a contour, and the coordinate sequence of the edge pixel points of the contour together constitutes the contour information of the obstacle.

[0029] Step S1125: Simplifying the tracked contour information, removing redundant pixel points in the contour, retaining key pixel points that can reflect the main shape features of the obstacle contour, and obtaining simplified obstacle contour information.

[0030] The tracked contour information contains a large number of edge pixel point coordinate sequences, some of which may be redundant and have less contribution to reflecting the main shape features of the obstacle contour. Too many pixel points will increase the burden of subsequent data processing. Therefore, the contour information needs to be simplified. The Douglas-Peucker algorithm can be used to simplify the contour information, which iteratively removes pixel points on the contour that deviate less from the current line segment, and retains pixel points that deviate more, until the number of pixel points on the contour is reduced to within a predetermined threshold range. Through the above method, redundant pixel points in the contour are removed, and key pixel points that can reflect the main shape features of the obstacle contour, such as turning points and endpoints, are retained, thereby obtaining simplified obstacle contour information.

[0031] Step S1126: By comparing with a common obstacle contour feature library, the similarity value of the simplified obstacle contour information and the common obstacle contour feature is calculated, the contour information with a similarity value greater than a predetermined similarity threshold is selected, and the contour information with a similarity value less than or equal to the predetermined similarity threshold and an area less than a predetermined area threshold is excluded, to obtain the final obstacle contour information in the image data.

[0032] In the intelligent warehouse center, common obstacles such as shelves, pallets, other robots, workers, etc. have specific contour features. In order to accurately identify the obstacles in the image data, a common obstacle contour feature library is constructed, which stores the standard contour feature information of various common obstacles. The simplified obstacle contour information is compared with each standard contour feature in the common obstacle contour feature library, and the similarity value between them is calculated. The similarity value can be calculated by using contour moments, shape descriptors, etc. The similarity degree is determined by comparing the shape parameters of the two. A preset similarity threshold is set, and the contour information with a similarity value greater than the threshold is selected. These contour information corresponds to the common obstacles in the warehouse environment. At the same time, for the contour information with an area less than the preset area threshold, it may be some small debris or noise, which is also excluded. Finally, the obstacle contour information in the image data is obtained.

[0033] Step S113: performing obstacle space position calculation processing on the point cloud data collected by the laser radar to obtain three-dimensional coordinate information of each obstacle in the robot coordinate system, and performing association matching processing on the three-dimensional coordinate information and the obstacle contour information to generate obstacle shape data containing spatial position as a second component of the obstacle shape data.

[0034] The point cloud data collected by the laser radar is a set of a large number of three-dimensional coordinate points, and each point cloud data point represents a sampling point on the surface of an object in the surrounding environment. In order to obtain the spatial position of each obstacle, the point cloud data collected by the laser radar needs to be processed for obstacle space position calculation. First, the point cloud data is preprocessed, including point cloud denoising, point cloud registration, etc. The point cloud denoising can use a statistical filtering algorithm to remove isolated points in the point cloud that deviate greatly from the distance of the surrounding point set; the point cloud registration is to unify the point cloud data collected at different times to the robot coordinate system. Then, a clustering-based segmentation algorithm such as the Euclidean clustering algorithm is used to segment the point cloud data, and the point cloud data points with similar spatial positions are clustered together to form different point cloud clusters, each of which corresponds to a potential obstacle. For each point cloud cluster, the three-dimensional coordinates of the center point are calculated as the three-dimensional coordinate information of the obstacle in the robot coordinate system. Next, the obtained three-dimensional coordinate information is associated and matched with the obstacle contour information extracted by the vision sensor. The correspondence between the position of the obstacle contour in the image and the projection position of the point cloud cluster in the robot coordinate system can be calculated to match the two, so that each obstacle contour information corresponds to a unique three-dimensional coordinate information. Finally, the three-dimensional coordinate information and the obstacle contour information are combined to generate obstacle shape data containing spatial position as a second component of the obstacle shape data.

[0035] Step S114: continuously sampling and analyzing the distance data collected by the ultrasonic sensor, calculating the distance change between the robot and the same obstacle at adjacent sampling time and the corresponding time interval, calculating the motion rate of the obstacle according to the distance change and the time interval, determining the motion direction according to the direction of the distance change, and integrating the motion direction and the motion rate information into obstacle motion state data.

[0036] The ultrasonic sensor measures the distance between the robot and the obstacle by transmitting and receiving ultrasonic signals. The distance data collected by the ultrasonic sensor changes over time. In order to obtain the motion state of the obstacle, the distance data collected by the ultrasonic sensor needs to be continuously sampled and analyzed. First, set the sampling time interval, and continuously collect distance data at multiple time points according to the time interval. For the distance data at each sampling time, combine the motion state of the robot itself (such as the change of the position and attitude of the robot), correct the distance data, and eliminate the influence of the motion of the robot itself on the distance measurement result. Then, for the same obstacle, the corresponding distance data at different sampling times is identified by feature matching method. Next, the distance change between the robot and the same obstacle at adjacent sampling times is calculated, that is, the distance value at the next time minus the distance value at the previous time. At the same time, record the time interval between adjacent sampling times. According to the distance change and the time interval, the motion rate of the obstacle is calculated, that is, the absolute value of the distance change divided by the time interval. According to the positive and negative directions of the distance change, the motion direction of the obstacle is determined. When the distance change is positive, it means that the obstacle is moving away from the robot; when the distance change is negative, it means that the obstacle is moving towards the robot. The motion direction and the motion rate information are integrated together to form the obstacle motion state data.

[0037] Step S115: performing terrain feature recognition processing on the path region in the image data collected by the vision sensor, extracting slope change information, flatness information and obstacle distribution density information in the path region, and combining the slope change information, flatness information and obstacle distribution density information into path terrain data.

[0038] The image data collected by the visual sensor contains the terrain information of the area where the robot travels. In order to extract these terrain features, it is necessary to first determine the path area in the image data. The path area in the image can be segmented from the background by pre-setting the region of interest (ROI) of the path area, or using a semantic segmentation algorithm. Then, the path area is subjected to terrain feature recognition processing. For the extraction of slope change information, the height change at different positions in the path area can be calculated by analyzing the gray value change rule of the pixel points in the path area, combined with the internal and external parameters of the camera, so as to obtain the slope change information. The extraction of flatness information can be realized by calculating the variance or standard deviation of the gray value in the path area. The smaller the variance or standard deviation, the higher the flatness of the path area. The extraction of obstacle distribution density information can be achieved by counting the number of obstacle contour information in the path area, and combining the area of the path area to calculate the number of obstacles per unit area, i.e. to obtain the obstacle distribution density information. The slope change information, flatness information and obstacle distribution density information extracted are combined together to form the path terrain data.

[0039] Step S116: performing spatio-temporal alignment processing on the first component of the obstacle shape data, the second component of the obstacle shape data, the obstacle motion state data and the path terrain data to generate an environment perception data set.

[0040] Since the first component of the obstacle shape data (structured shape description data from the visual sensor), the second component of the obstacle shape data (shape data containing spatial position), the obstacle motion state data (from the ultrasonic sensor) and the path terrain data (from the visual sensor) are collected by different sensors at different times, there may be differences in time and space between them. In order to ensure that these data can be used cooperatively for subsequent obstacle avoidance decision-making, spatio-temporal alignment processing is needed. In terms of time alignment, the system clock of the robot is used as the reference to record the time stamp of each data collection, and then interpolation or resampling methods are used to unify the data collected by different sensors to the same time sampling point. In terms of spatial alignment, all data are unified to the robot coordinate system to ensure that the related data of the same obstacle collected by different sensors can be corresponded in spatial position. For example, the spatial position corresponding to the obstacle contour information extracted by the visual sensor is accurately matched with the three-dimensional coordinate information calculated by the laser radar, and the distance data measured by the ultrasonic sensor is converted to position information in the robot coordinate system. After spatio-temporal alignment processing, the above data are integrated together to form an environment perception data set.

[0041] Step S120: feature correlation processing is performed on the set of environment perception data based on the pre-trained deep learning obstacle avoidance model, to generate environment obstacle correlation features reflecting the spatial correlation between obstacles and between obstacles and path terrain.

[0042] After obtaining the set of environment perception data, feature correlation processing needs to be performed on it by the pre-trained deep learning obstacle avoidance model to mine the spatial correlation between obstacles and between obstacles and path terrain. The deep learning obstacle avoidance model is trained on a large number of warehouse environment obstacle avoidance sample data and can learn the complex correlation patterns between various object and terrain features in the environment.

[0043] Step S121: obstacle shape data, obstacle motion state data and path terrain data in the set of environment perception data are input into the feature input layer of the deep learning obstacle avoidance model, and after feature dimension standardization processing is performed on each type of data, shape feature encoding processing is performed on the standardized dimension obstacle shape data in the feature extraction layer of the deep learning obstacle avoidance model to generate an obstacle shape feature vector, which contains obstacle contour complexity information, volume size information and surface texture information.

[0044] The obstacle shape data, obstacle motion state data and path terrain data in the set of environment perception data have different data types and feature dimensions, and direct input into the deep learning model may affect the training effect and feature extraction capability of the model. Therefore, the above data is first input into the feature input layer of the deep learning obstacle avoidance model. The feature input layer performs feature dimension standardization processing on each type of data, converting feature data of different scales and distributions into the same feature space, such as normalizing the value range of all feature data to 0-1, or standardizing it to have a mean of 0 and a variance of 1. After feature dimension standardization processing, the data enters the feature extraction layer of the deep learning obstacle avoidance model. For standardized dimension obstacle shape data, a shape feature encoding submodule in the feature extraction layer processes it. The shape feature encoding submodule contains multiple convolution layers and pooling layers, which extract local features such as contour edge features and texture detail features from obstacle shape data through convolution operations, and then reduce the dimension and abstract the features through pooling operations. After processing by multiple convolution and pooling layers, a high-dimensional feature map is obtained, which is then converted into a fixed-length vector, i.e. an obstacle shape feature vector, through a fully connected layer. The obstacle shape feature vector contains contour complexity information of the obstacle, which can be reflected by the number and distribution of turning points on the contour; volume size information, which is calculated from the volume of the obstacle's bounding rectangle or three-dimensional point cloud cluster; and surface texture information, which is obtained by analyzing the texture pattern and grayscale value variation of the obstacle surface.

[0045] Step S122: Perform motion feature encoding processing on the standardized obstacle motion state data to generate an obstacle motion feature vector, which contains stability information of the motion direction, change trend information of the motion speed, and prediction information of the motion trajectory.

[0046] For the standardized obstacle motion state data, the motion feature encoding submodule in the feature extraction layer processes it. The motion state data includes information such as the motion direction and motion speed of the obstacle, which changes over time. The motion feature encoding submodule uses a recurrent neural network (RNN) or its variants (such as LSTM, GRU) to process the time-series motion state data. RNN can capture the time-dependent relationship in sequence data by memorizing the motion state information at previous time points to analyze the motion features at the current time point. For example, LSTM can effectively handle long sequence dependency problems through a gating mechanism, avoiding gradient vanishing or gradient explosion. By inputting the motion direction and motion speed data of the obstacle at multiple consecutive time points into the LSTM network, the hidden layer state of the network is constantly updated to learn the stability information of the motion direction, i.e., the change amplitude of the motion direction over a period of time; the change trend information of the motion speed, such as whether the speed is increasing, decreasing, or remaining stable; and the prediction information of the motion trajectory based on the historical motion state within a certain period of time in the future. Finally, the learned motion features are converted into a fixed-length obstacle motion feature vector through the output layer or fully connected layer of the LSTM network.

[0047] Step S123: Perform terrain feature encoding processing on the standardized path terrain data to generate a path terrain feature vector, which contains change amplitude information of the path slope, fluctuation information of the road surface flatness, and density information of the obstacle distribution.

[0048] The path terrain data after standardization includes slope change information, flatness information and obstacle distribution density information. The terrain feature encoding submodule in the feature extraction layer processes the above data to generate a path terrain feature vector. First, the slope change information is analyzed to calculate the maximum change value and the average change value of the slope in the path area to reflect the change amplitude information of the path slope. For the road surface flatness information, the fluctuation information of the flatness, i.e., the change range and frequency of the flatness parameter, is extracted by calculating the distribution of the flatness parameter (such as the variance of the gray value) in the path area. The density information of the obstacle distribution is reflected by calculating the obstacle distribution density per unit length or unit area and the average distance between obstacles, etc. by counting the number and distribution of obstacles in the path area. The terrain feature encoding submodule can use a multi-layer perception (MLP) to process the above terrain feature parameters and map them to a high-dimensional feature space to generate a path terrain feature vector.

[0049] Step S124: input the obstacle shape feature vector and the obstacle motion feature vector into the first feature association layer of the deep learning obstacle avoidance model, calculate the association weight between the two feature vectors through the attention mechanism, perform feature fusion processing based on the association weight, and generate the obstacle self-association feature.

[0050] The obstacle shape feature vector reflects the static attributes of the obstacle, while the obstacle motion feature vector reflects the dynamic attributes of the obstacle. There is a close association between the two, for example, the shape of the obstacle may affect its movement. In order to capture the above association, the obstacle shape feature vector and the obstacle motion feature vector are input into the first feature association layer of the deep learning obstacle avoidance model. The first feature association layer uses an attention mechanism to calculate the association weight between the two feature vectors. The attention mechanism can make the model automatically focus on the parts with higher association in the two feature vectors, thereby better fusing their information.

[0051] Step S1241: input the obstacle shape feature vector and the obstacle motion feature vector into the feature preprocessing submodule of the first feature association layer for standardization to obtain the standardized obstacle shape feature vector and the standardized obstacle motion feature vector.

[0052] Before the attention mechanism calculation is performed, in order to ensure that the two feature vectors have the same scale and distribution, it is necessary to standardize them. The feature preprocessing submodule adopts a similar standardization method as the feature input layer, such as Z-score standardization, to convert each feature dimension of the obstacle shape feature vector and the obstacle motion feature vector to a distribution with a mean of 0 and a standard deviation of 1. For example, for each element in the obstacle shape feature vector, subtract the mean of the feature dimension and divide by the standard deviation of the feature dimension; the same processing is performed on the obstacle motion feature vector to obtain the standardized obstacle shape feature vector and the standardized obstacle motion feature vector.

[0053] Step S1242: In the attention weight calculation submodule of the first feature association layer, the standardized obstacle shape feature vector is taken as the query vector, and the standardized obstacle motion feature vector is taken as the key vector. The similarity matrix between the query vector and the key vector is calculated, and each element in the similarity matrix represents the similarity between the corresponding dimensions of the two feature vectors.

[0054] In the attention weight calculation submodule, the standardized obstacle shape feature vector is taken as the query vector (Q), and the standardized obstacle motion feature vector is taken as the key vector (K). In order to calculate the similarity between them, the dot product attention method is adopted, and the transpose of the query vector and the key vector is multiplied to obtain a similarity matrix. Each element in the similarity matrix represents the similarity between one dimension of the query vector and one dimension of the key vector. For example, if the dimension of the query vector is d1 and the dimension of the key vector is d2, the size of the similarity matrix is d1 x d2, where the element in the i-th row and j-th column represents the similarity between the i-th dimension of the query vector and the j-th dimension of the key vector.

[0055] Step S1243: The similarity matrix is subjected to softmax normalization processing to convert each element in the similarity matrix to a probability value between 0 and 1, and the sum of the probability values of all elements is 1, to obtain an attention weight matrix, and the elements in the attention weight matrix are the association weights between the two feature vectors.

[0056] In order to convert the elements in the similarity matrix into weight values with a probability significance, the similarity matrix is subjected to softmax normalization processing. The softmax function normalizes each row element in the similarity matrix, so that the sum of each row element is 1, and the value of each element is between 0 and 1. After softmax normalization processing, an attention weight matrix is obtained, and the elements in the attention weight matrix represent the attention degree of the corresponding dimension of the query vector to the corresponding dimension of the key vector, that is, the association weight between the corresponding dimensions of the two feature vectors. The higher the association weight, the closer the association between the features in the two vectors.

[0057] Step S1244: The normalized obstacle motion feature vector is taken as a value vector, the attention weight matrix is subjected to matrix multiplication operation with the value vector, and a weighted obstacle motion feature vector is obtained, which highlights the motion feature dimensions with high correlation to the obstacle shape feature.

[0058] The normalized obstacle motion feature vector is taken as a value vector (V), and the attention weight matrix (W) is subjected to matrix multiplication operation with the value vector (V), that is, WxV. Through the above operation, the motion feature dimensions with high correlation to the obstacle shape feature in the value vector are given higher weights, and a weighted obstacle motion feature vector is obtained. The weighted vector highlights the motion feature dimensions that have a greater influence on the obstacle shape feature or are closely associated with the obstacle shape feature, so that the motion feature can be better combined with the shape feature.

[0059] Step S1245: The normalized obstacle shape feature vector is subjected to feature splicing processing with the weighted obstacle motion feature vector, and the two vectors are combined along the feature dimension direction, and a spliced feature vector is obtained.

[0060] In order to fuse the obstacle shape feature and the weighted motion feature into an integral feature vector, feature splicing processing is performed. The normalized obstacle shape feature vector and the weighted obstacle motion feature vector are combined along the feature dimension direction, that is, the two vectors are connected head to tail to form a new feature vector. For example, if the dimension of the obstacle shape feature vector is d1 and the dimension of the weighted obstacle motion feature vector is d2, then the dimension of the spliced feature vector is d1+d2. The spliced feature vector contains the shape feature of the obstacle and the attention-weighted motion feature, and integrates the information of both.

[0061] Step S1246: The spliced feature vector is subjected to feature dimension reduction processing, and a principal component analysis algorithm is used to extract the main feature components in the spliced feature vector, and a reduced dimension feature vector is obtained.

[0062] The dimension of the spliced feature vector is high and may contain some redundant information, increasing the complexity and computational load of subsequent model processing. Therefore, feature dimension reduction processing is needed, and the principal component analysis (PCA) algorithm is used to extract the main feature components in the spliced feature vector. PCA decomposes the covariance matrix of the spliced feature vector to obtain eigenvalues and eigenvectors, selects the top k eigenvectors with larger eigenvalues as principal components, and projects the spliced feature vector onto these principal components to obtain a reduced dimension feature vector with dimension k. The selection of k can be determined according to the cumulative contribution rate of the principal components, for example, selecting the number of principal components with a cumulative contribution rate of more than 95% to ensure that the reduced dimension feature vector can retain most of the information of the original features.

[0063] Step S1247: Perform nonlinear transformation processing on the reduced dimension feature vector, and perform nonlinear mapping on each element in the feature vector through an activation function to enhance the expression ability of the feature vector for the correlation between the obstacles themselves, and obtain a transformed feature vector.

[0064] The reduced dimension feature vector is still linear, and in order to enhance the expression ability of the feature vector for the complex correlation between the obstacles themselves, nonlinear transformation processing is needed. An activation function such as a ReLU function, a tanh function, or a sigmoid function is used to perform nonlinear mapping on each element in the reduced dimension feature vector. For example, the ReLU function sets all elements less than 0 to 0 and keeps elements greater than 0 unchanged, which can introduce nonlinear characteristics and also alleviate the gradient vanishing problem. Through nonlinear transformation, the feature vector can learn more complex nonlinear correlation between the obstacle shape and motion, thereby obtaining a transformed feature vector.

[0065] Step S1248: Perform feature smoothing processing on the transformed feature vector, and use a moving average algorithm to perform smoothing calculation on the adjacent dimension feature values in the feature vector to obtain a smoothed feature vector.

[0066] In the transformed feature vector, the feature values of adjacent dimensions may have large fluctuations, and the above fluctuations may be caused by noise or unstable factors in the feature extraction process. In order to make the feature vector smoother and reduce the influence of fluctuations, a moving average algorithm is used to smooth the transformed feature vector. The moving average algorithm takes each element in the feature vector as the center and takes the average of the elements before and after it as the new value of the element. For example, a moving window with a window size of n is set, and for the i-th element in the feature vector, its smoothed value is the average of the element and its adjacent n elements (if the window exceeds the vector boundary, the boundary element is filled). Through the above method, the adjacent dimension feature values in the feature vector are smoothed to obtain a smoothed feature vector.

[0067] Step S1249: determining the smoothed feature vector as the obstacle self-correlation feature, which contains the correlation information between the obstacle shape and motion.

[0068] After the above series of processing, the smoothed feature vector has fused the obstacle shape feature and motion feature, and highlighted the correlation between the two through the attention mechanism. At the same time, after dimension reduction, nonlinear transformation and smoothing processing, it has good expression ability and stability. Therefore, the smoothed feature vector is determined as the obstacle self-correlation feature, which can comprehensively reflect the inherent correlation information between the obstacle shape and motion.

[0069] Step S125: inputting the obstacle self-correlation feature and the path terrain feature vector into the second feature correlation layer of the deep learning obstacle avoidance model, calculating the spatial position correlation degree, motion influence correlation degree and terrain adaptation correlation degree between the obstacle and the path terrain, performing feature integration processing based on the spatial position correlation degree, motion influence correlation degree and terrain adaptation correlation degree, and performing dimension compression processing on the integrated feature to generate the environment obstacle correlation feature.

[0070] The obstacle self-correlation feature reflects the correlation between the shape and motion of a single obstacle, while the path terrain feature vector reflects the terrain properties of the robot's travel path. There is also a close correlation between the obstacle and the path terrain. The role of the second feature correlation layer is to calculate these correlation degrees and perform feature integration. First, the spatial position correlation degree is calculated, i.e. the spatial position relationship between the three-dimensional coordinates of the obstacle in the robot coordinate system and the path terrain, such as the distance of the obstacle from the path center, the projection position of the obstacle on the path, etc. These parameters are used to quantify the spatial position correlation degree. The motion influence correlation degree considers the influence of the motion state of the obstacle on the path terrain, such as whether the moving obstacle will enter the path area, whether its motion trajectory will cross the path, etc. The terrain adaptation correlation degree refers to the adaptation degree of the shape and motion state of the obstacle to the path terrain feature, such as on a path with a large slope, an obstacle with a large volume and high motion rate may cause greater difficulty for the robot to avoid obstacles.

[0071] After the spatial position correlation degree, the motion influence correlation degree and the terrain adaptation correlation degree are calculated, the second feature correlation layer performs feature integration processing on the above correlation degree parameters and the obstacle self-correlation features and the path terrain feature vector. The above correlation degree parameters can be taken as additional feature dimensions, spliced with the obstacle self-correlation features and the path terrain feature vector, and a higher-dimensional integrated feature vector is formed. Then, the integrated feature vector is subjected to dimension compression processing, and a similar dimension reduction method such as principal component analysis or autoencoder can be used to compress the high-dimensional integrated feature vector to a suitable dimension, and finally the environmental obstacle correlation features are generated. The environmental obstacle correlation features comprehensively reflect the spatial correlation relationship between obstacles and between obstacles and path terrain.

[0072] Step S130: Obtain the current motion state data of the carrying robot, and construct an obstacle avoidance decision constraint condition combining the environmental obstacle correlation features, the obstacle avoidance decision constraint condition including robot motion speed limit, turning angle limit and path deviation range limit.

[0073] After the environmental obstacle correlation features are generated, in order to formulate a reasonable obstacle avoidance strategy, the current motion state of the carrying robot and the environmental constraints reflected by the environmental obstacle correlation features need to be considered. Therefore, first, the current motion state data of the robot is obtained, and then the obstacle avoidance decision constraint condition is constructed in combination with the environmental obstacle correlation features.

[0074] Step S131: Start the motion state sensor carried by the carrying robot, and collect the current motion parameter data of the robot, the motion parameter data including the current travel speed, the current turning angle, the current acceleration and the current position coordinate data of the robot.

[0075] The carrying robot usually carries multiple motion state sensors, such as a wheel speed encoder for measuring travel speed, a gyroscope and an accelerometer for measuring turning angle and acceleration, a global positioning system (GPS) or an indoor positioning system (such as UWB) for obtaining current position coordinate data. When the robot performs a carrying task, these motion state sensors are started in real time and collect data at a preset sampling frequency. The wheel speed encoder calculates the current travel speed by measuring the number of revolutions and the rotation speed of the robot drive wheel; the gyroscope determines the current turning angle by detecting the rotation angular velocity of the robot around the vertical axis; the accelerometer measures the acceleration of the robot in the forward, backward and turning directions; the GPS or UWB positioning system calculates the current position coordinate data of the robot in the global coordinate system or the local coordinate system by receiving satellite signals or indoor base station signals, and the above motion parameter data collectively constitute the original measurement data of the current motion state of the robot.

[0076] Step S132: Perform outlier detection processing on the motion parameter data to exclude abnormal motion parameters caused by interference in the sensor acquisition process, and retain the valid motion parameter data conforming to the actual motion state of the robot as the current motion state data of the transfer robot.

[0077] Due to the measurement error of the sensor itself, external environmental interference (such as electromagnetic interference, vibration), or noise in the data transmission process, etc., there may be outliers in the collected motion parameter data, which will affect the accurate judgment of the actual motion state of the robot, so outlier detection processing is needed. Statistical-based outlier detection methods can be used, such as the 3σ criterion, first calculate the mean and standard deviation of each motion parameter data sequence, then the data points outside the range of mean plus or minus 3 times the standard deviation are judged as outliers. Clustering-based methods can also be used, clustering the motion parameter data points, and the data points far from the cluster center are considered as outliers. For detected outliers, interpolation methods (such as linear interpolation, adjacent value interpolation) are used for replacement, or the sampling points where the outliers are directly removed, and the data sequence is smoothed. After outlier detection processing, the valid motion parameter data conforming to the actual motion state of the robot are retained as the current motion state data of the transfer robot.

[0078] Step S133: Extract the obstacle distribution density information and obstacle motion trend information in the environment obstacle correlation feature, analyze the restriction range of the obstacle distribution to the robot's travel space, and determine the boundary of the space region where the robot can move in the current environment.

[0079] The environment obstacle correlation feature contains obstacle distribution density information and obstacle motion trend information. Obstacle distribution density information can be extracted through specific dimensions or parameters in the environment obstacle correlation feature, such as the number of obstacles in a unit volume, the average distance between obstacles, etc. Obstacle motion trend information includes the motion direction of the obstacle, the trend of the motion rate, and the prediction information of the motion trajectory, etc. By analyzing these information, the distribution of obstacles around the robot and the motion situation can be determined, so as to judge the restriction range of the obstacles to the robot's travel space. For example, in the area where the obstacles are densely distributed, the space available for the robot to move is relatively small; and the obstacles whose motion trend is towards the robot's travel path will further compress the available space for the robot. Considering these factors, the boundary of the space region where the robot can safely move in the current environment is determined, which is usually a three-dimensional space range represented by coordinate values in the robot coordinate system.

[0080] Step S134: Based on the current travel speed in the current motion state data of the carrying robot, combined with the distance information of the space region boundary, the braking distance of the robot in an emergency is calculated, and according to the braking distance, the upper threshold and the lower threshold of the robot motion speed are determined to form the robot motion speed limit.

[0081] Step S1341: Extract the current travel speed in the current motion state data of the carrying robot, and record the speed value of the current travel speed as the current speed reference value.

[0082] From the carrying robot current motion state data after the abnormal value detection processing, the numerical value of the current travel speed is directly extracted, and the numerical value is recorded as the current speed reference value. The current speed reference value is a basic parameter for subsequent calculation of braking distance and determination of speed limit.

[0083] Step S1342: From the distance information of the space region boundary, the straight-line distance from the current position of the robot to the nearest space region boundary is extracted, and the distance value of the straight-line distance is taken as the boundary distance reference value.

[0084] The distance information of the space region boundary contains the distance of the robot to the movable space region boundary in each direction. In order to ensure that the robot can be safely braked in the most dangerous situation (i.e. moving towards the nearest boundary direction), the straight-line distance from the current position of the robot to the nearest space region boundary is extracted, and the distance value is recorded as the boundary distance reference value.

[0085] Step S1343: Obtain the braking performance parameters of the carrying robot, which include the maximum braking acceleration, braking response time and road surface friction coefficient of the robot braking system. The maximum braking acceleration, braking response time and road surface friction coefficient of the robot braking system are determined based on the hardware specifications of the robot and the current path terrain data.

[0086] The braking performance parameters of the carrying robot are the key basis for calculating the braking distance of the robot. The maximum braking acceleration depends on the hardware specifications of the robot braking system, such as the power of the brake motor, the material of the brake pad and the efficiency of the brake transmission mechanism, etc. These parameters can be obtained from the technical manual of the robot. The braking response time is the time interval from the issuance of the braking instruction to the start of the braking force generated by the braking system, which is also related to the hardware and control algorithm of the braking system. The road surface friction coefficient is related to the road flatness, material and other factors in the current path terrain data, for example, the friction coefficient of dry cement road is larger, while the friction coefficient of wet or oily road is smaller. By querying the hardware specification document of the robot and analyzing the current path terrain data, the specific values or ranges of these braking performance parameters can be obtained.

[0087] Step S1344: According to the brake response time and the current speed reference value, the uniform travel distance of the robot in the brake response stage is calculated, which is equal to the product of the brake response time and the current speed reference value, to obtain the response stage travel distance.

[0088] In the braking process, there is a brake response time, during which the brake system has not yet begun to generate braking force, and the robot still travels at a uniform speed of the current speed reference value. Therefore, the response stage travel distance is equal to the product of the brake response time and the current speed reference value. For example, if the brake response time is t1 and the current speed reference value is v, then the response stage travel distance s1 = t1 × v.

[0089] Step S1345: According to the maximum brake acceleration and the current speed reference value, the deceleration travel distance of the robot in the brake execution stage is calculated to obtain the execution stage travel distance.

[0090] After the brake response stage ends, the brake system starts to work, and the robot does uniform deceleration motion under the action of the maximum brake acceleration until the speed is reduced to zero. According to the formula of uniform deceleration motion, the execution stage travel distance s2 = v2 / (2 × a), where a is the maximum brake acceleration (taking the absolute value).

[0091] Step S1346: The response stage travel distance is added to the execution stage travel distance to obtain the total brake distance of the robot in the emergency situation, which represents the total distance traveled by the robot from triggering the brake instruction to completely stopping.

[0092] The total brake distance s = s1 + s2, that is, the sum of the response stage travel distance and the execution stage travel distance. The total brake distance reflects the total distance traveled by the robot from realizing the danger and triggering the brake instruction to completely stopping under the current motion state.

[0093] Step S1347: Considering the slope change information in the path terrain data, the total brake distance is subjected to slope correction processing. If the slope change information of the current path shows that the slope value is greater than 0, a first correction coefficient is calculated according to the uphill slope value, and the total brake distance is multiplied by the first correction coefficient; if the slope change information of the current path shows that the slope value is less than 0, a second correction coefficient is calculated according to the downhill slope value, and the total brake distance is multiplied by the second correction coefficient to obtain the corrected brake distance.

[0094] The slope of the path terrain has a significant impact on the braking distance. When going uphill, the component of gravity along the slope is opposite to the braking direction, which will reduce the braking distance; when going downhill, the component of gravity along the slope is the same as the braking direction, which will increase the braking distance. Therefore, the total braking distance needs to be corrected according to the slope change information in the path terrain data. The slope value is usually represented by the tangent value of the slope angle, i.e. the ratio of the height difference to the horizontal distance. When the slope value is greater than 0, it indicates uphill, and a first correction coefficient is calculated according to the size of the uphill slope value, which is less than 1, for example, the first correction coefficient = 1 / (1+k1×slope value), where k1 is an adjustment parameter of the uphill correction coefficient. When the slope value is less than 0, it indicates downhill, and a second correction coefficient is calculated according to the absolute value of the downhill slope value, which is greater than 1, for example, the second correction coefficient = 1 / (1-k2×|slope value|), where k2 is an adjustment parameter of the downhill correction coefficient. The total braking distance is multiplied by the corresponding correction coefficient to obtain the corrected braking distance, which is closer to the actual braking distance under the slope condition.

[0095] Step S1348: Determine a safety distance margin according to the boundary distance reference value and the corrected braking distance, the safety distance margin being the boundary distance reference value minus the corrected braking distance.

[0096] The safety distance margin refers to the difference between the distance from the current position of the robot to the boundary of the space region and the corrected braking distance, i.e. safety distance margin d = boundary distance reference value - corrected braking distance. The safety distance margin reflects the safety buffer space between the robot and the boundary of the space region after braking at the current speed.

[0097] Step S1349: If the safety distance margin is greater than a preset safety margin threshold, the braking distance at the current speed is safe enough, and the speed upper limit threshold is increased according to a preset speed increase ratio; if the safety distance margin is less than the preset safety margin threshold, the speed upper limit threshold is decreased according to a preset speed decrease ratio, so that the sum of the corrected braking distance and the safety distance margin does not exceed the boundary distance reference value.

[0098] The preset safety margin threshold is a minimum safety buffer distance set according to the safety requirement of the robot for obstacle avoidance. When the safety distance margin is greater than the safety margin threshold, it indicates that the robot has sufficient safety distance to brake at the current speed, and at this time the upper speed threshold can be appropriately increased to improve the running efficiency of the robot. The speed upscaling ratio can be determined according to the degree to which the safety distance margin exceeds the safety margin threshold. The more the excess, the greater the upscaling ratio, but the upper speed threshold after upscaling cannot exceed the maximum design speed of the robot. When the safety distance margin is less than the safety margin threshold, it indicates that the braking distance at the current speed may not be sufficient to ensure safety, and the upper speed threshold needs to be reduced. The speed downscaling ratio is also determined according to the degree to which the safety distance margin is insufficient. The more the deficiency, the greater the downscaling ratio, until the sum of the corrected braking distance and the safety distance margin does not exceed the boundary distance reference value, ensuring that the robot will not exceed the boundary of the movable space region after braking.

[0099] Step S13410: According to the minimum travel speed requirement of the robot and the flatness parameter in the path terrain data, if the road surface fluctuation amplitude corresponding to the flatness parameter in the path terrain data is less than a preset fluctuation threshold, the lower speed threshold is reduced by a preset first adjustment ratio; if the road surface fluctuation amplitude corresponding to the flatness parameter in the path terrain data is greater than or equal to the preset fluctuation threshold, the lower speed threshold is increased by a preset second adjustment ratio, to determine the lower speed threshold.

[0100] The lower speed threshold of the robot needs to consider the minimum travel speed requirement of the robot and the flatness of the path terrain. The minimum travel speed requirement of the robot is determined by the characteristics of its drive system and transmission system. Below this speed, the robot may not be able to travel stably or may experience stuttering. The road surface fluctuation amplitude corresponding to the flatness parameter of the path terrain reflects the flatness of the road surface. The smaller the fluctuation amplitude, the flatter the road surface, and the robot can travel stably at a lower speed. The larger the fluctuation amplitude, the less flat the road surface, and in order to avoid the robot from losing control due to bumps, the lower speed threshold needs to be appropriately increased. The preset first adjustment ratio and the preset second adjustment ratio are determined according to experiments or experience, and are used to adjust the minimum travel speed requirement according to the road surface fluctuation amplitude, to determine the final lower speed threshold.

[0101] Step S13411: The determined upper speed threshold and lower speed threshold and the corresponding constraint condition description are integrated as the robot motion speed limit.

[0102] The upper speed threshold and lower speed threshold determined in the above steps, as well as the constraint condition description related to the two thresholds (such as the basis for speed adjustment, safety distance requirement, etc.) are integrated together to form the robot motion speed limit. The robot motion speed limit clearly defines the speed range that the robot is allowed to travel in the current environment.

[0103] Step S135: Extract the obstacle space position correlation information in the environmental obstacle correlation feature, combine the steering mechanism performance parameters of the robot, calculate the shortest distance between the robot motion trajectory and each obstacle space position under different steering angles, screen out the steering angle range whose shortest distance is greater than the preset collision safety threshold, determine the maximum steering angle and the minimum steering angle in the steering angle range, and form the robot steering angle limit.

[0104] The obstacle space position correlation information in the environmental obstacle correlation feature reflects the spatial position relationship between the obstacles and the robot and between the obstacles. The steering mechanism performance parameters of the robot include the maximum steering angle, the steering angle velocity, the steering transmission ratio, etc., which determine the steering ability of the robot. In order to determine the steering angle limit of the robot, the obstacle space position correlation information needs to be extracted first to obtain the accurate three-dimensional coordinates of each obstacle in the robot coordinate system. Then, for different steering angles (from the minimum possible steering angle to the maximum possible steering angle, changing by a certain step), the kinematic model of the robot is used to calculate the motion trajectory of the robot under the steering angle. The motion trajectory can be represented by a series of discrete coordinate points. For each motion trajectory corresponding to a steering angle, the shortest distance between each point on the trajectory and each obstacle space position is calculated. Compare the above shortest distance with the preset collision safety threshold, and screen out the steering angles whose shortest distance is greater than the collision safety threshold. These steering angles are safe. Among the safe steering angles, find out the maximum steering angle and the minimum steering angle, which constitute the range of the robot steering angle limit.

[0105] Step S136: According to the path terrain data in the environmental perception data set and the path adaptation correlation information in the environmental obstacle correlation feature, determine the reference line of the current travel path of the robot, calculate the distance range of the robot deviating from the reference line in the process of avoiding obstacles, screen out the maximum distance and the minimum distance whose deviation distance will not cause the robot to contact the obstacle, and form the path deviation range limit.

[0106] The path region information in the path terrain data and the path adaptation correlation degree information in the environmental obstacle correlation features are used together to determine the reference line of the current travel path of the robot. The path reference line is usually the center line of the path region, or an ideal travel route planned according to the path terrain features and the distribution of obstacles. After the reference line is determined, the situation that the robot may deviate from the reference line in the process of avoiding obstacles needs to be considered. If the deviation distance is too large, the robot may exceed the path region or collide with other obstacles. Therefore, the distance range in which the robot is allowed to deviate from the reference line in the process of avoiding obstacles needs to be calculated. By analyzing the obstacle position information in the environmental obstacle correlation features and the path terrain data, the distance of the robot from the obstacle under different deviation distances is calculated, and the maximum distance and the minimum distance in which the robot does not come into contact with the obstacle under the deviation distance are screened out. The maximum deviation distance refers to the farthest distance in which the robot can deviate from the reference line without colliding with the obstacle, and the minimum deviation distance refers to the distance in which the robot needs to deviate from the reference line at least to avoid the obstacle. The two distance values constitute the path deviation range limit.

[0107] Step S137: logically integrating the robot motion speed limit, the robot turning angle limit, and the path deviation range limit, setting the priority relationship between the constraint conditions, and selecting the constraint condition that is preferentially satisfied according to the preset priority rule when the different constraint conditions conflict.

[0108] The robot motion speed limit, the turning angle limit, and the path deviation range limit jointly constitute the constraint conditions of the obstacle avoidance decision, but there may be conflicts between these constraint conditions. For example, in order to avoid a certain obstacle, the robot may need a large turning angle, but the turning angle at this time may exceed the turning angle limit; or in order to meet the speed limit, the robot needs to reduce the speed, but this may result in the inability to avoid a fast-moving obstacle in time. Therefore, the above constraint conditions need to be logically integrated and processed, and the priority relationship between the constraint conditions is set. The priority rule can be formulated according to the safety of obstacle avoidance and the priority of the task, for example, the safety priority is the highest, so the priority of the turning angle limit and the path deviation range limit directly related to collision avoidance is higher than that of the motion speed limit; under the premise of ensuring safety, the motion speed limit is considered to improve efficiency. When the different constraint conditions conflict, the constraint condition that is preferentially satisfied is selected according to the preset priority rule, for example, when the turning angle limit and the path deviation range limit conflict, the constraint condition that can ensure the maximum distance from the obstacle is preferentially satisfied.

[0109] Step S138: taking the logically integrated constraint condition as the final obstacle avoidance decision constraint condition.

[0110] After logical integration processing and priority setting, the obtained constraint conditions can coordinate the restrictions of all aspects and ensure the safety and feasibility of the robot in the obstacle avoidance process. The above logically integrated constraint conditions are used as the final obstacle avoidance decision constraint conditions to guide the subsequent preliminary obstacle avoidance path scheme generation and path adjustment processing.

[0111] Step S140: input the environmental obstacle correlation features and the obstacle avoidance decision constraint conditions into the decision output layer of the deep learning obstacle avoidance model to generate a preliminary obstacle avoidance path scheme.

[0112] The environmental obstacle correlation features provide the correlation information between the obstacles and the environment, and the obstacle avoidance decision constraint conditions specify the limitation range of the robot movement. Both of them are input into the decision output layer of the deep learning obstacle avoidance model to generate a preliminary obstacle avoidance path scheme.

[0113] Step S141: perform feature format conversion processing on the environmental obstacle correlation features to make them meet the input format requirements of the decision output layer of the deep learning obstacle avoidance model, and obtain the format-converted environmental obstacle correlation features.

[0114] The decision output layer of the deep learning obstacle avoidance model has specific requirements for the format of the input features (such as data type, dimension order, tensor shape, etc.). The environmental obstacle correlation features may adopt different data formats in the generation process, so feature format conversion processing is needed. For example, convert the data type of the environmental obstacle correlation features from floating point type to the specific precision floating point type or integer type required by the model; adjust the dimension order of the feature vector to make it consistent with the input tensor dimension order of the decision output layer; if the environmental obstacle correlation features are a multi-dimensional array, they need to be reshaped into the expected tensor shape of the decision output layer. Through these format conversion operations, the format-converted environmental obstacle correlation features are obtained to ensure that they can be correctly received and processed by the decision output layer.

[0115] Step S142: perform parameter encoding processing on the obstacle avoidance decision constraint conditions to convert the robot movement speed limit, turning angle limit and path deviation range limit into constraint parameter codes recognizable by the decision output layer, and obtain the encoded obstacle avoidance decision constraint conditions.

[0116] The obstacle avoidance decision constraint conditions (robot motion speed limit, turning angle limit, and path deviation range limit) are usually described in natural language or numerical range form and need to be converted into constraint parameter codes that can be recognized and processed by the decision output layer. Parameter coding processing can adopt one-hot coding, integer coding, or self-defined coding rules, etc. For example, the upper and lower threshold values of the speed limit are converted into floating-point number parameters; the range of the turning angle limit is converted into angle value parameters; and the path deviation range limit is converted into distance value parameters. At the same time, a specific parameter identifier is assigned to each constraint condition, so that the decision output layer can correctly parse and apply these constraint conditions. After parameter coding processing, the encoded obstacle avoidance decision constraint conditions are input into the decision output layer in the form of numerical values or codes.

[0117] Step S143: The formatted environment obstacle associated features and the encoded obstacle avoidance decision constraint conditions are simultaneously input into the constraint input submodule of the decision output layer of the deep learning obstacle avoidance model for data association processing to obtain associated comprehensive feature data.

[0118] The constraint input submodule of the decision output layer is responsible for receiving the formatted environment obstacle associated features and the encoded obstacle avoidance decision constraint conditions and performing data association processing thereon. The purpose of data association processing is to organically combine the constraint conditions with the environment features, so that the decision output layer can consider both the environment information and the constraint limit when generating the path. For example, the speed limit parameter in the obstacle avoidance decision constraint condition is associated with the obstacle motion feature in the environment obstacle associated feature, so that the speed can be dynamically adjusted according to the motion state of the obstacle when the path is generated. The constraint input submodule can add the constraint parameters as additional feature dimensions to the environment obstacle associated features, or integrate the constraint conditions into the processing process of the environment features through a gating mechanism, thereby obtaining the associated comprehensive feature data.

[0119] Step S144: In the path generation submodule of the decision output layer, a path search space is constructed based on the associated comprehensive feature data, and the path search space contains feature points corresponding to all possible paths of the robot from the current position to the target position.

[0120] The core task of the path generation submodule is to generate possible travel paths for the robot from the current position to the target position. Based on the associated comprehensive feature data, the path generation submodule first constructs a path search space. The path search space is an abstract space that contains all possible travel paths, where each point represents a feature point on the path (such as position coordinates, speed, turning angle, etc.). When constructing the path search space, the kinematic model and dynamic constraints of the robot need to be considered to ensure that the generated possible paths are physically achievable by the robot. For example, according to the minimum turning radius limit of the robot, the path in the path search space cannot contain a curve segment smaller than the minimum turning radius. The size and resolution of the path search space can be adjusted according to the complexity of the environment and the accuracy requirements of obstacle avoidance. The more complex the environment and the higher the accuracy requirements, the higher the resolution of the path search space needs to be.

[0121] Step S145: In combination with the encoded obstacle avoidance decision constraint conditions, the constraint satisfaction degree of each possible travel path in the path search space is evaluated, and the satisfaction degree of each possible travel path in terms of motion speed, turning angle, and path deviation to the constraint conditions is calculated.

[0122] For each possible travel path in the path search space, the satisfaction degree of the path to the obstacle avoidance decision constraint conditions needs to be evaluated. Constraint satisfaction degree evaluation is to check and quantitatively score each aspect of the possible travel path according to the encoded obstacle avoidance decision constraint conditions. In terms of motion speed, it is checked whether the speed of each point on the path is within the speed limit range, and the degree of speed exceeding or falling below the limit is calculated; in terms of turning angle, it is checked whether the turning angle of the path is within the turning angle limit range, and the deviation amount of the turning angle is calculated; in terms of path deviation, it is checked whether the distance of the path deviation from the reference line is within the path deviation range limit, and the size of the deviation distance is calculated. For the evaluation result of each aspect, a satisfaction degree score is given, and the higher the score, the better the degree of satisfying the constraint conditions. Then, the overall constraint satisfaction degree evaluation result of each possible travel path is obtained by synthesizing the satisfaction degree scores of each aspect.

[0123] Step S146: According to the constraint satisfaction degree evaluation result, a candidate path set that satisfies all obstacle avoidance decision constraint conditions is selected, and paths that do not satisfy any constraint condition are excluded.

[0124] According to the overall constraint satisfaction degree evaluation result, a constraint satisfaction degree threshold is set. Possible travel paths with an overall constraint satisfaction degree score greater than or equal to the threshold are selected, and these paths satisfy or basically satisfy the obstacle avoidance decision constraint conditions in terms of motion speed, turning angle, and path deviation, etc., to form a candidate path set. At the same time, paths with an overall constraint satisfaction degree score lower than the threshold are excluded, and these paths do not satisfy the constraint conditions in at least one aspect, which may have safety hazards or be unachievable.

[0125] Step S147: path cost calculation is performed on each candidate path in the candidate path set, the path cost including path length cost, motion energy consumption cost and path smoothness cost; the path length cost, motion energy consumption cost and path smoothness cost are normalized to a uniform scale; the normalized path length cost, motion energy consumption cost and path smoothness cost are weighted and summed according to preset weights to obtain a comprehensive path cost; the candidate paths are sorted according to the size of the comprehensive path cost.

[0126] In order to select the optimal path from the candidate path set, path cost calculation needs to be performed on each candidate path. Path cost is a comprehensive index reflecting the pros and cons of the path. Path length cost refers to the total length of the path, the shorter the path, the lower the cost; motion energy consumption cost refers to the energy consumed by the robot when driving along the path, which is related to factors such as path slope, speed change, number of turns, etc., the lower the energy consumption, the lower the cost; path smoothness cost refers to the smoothness of the path, the fewer the turns and undulations on the path, the higher the smoothness, the lower the cost. For the three costs, since their dimensions and value ranges are different, normalization processing is needed to convert them to a uniform scale between 0 and 1. The normalization method can use maximum and minimum normalization or Z-score normalization, etc. Then, according to the preset weights (these weights are determined according to the task requirements and performance indicators of the robot, such as higher path length cost weight for more emphasis on efficiency, higher motion energy consumption cost weight for more emphasis on energy saving), the normalized three costs are weighted and summed to obtain the comprehensive path cost of each candidate path. Finally, the candidate paths are sorted in order of the comprehensive path cost from small to large, the smaller the comprehensive path cost, the better the path.

[0127] Step S148: select the candidate path with the smallest path cost as the base path, and perform path node refinement processing on the base path to add transition nodes between the key turning points of the base path to obtain the refined path.

[0128] In the sorted candidate paths, the candidate path with the minimum comprehensive path cost is selected as the basic path. The basic path is usually connected by a series of key turning points (such as the starting point, the ending point, the obstacle avoidance point, etc.), and the path segments between these turning points can be straight lines or simple curves. In order to improve the accuracy and controllability of the path, the basic path needs to be refined by adding path nodes. In the basic path, transition nodes are added between key turning points according to a preset distance interval or angle interval, so that the path is smoother and the robot can more easily track the path. For example, a transition node is added at a certain distance on a straight line segment between two key turning points; for a curve segment, an appropriate number of transition nodes are added according to the curvature radius of the curve to ensure smooth transition of the curve. After the path node refinement, a refined path is obtained, which contains more path nodes and can more accurately describe the robot's trajectory.

[0129] Step S149: Perform collision detection processing on the refined path and the obstacle spatial position information in the environment obstacle association feature, calculate the distance value between each path node on the refined path and each obstacle spatial position, and determine whether each distance value is greater than a preset safety distance threshold. Keep the refined path whose distance value is greater than the preset safety distance threshold, and generate a preliminary obstacle avoidance path scheme.

[0130] Even the optimal path selected through constraint satisfaction degree evaluation and cost calculation may have a too close distance to the obstacle after refinement. Therefore, collision detection processing is needed. Using the obstacle spatial position information (three-dimensional coordinates of each obstacle) in the environment obstacle association feature, the straight-line distance between each path node on the refined path and each obstacle spatial position is calculated. Each distance value is compared with a preset safety distance threshold, which is the minimum safety distance set according to the size of the robot and the safety requirement of obstacle avoidance. If the distance value between all path nodes on the refined path and each obstacle is greater than the safety distance threshold, the path is safe, and the path is kept as a preliminary obstacle avoidance path scheme. If the distance value between any path node and a certain obstacle is less than or equal to the safety distance threshold, a suboptimal candidate path needs to be selected for refinement and collision detection until a path that meets the safety distance requirement is found, or a new path search space is generated for searching.

[0131] Step S1410: Add path node identifiers and corresponding motion parameter labels to the preliminary obstacle avoidance path scheme, so that the preliminary obstacle avoidance path scheme contains complete path information from the current position to the target position of the robot, including the coordinates of each path node, the corresponding travel speed and turning angle.

[0132] To make the preliminary obstacle avoidance path plan directly applicable to control the motion of the robot, path node identifiers and corresponding motion parameter tags need to be added in the plan. Path node identifiers are unique identifiers for each path node, used to distinguish different nodes. Motion parameter tags include the corresponding travel speed and steering angle of each path node, which are calculated according to the geometry of the path and the obstacle avoidance decision constraints. For example, in a straight line segment path node, the steering angle is 0; in a curve segment path node, the steering angle is calculated according to the curvature of the curve; the travel speed is adjusted according to the speed limit and the path smoothness requirement. After adding these identifiers and tags, the preliminary obstacle avoidance path plan contains the complete path information of the robot from the current position to the target position.

[0133] Step S150: According to the obstacle avoidance decision constraints, the preliminary obstacle avoidance path plan is processed for path adjustment to obtain the final obstacle avoidance strategy, which contains the robot travel speed adjustment parameters, steering operation instructions and path change node information.

[0134] The preliminary obstacle avoidance path plan may still not meet the obstacle avoidance decision constraints in some details, or there is further optimization space, so path adjustment processing is needed according to the obstacle avoidance decision constraints to obtain the final obstacle avoidance strategy.

[0135] Step S151: Extract all path node coordinates, corresponding node planned travel speed and planned steering angle in the preliminary obstacle avoidance path plan to form a preliminary path parameter set.

[0136] The preliminary obstacle avoidance path plan contains a large amount of path node information. In order to process path adjustment, the key parameters need to be extracted first. The coordinates of each path node (in the robot coordinate system or global coordinate system), the corresponding planned travel speed (i.e. the speed the robot should have when reaching the node) and the planned steering angle (i.e. the steering angle the robot should take at the node) are extracted. Organize the above parameters according to the order of the path nodes to form a preliminary path parameter set.

[0137] Step S152: Compare and analyze the preliminary path parameter set with the robot motion speed limit in the obstacle avoidance decision constraints to detect nodes in the preliminary path parameter set whose planned travel speed exceeds the speed limit range, and mark them as speed abnormal nodes.

[0138] The planned travel speed of each path node in the preliminary path parameter set is compared with the robot motion speed limit (upper threshold and lower threshold) in the obstacle avoidance decision constraint condition. For each node, if the planned travel speed is greater than the upper speed threshold or less than the lower speed threshold, the node is determined to be a speed abnormal node and is marked. The marking information includes the node identification, the planned travel speed value, and the extent of exceeding the limit (such as how much above the upper limit or how much below the lower limit). Through the above comparison analysis, the nodes in the preliminary path scheme whose speed does not meet the constraint condition are found.

[0139] Step S153: The planned travel speed of the speed abnormal node is adjusted and calculated, and the adjusted travel speed is determined according to the planned travel speeds of the path nodes before and after the speed abnormal node and the speed limit range, a robot travel speed adjustment parameter is generated, which contains the identification of the speed abnormal node, the speed value before adjustment, and the speed value after adjustment.

[0140] For the marked speed abnormal node, its planned travel speed needs to be adjusted. In the adjustment calculation, the planned travel speeds of the path nodes before and after the speed abnormal node and the speed limit range are comprehensively considered. For example, if the planned travel speed of the speed abnormal node is higher than the upper threshold, the speed of the previous node and the speed of the next node can be referred to, and the speed abnormal node can be adjusted to be within the upper threshold by using linear transition; if the speed abnormal node is located between two speed normal nodes, the speed of the previous node and the speed of the next node can also be referred to, and the speed abnormal node can be adjusted smoothly within the speed limit range according to the speed variation trend, so as to avoid the influence of speed mutation on the motion stability of the robot. The adjusted travel speed must be within the speed limit range. The identification of the speed abnormal node, the speed value before adjustment, and the speed value after adjustment are recorded to generate a robot travel speed adjustment parameter.

[0141] Step S154: The preliminary path parameter set after adjusting the speed is compared and analyzed with the steering angle limit in the obstacle avoidance decision constraint condition, and the nodes whose planned steering angle exceeds the angle limit range in the preliminary path parameter set are detected and marked as steering abnormal nodes.

[0142] After the speed of the speed abnormal node is adjusted, the preliminary path parameter set after adjusting the speed is obtained. Next, the planned steering angle of each path node in the set is compared with the steering angle limit (maximum steering angle and minimum steering angle) in the obstacle avoidance decision constraint condition. If the planned steering angle of a node is greater than the maximum steering angle or less than the minimum steering angle, the node is marked as a steering abnormal node. Similarly, the identification of the steering abnormal node, the planned steering angle value, and the extent of exceeding the limit are recorded.

[0143] Step S155: Adjust the planned turning angle corresponding to the turning abnormal node, combine the spatial position information of the obstacles around the turning abnormal node and the turning angle limit range, calculate the distance value between the robot motion trajectory corresponding to the adjusted turning angle and the surrounding obstacles, and determine the adjusted turning angle when the distance value is greater than the preset collision safety threshold, generate a turning operation instruction, which includes the identification of the turning abnormal node, the turning angle before adjustment and the turning angle after adjustment.

[0144] For the turning abnormal node, the spatial position information of the surrounding obstacles needs to be considered when adjusting the planned turning angle to ensure that the adjusted turning angle does not cause the robot to collide with the obstacles. First, obtain the spatial position information of the obstacles around the turning abnormal node (extracted from the environmental obstacle associated features). Then, try different turning angle values within the turning angle limit range. For each tried turning angle, calculate the robot's motion trajectory under that turning angle according to the robot's kinematic model. Next, calculate the shortest distance value between the motion trajectory and the surrounding obstacles. If the shortest distance value is greater than the preset collision safety threshold, the turning angle is safe. Select a suitable turning angle as the adjusted turning angle, for example, select the turning angle that maximizes the distance between the motion trajectory and the obstacles, or select the turning angle that has the smallest deviation from the original planned turning angle and is safe. Record the identification of the turning abnormal node, the turning angle before adjustment and the turning angle after adjustment, and generate a turning operation instruction.

[0145] Step S156: Compare and analyze the adjusted speed and turning preliminary path parameter set with the path deviation range limit in the obstacle avoidance decision constraint condition, detect the nodes whose path nodes deviate from the reference line beyond the range, and mark them as path deviation nodes.

[0146] After speed and turning adjustment, the adjusted speed and turning preliminary path parameter set is obtained. At this time, it is necessary to check whether the path nodes meet the path deviation range limit. Compare the coordinates of each path node with the reference line of the current path of the robot, and calculate the deviation distance of the node from the reference line. If the deviation distance is greater than the maximum distance of the path deviation range limit or less than the minimum distance (usually the minimum distance is negative, indicating the deviation on the other side of the reference line), the node is marked as a path deviation node. Record the identification, current deviation distance and deviation range limit of the path deviation node.

[0147] Step S157: path changing processing is performed on the path deviation node, the path segment between the path deviation node and the adjacent nodes before and after it is re-planned, the deviation distance of each node on the new path segment from the reference line and the distance value from the obstacle are calculated, and when the deviation distance is within the path deviation range limit and the distance value from the obstacle is greater than the preset safety distance threshold, the identification, the coordinates before changing and the coordinates after changing of the path deviation node are recorded to form path change node information.

[0148] For the path deviation node, the path segment between it and the adjacent nodes before and after it needs to be re-planned to correct the path deviation. When re-planning the path segment, the adjacent nodes before and after the path deviation node are taken as the starting point and the ending point, and a new path segment is generated within the path deviation range limit. The new path segment can be a straight segment, a circular arc segment or other smooth curve segment. For each node on the new path segment, the deviation distance of the node from the reference line is calculated to ensure that the distance is within the path deviation range limit. At the same time, the distance value of the node on the new path segment from the surrounding obstacles is calculated to ensure that all distance values are greater than the preset safety distance threshold. If the new path segment meets these conditions, the path segment is accepted, and the identification, the coordinates before changing and the coordinates after changing of the path deviation node are recorded to form path change node information. If not, the path segment needs to be re-planned until a path segment that meets the conditions is found.

[0149] Step S158: the robot travel speed adjustment parameters, the steering operation instructions and the path change node information are integrated and associated, and each adjustment parameter and instruction can be mapped to a specific path node in the preliminary obstacle avoidance path scheme through the path node identification.

[0150] For example, step S1581: a path node list in the preliminary obstacle avoidance path scheme is extracted, the path node list contains the unique identification, coordinate information, planned execution time and corresponding adjacent node identification of each path node, and a path node index table is formed.

[0151] The path nodes in the preliminary obstacle avoidance path scheme are arranged in the travel order to form a path node list. The detailed information of each path node is extracted from the list, including the unique identification (such as node number), the coordinate information in the robot coordinate system, the planned execution time (i.e. the time when the robot is expected to arrive at the node), and the identification of the previous node and the next node (adjacent node identification). The above information is organized into a table form, i.e. a path node index table.

[0152] Step S1582: Extract the identification of each speed abnormal node from the robot travel speed adjustment parameter, match the identification of the speed abnormal node with the node unique identification in the path node index table, determine the corresponding position and related information of each speed abnormal node in the path node index table, and establish a first association relationship between the speed adjustment parameter and the path node.

[0153] The identification of each speed abnormal node is contained in the robot travel speed adjustment parameter. The above identification is compared one by one with the node unique identification in the path node index table, and the matched node is found. For the matched node, its position (such as line number) in the path node index table and related information (such as coordinate information, planned execution time) are recorded. Through the above matching, a first association relationship between the speed adjustment parameter and the path node is established, and it is clear which speed adjustment parameter corresponds to which path node.

[0154] Step S1583: Extract the identification of each steering abnormal node from the steering operation instruction, match the identification of the steering abnormal node with the node unique identification in the path node index table, determine the corresponding position and related information of each steering abnormal node in the path node index table, and establish a second association relationship between the steering operation instruction and the path node.

[0155] Similarly, the identification of each steering abnormal node is extracted from the steering operation instruction, and is matched with the node unique identification in the path node index table. After the matched node is found, its position and related information in the index table are recorded, and a second association relationship between the steering operation instruction and the path node is established, so that the steering operation instruction can be accurately corresponded to a specific path node.

[0156] Step S1584: Extract the identification of each path change node from the path change node information, match the identification of the path change node with the node unique identification in the path node index table, determine the corresponding position and related information of each path change node in the path node index table, and establish a third association relationship between the path change node information and the path node.

[0157] The identification of the path change node is extracted from the path change node information, and is matched with the node unique identification in the path node index table. After the matching is successful, the position and related information of the node in the index table are recorded, and a third association relationship between the path change node information and the path node is established, so as to ensure that the path change information can be corresponded to the correct node.

[0158] Step S1585: Create an integrated association table, which is associated with the speed adjustment parameters in the first association relationship, the steering operation instructions in the second association relationship, and the path change node information in the third association relationship respectively, with the node unique identifier in the path node index table as the primary key. For nodes without adjustment parameters or instructions, the corresponding field is marked as null.

[0159] The primary key of the integrated association table is the node unique identifier in the path node index table. In the integrated association table, corresponding fields are set for the speed adjustment parameters (such as pre-adjustment speed value and post-adjustment speed value) in the first association relationship, the steering operation instructions (such as pre-adjustment steering angle and post-adjustment steering angle) in the second association relationship, and the path change node information (such as pre-change coordinates and post-change coordinates) in the third association relationship. For nodes in the path node index table without speed adjustment parameters, steering operation instructions, or path change node information, the corresponding fields in the integrated association table are marked as "null". Through the integrated association table, all adjustment information related to the node is centrally managed.

[0160] Step S1586: Perform format standardization processing on the integrated association table, and associate and match the standardized integrated association table with the adjusted path parameter set, and through the node unique identifier mapping, the adjustment information of each path node is mapped to the corresponding node parameter in the path parameter set.

[0161] The data in the integrated association table may have inconsistent formats (such as data type, precision, etc.), and needs to be standardized, for example, unified data type to floating point or string type, and the same number of decimal places. After standardization, the integrated association table is associated and matched with the adjusted path parameter set (the path parameter set after speed, steering, and path adjustment) through the node unique identifier. For each node unique identifier, the adjustment information (speed adjustment, steering adjustment, path change) of the node in the integrated association table is filled into the corresponding node parameter in the path parameter set, replacing the original planned parameter or adding new adjustment fields. Through the above association and matching, the path parameter set contains all the adjustment information.

[0162] Step S1587: The integrated association data after association and matching is taken as the result of the integrated association processing.

[0163] After the above steps, the integrated association data after association and matching contains the unique identifier of the path node and all the adjustment parameters and instruction information. The above data is taken as the result of the integrated association processing, which is used for subsequent final obstacle avoidance strategy generation.

[0164] Step S159: combine the adjusted robot travel speed parameter, steering operation instruction and path change node information after the integration and correlation processing with the adjusted path parameter set to generate the final obstacle avoidance strategy containing complete path information and operation instruction.

[0165] The adjusted parameter and instruction information after the integration and correlation processing and the adjusted path parameter set are both associated with specific path nodes. Combining these two parts of data, i.e. adding the adjusted parameter and instruction information to the corresponding nodes of the adjusted path parameter set, forms a complete data set containing the coordinates of all path nodes, the adjusted travel speed, the adjusted steering angle, the path change information and the corresponding operation instruction. This complete data set is the final obstacle avoidance strategy, which describes in detail the motion parameters and operation requirements of each path node of the robot from the current position to the target position.

[0166] Step S1510: add operation execution timing labels in the final obstacle avoidance strategy.

[0167] In order to enable the robot to execute the operation instructions in the final obstacle avoidance strategy in the correct order, operation execution timing labels need to be added in the final obstacle avoidance strategy. The operation execution timing label is usually the planned execution time of the path node, or the time offset relative to the starting time. Each path node operation instruction (such as speed adjustment, steering operation, path change) corresponds to an execution timing label, indicating when the robot should execute the operation. After adding the operation execution timing label, the final obstacle avoidance strategy not only contains the spatial information of the path and operation, but also contains the time information, enabling the robot control system to accurately execute each operation in time sequence.

[0168] Figure 2 A schematic diagram of exemplary hardware and software components of the deep learning-based obstacle avoidance strategy generation system 100 for a carrying robot according to some embodiments of the present application is shown, which can implement the idea of the present application. For example, the processor 120 can be used on the deep learning-based obstacle avoidance strategy generation system 100 for a carrying robot, and is used to perform the functions in the present application.

[0169] For example, the deep learning based transporting robot obstacle avoidance strategy generation system 100 can include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, a ROM, or a RAM, or any combination thereof. Exemplarily, the deep learning based transporting robot obstacle avoidance strategy generation system 100 can also include program instructions stored in a ROM, a RAM, or other types of non-transitory storage media, or any combination thereof. The methods of the present application can be implemented according to these program instructions. The deep learning based transporting robot obstacle avoidance strategy generation system 100 further includes an I / O interface 150 between the computer and other input and output devices.

[0170] In addition, the embodiment of the present application further provides a readable storage medium, wherein computer executable instructions are preset in the readable storage medium, and when a processor executes the computer executable instructions, the deep learning based transporting robot obstacle avoidance strategy generation method is realized.

[0171] It should be noted that, in order to simplify the expression of the present application and to help the understanding of one or more embodiments of the present application, in the foregoing description of the embodiments of the present application, various features are sometimes incorporated into one embodiment, drawing or description thereof.

Claims

1.A method for generating an obstacle avoidance strategy of a carrying robot based on deep learning, characterized in that, The method comprises: acquiring a set of environment perception data around the carrying robot, the set of environment perception data containing obstacle shape data, obstacle motion state data and path terrain data within the robot travel path range; performing feature correlation processing on the set of environment perception data based on a pre-trained deep learning obstacle avoidance model to generate environment obstacle correlation features, the environment obstacle correlation features reflecting spatial correlation relationships between obstacles and between obstacles and path terrain; acquiring current motion state data of the carrying robot, and constructing obstacle avoidance decision constraint conditions in combination with the environment obstacle correlation features, the obstacle avoidance decision constraint conditions containing robot motion speed limits, turning angle limits and path deviation range limits; inputting the environment obstacle correlation features and the obstacle avoidance decision constraint conditions into a decision output layer of the deep learning obstacle avoidance model to generate a preliminary obstacle avoidance path scheme; performing path adjustment processing on the preliminary obstacle avoidance path scheme according to the obstacle avoidance decision constraint conditions to obtain a final obstacle avoidance strategy, the final obstacle avoidance strategy containing robot travel speed adjustment parameters, turning operation instructions and path change node information. 2.The deep learning-based carrying robot obstacle avoidance strategy generation method according to claim 1, wherein, The acquiring of the set of environment perception data around the carrying robot, the set of environment perception data containing obstacle shape data, obstacle motion state data and path terrain data within the robot travel path range comprises: starting multiple types of environment perception devices carried by the carrying robot to collect initial environment data within the robot travel path range, the initial environment data containing image data collected by a visual sensor, point cloud data collected by a laser radar and distance data collected by an ultrasonic sensor; performing obstacle contour extraction processing on the image data collected by the visual sensor to obtain obstacle contour information in the image data, and converting the obstacle contour information into structured shape description data as a first component of the obstacle shape data; performing obstacle spatial position calculation processing on the point cloud data collected by the laser radar to obtain three-dimensional coordinate information of each obstacle in a robot coordinate system, and performing correlation matching processing on the three-dimensional coordinate information and the obstacle contour information to generate obstacle shape data containing spatial positions as a second component of the obstacle shape data; performing continuous sampling analysis processing on the distance data collected by the ultrasonic sensor to calculate distance change amounts and corresponding time intervals between the same obstacle and the robot at adjacent sampling time points, calculate a motion rate of the obstacle according to the distance change amounts and the time intervals, and determine a motion direction according to the direction of the distance change amounts, and integrate the motion direction and the motion rate information into the obstacle motion state data; performing terrain feature recognition processing on a path region in the image data collected by the visual sensor to extract slope change information, flatness information and obstacle distribution density information within the path region, and combining the slope change information, the flatness information and the obstacle distribution density information into the path terrain data; The first component of the obstacle shape data, the second component of the obstacle shape data, the obstacle motion state data and the path terrain data are spatiotemporally aligned to generate an environment perception data set. 3.The deep learning-based carrying robot obstacle avoidance strategy generation method according to claim 2, characterized in that, The image data collected by the visual sensor is subjected to obstacle contour extraction processing to obtain obstacle contour information in the image data, including: The image data collected by the visual sensor is converted into grayscale image data, and after noise removal processing of the grayscale image data, edge enhancement processing is performed on the denoised grayscale image data to obtain edge-enhanced grayscale image data; The edge-enhanced grayscale image data is subjected to binarization processing to obtain binarized image data, and the binarized image data is subjected to connected region analysis processing to identify all connected white regions in the binarized image data, each connected white region corresponding to a potential obstacle, to obtain a connected region set; Each connected region in the connected region set is subjected to region morphological processing, a dilation algorithm is used to fill small cavities inside the connected region, and an erosion algorithm is used to eliminate small protrusions on the edge of the connected region, to obtain morphologically optimized connected regions; Each morphologically optimized connected region is subjected to contour tracking processing, and a chain code tracking algorithm is used to record the coordinate sequence of the edge pixel points of the connected region, and the coordinate sequence constitutes the contour information of the obstacle; The contour information obtained by tracking is subjected to simplification processing to remove redundant pixel points in the contour and retain key pixel points that can reflect the main shape features of the obstacle contour, to obtain simplified obstacle contour information; The simplified obstacle contour information is compared with a common obstacle contour feature library to calculate a similarity value of the simplified obstacle contour information and the common obstacle contour feature, and the contour information with a similarity value greater than a preset similarity threshold is selected, and the contour information with a similarity value less than or equal to the preset similarity threshold and an area less than a preset area threshold is excluded, to obtain the final obstacle contour information in the image data. 4.The method of claim 1, wherein, The environment perception data set is subjected to feature correlation processing based on the pre-trained deep learning obstacle avoidance model to generate environment obstacle correlation features, including: The obstacle shape data, the obstacle motion state data and the path terrain data in the environment perception data set are input into the feature input layer of the deep learning obstacle avoidance model, and after feature dimension standardization processing of each type of data, the standardized dimension obstacle shape data is subjected to shape feature encoding processing in the feature extraction layer of the deep learning obstacle avoidance model to generate an obstacle shape feature vector, the obstacle shape feature vector containing obstacle contour complexity information, volume size information and surface texture information; The standardized dimension obstacle motion state data is subjected to motion feature encoding processing to generate an obstacle motion feature vector, the obstacle motion feature vector containing obstacle motion direction stability information, motion rate trend information and motion trajectory prediction information; The path terrain data after the standardized dimension is subjected to terrain feature coding processing to generate a path terrain feature vector, which contains information about the change range of path slope, fluctuation information about road flatness, and information about the density of obstacle distribution; The obstacle shape feature vector and the obstacle motion feature vector are input into a first feature association layer of the deep learning obstacle avoidance model, the association weight between the two feature vectors is calculated through an attention mechanism, feature fusion processing is performed based on the association weight, and an obstacle self-association feature is generated; The obstacle self-association feature and the path terrain feature vector are input into a second feature association layer of the deep learning obstacle avoidance model, the spatial position association degree, the motion influence association degree and the terrain adaptation association degree between the obstacle and the path terrain are calculated, feature integration processing is performed based on the spatial position association degree, the motion influence association degree and the terrain adaptation association degree, and dimension compression processing is performed on the integrated feature to generate an environment obstacle association feature. 5.The deep learning-based carrying robot obstacle avoidance strategy generation method according to claim 4, characterized in that, The obstacle shape feature vector and the obstacle motion feature vector are input into a first feature association layer of the deep learning obstacle avoidance model, the association weight between the two feature vectors is calculated through an attention mechanism, feature fusion processing is performed based on the association weight, and an obstacle self-association feature is generated, including: The obstacle shape feature vector and the obstacle motion feature vector are input into a feature preprocessing submodule of the first feature association layer for standardization processing to obtain a standardized obstacle shape feature vector and a standardized obstacle motion feature vector; In an attention weight calculation submodule of the first feature association layer, the standardized obstacle shape feature vector is taken as a query vector, and the standardized obstacle motion feature vector is taken as a key vector, a similarity matrix between the query vector and the key vector is calculated, and each element in the similarity matrix represents the similarity between the corresponding dimensions of the two feature vectors; The similarity matrix is subjected to softmax normalization processing to convert each element in the similarity matrix into a probability value between 0 and 1, and the sum of the probability values of all elements is 1, thereby obtaining an attention weight matrix, and the elements in the attention weight matrix are the association weights between the two feature vectors; The standardized obstacle motion feature vector is taken as a value vector, the attention weight matrix and the value vector are subjected to matrix multiplication to obtain a weighted obstacle motion feature vector, and the weighted obstacle motion feature vector highlights the motion feature dimensions with high association degree with the obstacle shape feature; The standardized obstacle shape feature vector and the weighted obstacle motion feature vector are subjected to feature splicing processing, and the two vectors are combined along the feature dimension direction to obtain a spliced feature vector; The spliced feature vector is subjected to feature dimension reduction processing, and a principal component analysis algorithm is used to extract the main feature components in the spliced feature vector to obtain a reduced feature vector; The feature vector after dimension reduction is subjected to nonlinear transformation processing, and each element in the feature vector is subjected to nonlinear mapping through an activation function to enhance the expression ability of the feature vector to the correlation between the obstacles themselves, and a transformed feature vector is obtained; The transformed feature vector is subjected to feature smoothing processing, and a moving average algorithm is used to smooth the adjacent dimension feature values in the feature vector to obtain a smoothed feature vector; The smoothed feature vector is determined as the obstacle self-correlation feature, which contains the correlation information between the obstacle shape and movement. 6.The deep learning-based carrying robot obstacle avoidance strategy generation method of claim 1, wherein, The current motion state data of the carrying robot is acquired, and the environmental obstacle correlation feature is combined to construct an obstacle avoidance decision constraint condition, which includes: A motion state sensor carried by the carrying robot is started to collect the current motion parameter data of the robot, which includes the current travel speed, current steering angle, current acceleration, and current position coordinate data of the robot; The motion parameter data is subjected to outlier detection processing to exclude abnormal motion parameters generated due to interference in the sensor collection process, and the valid motion parameter data conforming to the actual motion state of the robot is retained as the current motion state data of the carrying robot; The obstacle distribution density information and obstacle movement trend information in the environmental obstacle correlation feature are extracted, the restriction range of the obstacle distribution on the robot travel space is analyzed, and the space region boundary available for the robot in the current environment is determined; Based on the current travel speed in the current motion state data of the carrying robot, the distance information of the space region boundary is combined to calculate the braking distance of the robot in an emergency, and the upper and lower threshold values of the robot motion speed are determined according to the braking distance to form a robot motion speed limit; The obstacle space position correlation information in the environmental obstacle correlation feature is extracted, and the steering mechanism performance parameters of the robot are combined to calculate the shortest distance between the robot motion trajectory and each obstacle space position under different steering angles, and the steering angle range with the shortest distance greater than a preset collision safety threshold value is screened out to determine the maximum and minimum steering angles in the steering angle range, and a robot steering angle limit is formed; According to the path terrain data in the environmental perception data set and the path adaptation correlation information in the environmental obstacle correlation feature, the reference line of the current travel path of the robot is determined, the distance range of the robot deviating from the reference line in the process of avoiding obstacles is calculated, the maximum and minimum distances deviating from the reference line without causing the robot to contact the obstacles are screened out, and a path deviation range limit is formed; The robot motion speed limit, robot steering angle limit, and path deviation range limit are subjected to logical integration processing, and the priority relationship between the constraint conditions is set, and when different constraint conditions conflict, the constraint condition that is preferentially satisfied is selected according to the preset priority rule; The constraint condition after logical integration is used as the final obstacle avoidance decision constraint condition. 7.The deep learning-based carrying robot obstacle avoidance strategy generation method according to claim 6, characterized in that, The current travel speed in the current motion state data of the carrying robot is extracted, and a speed value of the current travel speed is recorded as a current speed reference value. A straight-line distance from the current position of the robot to the nearest space region boundary is extracted from the distance information of the space region boundary, and a distance value of the straight-line distance is taken as a boundary distance reference value. The braking performance parameters of the carrying robot are obtained, including the maximum braking acceleration, braking response time, and road surface friction coefficient of the robot braking system, which are determined based on the hardware specifications of the robot and the current path terrain data. The braking response time and the current speed reference value are used to calculate the uniform travel distance of the robot in the braking response stage, which is equal to the product of the braking response time and the current speed reference value, to obtain a response stage travel distance. The maximum braking acceleration and the current speed reference value are used to calculate the deceleration travel distance of the robot in the braking execution stage, to obtain an execution stage travel distance. The response stage travel distance and the execution stage travel distance are added to obtain the total braking distance of the robot in an emergency, which represents the total distance traveled by the robot from the start of the triggered braking instruction to complete stop. The slope correction processing is performed on the total braking distance considering the slope change information in the path terrain data. If the slope change information of the current path shows that the slope value is greater than 0, a first correction coefficient is calculated according to the uphill slope value, and the total braking distance is multiplied by the first correction coefficient. If the slope change information of the current path shows that the slope value is less than 0, a second correction coefficient is calculated according to the downhill slope value, and the total braking distance is multiplied by the second correction coefficient to obtain the corrected braking distance. The boundary distance reference value and the corrected braking distance are used to determine a safety distance margin, which is the boundary distance reference value minus the corrected braking distance. If the safety distance margin is greater than a preset safety margin threshold, the braking distance at the current speed is sufficient for safety, and the speed upper limit threshold is increased according to a preset speed upscaling ratio. If the safety distance margin is less than the preset safety margin threshold, the speed upper limit threshold is decreased according to a preset speed downscaling ratio, so that the sum of the corrected braking distance and the safety distance margin does not exceed the boundary distance reference value. ​ According to the minimum travel speed requirement of the robot and the flatness parameter in the path terrain data, if the road surface fluctuation amplitude corresponding to the flatness parameter in the path terrain data is less than a preset fluctuation threshold, the lower limit threshold of the speed is reduced by a preset first adjustment ratio; if the road surface fluctuation amplitude corresponding to the flatness parameter in the path terrain data is greater than or equal to the preset fluctuation threshold, the lower limit threshold of the speed is increased by a preset second adjustment ratio, and the lower limit threshold of the speed is determined; The determined upper limit threshold of the speed, the lower limit threshold of the speed and the corresponding constraint condition are integrated as a robot motion speed limit. 8.The deep learning-based carrying robot obstacle avoidance strategy generation method of claim 1, wherein, The environmental obstacle associated features and the obstacle avoidance decision constraint condition are input into the decision output layer of the deep learning obstacle avoidance model to generate a preliminary obstacle avoidance path scheme, including: The environmental obstacle associated features are subjected to feature format conversion processing to meet the input format requirements of the decision output layer of the deep learning obstacle avoidance model, and the format-converted environmental obstacle associated features are obtained; The obstacle avoidance decision constraint condition is subjected to parameter encoding processing to convert the robot motion speed limit, the steering angle limit and the path deviation range limit into constraint parameter codes recognizable by the decision output layer, and the encoded obstacle avoidance decision constraint condition is obtained; The format-converted environmental obstacle associated features and the encoded obstacle avoidance decision constraint condition are simultaneously input into the constraint input sub-module of the decision output layer of the deep learning obstacle avoidance model for data association processing, and the format-converted environmental obstacle associated features are obtained; In the path generation sub-module of the decision output layer, a path search space is constructed based on the format-converted environmental obstacle associated features, and the path search space contains feature points corresponding to all possible travel paths of the robot from the current position to the target position; The constraint satisfaction degree of each possible travel path in the path search space is evaluated in combination with the encoded obstacle avoidance decision constraint condition, and the satisfaction degree of each possible travel path in terms of motion speed, steering angle and path deviation to the constraint condition is calculated; According to the constraint satisfaction degree evaluation result, a candidate path set satisfying all obstacle avoidance decision constraint conditions is selected, and paths not satisfying any constraint condition are excluded; The path cost of each candidate path in the candidate path set is calculated, the path cost includes path length cost, motion energy consumption cost and path smoothness cost, the path length cost, motion energy consumption cost and path smoothness cost are normalized to a unified scale, the normalized path length cost, motion energy consumption cost and path smoothness cost are weighted and summed according to a preset weight, and a comprehensive path cost is obtained, and the candidate paths are sorted according to the size of the comprehensive path cost; The candidate path with the minimum path cost is selected as a basic path, and the basic path is subjected to path node refinement processing, and transition nodes are added between key turning points of the basic path to obtain a refined path. The refined path is subjected to collision detection processing with the obstacle spatial position information in the environment obstacle association feature, distance values of each path node on the refined path and each obstacle spatial position are calculated, whether each distance value is greater than a preset safety distance threshold is judged, the refined path with all distance values greater than the preset safety distance threshold is reserved, and a preliminary obstacle avoidance path scheme is generated; Path node identifiers and corresponding motion parameter tags are added in the preliminary obstacle avoidance path scheme, so that the preliminary obstacle avoidance path scheme contains complete path information of the robot from the current position to the target position, including coordinates of each path node, corresponding travel speed and turning angle. 9.The deep learning-based carrying robot obstacle avoidance strategy generation method of claim 1, wherein, The preliminary obstacle avoidance path scheme is subjected to path adjustment processing according to the obstacle avoidance decision constraint condition, and a final obstacle avoidance strategy is obtained, including: All path node coordinates, planned travel speeds and planned turning angles of corresponding nodes in the preliminary obstacle avoidance path scheme are extracted to form a preliminary path parameter set; The preliminary path parameter set is subjected to comparison and analysis with a robot motion speed limit in the obstacle avoidance decision constraint condition, nodes with planned travel speeds exceeding the speed limit range in the preliminary path parameter set are detected and marked as speed abnormal nodes; The planned travel speed of the speed abnormal node is subjected to adjustment calculation, the adjusted travel speed is determined according to the planned travel speeds of path nodes before and after the speed abnormal node and the speed limit range, a robot travel speed adjustment parameter is generated, and the robot travel speed adjustment parameter contains the identifier of the speed abnormal node, the speed value before adjustment and the speed value after adjustment; The preliminary path parameter set after adjustment of the speed is subjected to comparison and analysis with a turning angle limit in the obstacle avoidance decision constraint condition, nodes with planned turning angles exceeding the angle limit range in the preliminary path parameter set are detected and marked as turning abnormal nodes; The planned turning angle of the turning abnormal node is subjected to adjustment calculation, the distance value of the robot motion trajectory corresponding to the adjusted turning angle and the surrounding obstacle is calculated in combination with the spatial position information of the surrounding obstacle of the turning abnormal node and the turning angle limit range, the adjusted turning angle is determined when the distance value is greater than a preset collision safety threshold, a turning operation instruction is generated, and the turning operation instruction contains the identifier of the turning abnormal node, the turning angle before adjustment and the turning angle after adjustment; The preliminary path parameter set after adjustment of the speed and the turning is subjected to comparison and analysis with a path deviation range limit in the obstacle avoidance decision constraint condition, nodes with path nodes deviating from the reference line exceeding the range are detected in the preliminary path parameter set, and the nodes are marked as path deviation nodes; The path deviation node is subjected to path change processing, a path segment between the path deviation node and adjacent nodes before and after the path deviation node is re-planned, a deviation distance of each node on the new path segment from the reference line and a distance value of each node from the obstacle are calculated, the identifier of the path deviation node, the coordinate before change and the coordinate after change are recorded when the deviation distance conforms to the path deviation range limit and the distance value from the obstacle is greater than a preset safety distance threshold, and path change node information is formed. The robot travel speed adjustment parameter, the steering operation instruction and the path change node information are integrated and associated, and each adjustment parameter and instruction can be corresponded to a specific path node in the preliminary obstacle avoidance path scheme through path node identification mapping; The robot travel speed adjustment parameter, the steering operation instruction and the path change node information after the integrated and associated processing are combined with the adjusted path parameter set to generate a final obstacle avoidance strategy containing complete path information and operation instructions; An operation execution timing label is added in the final obstacle avoidance strategy. 10.A system for generating an obstacle avoidance strategy for a transport robot based on deep learning, characterized in that, A processor and a memory are included, the memory is connected with the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to realize the deep learning-based obstacle avoidance strategy generation method for the carrying robot in any one of claims 1-9.

Images