Road inspection robot obstacle avoidance method and system based on instance segmentation

Abstract

The invention relates to a road inspection robot obstacle avoidance method and system based on instance segmentation, and the method comprises the following steps: obtaining a real-time video of an inspection region in a road inspection process, carrying out the obstacle labeling of road images in the real-time video frame by frame, and dividing the labeled obstacle images into a training set and a test set; constructing a real-time end-to-end target detection model, wherein the real-time end-to-end target detection model comprises an image encoder module, a prompt encoder module, a mask decoder module, a memory attention module, a memory decoder module and a memory bank; and training a real-time end-to-end target detection model by using the training set to obtain a trained real-time end-to-end target detection model for performing frame-by-frame real-time detection on each frame of image in the real-time video of the inspection area in the road inspection process. According to the invention, obstacles in the inspection process can be detected more accurately and efficiently in real time.

Description

Technical Field

[0001] The present invention relates to the field of prefabricated factories and instance segmentation, and in particular to an obstacle avoidance method and system for a road inspection robot based on instance segmentation. Background Art

[0002] In prefabricated factories, obstacles on factory roads will affect the transportation safety of prefabricated components. Inspections can not only identify and deal with potential safety hazards in a timely manner to avoid personal injury or property loss, but also help ensure that the functional performance of prefabricated roads meets the design requirements and improve the overall efficiency and comfort of vehicle driving. Conventional manual inspections are characterized by low work efficiency and high costs. At the same time, manual inspections have low accuracy and cannot be quantified. The results of inspections are easily affected by personal experience, skills, and the working status of the day, resulting in low stability and reliability of the inspection results. At the same time, manual inspections are also cost-effective. At the same time, manual inspections in dangerous areas pose safety risks.

[0003] Compared with traditional manual inspection, robot inspection can work in dangerous environments such as high temperature, high voltage, and strong electricity, avoiding the risk of personal injury to inspection personnel and reducing the safety production risks caused by improper equipment operation. The traditional inspection robot obstacle avoidance adopts the target detection algorithm, which only provides a rectangular bounding box to locate the object, that is, the target position can only be located through the detection box, and the object dimension cannot be accurately determined, and the specific shape information of the object cannot be provided. The method of the present invention can more accurately segment the outline of the object through the new instance segmentation, and can distinguish and separately mark the overlapping or touching objects in the image, avoiding the deficiency that the target detection may treat them as a whole when processing overlapping objects, and it is difficult to distinguish the individual instances. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above technologies and provide a road inspection robot obstacle avoidance method and system based on instance segmentation with stronger real-time performance and higher accuracy.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] In a first aspect, the present invention provides a method for avoiding obstacles of a road inspection robot based on instance segmentation, the method comprising the following steps:

[0007] Step 1: obtain real-time video of the inspection area during the road inspection process, and mark obstacles frame by frame on the road images in the real-time video, and divide the marked obstacle images into a training set and a test set;

[0008] Step 2: Build a real-time end-to-end object detection model.

[0009] The real-time end-to-end object detection model includes an image encoder module, a prompt encoder module, a mask decoder module, a memory attention module, a memory decoder module and a memory bank. The image encoder module adopts a VIM model and has recognition conditional conditioning. The VIM model is used to divide the input first image into blocks and extract feature vectors for each block of the image. The image encoder outputs a first image embedding and a first recognition conditional conditioning.

[0010] The prompt encoder module includes a convolution block and a CLIP, the mask prompt is input into the convolution block, the text prompt is input into the CLIP, and the prompt encoder module finally outputs a target token and a prompt token, and the target token and the prompt token are combined to form a prompt embedding;

[0011] The hint embedding is input into a mask decoder, which includes a first CBAM attention layer, a second CBAM attention layer, a first MLP, a third CBAM attention layer, a fourth CBAM attention layer, a convolution operation, a second MLP, and outputs an instance segmentation map;

[0012] The instance segmentation map output by the mask decoder is then input into the memory decoder module, and after three convolution operations, a feature map and an object point vector are obtained, which are then summed with the first image embedding to realize the combination of the feature map and object point output by the three convolution operations with the first image embedding to become the first memory feature; the first memory feature is input into the memory bank for storage; at this point, the processing of the first image is completed;

[0013] Then, the second image is processed, and the image encoder module outputs the second image embedding and the second recognition condition conditioned; the second image embedding is input into the memory attention module, and the memory attention module includes a CBAM attention mechanism and a cross attention mechanism connected in sequence. The output result of the CBAM attention mechanism is input into the cross attention mechanism as a query. At the same time, the first memory feature stored in the memory bank is input into the cross attention mechanism as a key and a value. After the operation of the cross attention mechanism, an updated second image embedding is output;

[0014] Finally, the updated second image embedding is input into the second CBAM attention mechanism layer and the third CBAM attention mechanism layer of the mask decoder respectively. The mask decoder outputs an instance segmentation map, which is then input into the memory decoder module. In the memory decoder module, the instance segmentation map output by the mask decoder is processed by the convolution operation and combined with the second image embedding to become the second memory feature, which is then input into the memory bank for storage. At this point, the processing of the second image is completed;

[0015] Following the above process, all images are processed;

[0016] The workflow of the VIM model is as follows: first, the input image F is divided into multiple non-overlapping blocks of fixed size, and then the position information of each image block is embedded into its corresponding pixel vector representation information, and the pixel vector representation information of each image block is flattened and arranged into a sequence to obtain the original pixel sequence information and the corresponding position information of each image block; the original pixel sequence information of each image block is linearly transformed through a linear layer to adjust its data dimension, and the combined embedding features of the primary pixel sequence information and the corresponding position information of each image block are obtained; the linear transformation is expressed as Y=TW Y +b Y , where T is the original pixel sequence information of the image block, W Y is the weight matrix, b Y is the bias vector, and Y is the primary pixel sequence information obtained after transformation;

[0017] Then, the primary pixel sequence information of the image block is batch normalized through the normalization layer;

[0018] Afterwards, the primary pixel sequence information Y(B, M, D) of the image block is processed by the normalization layer and the results are projected into a sequence X(B, M, E) and a sequence Z(B, M, E) with a dimension of E using two different projection layers respectively;

[0019] Where B is the batch size; M is the sequence length; D is the original feature dimension; E is the expanded state dimension, which is set to 384;

[0020] The two different projection layers are two different linear layers, which project the input sequence to generate sequences X and Z; the linear layer is composed of a weight matrix and a bias term, and for sequences X and Z, there are weight matrices Wx and Wz and bias vectors bx and bz respectively; the input sequence is multiplied by the weight matrix Wx, and the bias bx is added to obtain the sequence X,

[0021] X=Wx×Input+bx,

[0022] Input is the input sequence, Wx is a D×E matrix; the sequence Z is obtained in the same way. The dimensions of the weight matrix and bias vector of the corresponding linear layer are the same as those of the sequence X, but the element values ​​are different;

[0023] Then, the sequence X is processed in both forward and backward directions to capture local dependencies in the vector. For the forward processing, a one-dimensional forward convolution operation is first applied to the sequence X to obtain X' o , and then X'o Linear projection to three different linear layers, respectively, to obtain sequence B o :(B,M,N), o:(B,M,N) and Δ o , where the sequence Δ o Use a soft-add activation function to ensure it is positive;

[0024] Δ o =log(1+exp(X' o ))

[0025] Where N is the dimension of the hidden state;

[0026] Then the sequence Δ o Apply the Hadamard product according to the following formula and A o 、B o Multiply to get the conversion parameter and

[0027]

[0028] Finally, use sequence C o , conversion parameters Conversion parameters And the output of the forward convolution X' o Through the spatial state model SSM, the forward processing result y of X is calculated forward ;

[0029] Apply a one-dimensional backward convolution operation and a spatial state model to obtain the backward processing result y of the sequence X backward ;

[0030] Will get y from the sequence X forward ,y backward The results obtained after the activation function processing of the sequence Z are added separately, and then the two results are added together. After the result is processed by the projection layer, it is residually connected with the primary pixel sequence information Y of the image block, and then the final encoded pixel sequence information T is generated. l , T l It will be sent to the multi-layer perceptron MLP for processing. l Combined with the corresponding image block position information, the image embedding of the image block is obtained, which is the output of the image encoder;

[0031] Step 3: Use the training set to train a real-time end-to-end target detection model. During training, the mask is used as a prompt input to prompt the encoder module to obtain a trained real-time end-to-end target detection model, which is used to perform real-time detection frame by frame on each frame of the real-time video in the inspection area during the road inspection process.

[0032] Furthermore, the mask decoder module is used to combine the image embedding output by the image encoder and the target token and the prompt token output by the prompt encoder as input, and finally output the predicted segmentation mask. The specific workflow is as follows:

[0033] First, the target token and the prompt token are combined into the prompt embedding F0, and F0 is input into the first CBAM attention mechanism layer as the query, key, and value. The first CBAM attention mechanism layer outputs the one-dimensional first output information F1;

[0034] The first CBAM attention mechanism layer will update the representations of these tokens internally, and each token will be updated according to the information of other tokens; for each token, its attention score with all other tokens in the set (including itself) is calculated, the attention score is normalized, and the normalized attention score is used as a weight to perform a weighted sum of the representations of other tokens; the updated token representation is added to the original token representation through a residual connection, and then the updated token representation is normalized using layer normalization to obtain the one-dimensional first output information F1;

[0035] Then, the first output information F1 is input as Query into the second CBAM attention mechanism layer. At the same time, the encoding sequence information in the updated image embedding feature information output by the memory attention module is input as Key and Value into the second CBAM attention mechanism layer. The Query, Key and Value are combined into feature information of a three-dimensional matrix. The feature information is processed by the second CBAM attention mechanism layer to obtain a one-dimensional second output information F2.

[0036] The second output information F2 is input into the first MLP layer, and after being processed by the first MLP layer, the third output information F3 is obtained; wherein the part of the third output information F3 corresponding to the first output information F1 is a mask prediction token related to the image segmentation task, and the part corresponding to the encoded pixel sequence information in the image embedding feature information is a specific image feature representation used to predict the segmentation mask;

[0037] Then, the third output information F3 is input as Query to the third CBAM attention mechanism layer, and the output of the cross attention mechanism of memory attention is input as Key and Value to the third CBAM attention mechanism layer, and then after processing and updating the information, the one-dimensional fourth output information F4 is obtained;

[0038] The fourth output information F4 is used as the key and value and the mask prediction token in the third output information F3 is used as the query to be input into the fourth CBAM attention mechanism layer again. After processing, the mask of each target point is output to obtain the one-dimensional fifth output information F5; wherein the part of F5 corresponding to the first output information F1 is the mask prediction token related to the image segmentation task, and the part corresponding to the encoded pixel sequence information in the image embedding feature information is the specific image feature representation used to predict the segmentation mask;

[0039] At the same time, the fourth output information F4 undergoes two convolutions to obtain an output feature F6 of the same size as the original image F;

[0040] The part of the fifth output information F5 corresponding to the mask prediction token is input into the second MLP layer, and the output of the second MLP layer is dot-multiplied with the output feature F6 to obtain the final instance segmentation map.

[0041] Furthermore, a width threshold of the obstacle, a first distance threshold and a second distance threshold between the obstacle and the robot are set. The inspection robot uses the trained real-time end-to-end target detection model to obtain in real time whether there is an obstacle in the real-time video frame by frame within the camera acquisition range. If there is an obstacle, the width of the obstacle is obtained. If the obstacle width is greater than the set obstacle width threshold and the distance between the obstacle and the robot is less than the first distance threshold, or if the obstacle width is not greater than the set obstacle width threshold and the distance between the obstacle and the robot is less than the second distance threshold, the inspection robot automatically adjusts the route to avoid obstacles.

[0042] The first distance threshold is greater than the second distance threshold.

[0043] Furthermore, the width threshold of the obstacle is 3 m, the first distance threshold between the obstacle and the robot is 2 m, and the second distance threshold is 1 m.

[0044] In a second aspect, the present invention provides a road inspection robot obstacle avoidance system based on instance segmentation, which executes the obstacle avoidance method, and the system comprises:

[0045] Image acquisition module, used to obtain real-time video of the inspection area during road inspection;

[0046] An image processing module is used to mark obstacles frame by frame on the road image in the real-time video collected by the image acquisition module, and mark the category to which the obstacles belong;

[0047] The obstacle image includes at least one of a tree obstacle image, a rock obstacle image, a construction material obstacle image, a road maintenance facility obstacle image, and an animal obstacle image;

[0048] Real-time end-to-end object detection model, used to achieve real-time instance segmentation of obstacles and obtain the width of obstacles;

[0049] Distance detection unit: used to measure the real-time distance between obstacles and the robot;

[0050] Obstacle avoidance unit: According to the width of the obstacle and the real-time distance measured by the distance detection unit, the inspection robot automatically adjusts its route to avoid obstacles.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention creatively utilizes a real-time end-to-end target detector to realize rapid and high-precision detection of obstacles such as trees, rocks, construction materials, road maintenance facilities, etc., especially moving obstacles such as animals, by road inspection robots. It can more accurately and efficiently detect obstacles in the inspection process in real time and remind staff to deal with them in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a structural diagram of the real-time end-to-end target detection model in the present invention.

[0054] Figure 2 It is a structural schematic diagram of the VIM model in the present invention.

[0055] Figure 3 It is a structural diagram of the CBAM attention mechanism. DETAILED DESCRIPTION

[0056] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the drawings and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0057] The obstacle avoidance method of a road inspection robot based on instance segmentation of the present invention comprises the following steps:

[0058] Step 1: obtain real-time video of the inspection area during the road inspection process, and mark obstacles frame by frame on the road images in the real-time video, and divide the marked obstacle images into a training set and a test set;

[0059] Step 2: Build a real-time end-to-end object detection model.

[0060] The real-time end-to-end object detection model (see Figure 1) includes an image encoder module, a prompt encoder module, a mask decoder module, a memory attention module, a memory decoder module, and a memory bank, wherein the image encoder module adopts a VIM model and has a recognition condition conditioned, uses the VIM model to divide the input first image into blocks, and extracts a feature vector for each block of the image, and the image encoder outputs a first image embedding and a first recognition condition conditioned;

[0061] The prompt encoder module includes a convolution block and a CLIP, the mask prompt is input into the convolution block, the text prompt is input into the CLIP, and the prompt encoder module finally outputs a target token and a prompt token, and the target token and the prompt token are combined to form a prompt embedding;

[0062] The hint embedding is input into a mask decoder, which includes a first CBAM attention layer, a second CBAM attention layer, a first MLP, a third CBAM attention layer, a fourth CBAM attention layer, a convolution operation, a second MLP, and outputs an instance segmentation map;

[0063] The instance segmentation map output by the mask decoder is then input into the memory decoder module, and after three convolution operations, a feature map and an object point vector are obtained, which are then summed with the first image embedding to realize the combination of the feature map and object point output by the three convolution operations with the first image embedding to become the first memory feature; the first memory feature is input into the memory bank for storage; at this point, the processing of the first image is completed;

[0064] Then, the second image is processed, and the image encoder module outputs the second image embedding and the second recognition condition conditioned; the second image embedding is input into the memory attention module, and the memory attention module includes a CBAM attention mechanism and a cross attention mechanism connected in sequence. The output result of the CBAM attention mechanism is input into the cross attention mechanism as a query. At the same time, the first memory feature stored in the memory bank is input into the cross attention mechanism as a key and a value. After the operation of the cross attention mechanism, an updated second image embedding is output;

[0065] Finally, the updated second image embedding is input into the second CBAM attention mechanism layer and the third CBAM attention mechanism layer of the mask decoder respectively. The mask decoder outputs an instance segmentation map, which is then input into the memory decoder module. In the memory decoder module, the instance segmentation map output by the mask decoder is processed by the convolution operation and combined with the second image embedding to become the second memory feature, which is then input into the memory bank for storage. At this point, the processing of the second image is completed;

[0066] Following the above process, all images are processed;

[0067] The VIM model (see Figure 2 ) is as follows: first, the input image F is divided into multiple non-overlapping blocks of fixed size, and then the position information of each image block is embedded into its corresponding pixel vector representation information, and the pixel vector representation information of each image block is flattened and arranged into a sequence to obtain the original pixel sequence information of each image block and the corresponding position information; the original pixel sequence information of each image block is linearly transformed through a linear layer to adjust its data dimension, and the combined embedding feature of the primary pixel sequence information and the corresponding position information of each image block is obtained; the linear transformation is expressed as Y = TW Y +b Y , where T is the original pixel sequence information of the image block, W Y is the weight matrix, b Y is the bias vector, and Y is the primary pixel sequence information obtained after transformation;

[0068] Then, the primary pixel sequence information of the image block is batch normalized through the normalization layer;

[0069] Afterwards, the primary pixel sequence information Y(B, M, D) of the image block is processed by the normalization layer and the results are projected into a sequence X(B, M, E) and a sequence Z(B, M, E) with a dimension of E using two different projection layers respectively;

[0070] Where B is the batch size; M is the sequence length; D is the original feature dimension; E is the expanded state dimension, which is set to 384;

[0071] The two different projection layers are two different linear layers, which project the input sequence to generate sequences X and Z; the linear layer is composed of a weight matrix and a bias term, and for sequences X and Z, there are weight matrices Wx and Wz and bias vectors bx and bz respectively; the input sequence is multiplied by the weight matrix Wx, and the bias bx is added to obtain the sequence X,

[0072] X=Wx×Input+bx,

[0073] Input is the input sequence, Wx is a D×E matrix; the sequence Z is obtained in the same way. The dimensions of the weight matrix and bias vector of the corresponding linear layer are the same as those of the sequence X, but the element values ​​are different;

[0074] Then, the sequence X is processed in both forward and backward directions to capture local dependencies in the vector. For the forward processing, a one-dimensional forward convolution operation is first applied to the sequence X to obtain X' o , and then X' o Linear projection to three different linear layers, respectively, to obtain sequence B o :(B,M,N),C o :(B, M, N) and Δ o , where the sequence Δ o Use a soft-add activation function to ensure it is positive;

[0075] Δ o =log(1+exp(X' o ))

[0076] Where N is the dimension of the hidden state;

[0077] Then the sequence Δ o Apply the Hadamard product according to the following formula and A o 、B o Multiply to get the conversion parameter and

[0078]

[0079] Finally, use sequence C o , conversion parameters Conversion parameters And the output of the forward convolution X' o Through the spatial state model SSM, the forward processing result y of X is calculated forward ;

[0080] Apply a one-dimensional backward convolution operation and a spatial state model to obtain the backward processing result y of the sequence X backward ;

[0081] Will get y from the sequence X forward ,y backward The results obtained after the activation function processing of the sequence Z are added separately, and then the two results are added together. After the result is processed by the projection layer, it is residually connected with the primary pixel sequence information Y of the image block, and then the final encoded pixel sequence information T is generated. l , T l It will be sent to the multi-layer perceptron MLP for processing. lCombined with the corresponding image block position information, the image embedding of the image block is obtained, which is the output of the image encoder;

[0082] Step 3: Use the training set to train a real-time end-to-end target detection model. During training, the mask is used as a prompt input to prompt the encoder module to obtain a trained real-time end-to-end target detection model, which is used to perform real-time detection frame by frame on each frame of the real-time video in the inspection area during the road inspection process.

[0083] When used in actual detection, first, the real-time video enters the image encoder module, which includes the VIM model. The VIM model is used to segment the first frame of the input video into blocks and extract feature vectors for each block. The image encoder finally outputs the first image embedding and a conditioned (recognition condition) (this conditioned is an identification point, included in the image embedding, which can locate the objects to be segmented and avoid objects overlapping and being recognized as one object). The image embedding captures a high-level representation of all important visual features in the first frame.

[0084] At this point, the prompt encoder module converts the input prompt into a vector sequence that the model can understand. In this implementation example, the prompt encoder inputs a text prompt, and the prompt encoder module outputs a vector sequence as an encoder through CLIP. The prompt encoder module outputs target tokens (output tokens) and prompt tokens (Prompttokens), and the target tokens (output tokens) and prompt tokens (Prompt tokens) are combined to form a prompt embedding (prompt embedding).

[0085] Next, the prompt embedding is input into the mask decoder. The mask decoder includes a prompt embedding operation, a first CBAM attention mechanism layer, a second CBAM attention mechanism layer, a first MLP, a third CBAM attention mechanism layer, a fourth CBAM attention mechanism layer, a convolution operation, and a second MLP. Through the attention mechanism, the MLP layer, and the convolution operations, an instance segmentation map is output.

[0086] Then, the instance segmentation map output by the mask decoder is input into the memory decoder module (Memorydecoder), and after three convolution operations, a feature map and a target point vector object point (essentially a vector used to predict the direction of the segmented part in the next frame) are obtained. Then, the feature map and the object point output by the three convolution operations are summed with the first image embedding, and the feature map and object point output by the three convolution operations are combined with the first image embedding to become the first memory feature; the first memory feature is input into the memory bank (Memory Bank) for storage. At this point, the processing of the first frame of the image is completed.

[0087] Then, the second frame of image processing begins. The image encoder module outputs the second image embedding, the second conditioned. The second image embedding and conditioned will be input into the memory attention module, which includes the CBAM attention mechanism and the cross attention mechanism. It first passes through the CBAM attention mechanism and then enters the cross attention mechanism as the query. At the same time, the first memory feature stored in the memory bank will also be input into the cross attention mechanism as the key and value. After the cross attention mechanism is operated, an updated second image embedding is output.

[0088] Finally, the updated second image embedding is input into the mask decoder, and through the attention mechanism, MLP layer, and convolution operations, an instance segmentation map is output. Then, it is input into the memory decoder module, where the instance segmentation map output by the mask decoder is processed by the convolution operation and combined with the second image embedding to become the second memory feature, and then input into the memory bank for storage. At this point, the processing of the second frame image is completed.

[0089] According to the above process, the real-time detection of the real-time video is completed until the last frame of the real-time video is processed.

[0090] The present invention adopts the VIM (vision mamba) model as an image encoder to extract the feature representation of the input image. The VIM (vision mamba) model includes an image segmentation operation, batch normalization, two projection layers, forward convolution and linear layers, backward convolution and linear layers, a spatial state module, and MLP:

[0091] Image segmentation operation: The input image F will be divided into multiple non-overlapping blocks (patches) of fixed size, and then the position information of each image block will be embedded into its corresponding pixel vector representation information, and the pixel vector representation information of each image block will be flattened and arranged into a sequence to obtain the original pixel sequence information and corresponding position information of each image block. The original pixel sequence information of each image block is linearly transformed through a linear layer to adjust its data dimension to obtain the combined embedding features of the primary pixel sequence information and the corresponding position information of each image block. The above linear transformation is defined by a weight matrix and a bias term, where the input vector is multiplied by the weight matrix and then the bias term is added. The linear transformation can be expressed as Y=TW Y +b Y , where T is the original pixel sequence information of the image block, W Y is the weight matrix, b Y is the bias vector, and Y is the primary pixel sequence information obtained after transformation.

[0092] Batch normalization: The primary pixel sequence information of the image block is normalized through the normalization layer, using batch normalization processing.

[0093] Two projection layers: The primary pixel sequence information Y(B, M, D) of the image block is processed by the normalization layer and the results are projected into sequences X(B, M, E) and Z(B, M, E) of dimension E (extended state dimension, set to 384) using two different projection layers, where B is the batch size, M is the sequence length, and D is the original feature dimension.

[0094] The two different projection layers are two different linear layers (or fully connected layers) that project the input sequence to generate sequences X and Z. The linear layer consists of a weight matrix and a bias term. For sequences X and Z, there are weight matrices Wx and Wz, and bias vectors bx and bz, respectively. Multiply the input sequence with the weight matrix Wx and add the bias vector bx to get the sequence X.

[0095] X=Wx×Input+bx,

[0096] Input is the input sequence, Wx is a D×E matrix, E is the expanded state dimension, and D is the original feature dimension. The sequence Z is obtained in the same way. The dimensions of the weight matrix and bias vector of the corresponding linear layer are the same as those of the sequence X, but the element values ​​are different.

[0097] The sequence X is processed in both forward and backward directions to capture local dependencies in the vector: For forward processing, a one-dimensional forward convolution operation is first applied to the sequence X to obtain X' o , and then X' o Linear projection to three different linear layers, respectively, to obtain sequence B o :(B,M,N),C o :(B, M, N) and Δ o , where the sequence Δ o Use the softplus activation function to ensure it is positive.

[0098] Δ o =log(1+exp(X' o ))

[0099] Sequence Δ o With sequence A o (E, N) is multiplied by the bias vector bx and then the Hadamard product is performed to obtain the conversion parameters Sequence Δ o With sequence B o (B, M, N) is multiplied by the bias vector bz and then the Hadamard product is performed to obtain the conversion parameters

[0100]

[0101] Where N is the dimension of the hidden state; the sequence A o (E, N) are learnable parameters.

[0102] The spatial state model (SSM) is a one-dimensional function or sequence x(t)(x(t)∈R) through the hidden state h(t)(h(t)∈R N ) to y(t) (y(t)∈R).

[0103] Use parameter C o , And the output of the forward convolution X' o Combined with the evolution parameter α(α∈R N*N ), projection parameter β(β∈R N*1 ) The forward processing result y of sequence X is calculated by the spatial state model forward , and the specific process is expressed by the formula:

[0104] h(X)=α+β×X

[0105]

[0106] y forward =C o ×h'(X)

[0107] In the above formula, X is the sequence X; α is the evolution matrix of the hidden state, and β is the influence matrix of the input signal x(t) on the hidden state. Both are trainable parameter matrices.

[0108] The backward processing result y of sequence X backward The obtained process is the same as the forward processing result y of sequence X forward The process of obtaining is similar, except that a one-dimensional backward convolution operation is applied.

[0109] Will get y from the sequence X forward ,y backward The results obtained after the activation function processing of the sequence Z are added separately, and then the two results are added together. After the result is processed by the projection layer, it is residually connected (added) with the primary pixel sequence information Y of the image block, and then the final encoded pixel sequence information T is generated. l , T l It will be sent to MLP (Multi-layer Perceptron) for processing. l Combined with the corresponding image block position information, the embedded feature information of the image block is obtained, which is image embedding.

[0110] The prompt encoder module in the present invention is used to convert the input prompt into a vector sequence that the model can understand. There are two types of prompts: text and mask. Mask prompts are used during training, and the mask is processed by convolution to obtain the corresponding target token and instruction token. In actual use, mask prompts do not work, and only text prompts are input. The input text prompts are processed by the ClIP model to obtain the target token and prompt instruction.

[0111] The memory attention module (Memory attention) is composed of the CBAM attention mechanism and the Cross attention cross attention mechanism. When the image is embedded into the memory attention module (Memory attention), it will first enter the CBAM attention mechanism to realize the adaptive refinement of its own features, and then enter the Cross attention cross attention mechanism together with the memory features stored in the memory bank of the previous frame. At this time, the features output by CBAM are used as the Query, and the memory features of the previous frame are used as the Key and Value. Finally, the Memory Attention module (Memory attention) will embed an updated image into the second and third CBAM attention mechanism layers of the Mask Decoder (Mask Decoder).

[0112] The CBAM attention mechanism (see Figure 3) includes a channel attention module and a spatial attention module. The output of the previous level is used as the query, key, and value input. The three are arranged into a matrix and input into the channel attention module to emphasize the features in different channels. The channel attention module uses average pooling and maximum pooling to compress features in the spatial dimension and generate two different spatial context descriptors F c-avg and F c-max The two descriptors are then input into a shared multi-layer perceptron (MLP) to generate the channel attention feature M c , the formula is as follows:

[0113] M c =σ(W1(W0(F c-avg ))+W1(W0(F c-max ))) +b

[0114] Where σ refers to the sigmoid function, W0∈R C / r*C ,W1∈R C*C / r ; R is the hidden activation size, r is the reduction rate, and C is the number of channels. W0 and W1 are the trainable parameters of the shared multi-layer perceptron MLP, and b is a parameter;

[0115] Then, the spatial region in the feature is emphasized by the spatial attention module. The spatial attention module applies average pooling and maximum pooling in the channel dimension to generate two 2D maps F s-avg and F s-max The two 2D maps are then concatenated in the channel dimension and passed through a convolutional layer to generate the spatial attention feature M s The formula is as follows:

[0116] M s =σ(f 7*7 ([F s-avg ; F s-max ])) +b

[0117] σ refers to the sigmoid function, f 7*7 represents the convolution operation of the filter 7*7, b is the parameter, M s With M c The b parameter is the same, making the two consistent.

[0118] First, the channel attention feature M cMultiply it with the input feature of the channel attention module to get the refined feature F', use the feature F' as the input of the spatial attention module, and then use the spatial attention feature M s Multiply it with F' to get the final adaptively refined feature F1. The original features are refined using channel attention features and spatial attention features, and the refinement process is completed by element-wise multiplication.

[0119] The mask decoder module is used to combine the image embedding output by the image encoder with the target tokens (outputtokens) and prompt tokens (Prompttokens) output by the prompt encoder as input, and finally output the predicted segmentation mask. The specific workflow is as follows:

[0120] First, the target tokens (output tokens) and prompt tokens (Prompt tokens) are combined into prompt embedding (Prompt embedding) F0, and F0 is input into the first CBAM attention mechanism layer as query, key, and value. The first CBAM attention mechanism layer outputs the one-dimensional first output information F1.

[0121] The first CBAM attention mechanism layer will update the representations of these tokens internally, and each token will be updated based on the information of other tokens. For each token, calculate its attention score with all other tokens in the set (including itself). This is achieved by calculating the dot product between tokens. The attention scores are normalized using the softmax function of the channel attention module in the CBAM attention mechanism to ensure that the sum of all scores is 1. This step ensures that the model can weigh the relative importance of different tokens. The normalized attention scores are used as weights to perform a weighted summation of the representations of other tokens. In this way, the updated representation of each token not only contains its own information, but also incorporates the information of all other tokens. The updated token representation is added to the original token representation through a residual connection to enhance the learning ability of the model and avoid the gradient vanishing problem in deep networks. Next, the updated token representation is normalized using layer normalization to obtain the one-dimensional first output information F1;

[0122] Then, the first output information F1 is input as Query into the second CBAM attention mechanism layer. At the same time, the encoding sequence information in the updated image embedding feature information output by the memory attention module is input as Key and Value into the second CBAM attention mechanism layer. The above Query, Key and Value are combined into feature information of a three-dimensional matrix. The feature information is processed by the second CBAM attention mechanism layer to obtain a one-dimensional second output information F2.

[0123] The second output information F2 is input into the first MLP layer, and after being processed by the first MLP layer, the third output information F3 is obtained; wherein the part of the third output information F3 corresponding to the first output information F1 is the mask prediction token related to the image segmentation task, and the part corresponding to the encoded pixel sequence information in the image embedded feature information is the specific image feature representation used to predict the segmentation mask.

[0124] Then, the third output information F3 is input as Query to the third CBAM attention mechanism layer, and the output of the cross attention mechanism of memory attention is input as Key and Value to the third CBAM attention mechanism layer, and then after processing and updating the information, the one-dimensional fourth output information F4 is obtained;

[0125] The fourth output information F4 is used as the key and value and the mask prediction token in the third output information F3 is used as the query to input into the fourth CBAM attention mechanism layer again. After processing, the mask of each target point is output to obtain the one-dimensional fifth output information F5; the part of F5 corresponding to the first output information F1 is the mask prediction token related to the image segmentation task, and the part corresponding to the encoded pixel sequence information in the image embedding feature information is the specific image feature representation used to predict the segmentation mask.

[0126] At the same time, the fourth output information F4 undergoes two convolutions to obtain an output feature F6 of the same size as the original image F (the initial input image);

[0127] The part of the fifth output information F5 corresponding to the mask prediction token is input to the second MLP layer, and the output of the second MLP layer is multiplied by the output feature F6 to obtain the final instance segmentation map. After the second MLP layer is processed, it is also an instance segmentation map with each detected object.

[0128] After training, the mask decoder will eventually output an instance segmentation image of the original image F. The output of the mask decoder is used as the output of the real-time end-to-end target detection model. According to the output result of the model combined with the pixel block size of the segmented instance, the width of the obstacle is accurately calculated. The training in the present invention uses a 1024×1024 pixel annotated image, which is converted using an actual ratio of 1:20, that is, 1 pixel is 0.05m.

[0129] The memory decoder module consists of three 3x3 convolution operations and a summation layer. The instance segmentation image output by the mask decoder will be input into the memory decoder module, where it first enters the convolution layer for convolution to obtain an object point. Then, the object point is input into the summation layer, where the object point is summed with the image embedding output by the picture encoder module to obtain a memory feature. Finally, this memory feature is input into the memory bank module for storage.

[0130] Example 1

[0131] This embodiment is a road inspection robot obstacle avoidance system based on instance segmentation, which executes the obstacle avoidance method. The system includes:

[0132] Image acquisition module, used to obtain real-time video of the inspection area during road inspection;

[0133] An image processing module is used to mark obstacles frame by frame on the road image in the real-time video collected by the image acquisition module, and mark the category to which the obstacles belong;

[0134] The obstacle image includes at least one of a tree obstacle image, a rock obstacle image, a construction material obstacle image, a road maintenance facility obstacle image, an animal obstacle image, etc.;

[0135] Real-time end-to-end object detection model, used to achieve real-time instance segmentation of obstacles and obtain the width of obstacles;

[0136] Distance detection unit: used to measure the real-time distance between obstacles and the robot;

[0137] Obstacle avoidance unit: According to the width of the obstacle and the real-time distance measured by the distance detection unit, the inspection robot automatically adjusts its route to avoid obstacles.

[0138] The specific steps are:

[0139] (1) Obtaining a video of the road being inspected during the inspection process, and performing image color enhancement processing on the video image frame by frame;

[0140] (1.1) Using the image acquisition module, approximately 20,000 images of the road in the inspection area during the inspection process of the inspection robot are obtained;

[0141] (1.2) Convert the format of the road image into a standard format of 1024*1024 to obtain a standard format image;

[0142] (1.3) low-pass filtering the acquired standard format image using a Gaussian function to obtain a filtered image;

[0143] (1.4) Convert the acquired standard format image and the filtered image into the logarithmic domain and perform a difference (i.e., subtract the low-frequency components in the image) to obtain a reflection image in the logarithmic domain;

[0144] (1.5) Perform image summation on the pixel level on the logarithmic domain reflection image to obtain the result;

[0145] (1.6) Color restoration:

[0146] (1.6.1) At the channel level, for each channel, add the pixel values ​​of the channel in the original image to get the sum of the channel, calculate the maximum value of the sum of all channels, and use it as the denominator of the normalization factor. For the sum of each channel, divide it by the denominator of the normalization factor to get the normalization factor;

[0147] (1.6.2) Normalize the weight matrix (the weight matrix is ​​set manually according to different applications. Here, a uniform weight matrix is ​​used, that is, the weight of each channel is equal), convert it to the logarithmic domain, then multiply it by the nonlinear factor of color restoration (here taken as 2.0), and then divide it by the normalization factor to obtain the image color gain result.

[0148] (1.6.3) Recombine (multiply) the result obtained in (1.5) according to the weight matrix and the result of the image color gain to obtain the image result after color restoration;

[0149] (1.7) Image restoration: Multiply the color restored image result obtained in (1.6.3) by the gain of the image pixel value change range, and add the offset of the image pixel value change range to obtain the final result.

[0150] The gain of the image pixel value change range is a multiplication factor used to adjust the brightness and contrast of the image. It is applied to each pixel value of the image result after color restoration. The offset of the image pixel value change range is a constant value used to shift the image after the multiplication factor. It is added to each pixel value of the image result after color restoration. The final image restoration result is obtained by multiplying the multiplication factor by the image result after color restoration and adding the offset to the multiplication result.

[0151] (2) Preprocess the acquired image first, and then label the obstacle type on the preprocessed obstacle image.

[0152] (2.1) Image data preprocessing: Sliding window technology is introduced. A point in the image is randomly selected as the center point of the square sliding window. A square sliding window of constant size is used to slide from the center point on the image in the order from left to right and from top to bottom. Each time the sliding window is slid, the image area covered by the sliding window is saved as a sub-image. The size range of the square sliding window is 100-300 pixels, and the sliding step range of the square sliding window is set to 50%-70% of its size. If the side length of the remaining uncovered image in the current row or column is less than the sliding step of the square sliding window, the part exceeding the sliding step is filled with 0, and then continues to slide downward with the original sliding step until the square sliding window covers all areas of the image, stops sliding, and collects all sub-images.

[0153] (2.2) All collected sub-images are preprocessed. First, they are labeled. Sub-images with obstacles are labeled as 1, which are positive samples; sub-images without obstacles are labeled as 0, which are negative samples.

[0154] (2.3) The labeled data set is subjected to data enhancement processing. The original image is repeated three times in steps ①, ②, and ③ to obtain three difference images. The difference is that the sigmoid parameter of the Gaussian blur is different each time, which is set to 20, 100, and 180 respectively. Finally, the difference images obtained with the three different sigmoid parameters are weighted and averaged to obtain the enhanced data set.

[0155] Finally, the enhanced dataset and the labeled dataset were randomly assigned as training sets and test sets in a ratio of 7 to 3.

[0156] (3) Build and train a real-time end-to-end object detection model

[0157] A real-time end-to-end target detection model is constructed as described above and trained, using transfer learning during the training process: In this embodiment, a dataset of 11 million natural images and 1.1 billion masks is selected for pre-training, with masks used as prompts.

[0158] After that, the pre-trained parameters are used as the basis to train the real-time end-to-end object detection model again using the aforementioned training set. All images are iteratively trained in sequence. After 100,000 iterations, the pixel-level training loss is calculated using the mean square error loss function, and the trainable parameters of the network are updated once through the back propagation of the training loss. All images are iteratively trained in sequence. After training all the graphics in the training set, a round of training is completed. The trainable parameters of the network that completed the previous round are used as the initial trainable parameters of the next round. When the training gradient value is within ±1e －5 When the training iterations reach 100,000 or within the specified range, stop training and extract the network parameters with the smallest training loss to obtain a real-time end-to-end object detection model that has completed training.

[0159] Then the images in the test set and the corresponding annotations are input into the trained real-time end-to-end object detection model, and the accuracy of the segmentation results is calculated. When the pixel accuracy is 95%-100%, it is proved that the trained real-time end-to-end object detection model is the optimal segmentation model. Otherwise, the initialization method of the trainable parameters of the real-time end-to-end object detection model is adjusted, the values ​​of the fixed parameters of the network are adjusted, and the training and testing processes of the segmentation model are repeated until the optimal segmentation model is obtained.

[0160] The training was done on the same training set and the performance was tested on the same test set. The performance comparison of different network models is shown in Table 1.

[0161] Table 1 Performance comparison of different models

[0162]

[0163] Mean PixelAccuracy (mPA) is a commonly used evaluation metric in image segmentation tasks. It measures the consistency between the mask predicted by the segmentation model and the true mask. Pixel accuracy is the correct proportion of each pixel point, which is an intuitive indicator of image segmentation quality. The obstacle recognition measurement accuracy usually refers to the accuracy of obstacle instance segmentation in image analysis or computer vision tasks. The average FPS is the number of images that can be processed per second.

[0164] (4) Input the video image to be identified into the trained real-time end-to-end object detection model to detect obstacles and avoid them.

[0165] (4.1) The inspection robot is equipped with a binocular camera with a resolution of 1080P and a sensor pixel of 2 million. The camera is located in the middle of the top of the inspection robot, 1.6 meters above the ground. The camera is tilted toward the ground and the angle formed by the camera and the ground is 10°. During inspection, the industrial camera will capture images of the area 8 meters long and 5 meters wide in front of the inspection robot.

[0166] A binocular camera is an imaging system equipped with two cameras that simulates human binocular vision and obtains depth information by capturing two slightly different perspectives of the same scene. It can automatically obtain the depth information of objects in the scene by comparing the parallax of the images captured by the two cameras. Based on the obtained distance and built-in measurement scale, combined with the target obstacle width output by the real-time end-to-end target detection model, it can accurately measure the size of the target.

[0167] (4.2) If the image is identified as one without obstacles, the inspection robot continues the inspection;

[0168] (4.3) If an obstacle image is identified, the inspection robot will sound an alarm to remind the staff to clear the obstacle in time until the inspection is completed. If there is an obstacle and its width is greater than 3m and it is 2 meters away from the robot, the inspection robot will automatically adjust its route to avoid the obstacle. Or if the obstacle is less than 3 meters wide and the inspection robot is less than 1 meter away from the obstacle, the inspection robot will automatically adjust its route to avoid the obstacle.

[0169] Obstacle images specifically include images of trees, rocks, construction materials, road maintenance facilities, animals, etc.

[0170] The present invention creatively uses a real-time end-to-end object detector to realize the detection of obstacles such as trees, rocks, construction materials, road maintenance facilities, etc. by road inspection robots, especially obstacles that move such as animals, and can more accurately and efficiently detect obstacles in the inspection process in real time and automatically avoid them. The method of the present invention can more accurately segment the contours of objects, distinguish and individually mark objects that overlap or touch each other in the image, and can better distinguish individual instances.

[0171] The method of the present invention is suitable for high-resolution image processing and meets the training requirements of large-scale visual data sets.

[0172] The mask decoder module in the present invention uses the CBAM attention mechanism to independently process channel and spatial dimensions, reducing the computational complexity and amount, avoiding complex attention weight calculations and high-dimensional matrix multiplication operations, enabling the network to better capture key information in the image, and helping to improve the accuracy of obstacle detection during patrol inspections.

[0173] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A road inspection robot obstacle avoidance method based on instance segmentation, characterized in that: The method comprises the following steps: Step 1: obtain real-time video of the inspection area during the road inspection process, and mark obstacles frame by frame on the road images in the real-time video, and divide the marked obstacle images into a training set and a test set; Step 2: Build a real-time end-to-end object detection model. The real-time end-to-end object detection model includes an image encoder module, a prompt encoder module, a mask decoder module, a memory attention module, a memory decoder module and a memory bank. The image encoder module adopts a VIM model and has recognition conditional conditioning. The VIM model is used to divide the input first image into blocks and extract feature vectors for each block of the image. The image encoder outputs a first image embedding and a first recognition conditional conditioning. The prompt encoder module includes a convolution block and a CLIP, the mask prompt is input into the convolution block, the text prompt is input into the CLIP, and the prompt encoder module finally outputs a target token and a prompt token, and the target token and the prompt token are combined to form a prompt embedding; The hint embedding is input into a mask decoder, which includes a first CBAM attention layer, a second CBAM attention layer, a first MLP, a third CBAM attention layer, a fourth CBAM attention layer, a convolution operation, a second MLP, and outputs an instance segmentation map; The instance segmentation map output by the mask decoder is then input into the memory decoder module, and after three convolution operations, a feature map and an object point vector are obtained, which are then summed with the first image embedding to realize the combination of the feature map and object point output by the three convolution operations with the first image embedding to become the first memory feature; the first memory feature is input into the memory bank for storage; at this point, the processing of the first image is completed; Then, the second image is processed, and the image encoder module outputs the second image embedding and the second recognition condition conditioned; the second image embedding is input into the memory attention module, and the memory attention module includes a CBAM attention mechanism and a cross attention mechanism connected in sequence. The output result of the CBAM attention mechanism is input into the cross attention mechanism as a query. At the same time, the first memory feature stored in the memory bank is input into the cross attention mechanism as a key and a value. After the operation of the cross attention mechanism, an updated second image embedding is output; Finally, the updated second image embedding is input into the second CBAM attention mechanism layer and the third CBAM attention mechanism layer of the mask decoder respectively. The mask decoder outputs an instance segmentation map, which is then input into the memory decoder module. In the memory decoder module, the instance segmentation map output by the mask decoder is processed by the convolution operation and combined with the second image embedding to become the second memory feature, which is then input into the memory bank for storage. At this point, the processing of the second image is completed; Following the above process, all images are processed; The workflow of the VIM model is as follows: first, the input image F is divided into multiple non-overlapping blocks of fixed size, and then the position information of each image block is embedded into its corresponding pixel vector representation information, and the pixel vector representation information of each image block is flattened and arranged into a sequence to obtain the original pixel sequence information and the corresponding position information of each image block; the original pixel sequence information of each image block is linearly transformed through a linear layer to adjust its data dimension, and the combined embedding features of the primary pixel sequence information and the corresponding position information of each image block are obtained; the linear transformation is expressed as Y=TW Y +b Y , where T is the original pixel sequence information of the image block, W Y is the weight matrix, b Y is the bias vector, and Y is the primary pixel sequence information obtained after transformation; Then, the primary pixel sequence information of the image block is batch normalized through the normalization layer; Afterwards, the primary pixel sequence information Y(B, M, D) of the image block is processed by the normalization layer and the results are projected into a sequence X(B, M, E) and a sequence Z(B, M, E) with a dimension of E using two different projection layers respectively; Where B is the batch size; M is the sequence length; D is the original feature dimension; E is the expanded state dimension, which is set to 384; The two different projection layers are two different linear layers, which project the input sequence to generate sequences X and Z; the linear layer is composed of a weight matrix and a bias term, and for sequences X and Z, there are weight matrices Wx and Wz and bias vectors bx and bz respectively; the input sequence is multiplied by the weight matrix Wx, and the bias bx is added to obtain the sequence X, X=Wx×Input+bx, Input is the input sequence, Wx is a D×E matrix; the sequence Z is obtained in the same way. The dimensions of the weight matrix and bias vector of the corresponding linear layer are the same as those of the sequence X, but the element values ​​are different; Then, the sequence X is processed in both forward and backward directions to capture local dependencies in the vector. For the forward processing, a one-dimensional forward convolution operation is first applied to the sequence X to obtain X' o , and then X' o Linear projection to three different linear layers, respectively, to obtain sequence B o :(B,M,N),C o :(B, M, N) and Δ o , where the sequence Δ o Use a soft-add activation function to ensure it is positive; Δ o =log(1+exp(X' o )) Where N is the dimension of the hidden state; Then the sequence Δ o Apply the Hadamard product according to the following formula and A o 、B o Multiply to get the conversion parameter and Finally, use sequence C o , conversion parameters Conversion parameters And the output of the forward convolution X' o Through the spatial state model SSM, the forward processing result y of X is calculated forward ; Apply a one-dimensional backward convolution operation and a spatial state model to obtain the backward processing result y of the sequence X backward ; Will get y from the sequence X forward ,y backward The results obtained after the activation function processing of the sequence Z are added separately, and then the two results are added together. After the result is processed by the projection layer, it is residually connected with the primary pixel sequence information Y of the image block, and then the final encoded pixel sequence information T is generated. l , T l It will be sent to the multi-layer perceptron MLP for processing. l Combined with the corresponding image block position information, the image embedding of the image block is obtained, which is the output of the image encoder; Step 3: Use the training set to train a real-time end-to-end target detection model. During training, the mask is used as a prompt input to prompt the encoder module to obtain a trained real-time end-to-end target detection model, which is used to perform real-time detection frame by frame on each frame of the real-time video in the inspection area during the road inspection process.

2. The method according to claim 1, characterized in that The mask decoder module is used to combine the image embedding output by the image encoder and the target token and prompt token output by the prompt encoder as input, and finally output the predicted segmentation mask. The specific workflow is as follows: First, the target token and the prompt token are combined into the prompt embedding F0, and F0 is input into the first CBAM attention mechanism layer as the query, key, and value. The first CBAM attention mechanism layer outputs the one-dimensional first output information F1; The first CBAM attention mechanism layer updates the representations of these tokens internally, with each token being updated based on information from other tokens; for each token, it calculates its attention score with all other tokens in the set (including itself), normalizes the attention score, uses the normalized attention score as a weight, and performs a weighted sum of the representations of other tokens; The updated token representation is added to the original token representation through a residual connection, and then the updated token representation is normalized using layer normalization to obtain a one-dimensional first output information F1; Then, the first output information F1 is input as Query into the second CBAM attention mechanism layer. At the same time, the encoding sequence information in the updated image embedding feature information output by the memory attention module is input as Key and Value into the second CBAM attention mechanism layer. The Query, Key and Value are combined into feature information of a three-dimensional matrix. The feature information is processed by the second CBAM attention mechanism layer to obtain a one-dimensional second output information F2. The second output information F2 is input into the first MLP layer, and after being processed by the first MLP layer, the third output information F3 is obtained; wherein the part of the third output information F3 corresponding to the first output information F1 is a mask prediction token related to the image segmentation task, and the part corresponding to the encoded pixel sequence information in the image embedding feature information is a specific image feature representation used to predict the segmentation mask; Then, the third output information F3 is input as Query to the third CBAM attention mechanism layer, and the output of the cross attention mechanism of memory attention is input as Key and Value to the third CBAM attention mechanism layer, and then after processing and updating the information, the one-dimensional fourth output information F4 is obtained; The fourth output information F4 is used as the key and value and the mask prediction token in the third output information F3 is used as the query to be input into the fourth CBAM attention mechanism layer again. After processing, the mask of each target point is output to obtain the one-dimensional fifth output information F5; wherein the part of F5 corresponding to the first output information F1 is the mask prediction token related to the image segmentation task, and the part corresponding to the encoded pixel sequence information in the image embedding feature information is the specific image feature representation used to predict the segmentation mask; At the same time, the fourth output information F4 undergoes two convolutions to obtain an output feature F6 of the same size as the original image F; The part of the fifth output information F5 corresponding to the mask prediction token is input into the second MLP layer, and the output of the second MLP layer is dot-multiplied with the output feature F6 to obtain the final instance segmentation map.

3. The method according to claim 1, characterized in that: The width threshold of the obstacle, the first distance threshold and the second distance threshold between the obstacle and the robot are set. The inspection robot uses the trained real-time end-to-end target detection model to obtain in real time whether there is an obstacle in the real-time video frame by frame within the camera acquisition range. If there is an obstacle, the width of the obstacle is obtained. If the obstacle width is greater than the set obstacle width threshold and the distance between the obstacle and the robot is less than the first distance threshold, or if the obstacle width is not greater than the set obstacle width threshold and the distance between the obstacle and the robot is less than the second distance threshold, the inspection robot automatically adjusts the route to avoid obstacles. The first distance threshold is greater than the second distance threshold.

4. The method according to claim 1, characterized in that: The width threshold of the obstacle is 3m, the first distance threshold between the obstacle and the robot is 2m, and the second distance threshold is 1m.

5. A road inspection robot obstacle avoidance system based on instance segmentation, characterized in that: The obstacle avoidance method according to any one of claims 1 to 4 is implemented, wherein the system comprises: Image acquisition module, used to obtain real-time video of the inspection area during road inspection; An image processing module is used to mark obstacles frame by frame on the road image in the real-time video collected by the image acquisition module, and mark the category to which the obstacles belong; The obstacle image includes at least one of a tree obstacle image, a rock obstacle image, a construction material obstacle image, a road maintenance facility obstacle image, and an animal obstacle image; Real-time end-to-end object detection model, used to achieve real-time instance segmentation of obstacles and obtain the width of obstacles; Distance detection unit: used to measure the real-time distance between obstacles and the robot; Obstacle avoidance unit: According to the width of the obstacle and the real-time distance measured by the distance detection unit, the inspection robot automatically adjusts its route to avoid obstacles.

Images