Obstacle detection method and device, electronic equipment and storage medium
By using the obstacle detection model of backbone network, depth map encoder, visual fusion attention network and detection head in agricultural machinery, the captured image and depth image are characterized, and obstacle recognition in agricultural machinery unmanned driving technology is solved The problems of low accuracy and insufficient generalization ability achieve efficient and robust obstacle detection and positioning.
Patent Information
- Application Number
- CN202411011852.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
AI Technical Summary
The existing agricultural machinery unmanned driving technology has problems such as light sensitivity, low recognition accuracy, limited lidar recognition capabilities and insufficient generalization capabilities of deep learning algorithms in terms of obstacle recognition, especially in complex farmland environments, which are difficult to adapt to change. obstacles and environment.
Adopt the obstacle detection model composed of backbone network, depth map encoder, visual fusion attention network and detection head to feature the captured image and depth image, enhance the attention layer through gate mechanism and depth information to improve the recognition accuracy and robust, capable of detecting multiple unknown obstacles without being restricted by predefined categories.
Significantly improve the accuracy and robustness of obstacle detection, enhance the adaptability to complex environments, realize real-time efficient obstacle detection and positioning in agricultural machinery, and break through the traditional algorithms that are limited by predefined target categories. limitation.
Smart Images

Figure CN118968252A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of target detection, and in particular to a method, device, electronic device and storage medium for detecting an obstacle. Background Art
[0002] With the development trend of modern agriculture towards intelligence and automation, unmanned driving technology has been widely studied and applied to agricultural machinery to improve work efficiency and reduce labor costs. Especially in complex farmland environments, the ability to perceive obstacles has become a key technology to ensure crop safety and the automated operation of agricultural machinery. At present, in the field of unmanned driving of agricultural machinery, obstacle recognition is mainly carried out in the following ways: method 1: using a visual system; method 2: using a radar sensor; method 3: using deep learning algorithms such as target detection and panoramic segmentation algorithms for obstacle recognition.
[0003] However, the first method is very sensitive to lighting conditions. The image quality will be significantly reduced in strong or weak light environments. When obstacles are similar to crops in color or shape, it is difficult for the visual system to accurately distinguish them, resulting in reduced recognition accuracy. The second method is that the lidar has limited recognition ability for surface materials or colors. Therefore, in farmland environments, many obstacles (such as stones and earth piles) are similar in material to the surrounding environment and are difficult to accurately distinguish by lidar alone. The algorithm used in the third method usually requires pre-definition and training of specific target categories. Objects that do not appear or are marked in the training data are difficult for the model to correctly identify. Secondly, in agricultural scenarios, due to the variability of the environment and the diversity of obstacles, it is very difficult to obtain a large amount of high-quality labeled data, which limits the generalization ability of the model. Therefore, how to detect obstacles has become an urgent problem to be solved. Summary of the invention
[0004] In view of this, the purpose of the present application is to provide an obstacle detection method, device, electronic device and storage medium, which can obtain the detection result of whether there are obstacles in the monitoring area by inputting the captured image and the depth image into the obstacle detection model composed of the backbone network, the depth map encoder, the visual fusion attention network and the detection head. It can detect a variety of unknown obstacles without being restricted by predefined categories, and utilize the data complementary characteristics of the captured image and the depth image through the structure of the obstacle detection model to adaptively fuse the features of the captured image and the depth image, thereby significantly improving the accuracy and robustness of obstacle detection.
[0005] This application mainly includes the following aspects:
[0006] In a first aspect, an embodiment of the present application provides a method for detecting an obstacle, the method comprising:
[0007] Acquire captured images and depth images in the monitoring area;
[0008] The captured image and the depth image are input into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model includes a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
[0009] Furthermore, the step of inputting the captured image and the depth image into an obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model includes:
[0010] Inputting the captured image into the backbone network to obtain a multi-scale feature image of the captured image;
[0011] Inputting the depth image into the depth map encoder to obtain a feature image of the depth image;
[0012] Inputting the feature image of the depth image and the multi-scale feature images of the captured image into the visual fusion attention network for feature fusion to obtain a target feature image corresponding to each scale;
[0013] The target feature image corresponding to each scale is input into the detection head for obstacle detection, and a detection result of whether there is an obstacle in the monitoring area output by the detection head is obtained.
[0014] Furthermore, the depth map encoder includes a convolution layer, a pooling layer and a residual block; the step of inputting the depth image into the depth map encoder to obtain a feature image of the depth image includes:
[0015] Inputting the depth image into the convolutional layer to extract features, thereby obtaining a first feature image;
[0016] Inputting the first feature image into the pooling layer for downsampling to obtain a second feature image;
[0017] The second feature image is input into the residual block for feature extraction to obtain the feature image of the depth image.
[0018] Furthermore, the visual fusion attention network includes a depth information enhanced attention layer, a gating mechanism, a gating mechanism, a first fusion attention layer and a second fusion attention layer; the step of inputting the feature image of the depth image and the multi-scale feature image of the captured image into the visual fusion attention network for feature fusion to obtain the target feature image corresponding to each scale includes:
[0019] Inputting the feature image of the depth image into a gating mechanism to obtain a fourth feature image;
[0020] Inputting the fourth feature image and the feature image of the corresponding scale of the captured image into the corresponding depth information enhanced attention layer to obtain a fifth feature image;
[0021] Inputting the fifth feature image and the feature image of the corresponding scale of the captured image into the first fusion attention layer to obtain a first fused image and a fused image corresponding to each scale;
[0022] Fusing the first fused image with the feature image of the depth image after passing through a gating mechanism to obtain a second fused image;
[0023] Inputting the second fused image into a gating mechanism to obtain a sixth feature image;
[0024] Input the sixth feature image and the corresponding fused image into the corresponding depth information enhanced attention layer through a gating mechanism to obtain a seventh feature image;
[0025] The seventh feature image and the fused image corresponding to the preset scale are input into the second fusion attention layer to obtain the target feature image corresponding to each scale.
[0026] Furthermore, the obstacle detection model is trained by the following steps:
[0027] Acquire training sample images; wherein the training sample images are multiple video frames obtained by extracting frames from a continuous video at a preset time interval, each video frame having a label, and the label indicates whether the video frame has an obstacle;
[0028] Performing data enhancement processing on the training sample image to update the training sample image; wherein the data enhancement processing includes cropping, rotating, translating, reducing and enlarging;
[0029] Inputting the updated training sample image and the depth image corresponding to the training sample image into the constructed obstacle detection model for pre-training to obtain pre-training weights;
[0030] Using the pre-trained weights as model parameters to train the obstacle detection model, and obtaining a prediction result of the training sample image output by the obstacle detection model;
[0031] Based on the prediction result and the label, a loss function is obtained;
[0032] Determining whether the loss function converges;
[0033] If not converged, adjusting the model parameters to continue training the obstacle detection model;
[0034] If converged, a trained obstacle detection model is obtained.
[0035] Furthermore, the step of obtaining a loss function based on the prediction result and the label includes:
[0036] Based on the label, if the label indicates that there is an obstacle in the training sample image, the label includes a real bounding box marking the obstacle;
[0037] Determining a position loss value between the predicted bounding box and the true bounding box based on the predicted bounding box of the obstacle in the predicted result;
[0038] Based on the predicted existence probability of the obstacle in the prediction result, determining a confidence loss value between the predicted existence probability and a label of the corresponding training sample image;
[0039] Obtaining a preset weight, and determining the product of the preset weight and the confidence loss value as a first parameter;
[0040] The sum of the first parameter and the position loss value is determined as a loss function.
[0041] Furthermore, the depth map encoder further includes an Inception structure and an attention mechanism; the step of inputting the second feature image into the residual block for feature extraction to obtain the feature image of the depth image includes:
[0042] Inputting the second feature image into the residual block for feature extraction to obtain a feature image output by the residual block;
[0043] Inputting the feature image output by the residual block into the Inception structure for feature extraction to obtain the feature image output by the Inception structure;
[0044] The feature image output by the Inception structure is input into the attention mechanism for feature extraction to obtain the feature image of the depth image output by the attention mechanism.
[0045] In a second aspect, an embodiment of the present application further provides an obstacle detection device, the detection device comprising:
[0046] An acquisition module is used to acquire the captured images and depth images in the monitoring area;
[0047] A detection module is used to input the captured image and the depth image into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model includes a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
[0048] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the obstacle detection method as described above are performed.
[0049] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the obstacle detection method as described above are executed.
[0050] An obstacle detection method, device, electronic device and storage medium provided in an embodiment of the present application, the detection method comprising: acquiring a captured image and a depth image in a monitoring area; inputting the captured image and the depth image into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model comprises a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
[0051] In this way, the technical solution provided by the present application can obtain the detection result of whether there are obstacles in the monitoring area by inputting the captured image and the depth image into the obstacle detection model composed of the backbone network, the depth map encoder, the visual fusion attention network and the detection head. It can detect a variety of unknown obstacles without being restricted by predefined categories. The data complementarity of the captured image and the depth image is utilized through the structure of the obstacle detection model to adaptively fuse the features of the captured image and the depth image, thereby significantly improving the accuracy and robustness of obstacle detection.
[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0054] Figure 1 A flow chart of an obstacle detection method provided by an embodiment of the present application is shown;
[0055] Figure 2 A flow chart showing another obstacle detection method provided by an embodiment of the present application;
[0056] Figure 3 A schematic diagram of an obstacle detection model detection process provided by an embodiment of the present application is shown;
[0057] Figure 4 A schematic diagram of a visual fusion attention network processing flow provided in an embodiment of the present application is shown;
[0058] Figure 5 A schematic diagram showing a depth information enhanced attention layer processing flow provided in an embodiment of the present application;
[0059] Figure 6 A schematic diagram of a fusion attention layer processing flow provided in an embodiment of the present application is shown;
[0060] Figure 7 One of the structural schematic diagrams of an obstacle detection device provided in an embodiment of the present application is shown;
[0061] Figure 8 A second structural schematic diagram of an obstacle detection device provided in an embodiment of the present application is shown;
[0062] Fig. 9 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0063] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.
[0064] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0065] In order to enable those skilled in the art to use the contents of this application, the following implementation methods are provided in combination with the specific application scenario "obstacle detection". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.
[0066] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario where obstacle detection is required. The embodiments of the present application are not limited to specific application scenarios. Any solution using an obstacle detection method, device, electronic device and storage medium provided by the embodiments of the present application is within the protection scope of the present application.
[0067] It is worth noting that with the development trend of modern agriculture towards intelligence and automation, unmanned driving technology has been widely studied and applied to agricultural machinery to improve work efficiency and reduce labor costs. Especially in complex farmland environments, the ability to perceive "foreign objects" has become a key technology to ensure crop safety and automated operation of agricultural machinery. Traditional visual systems mainly identify obstacles through features such as color and shape, but are easily affected by light and background, resulting in unstable recognition rates; although laser radars have accurate ranging, they are expensive; and existing deep learning algorithms often rely on predefined target categories and are difficult to adapt to the variability of farmland environments. These limitations restrict the further development of unmanned driving technology for agricultural machinery. The following will introduce three representative technical solutions in detail and point out their main shortcomings, which are precisely the technical problems to be solved by the present invention.
[0068] Here, technical solution 1: obstacle recognition based on traditional visual system. In this technical solution, agricultural machinery uses a camera mounted on the vehicle body to capture real-time images of farmland, and then uses traditional image processing technology to identify obstacles. This method usually uses visual features such as color, shape, and texture to distinguish obstacles from the background. Commonly used image processing algorithms include edge detection, color segmentation, template matching, etc.; for example, by analyzing the color difference in the image, green crops and obstacles of other colors can be distinguished, and using edge detection algorithms, the outline between obstacles and background can be identified. However, this vision-based method has obvious limitations; first, it is very sensitive to lighting conditions. In strong or weak light environments, the image quality will be significantly reduced, resulting in reduced recognition accuracy. Secondly, when obstacles are similar to crops in color or shape (such as green weeds or crop residues), it is difficult for the visual system to accurately distinguish them; in addition, complex farmland backgrounds (such as soil texture, shadow changes, etc.) will also interfere with recognition performance. These factors limit the actual application effect of the visual system in unmanned driving of agricultural machinery. Technical Solution 2: Obstacle detection based on radar sensors. Radar sensors, especially laser radar (LIDAR), are also used to detect obstacles in unmanned agricultural machinery. Lidar determines the distance and position of obstacles by emitting laser beams and measuring their reflection time. Specifically, Lidar emits one or more laser beams. When the laser encounters an obstacle, it will be reflected back to the sensor. By measuring the time difference between laser emission and reception, the precise distance of the obstacle can be calculated. At the same time, through lateral scanning, Lidar can obtain the three-dimensional spatial information of the obstacle. Although Lidar is very accurate in measuring distance, it is expensive and relatively weak in identifying the type of obstacle. Lidar mainly judges based on the geometric shape of the obstacle and has limited ability to recognize surface material or color. In farmland environments, many obstacles (such as stones and earth piles) are similar in material to the surrounding environment and are difficult to accurately distinguish by relying solely on Lidar. In addition, Lidar's measurement range and angular resolution are also limited.Technical Solution 3: Environmental Perception Algorithm Based on Deep Learning. In recent years, deep learning technology has been widely used in the field of environmental perception. These algorithms learn how to identify different types of objects through a large amount of training data. Typical deep learning models, such as convolutional neural networks (CNNs), can automatically extract high-level features from images and classify targets based on these features. During the training phase, the model accepts a large amount of labeled image data and continuously adjusts internal parameters through optimization algorithms to minimize the error between the predicted results and the true labels. A fully trained model can achieve a high recognition accuracy on new images. However, these traditional deep learning environmental perception algorithms also have Some problems. First, these algorithms usually require pre-definition and training of specific target categories. It is difficult for the model to correctly identify objects that do not appear or are labeled in the training data. This limitation of "closed set" recognition makes the algorithm less adaptable when facing unknown obstacles. Second, the performance of deep learning models depends largely on the quality and quantity of training data. In agricultural scenarios, due to the variability of the environment and the diversity of obstacles, it is very difficult to obtain a large amount of high-quality labeled data, which limits the generalization ability of the model. In addition, the complex model structure and a large number of parameters also bring challenges to real-time reasoning, and it is difficult to achieve efficient deployment on embedded systems with limited computing resources.
[0069] Based on this, the present application proposes an obstacle detection method, device, electronic device and storage medium. The detection method includes: obtaining a captured image and a depth image in a monitoring area; inputting the captured image and the depth image into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model includes a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
[0070] In this way, the technical solution provided by the present application can obtain the detection result of whether there are obstacles in the monitoring area by inputting the captured image and the depth image into the obstacle detection model composed of the backbone network, the depth map encoder, the visual fusion attention network and the detection head. It can detect a variety of unknown obstacles without being restricted by predefined categories. The data complementarity of the captured image and the depth image is utilized through the structure of the obstacle detection model to adaptively fuse the features of the captured image and the depth image, thereby significantly improving the accuracy and robustness of obstacle detection.
[0071] To facilitate understanding of the embodiments of the present application, a method for detecting an obstacle disclosed in the embodiments of the present application is first introduced in detail.
[0072] See also Figure 1 , Figure 1A flowchart of an obstacle detection method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown in , the detection methods include:
[0073] S101, acquiring a captured image and a depth image in a monitoring area;
[0074] In this step, this embodiment proposes a FusionYOLO algorithm, which aims to solve the above technical problems by fusing vision (captured images) and depth information (depth images) and adopting novel network structures and training strategies. Specifically, the algorithm first uses a binocular camera to simultaneously acquire captured images (RGB images) and depth images of the monitoring area (such as farmland environment). RGB images provide rich color and texture details, while depth images provide spatial position information of objects. By fusing two complementary data modalities, the accuracy and robustness of obstacle recognition can be significantly improved. At the same time, the algorithm also breaks through the category limitations and directly performs reasoning and prediction of multiple obstacles.
[0075] S102: Input the captured image and the depth image into a pre-trained obstacle detection model to obtain a detection result output by the obstacle detection model as to whether there is an obstacle in the monitoring area.
[0076] It should be noted that the obstacle detection model needs to be trained before executing step S102. The obstacle detection model is trained by the following steps:
[0077] 1. Obtain training sample images; wherein the training sample images are multiple video frames obtained by extracting frames from a continuous video at a preset time interval, each video frame having a label, and the label indicates whether the video frame has an obstacle;
[0078] In this step, it is necessary to collect a large amount of monitoring environment, such as continuous video of farmland environment, and extract frames at fixed time intervals. At the same time, the RGB image and the corresponding depth image of each video frame should be aligned, and the foreign objects in the RGB image should be marked with bounding boxes. Unlike traditional target detection data sets, the detection method provided in this embodiment only needs to mark the bounding boxes of foreign objects without giving specific category labels. This annotation method can greatly reduce the workload of data preparation and is more in line with the needs of practical applications.
[0079] As an example, traditional target detection algorithms, such as the YOLO series, use instance annotation Among them, the bounding box {B i} and the category label {c i In this embodiment, the example is restated as Only the bounding box needs to be marked. Since it is based on open-set target detection in agricultural autonomous driving scenarios and is assisted by a depth camera, the bounding boxes of targets in the scene with depth features that affect the driving of the vehicle are marked.
[0080] 2. performing data enhancement processing on the training sample image to update the training sample image; wherein the data enhancement processing includes cropping, rotating, translating, reducing and enlarging;
[0081] In this step, in order to improve the generalization ability and robustness of the network, the training data is enhanced in various forms, including random cropping, rotation, translation, scaling, etc. This can simulate different shooting angles and distances, allowing the network to adapt to more variable scenes. At the same time, some specific enhancement methods are also introduced, such as random occlusion and blur, to simulate common interference factors in the monitoring environment.
[0082] 3. Inputting the updated training sample image and the depth image corresponding to the training sample image into the constructed obstacle detection model for pre-training to obtain pre-training weights;
[0083] In this step, before training begins, the weights of the network need to be initialized. A good initialization strategy can accelerate convergence and help the network find a better solution. This embodiment adopts a pre-training method, that is, the network is first trained on a large-scale general data set. Since the input of this algorithm is a one-to-one corresponding depth image and RGB image, and the gate structure in the algorithm structure will be affected by the timing, the best pre-training data is a large number of continuous video frames and corresponding depth images for training to obtain pre-training weights. The scenes can be diversified, not just farmland scenes.
[0084] 4. Using the pre-trained weights as model parameters to train the obstacle detection model, and obtaining a prediction result of the training sample image output by the obstacle detection model;
[0085] 5. Obtaining a loss function based on the prediction result and the label;
[0086] It should be noted that the steps to obtain the loss function based on the prediction results and labels include:
[0087] 1) Based on the label, if the label indicates that there is an obstacle in the training sample image, the label includes a real bounding box marking the obstacle;
[0088] 2) Based on the predicted bounding box of the obstacle in the prediction result, determining the position loss value between the predicted bounding box and the real bounding box;
[0089] 3) Based on the predicted existence probability of the obstacle in the prediction result, determine the confidence loss value between the predicted existence probability and the label of the corresponding training sample image;
[0090] 4) obtaining a preset weight, and determining the product of the preset weight and the confidence loss value as a first parameter;
[0091] 5) The sum of the first parameter and the position loss value is determined as a loss function.
[0092] In the above steps 1) to 5), since the goal is to detect foreign objects rather than identify specific target categories, the traditional classification loss function is no longer applicable. This embodiment designs a new loss function as shown below:
[0093] Loss = λL conf + bbox ;
[0094] Among them, Loss is the loss function, λ is the preset weight, L conf is the confidence loss value, L bbox is the position loss value; the loss function provided in this embodiment is an improvement on the basic loss function of YOLOV8, which consists of two parts: position loss and confidence loss. Position loss is used to measure the difference between the predicted bounding box and the true bounding box, and confidence loss is used to measure the difference between the predicted obstacle probability and the true label. Through the weighted combination of these two losses, the network can simultaneously learn the position and existence probability of the obstacle. However, due to the cancellation of the category loss and the "open set" detection of the type of algorithm target, this algorithm further adjusts the weight of the confidence loss and adds the λ hyperparameter to the confidence loss to meet a more accurate grasp of the obstacles.
[0095] 6. Determine whether the loss function converges;
[0096] 7. If not converged, adjusting the model parameters to continue training the obstacle detection model;
[0097] 8. If converged, the trained obstacle detection model is obtained.
[0098] Here, during the training process, fine-tuning is performed on the data set of the monitoring environment. This method can inherit the general features of the pre-trained model and adapt to the particularity of the monitoring environment. Mini-batch methods can be used to extract images from the training set and send them to the network for training. During the training process, the performance of the model needs to be evaluated regularly on the validation set to monitor the progress of the training. The general mean average precision mAP (mean Average Precision) is used as the evaluation indicator, which comprehensively considers the accuracy and completeness of the detection. When the performance of the model on the validation set reaches saturation, the training can be stopped and the current model parameters are saved. After the model training is completed, it needs to be deployed to the actual system (such as an agricultural machinery system). In order to achieve real-time foreign body detection, the model is optimized and accelerated, including pruning, quantization, compilation, etc. This can reduce the computational workload and storage space of the model while maintaining performance, so that it can run efficiently on embedded devices.
[0099] In step S102, the obstacle detection model includes a backbone network, a depth map encoder, a visual fusion attention network, and a detection head; the depth map encoder includes a convolutional layer, a pooling layer, and a residual block; see Figure 3 , Figure 3 A schematic diagram of an obstacle detection model detection process provided in an embodiment of the present application, such as Figure 3 The figure shows the overall architecture of the FusionYOLO algorithm, which is used for obstacle detection in obstacle detection models. It is mainly developed based on YOLOv8 and consists of a YOLO backbone network, a depth map encoder, a reparameterizable visual fusion attention network combined with a gating mechanism, and a detection head of the target detection algorithm. The captured image (RGB image) is feature extracted through the backbone network (original YOLOV8 Backbone), and the depth image is feature extracted through the depth map encoder. The two features are fully fused and interacted through the visual fusion attention network, and the detection head performs inference prediction to obtain the detection result.
[0100] Among them, the visual fusion attention network includes a depth information enhanced attention layer, a gating mechanism, a first fusion attention layer, and a second fusion attention layer. In terms of the fusion strategy of the obstacle detection model, the FusionYOLO algorithm adopts an innovative visual fusion attention network (Depth-RGB Fusion Attention Layer Network) combined with a gating mechanism. This structure introduces a gating mechanism at different stages of feature extraction. The gating mechanism can adaptively adjust the importance of deep features, enabling the network to flexibly utilize the two types of information according to the characteristics of different scenes and targets. The single gating signal makes the model structure simple and the computational efficiency higher. This sophisticated feature interaction and fusion method greatly enhances the algorithm's adaptability to complex environments.
[0101] Please note that Figure 2 , Figure 2 A flowchart of another obstacle detection method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the step of inputting the captured image and the depth image into the obstacle detection model to obtain the detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model includes:
[0102] S201, inputting the captured image into the backbone network to obtain a multi-scale feature image of the captured image;
[0103] S202, inputting the depth image into the depth map encoder to obtain a feature image of the depth image;
[0104] It should be noted that the step of inputting the depth image into the depth image encoder to obtain the feature image of the depth image includes:
[0105] S2021, inputting the depth image into the convolution layer to perform feature extraction to obtain a first feature image;
[0106] S2022, inputting the first feature image into the pooling layer for down-sampling to obtain a second feature image;
[0107] S2023: Input the second feature image into the residual block to perform feature extraction to obtain a feature image of the depth image.
[0108] In step S2021 to step S2023, for the input depth image, a lightweight depth map encoder is used to extract its features, and the depth map encoder consists of a series of convolutional layers and pooling layers. For example, the depth image is initially feature extracted through a 3x3 convolutional layer, and then downsampled through a maximum pooling layer. Next, two residual blocks are used to further refine the features. As an example, each residual block can contain two 3x3 convolutional layers and a cross-layer residual connection. This structure can effectively increase the depth and expression ability of the network while avoiding the problem of gradient disappearance.
[0109] It should be noted that the depth map encoder also includes an Inception structure and an attention mechanism; the step of inputting the second feature image into the residual block for feature extraction to obtain the feature image of the depth image includes:
[0110] (1) Inputting the second feature image into the residual block for feature extraction to obtain a feature image output by the residual block;
[0111] (2) Inputting the feature image output by the residual block into the Inception structure for feature extraction to obtain the feature image output by the Inception structure;
[0112] (3) Inputting the feature image output by the Inception structure into the attention mechanism for feature extraction, thereby obtaining the feature image of the depth image output by the attention mechanism.
[0113] In the above steps (1) to (3), in the design of the depth map encoder, in addition to using convolutional layers and residual blocks, you can also try to introduce other feature extraction modules, such as Inception structure, attention mechanism, etc., to further improve the expression ability of deep features.
[0114] S203, inputting the feature image of the depth image and the multi-scale feature images of the captured image into the visual fusion attention network for feature fusion to obtain a target feature image corresponding to each scale;
[0115] In this step, the structure of the visual fusion attention network, after top-down or bottom-up fusion, establishes a feature pyramid {P3, P4, P5} with multi-scale image features {C3, C4, C5}. In addition, it is proposed to combine the gating mechanism (Gate mechanism), the depth information enhanced attention layer DA-CSPLayer (Depth Attention CSPLayer) and the fusion attention layer DRFALayer (Depth-RGB fusion attention Layer) to further enhance the interaction between the depth image features and the RGB image, and improve the representation ability of the depth image in the RGB image. When the features of the depth image are fused to the RGB image, a gating mechanism is also added to select the depth information features and filter the information.
[0116] It should be noted that the step of inputting the feature image of the depth image and the multi-scale feature images of the captured image into the visual fusion attention network for feature fusion to obtain the target feature image corresponding to each scale includes:
[0117] S2031, inputting the feature image of the depth image into a gating mechanism to obtain a fourth feature image;
[0118] In this step, the formula for feature selection and information filtering of the gating mechanism for the depth image is as follows:
[0119] y=(σ(W g x+b g ))⊙φ(W f x+b f );
[0120] Among them, x is the feature vector containing depth information, W g and b g are the weight matrix and bias, which are used to control the gate. ⊙ represents the Hadamard product, which is often used to implement the gating mechanism in neural networks, that is, to modulate another signal by an activation output. f and b f It is another set of weight matrices and biases, which refers to the weight data of the depth image, which is used to transform the input data and finally obtain the gated output y, where y represents the weight of the controlled feature after processing by the gating mechanism, so as to obtain the fourth feature image.
[0121] S2032, inputting the fourth feature image and the feature image of the corresponding scale of the captured image into the corresponding depth information enhanced attention layer to obtain a fifth feature image;
[0122] In this step, there is no limit on the number of depth information enhancement attention layers. For example, see Figure 4 , Figure 4 A schematic diagram of a visual fusion attention network processing flow provided in an embodiment of the present application, such as Figure 4 As shown, the fourth feature image is compared with the feature image of the corresponding scale of the captured image (such as Figure 4 The fifth feature image output by the corresponding depth information enhanced attention layer is obtained. The fifth feature image output by the layer and the feature image of the corresponding scale of the captured image (such as Figure 4 C3 in the image is input again to the next depth information enhanced attention layer to continue to obtain the fifth feature image output by the depth information enhanced attention layer (the fifth feature image here only refers to the feature images output by each depth information enhanced attention layer as the fifth feature image, and does not represent the same feature image), and so on, the fifth feature image output by each depth information enhanced attention layer can be obtained.
[0123] Here, the depth information enhanced attention layer uses the cross-stage partial layer (CSPLayer) to guide the merging of the depth map features into the multi-scale image features to form a depth-guided CSPLayer. The specific formula is as follows:
[0124]
[0125] in, is the RGB image feature (here refers to Figure 4 C3, C4 and C5 in the figure), W here represents the fourth feature image, δ represents the Sigmoid activation function, and the Max-Sigmoid activation function after the bottleneck block is used to aggregate the two image features, and the updated X l ′ Connect with the cross-stage features as the output, that is, X l ′ Enhance the features of the attention layer output for depth information, here it refers to the fifth feature image.
[0126] S2033, inputting the fifth feature image and the feature image of the scale corresponding to the captured image into the first fusion attention layer to obtain a first fused image and a fused image corresponding to each scale;
[0127] In this step, if Figure 4 As shown, the fifth feature image obtained by each depth information enhancement attention layer in the above steps and the feature image of the corresponding scale of the captured image (such as Figure 4C5 in the figure, where the feature images of the scale corresponding to the captured image can be preset and is not limited to C5) are input into the first fusion attention layer for feature fusion, and the first fused image after the fusion of all feature images and the fused image corresponding to each scale can be obtained. The fusion here can be splicing.
[0128] Here, in order to enhance the deep features with image perception information, a fusion attention layer (e.g. Figure 4 The first fusion attention layer or the second fusion attention layer in the fusion layer is used to fuse the overall features. The specific formula is as follows:
[0129]
[0130] MultiHead-Attention(Q,K,V)=oncat(head1,…,head i )W O ;
[0131] Among them, MultiHead-Attention represents the multi-head attention mechanism. Yes X l The 3×3 area obtained by the maximum pooling layer of multi-scale features has a total of 27 token image blocks (patch tokens). C represents the channel, and W here is the feature output by the depth information enhancement attention layer, which refers to the fifth feature image, that is, the above X l ′ , Q, K, V are respectively In the formula, W corresponds to q, as the query matrix (text embedding), Corresponding to K and V, as the key matrix (image patch features) and value matrix. O It is learned through back propagation and is a linear transformation matrix used to map the concatenation results of multi-head attention to the output space of the model, allowing the model to adaptively adjust how to combine the outputs of multiple heads. l is the output of the l-layer fusion attention layer, where l∈{3,4,5}. In the above formula, hd i Represents the output of the i-th attention head, corresponding to the formula:
[0132]
[0133]
[0134] in, is the learnable weight matrix of the i-th attention head. is the result of linear transformation. Projecting the original Q, K, V into different subspaces allows each attention head to focus on different aspects of the input. Attention() is the attention calculation function, d k is the dimension of the key vector.
[0135] S2034, fusing the first fused image with the feature image of the depth image through a gating mechanism to obtain a second fused image;
[0136] In this step, if Figure 4 As shown, the first fused image is fused with the feature image of the depth image after features are extracted by a gating mechanism to obtain a second fused image.
[0137] S2035, inputting the second fused image into a gating mechanism to obtain a sixth feature image;
[0138] S2036, inputting the sixth feature image and the corresponding fused image into the corresponding depth information enhanced attention layer through a gating mechanism to obtain a seventh feature image;
[0139] In this step, if Figure 4 As shown, the sixth feature image and the corresponding fusion image are subjected to a gating mechanism (such as Figure 4 The fused image corresponding to C5 in the figure) is input into the corresponding depth information enhanced attention layer to obtain the seventh feature image output by the depth information enhanced attention layer. The seventh feature image output by the depth information enhanced attention layer can also be combined with the corresponding fused image (such as Figure 4 The fused image corresponding to C4 in ( ), or also includes Figure 4 The fused image corresponding to C3 is input into the next depth information enhanced attention layer to continue feature fusion, and the seventh feature image output by the depth information enhanced attention layer is obtained (the seventh feature image here only refers to the feature images output by each depth information enhanced attention layer after the first fused attention layer, and does not represent the same feature image). The seventh feature image output by this layer can also be input into the previous depth information enhanced attention layer and fused with other features to obtain the seventh feature image output by the depth information enhanced attention layer. By analogy, the seventh feature image output by each depth information enhanced attention layer after the first fused attention layer can be obtained.
[0140] Here, please see Figure 5 , Figure 5 A schematic diagram of a depth information enhanced attention layer processing flow provided in an embodiment of the present application, such as Figure 5As shown in the figure, it is a structure of a depth information enhanced attention layer (DA-CSPLayer). The processing flow of the DA-CSPLayer structure is as follows: the feature image is input into the convolutional bottleneck layer (Conv Bottleneck) to extract features, and then it is fused with the features of the depth image through the output layer (Max-Sigmod), and the fused feature image is spliced with the input feature image and then output; the S circle indicates separation, and the C circle indicates splicing.
[0141] S2037. Input the seventh feature image and the fused image corresponding to the preset scale into the second fusion attention layer to obtain the target feature image corresponding to each scale.
[0142] In this step, the preset scale (such as Figure 4 The C3 scale in the example, where the preset scale can be set and is not limited to C3), the corresponding fused image and each seventh feature image are input into the second fusion attention layer for fusion. The fusion here can be splicing to obtain the target feature image (P3, P4 and P5) corresponding to each scale output by the second fusion attention layer.
[0143] Here, please see Figure 6 , Figure 6 A schematic diagram of a fusion attention layer processing flow provided in an embodiment of the present application, such as Figure 6 As shown in the figure, it is a structure of a fusion attention layer (DRFALayer). The processing flow of the DRFALayer structure is: multiple feature images are spliced and input into the convolution bottleneck layer (Conv Bottleneck) to extract features, and then input into the multi-head cross-attention layer (multi-head cross-attention, MHCA) to fuse with the features of the depth image.
[0144] S204: Input the target feature image corresponding to each scale into the detection head to perform obstacle detection, and obtain a detection result output by the detection head as to whether there is an obstacle in the monitoring area.
[0145] In this step, the detection result may be that there is an obstacle in the monitoring area and a predicted boundary box of the obstacle (ie, the location of the obstacle), or there is no obstacle in the monitoring area.
[0146] In summary, this embodiment makes full use of the complementary characteristics of the two data by fusing RGB images and depth images. RGB images provide rich color and texture details, and depth images provide spatial position information of objects. The combination of the two can significantly improve the accuracy and robustness of obstacle recognition and enhance the adaptability of the algorithm to complex monitoring environments. This embodiment also adopts an "open set" recognition strategy to train the network to detect and locate various obstacles, breaking through the limitations of traditional algorithms that are limited to predefined target categories. This generalized foreign body detection capability enables the algorithm to flexibly respond to various unknown obstacles in the monitoring environment, greatly improving the flexibility of the system. This embodiment also designs a visual fusion attention network structure for adaptive fusion of RGB features and depth features, and the gating mechanism can flexibly adjust the importance of the two features according to the characteristics of different scenes and targets. Thanks to the improvement in computing efficiency brought about by feature fusion and attention mechanism, the detection method provided in this embodiment can realize real-time obstacle detection and positioning on embedded hardware, which provides reliable and efficient environmental perception capabilities for unmanned driving scenarios such as agricultural machinery. Therefore, the FusionYOLO algorithm provided in this embodiment has made significant progress in improving recognition accuracy, enhancing environmental adaptability, and realizing real-time and efficient detection. It is expected to promote a higher level of unmanned driving in fields such as agricultural machinery, so as to improve the level of intelligent agricultural production.
[0147] An embodiment of the present application provides a method for detecting an obstacle, the method comprising: acquiring a captured image and a depth image in a monitoring area; inputting the captured image and the depth image into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model comprises a backbone network, a depth map encoder, a visual fusion attention network, and a detection head.
[0148] In this way, the technical solution provided by the present application can obtain the detection result of whether there are obstacles in the monitoring area by inputting the captured image and the depth image into the obstacle detection model composed of the backbone network, the depth map encoder, the visual fusion attention network and the detection head. It can detect a variety of unknown obstacles without being restricted by predefined categories. The data complementarity of the captured image and the depth image is utilized through the structure of the obstacle detection model to adaptively fuse the features of the captured image and the depth image, thereby significantly improving the accuracy and robustness of obstacle detection.
[0149] Based on the same inventive concept, the embodiment of the present application also provides an obstacle detection device corresponding to the above-mentioned obstacle detection method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0150] See also Figure 7 , Figure 8 , Figure 7 One of the structural schematic diagrams of an obstacle detection device provided in an embodiment of the present application, Figure 8 The second structural diagram of an obstacle detection device provided in an embodiment of the present application is as follows: Figure 7 As shown in , the detection device 710 includes:
[0151] An acquisition module 711 is used to acquire a captured image and a depth image in a monitoring area;
[0152] The detection module 712 is used to input the captured image and the depth image into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model includes a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
[0153] Optionally, the detection module 712 is specifically used for:
[0154] Inputting the captured image into the backbone network to obtain a multi-scale feature image of the captured image;
[0155] Inputting the depth image into the depth map encoder to obtain a feature image of the depth image;
[0156] Inputting the feature image of the depth image and the multi-scale feature images of the captured image into the visual fusion attention network for feature fusion to obtain a target feature image corresponding to each scale;
[0157] The target feature image corresponding to each scale is input into the detection head for obstacle detection, and a detection result of whether there is an obstacle in the monitoring area output by the detection head is obtained.
[0158] Optionally, the depth map encoder includes a convolution layer, a pooling layer, and a residual block; when the detection module 712 is used to input the depth image into the depth map encoder to obtain a feature image of the depth image, the detection module 712 is specifically used to:
[0159] Inputting the depth image into the convolutional layer to extract features, thereby obtaining a first feature image;
[0160] Inputting the first feature image into the pooling layer for downsampling to obtain a second feature image;
[0161] The second feature image is input into the residual block for feature extraction to obtain the feature image of the depth image.
[0162] Optionally, the visual fusion attention network includes a depth information enhanced attention layer, a gating mechanism, a gating mechanism, a first fusion attention layer, and a second fusion attention layer; when the detection module 712 is used to input the feature image of the depth image and the multi-scale feature image of the captured image into the visual fusion attention network for feature fusion to obtain the target feature image corresponding to each scale, the detection module 712 is specifically used to:
[0163] Inputting the feature image of the depth image into a gating mechanism to obtain a fourth feature image;
[0164] Inputting the fourth feature image and the feature image of the corresponding scale of the captured image into the corresponding depth information enhanced attention layer to obtain a fifth feature image;
[0165] Inputting the fifth feature image and the feature image of the corresponding scale of the captured image into the first fusion attention layer to obtain a first fused image and a fused image corresponding to each scale;
[0166] Fusing the first fused image with the feature image of the depth image after passing through a gating mechanism to obtain a second fused image;
[0167] Inputting the second fused image into a gating mechanism to obtain a sixth feature image;
[0168] Input the sixth feature image and the corresponding fused image into the corresponding depth information enhanced attention layer through a gating mechanism to obtain a seventh feature image;
[0169] The seventh feature image and the fused image corresponding to the preset scale are input into the second fusion attention layer to obtain the target feature image corresponding to each scale.
[0170] Optional, such as Figure 8 As shown, the detection device 710 further includes a training module 713, and the training module 713 is used to:
[0171] Acquire training sample images; wherein the training sample images are multiple video frames obtained by extracting frames from a continuous video at a preset time interval, each video frame having a label, and the label indicates whether the video frame has an obstacle;
[0172] Performing data enhancement processing on the training sample image to update the training sample image; wherein the data enhancement processing includes cropping, rotating, translating, reducing and enlarging;
[0173] Inputting the updated training sample image and the depth image corresponding to the training sample image into the constructed obstacle detection model for pre-training to obtain pre-training weights;
[0174] Using the pre-trained weights as model parameters to train the obstacle detection model, and obtaining a prediction result of the training sample image output by the obstacle detection model;
[0175] Based on the prediction result and the label, a loss function is obtained;
[0176] Determining whether the loss function converges;
[0177] If not converged, adjusting the model parameters to continue training the obstacle detection model;
[0178] If converged, a trained obstacle detection model is obtained.
[0179] Optionally, when the training module 713 is used to obtain a loss function based on the prediction result and the label, the training module 713 is specifically used to:
[0180] Based on the label, if the label indicates that there is an obstacle in the training sample image, the label includes a real bounding box marking the obstacle;
[0181] Determining a position loss value between the predicted bounding box and the true bounding box based on the predicted bounding box of the obstacle in the predicted result;
[0182] Based on the predicted existence probability of the obstacle in the prediction result, determining a confidence loss value between the predicted existence probability and a label of the corresponding training sample image;
[0183] Obtaining a preset weight, and determining the product of the preset weight and the confidence loss value as a first parameter;
[0184] The sum of the first parameter and the position loss value is determined as a loss function.
[0185] Optionally, the depth map encoder further includes an Inception structure and an attention mechanism; when the detection module 712 is used to input the second feature image into the residual block for feature extraction to obtain the feature image of the depth image, the detection module 712 is specifically used to:
[0186] Inputting the second feature image into the residual block for feature extraction to obtain a feature image output by the residual block;
[0187] Inputting the feature image output by the residual block into the Inception structure for feature extraction to obtain the feature image output by the Inception structure;
[0188] The feature image output by the Inception structure is input into the attention mechanism for feature extraction to obtain the feature image of the depth image output by the attention mechanism.
[0189] An embodiment of the present application provides an obstacle detection device, the detection device comprising: an acquisition module, used to acquire a captured image and a depth image in a monitoring area; a detection module, used to input the captured image and the depth image into a pre-trained obstacle detection model, and obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model comprises a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
[0190] In this way, the technical solution provided by the present application can obtain the detection result of whether there are obstacles in the monitoring area by inputting the captured image and the depth image into the obstacle detection model composed of the backbone network, the depth map encoder, the visual fusion attention network and the detection head. It can detect a variety of unknown obstacles without being restricted by predefined categories. The data complementarity of the captured image and the depth image is utilized through the structure of the obstacle detection model to adaptively fuse the features of the captured image and the depth image, thereby significantly improving the accuracy and robustness of obstacle detection.
[0191] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown in , the electronic device 900 includes a processor 910 , a memory 920 and a bus 930 .
[0192] The memory 920 stores machine-readable instructions executable by the processor 910. When the electronic device 900 is running, the processor 910 communicates with the memory 920 via the bus 930. When the machine-readable instructions are executed by the processor 910, the above-mentioned Figure 1 to Figure 2 The steps of the obstacle detection method in the illustrated method embodiment and the specific implementation thereof can be found in the method embodiment, which will not be described in detail here.
[0193] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 to Figure 2The steps of the obstacle detection method in the illustrated method embodiment and the specific implementation thereof can be found in the method embodiment, which will not be described in detail here.
[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0195] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0197] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0198] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0199] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for detecting an obstacle, characterized in that: The detection method comprises: Acquire captured images and depth images in the monitoring area; The captured image and the depth image are input into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model includes a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
2. The detection method according to claim 1, characterized in that: The step of inputting the captured image and the depth image into an obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model includes: Inputting the captured image into the backbone network to obtain a multi-scale feature image of the captured image; Inputting the depth image into the depth map encoder to obtain a feature image of the depth image; Inputting the feature image of the depth image and the multi-scale feature images of the captured image into the visual fusion attention network for feature fusion to obtain a target feature image corresponding to each scale; The target feature image corresponding to each scale is input into the detection head for obstacle detection, and a detection result of whether there is an obstacle in the monitoring area output by the detection head is obtained.
3. The detection method according to claim 2, characterized in that: The depth map encoder includes a convolution layer, a pooling layer and a residual block; the step of inputting the depth image into the depth map encoder to obtain a feature image of the depth image includes: Inputting the depth image into the convolutional layer to extract features, thereby obtaining a first feature image; Inputting the first feature image into the pooling layer for downsampling to obtain a second feature image; The second feature image is input into the residual block for feature extraction to obtain the feature image of the depth image.
4. The detection method according to claim 2, characterized in that: The visual fusion attention network includes a depth information enhanced attention layer, a gating mechanism, a first fusion attention layer and a second fusion attention layer; the step of inputting the feature image of the depth image and the multi-scale feature images of the captured image into the visual fusion attention network for feature fusion to obtain a target feature image corresponding to each scale includes: Inputting the feature image of the depth image into a gating mechanism to obtain a fourth feature image; Inputting the fourth feature image and the feature image of the corresponding scale of the captured image into the corresponding depth information enhanced attention layer to obtain a fifth feature image; Inputting the fifth feature image and the feature image of the corresponding scale of the captured image into the first fusion attention layer to obtain a first fused image and a fused image corresponding to each scale; Fusing the first fused image with the feature image of the depth image after passing through a gating mechanism to obtain a second fused image; Inputting the second fused image into a gating mechanism to obtain a sixth feature image; Input the sixth feature image and the corresponding fused image into the corresponding depth information enhanced attention layer through a gating mechanism to obtain a seventh feature image; The seventh feature image and the fused image corresponding to the preset scale are input into the second fusion attention layer to obtain the target feature image corresponding to each scale.
5. The detection method according to claim 1, characterized in that: Train the obstacle detection model by following these steps: Acquire training sample images; wherein the training sample images are multiple video frames obtained by extracting frames from a continuous video at a preset time interval, each video frame having a label, and the label indicates whether the video frame has an obstacle; Performing data enhancement processing on the training sample image to update the training sample image; wherein the data enhancement processing includes cropping, rotating, translating, reducing and enlarging; Inputting the updated training sample image and the depth image corresponding to the training sample image into the constructed obstacle detection model for pre-training to obtain pre-training weights; Using the pre-trained weights as model parameters to train the obstacle detection model, and obtaining a prediction result of the training sample image output by the obstacle detection model; Based on the prediction result and the label, a loss function is obtained; Determining whether the loss function converges; If not converged, adjusting the model parameters to continue training the obstacle detection model; If converged, a trained obstacle detection model is obtained.
6. The detection method according to claim 5, characterized in that: The step of obtaining a loss function based on the prediction result and the label includes: Based on the label, if the label indicates that there is an obstacle in the training sample image, the label includes a real bounding box marking the obstacle; Determining a position loss value between the predicted bounding box and the true bounding box based on the predicted bounding box of the obstacle in the predicted result; Based on the predicted existence probability of the obstacle in the prediction result, determining a confidence loss value between the predicted existence probability and a label of the corresponding training sample image; Obtaining a preset weight, and determining the product of the preset weight and the confidence loss value as a first parameter; The sum of the first parameter and the position loss value is determined as a loss function.
7. The detection method according to claim 3, characterized in that: The depth map encoder further includes an Inception structure and an attention mechanism; the step of inputting the second feature image into the residual block for feature extraction to obtain the feature image of the depth image includes: Inputting the second feature image into the residual block for feature extraction to obtain a feature image output by the residual block; Inputting the feature image output by the residual block into the Inception structure for feature extraction to obtain the feature image output by the Inception structure; The feature image output by the Inception structure is input into the attention mechanism for feature extraction to obtain the feature image of the depth image output by the attention mechanism.
8. An obstacle detection device, characterized in that: The detection device comprises: An acquisition module is used to acquire the captured images and depth images in the monitoring area; A detection module is used to input the captured image and the depth image into a pre-trained obstacle detection model to obtain a detection result of whether there is an obstacle in the monitoring area output by the obstacle detection model; wherein the obstacle detection model includes a backbone network, a depth map encoder, a visual fusion attention network and a detection head.
9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the obstacle detection method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the obstacle detection method according to any one of claims 1 to 7 are executed.
Citation Information
Patent Citations
Vehicle-mounted barrier detection method based on radar data and image data fusion and deep learning
CN108229366A
Salient target detection method and device
CN116310394A
Multi-modal feature target detection method based on dynamic convolution and attention mechanism
CN116452937A
Image enhancement method and device
CN116547696A
Display screen foreign matter detection method and device, model training method and device, equipment and medium
CN118229781A
Cited By
Subway work area steel support foreign matter detection method and device
CN119180731A
A foreign body detection method and device for steel support in subway work area
CN119180731B
