Roadside weed segmentation method, system and medium based on vehicle-mounted recorder

Abstract

The application discloses a roadside near weed segmentation method and system based on a vehicle-mounted recorder and a medium, the method first acquires a video shot by the vehicle-mounted recorder and decodes to obtain a video frame picture; the video frame picture is input into a target detection model for detection, a weed position is located and a rectangular frame is labeled to obtain a weed labeled picture; a target segmentation model is used for segmenting the weed labeled picture to obtain a weed binary graph; a self-supervised monocular depth estimation model is used for depth estimation on the video frame picture to obtain a weed depth graph; a fusion algorithm is used for information fusion of the weed binary graph and the weed depth graph to obtain a fusion mask picture; after a hole elimination operation on the weed binary graph, the weed binary graph and the video frame picture are superimposed with a mask to obtain a weed segmentation result; and after height judgment on the fusion mask picture, the fusion mask picture and the video frame picture are superimposed to obtain a near weed segmentation result. The application realizes automatic segmentation and detection of roadside near weeds, greatly improves the efficiency of highway maintenance, and reduces the labor cost.

Description

Technical Field

[0001] This invention belongs to the technical field of image detection and segmentation and greening maintenance, specifically relating to a method, system and medium for segmenting roadside weeds based on a vehicle recorder. Background Technology

[0002] Highway maintenance is a crucial task for ensuring traffic safety and the sustainable development of highways. Roadside weeds not only affect the aesthetics of highways but can also obstruct visibility and increase the risk of traffic accidents. Therefore, timely detection and removal of roadside weeds are essential for highway maintenance.

[0003] With technological advancements and cost reductions, in-vehicle dashcams have become widely used in modern vehicles. These dashcams, equipped with cameras mounted on the vehicle, record image data in real time during driving, providing strong support for road condition monitoring and accident recording. Meanwhile, computer vision and image segmentation technologies have made significant progress in recent years. The rise of deep learning algorithms has provided powerful tools for image segmentation, enabling the extraction of semantic information from images and achieving more accurate segmentation results.

[0004] Currently, in research on intelligent discovery and management of roadside greenery, most users cannot obtain greenery information quickly and accurately. Based on this problem, this application proposes a method for segmenting nearby roadside weeds based on a vehicle-mounted recorder. By utilizing image data acquired by the recorder and combining computer vision and image segmentation techniques, it achieves automatic segmentation and detection of nearby roadside weeds, significantly improving the efficiency of highway maintenance, reducing labor costs, and providing timely image information to relevant departments to take appropriate maintenance measures, thus supporting highway maintenance and traffic safety. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method, system, and medium for segmenting roadside weeds based on a vehicle recorder. This method can detect and segment roadside weeds, assisting road management personnel in better managing and maintaining roadside greenery and promptly addressing issues such as weeds obscuring signs, encroaching on roads, and excessively long weeds.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] On the one hand, a method for segmenting roadside weeds based on a vehicle-mounted recorder is provided, including the following steps:

[0008] The system acquires and decodes video footage captured by a vehicle dashcam to obtain video frame images.

[0009] The video frame images are input into the object detection model for detection, the location of weeds is located and a rectangular box is marked, resulting in a weed-marked image;

[0010] The target segmentation model is used to segment the image labeled with weeds to obtain the accurate location of the weeds. At the same time, redundant information in the rectangle is removed and the pixels at the location of the weeds are retained to obtain a binary image of the weeds.

[0011] A self-supervised monocular depth estimation model was used to estimate the depth of video frame images to obtain a weed depth map.

[0012] A fusion algorithm is used to fuse information from the binary image of weeds and the depth image of weeds to obtain a fused mask image.

[0013] After removing holes from the binary image of weeds, a mask is overlaid on the video frame image to obtain the weed segmentation result; at the same time, the height of the fused mask image is determined and then overlaid on the video frame image to obtain the near-field weed segmentation result.

[0014] As a preferred technical solution, the target detection model is built based on the YOLOv5 model and adopts an anchor-based and one-stage structure, including a BackBone layer, a CSP PAN layer, and a Head layer.

[0015] As a preferred technical solution, the step of locating the weed positions and marking them with rectangular frames to obtain a weed-marked image specifically involves:

[0016] The video frame images are input into the BackBone layer for feature extraction and pooling operations, and then input into the CSP PAN layer for upsampling and downsampling operations to obtain feature maps of three different sizes.

[0017] The Head layer classifies weeds on feature maps of three different sizes and annotates them with candidate boxes of different sizes to obtain weed-annotated images. At the same time, it uses a weighted non-maximum method to filter candidate boxes and outputs target classification and bounding box regression.

[0018] As a preferred technical solution, the target segmentation model is constructed based on the SAM model, which includes an image encoder, a cue encoder, and a mask decoder.

[0019] As a preferred technical solution, the self-supervised monocular depth estimation model is constructed based on a hybrid CNN and Transformer, including Stage1, Stage2, Stage3, Stage4, average pooling layer, 3 Concat layers, 3 upsampling layers and 3 prediction heads;

[0020] The three concat layers are concat1, concat2, and concat3; the three upsampling layers are upsampling layer 1, upsampling layer 2, and upsampling layer 3; and the three prediction heads are prediction head 1, prediction head 2, and prediction head 3.

[0021] Stage 1 is a Stem Conv convolutional layer, which is connected to Stage 2 via Concat 1 and the average pooling layer.

[0022] Stage 2, Stage 3 and Stage 4 all include a downsampling layer, a continuously dilated convolutional layer (CDC), and a local global feature interaction layer (LGFI).

[0023] The downsampling layer, Stage 2, and average pooling layer of Stage 2 are connected to Stage 3 via Concat 2;

[0024] The downsampling layer, Stage 3, and average pooling layer of Stage 3 are connected to Stage 4 via Concat 3;

[0025] The three upsampling layers are connected sequentially, and each of them is connected to one of the three prediction heads;

[0026] Stage 4 and Stage 3 are connected to upsampling layer 1 and output through prediction head 1; Stage 2 and upsampling layer 1 are connected to upsampling layer 2 and output through prediction head 2; Upsampling layer 2 and upsampling layer 3 are connected and output through prediction head 3.

[0027] As a preferred technical solution, obtaining the fused mask image specifically involves:

[0028] The weed depth map is converted from RGB to grayscale. An empirical threshold is set. After conversion, the pixel values ​​of pixels below the empirical threshold are set to 0, and the values ​​of the remaining pixels are set to 255, thus obtaining the near-field segmentation map.

[0029] The formula for converting an RGB image to a grayscale image is as follows:

[0030] Gray=R*0.299+G*0.587+B*0.114;

[0031] Where R, G, and B are the R channel value, G channel value, and B channel value of a pixel in the weed depth map, respectively; Gray is the converted grayscale value, with a value range of 0 to 255.

[0032] The weed binary image is de-holeed to obtain a weed morphology image. A fusion algorithm is then used to fuse the weed morphology image with the nearby segmentation image, retaining the overlapping information in the two images to obtain a fused mask image.

[0033] As a preferred technical solution, obtaining the near-field segmentation image specifically involves:

[0034] After converting the weed depth map to a grayscale image, an empirical threshold value is set;

[0035] Set up a queue of length m to store images. Insert each image into the queue from the tail of the queue. If the queue is full, discard the image at the head of the queue and insert the new image from the tail of the queue, ensuring that the queue length does not exceed m.

[0036] Each time an image is inserted into the queue, the average pixel value of the inserted image is calculated using the formula: Average pixel value of the image = (sum of all pixels in the image) / size of the image;

[0037] Determine whether the average pixel value of the image is within ±10% of an empirical threshold. If it is, calculate the empirical value and binarize the image based on the empirical value to obtain a near-field segmentation map; otherwise, binarize the image based on the empirical threshold to obtain a near-field segmentation map. The empirical value calculation formula is: empirical value = empirical value * 0.8 + average pixel value * 0.2.

[0038] The obtained weed morphological diagram is as follows:

[0039] Construct a two-dimensional matrix of size n*n as the kernel, and assign all values ​​to it the value 255;

[0040] Use the kernel to slide through each pixel in the weed binary image and align the center of the kernel with the current pixel; if all elements in the kernel match a corresponding image pixel, set the center pixel of the kernel to 255, otherwise set it to 0, to obtain the traversed weed binary image;

[0041] Set all kernel values ​​to 0, use the kernel to slide through each pixel in the traversed weed binary image again, and align the center of the kernel with the current pixel; if all elements in the kernel match a corresponding image pixel, set the center pixel of the kernel to 0, otherwise set it to 255, and obtain the weed morphology image.

[0042] As a preferred technical solution, the height determination of the fused mask image is as follows:

[0043] Get the number of pixels with a value of 255 in each column of the fused mask image, and compare it with half the height of the fused mask image. If it is greater than half, keep the pixels in the current column; otherwise, do not keep them.

[0044] The height determination is performed in parallel from both the left and right sides of the fused mask image; when 30 consecutive columns do not meet the conditions or reach the center line of the fused mask image, the process is considered complete, and the mask image with the retained results is output.

[0045] The mask image of the preserved result is superimposed with the video frame image to obtain the segmentation result of the nearby weeds.

[0046] On the other hand, a roadside near-field weed segmentation system based on a vehicle-mounted recorder is provided, which is applied to the above-mentioned roadside near-field weed segmentation method based on a vehicle-mounted recorder. It includes a video input module, a weed detection module, a weed segmentation module, an image monocular depth estimation module, an information fusion module, and a near-field weed information output module.

[0047] The video input module is used to acquire and decode the video captured by the vehicle dashcam to obtain video frame images;

[0048] The weed detection module is connected to the video input module and is used to input video frame images into the target detection model for detection, locate the weed positions and mark them with rectangular boxes to obtain weed-marked images.

[0049] The weed segmentation module is connected to the weed detection module and is used to segment the weed-labeled image using the target segmentation model to obtain the accurate location of the weeds. At the same time, redundant information in the rectangular box is removed, and the pixels at the location of the weeds are retained to obtain a binary image of the weeds.

[0050] The image monocular depth estimation module is connected to the video input module and is used to perform depth estimation on video frame images using a self-supervised monocular depth estimation model to obtain a weed depth map.

[0051] The information fusion module is connected to the weed segmentation module and the image monocular depth estimation module respectively, and is used to use a fusion algorithm to fuse the information of the weed binary image and the weed depth image to obtain a fused mask image.

[0052] The near-field weed information output module is connected to the information fusion module. It is used to perform mask overlay on the weed binary image after hole removal operation and the video frame image to obtain the weed segmentation result; at the same time, the height of the fused mask image is determined and then overlaid on the video frame image to obtain the near-field weed segmentation result.

[0053] In another aspect, a computer-readable storage medium is provided, storing a program that, when executed, implements the aforementioned method for separating roadside weeds based on a vehicle recorder.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] 1. The method for segmenting roadside weeds in this invention obtains weed information through weed detection and segmentation, obtains weed depth information through depth estimation, and then fuses the weed information and weed depth information through a fusion algorithm to obtain the segmentation result of nearby weeds. This achieves automatic segmentation and detection of roadside weeds, greatly improving the efficiency of highway maintenance, reducing labor costs, and providing timely image information to relevant departments so as to take corresponding maintenance measures, thus supporting highway maintenance and traffic safety.

[0056] 2. This invention uses video captured by a vehicle-mounted recorder to detect and segment nearby weeds, eliminating the need for additional cameras or other monitoring equipment. It is convenient, fast, and has high coverage, thus enabling resource reuse.

[0057] 3. This invention adopts a target detection model + target segmentation model, which greatly improves detection efficiency while ensuring the accuracy of weed detection; at the same time, it adopts a self-supervised monocular depth estimation model to add depth estimation, avoiding the problem of not being able to obtain the distance of weeds, and solving the problems of not being able to distinguish between near and far weeds and not being able to make subsequent judgments on near weeds.

[0058] 4. This application employs a morphological method that eliminates holes to avoid misidentification of weeds due to redundant information, thereby further ensuring the accuracy of the results.

[0059] 5. The near-field weed segmentation results obtained by the method of this application can help road management personnel to better manage and maintain the greenery along the roadside, and promptly address issues such as weeds obscuring signs, encroaching on the road, and excessively long weeds. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of a method for segmenting roadside weeds based on a vehicle recorder in an embodiment of the present invention.

[0062] Figure 2 This is a structural diagram of the self-supervised monocular depth estimation model in an embodiment of the present invention.

[0063] Figure 3 This is a flowchart illustrating information fusion in an embodiment of the present invention.

[0064] Figure 4 This is a flowchart illustrating the weed segmentation results and the nearby weed segmentation results obtained in an embodiment of the present invention.

[0065] Figure 5 This is a comparison chart showing the results of using and not using the fusion algorithm in the embodiments of the present invention.

[0066] Figure 6 This is a structural diagram of a roadside weed separation system based on a vehicle recorder, as described in an embodiment of the present invention. Detailed Implementation

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0068] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0069] like Figure 1 As shown, this embodiment of the method for segmenting roadside weeds based on a vehicle dashcam includes the following steps:

[0070] Step 1: Acquire the video captured by the vehicle dashcam and decode it to obtain video frame image A;

[0071] Step 2: Input the video frame images into the target detection model for detection, locate the weed positions and mark them with rectangular boxes, and obtain the weed-marked image B;

[0072] Step 3: Use the target segmentation model to segment the image labeled with weeds to obtain the accurate location of the weeds. At the same time, remove redundant information in the rectangular box and retain the pixels at the location of the weeds to obtain the binary image C of the weeds.

[0073] Step 4: Use a self-supervised monocular depth estimation model to estimate the depth of the video frame images to obtain the weed depth map D;

[0074] Step 5: Use a fusion algorithm to fuse the information of the weed binary image C and the weed depth image D to obtain the fused mask image F;

[0075] Step 6: After removing holes from the binary image of weeds C, perform mask overlay with the video frame image A to obtain the weed segmentation result H; at the same time, perform height determination on the fused mask image F and overlay it with the video frame image A to obtain the near weed segmentation result I.

[0076] Furthermore, the object detection model in this application is built based on the YOLOv5 model and adopts an anchor-based and one-stage structure, including a BackBone layer, a CSP PAN layer, and a Head layer;

[0077] The BackBone layer mainly consists of the Focus, BottleneckCSP, and SSP networks, used for feature extraction from the input image. The CSP PAN layer uses the BottleneckCSP2 module to reduce the number of model parameters, and obtains three different sizes of feature maps through upsampling and downsampling operations. The Head layer annotates candidate boxes on the three different sized feature maps and outputs object classification and bounding box regression.

[0078] In this application, a self-built dataset is used to train the object detection model. The dataset consists of video recordings of roads from a vehicle dashcam, which, after sorting and filtering, yielded 20,000 video frames, with the image perspective primarily in front of the vehicle. During model training, the batch size was set to 16, and the initial learning rate was 0.001. The training method involved two steps: freezing and unfreezing the network layers. First, the backbone layer was frozen for training, with 50 epochs (Epo). After unfreezing, another 50 epochs were trained, for a total of 100 epochs. Finally, the trained object detection model was obtained.

[0079] Furthermore, after training the object detection model, the model is used to locate the weeds and label them with bounding boxes, resulting in labeled weed images, specifically:

[0080] The video frame images are input into the BackBone layer for feature extraction and pooling operations, and then input into the CSP PAN layer for upsampling and downsampling operations to obtain feature maps of three different sizes.

[0081] The Head layer classifies weeds on feature maps of three different sizes and annotates them with candidate boxes of different sizes to obtain weed-annotated images. At the same time, it uses a weighted non-maximum method to filter candidate boxes and outputs target classification and bounding box regression.

[0082] Furthermore, the target segmentation model is trained based on a deep learning network and can generate segmentation results from a given image labeled with weeds. In this application, the target segmentation model is built based on a SAM model, including an image encoder, a cue encoder, and a mask decoder. The image encoder generates a one-time image embedding that can be applied before the cue model.

[0083] The cue encoder encodes background points, masks, bounding boxes, or text into embedding vectors in real time. This application considers two sets of cues: sparse (points, boxes, text) and dense (masks). Points and boxes are represented by positional encoding, and learned embeddings are added for each cue type; free-form text cues are represented by an off-the-shelf text encoder from CLIP (a pre-trained model based on contrastive language-image pairs); dense cues, such as masks, are represented by embedding convolutions and summed element-wise with the image embeddings.

[0084] The mask decoder maps image embeddings, cue embeddings, and output tags to a mask; all embeddings are updated by the decoder block, which uses cue self-attention and cross-attention in both directions (from cue to image embedding and back).

[0085] The SAM model has the advantages of being plug-and-play without training and can be used as a downstream task to directly segment the weed region within the candidate box and obtain a binary image of the weeds.

[0086] Furthermore, such as Figure 2 As shown, the self-supervised monocular depth estimation model is built based on a hybrid CNN and Transformer, including Stage1, Stage2, Stage3, Stage4, average pooling layer, 3 Concat layers, 3 upsampling layers and 3 prediction heads;

[0087] The three concat layers are Concat1, Concat2, and Concat3; the three upsampling layers are Upsampling Layer 1, Upsampling Layer 2, and Upsampling Layer 3; and the three prediction heads are Prediction Head 1, Prediction Head 2, and Prediction Head 3.

[0088] Stage 1 is a Stem Conv convolutional layer, which is connected to Stage 2 via Concat 1 and the average pooling layer.

[0089] Stage 2, Stage 3, and Stage 4 all include a downsampling layer, a continuously dilated convolutional layer (CDC), and a local-global feature interaction layer (LGFI). The continuously dilated convolutional layer (CDC) is used to extract rich multi-scale local features, and the local-global feature interaction layer (LGFI) uses a self-attention mechanism to encode random global information into local features.

[0090] The downsampling layer, Stage2, and average pooling layer of Stage2 are connected to Stage3 via Concat2;

[0091] The downsampling layer, Stage3, and average pooling layer of Stage3 are connected to Stage4 via Concat3;

[0092] The three upsampling layers are connected sequentially, and each layer is connected to one of the three prediction heads.

[0093] Stage 4 and Stage 3 are connected to upsampling layer 1 and output through prediction head 1; Stage 2 and upsampling layer 1 are connected to upsampling layer 2 and output through prediction head 2; upsampling layer 2 and upsampling layer 3 are connected and output through prediction head 3.

[0094] Furthermore, such as Figure 3 As shown, this application fuses depth information based on weed segmentation to obtain a fused mask image F, avoiding interference from distant weeds in the judgment. Specifically:

[0095] The weed depth map D is converted from an RGB image to a grayscale image. An empirical threshold is set, and after conversion, pixels with values ​​below the threshold are set to 0, while the remaining pixels are set to 255, resulting in a near-field segmentation image. The formula for converting the RGB image to grayscale is as follows:

[0096] Gray=R*0.299+G*0.587+B*0.114;

[0097] Where R, G, and B are the R channel value, G channel value, and B channel value of a pixel in the weed depth map, respectively; Gray is the converted grayscale value, with a value range of 0 to 255.

[0098] The weed binary image C is subjected to hole removal operation to obtain the weed morphology image E. The weed morphology image E is then fused with the nearby segmentation image using a fusion algorithm, retaining the overlapping information in the two images to obtain the fused mask image F.

[0099] Furthermore, this application obtains the near-field segmentation map based on empirical values, specifically as follows:

[0100] After converting the weed depth map D into a grayscale image, an empirical threshold value is set.

[0101] Set up a queue of length m to store images. Insert each image into the queue from the tail of the queue. If the queue is full, discard the image at the head of the queue and insert the new image from the tail of the queue, ensuring that the queue length does not exceed m.

[0102] Each time an image is inserted into the queue, the average pixel value of the inserted image is calculated using the formula: Average pixel value of the image = (sum of all pixels in the image) / size of the image;

[0103] Determine whether the average pixel value of the image is within ±10% of the empirical threshold. If it is, calculate the empirical value and binarize the image based on the empirical value to obtain the near segmentation map; otherwise, binarize the image based on the empirical threshold to obtain the near segmentation map. The empirical value is calculated as follows: empirical value = empirical value * 0.8 + average pixel value * 0.2.

[0104] The morphological image of the weeds, E, was obtained by eliminating the holes. Specifically:

[0105] Construct a two-dimensional matrix of size n*n as the kernel, and assign all values ​​to it the value 255;

[0106] The kernel is used to slide through each pixel of the weed binary image and the center of the kernel is aligned with the current pixel. If all elements in the kernel match a corresponding image pixel, the center pixel of the kernel is set to 255, otherwise it is set to 0. This operation can cause pixels to expand and fill holes.

[0107] The kernel is set to 0 for all values. The kernel is then used to slide through each pixel in the traversed binary image of weeds, and the center of the kernel is aligned with the current pixel. If all elements in the kernel match a corresponding image pixel, the center pixel of the kernel is set to 0; otherwise, it is set to 255. This operation can smooth the boundary of the white mask. After the above calculation, some holes in the image can be eliminated, avoiding the impact and errors on subsequent processing, and the weed morphology image E is obtained.

[0108] In this embodiment, the empirical threshold is set to 150, the queue length m = 10, and the constructed kernel size is 5*5.

[0109] Furthermore, such as Figure 4 As shown, height determination is performed on the fused mask image F to obtain the near-field weed segmentation result I, specifically:

[0110] Obtain the number of pixels with a value of 255 in each column of the fused mask image F, and compare them with half the height of the fused mask image. If the value is greater, keep the pixel.

[0111] The height determination is performed in parallel from the left and right sides of the fused mask image. The process ends when 30 consecutive columns fail to meet the conditions or reach the center line of the fused mask image, and the mask image with the retained results is output.

[0112] The mask image of the preserved result is superimposed with the video frame image to obtain the near-field weed segmentation result I.

[0113] In this embodiment, the mask in weed segmentation result H is blue; the mask in near weed segmentation result I is red.

[0114] Figure 5 The image shows a comparison between the proposed method and a method without depth information fusion. As can be seen, there are no excessively long weeds in the video frame. Using only image segmentation would not yield accurate results because it would simultaneously classify both distant and nearby weeds as excessively long. However, the proposed method incorporates depth information—namely, monocular depth estimation followed by information fusion—which allows for the determination of the weeds' distance and analysis to identify cases where excessively long weeds do not exist in the image, effectively reducing erroneous analysis results.

[0115] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0116] Based on the same idea as the roadside weed segmentation method based on a vehicle-mounted recorder in the above embodiments, the present invention also provides a roadside weed segmentation system based on a vehicle-mounted recorder, which can be used to execute the above-described roadside weed segmentation method based on a vehicle-mounted recorder. For ease of explanation, the structural diagram of the roadside weed segmentation system based on a vehicle-mounted recorder embodiment only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0117] like Figure 6 As shown, another embodiment of the present invention provides a roadside near-field weed segmentation system based on a vehicle recorder, including a video input module, a weed detection module, a weed segmentation module, an image monocular depth estimation module, an information fusion module, and a near-field weed information output module;

[0118] The video input module is used to acquire and decode the video captured by the vehicle dashcam to obtain video frame images.

[0119] The weed detection module is connected to the video input module and is used to input video frame images into the target detection model for detection, locate the weed positions and mark them with rectangular boxes to obtain weed-marked images;

[0120] The weed segmentation module is connected to the weed detection module. It is used to segment the weed-labeled image using the target segmentation model to obtain the accurate location of the weeds. At the same time, it removes redundant information in the rectangular box, retains the pixels at the location of the weeds, and obtains a binary image of the weeds.

[0121] The image monocular depth estimation module is connected to the video input module and is used to perform depth estimation on video frame images using a self-supervised monocular depth estimation model to obtain a weed depth map.

[0122] The information fusion module is connected to the weed segmentation module and the image monocular depth estimation module, respectively. It is used to use a fusion algorithm to fuse the information of the weed binary image and the weed depth image to obtain a fused mask image.

[0123] The near-field weed information output module is connected to the information fusion module. It is used to perform mask overlay on the binary image of weeds after hole removal and video frame image to obtain the weed segmentation result; at the same time, the height of the fused mask image is determined and then overlaid on the video frame image to obtain the near-field weed segmentation result.

[0124] Furthermore, in the above embodiments of the roadside weed segmentation system based on a vehicle recorder, the logical division of each program module is merely an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the roadside weed segmentation system based on a vehicle recorder can be divided into different program modules to complete all or part of the functions described above.

[0125] This invention also provides a computer-readable storage medium storing a program that, when executed by an executable device, implements a method for segmenting roadside weeds based on a vehicle recorder, specifically:

[0126] The system acquires and decodes video footage captured by a vehicle dashcam to obtain video frame images.

[0127] The video frame images are input into the object detection model for detection, the location of weeds is located and a rectangular box is marked, resulting in a weed-marked image;

[0128] The target segmentation model is used to segment the image labeled with weeds to obtain the accurate location of the weeds. At the same time, redundant information in the rectangle is removed and the pixels at the location of the weeds are retained to obtain a binary image of the weeds.

[0129] A self-supervised monocular depth estimation model was used to estimate the depth of video frame images to obtain a weed depth map.

[0130] A fusion algorithm is used to fuse information from the binary image of weeds and the depth image of weeds to obtain a fused mask image.

[0131] After removing holes from the binary image of weeds, a mask is overlaid on the video frame image to obtain the weed segmentation result; at the same time, the height of the fused mask image is determined and then overlaid on the video frame image to obtain the near-field weed segmentation result.

[0132] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for segmenting roadside weeds based on a vehicle-mounted recorder, characterized in that, Includes the following steps: The system acquires and decodes video footage captured by a vehicle dashcam to obtain video frame images. The video frame images are input into the object detection model for detection, the location of weeds is located and a rectangular box is marked, resulting in a weed-marked image; The target segmentation model is used to segment the image labeled with weeds to obtain the accurate location of the weeds. At the same time, redundant information in the rectangle is removed and the pixels at the location of the weeds are retained to obtain a binary image of the weeds. A self-supervised monocular depth estimation model was used to estimate the depth of video frame images to obtain a weed depth map. A fusion algorithm is used to fuse the binary image and depth image of weeds to obtain a fused mask image, specifically: The weed depth map is converted from RGB to grayscale. An empirical threshold is set. After conversion, the pixel values ​​of pixels below the empirical threshold are set to 0, and the values ​​of the remaining pixels are set to 255, thus obtaining the near-field segmentation map. The formula for converting an RGB image to a grayscale image is as follows: Gray = R*0.299 + G*0.587 + B*0.114, Where R, G, and B are the R channel value, G channel value, and B channel value of a pixel in the weed depth map, respectively; Gray is the converted grayscale value, with a value range of 0 to 255. The weed binary image is de-holeed to obtain a weed morphology image. A fusion algorithm is then used to fuse the weed morphology image with the nearby segmentation image, retaining the overlapping information in the two images to obtain a fused mask image. The obtained near-field segmentation map is specifically as follows: After converting the weed depth map to a grayscale image, an empirical threshold value is set; Set up a queue of length m to store images. Insert each image into the queue from the tail of the queue. If the queue is full, discard the image at the head of the queue and insert the new image from the tail of the queue, ensuring that the queue length does not exceed m. Each time an image is inserted into the queue, the average pixel value of the inserted image is calculated using the formula: Average pixel value of the image = (sum of all pixels in the image) / size of the image; Determine whether the average pixel value of the image is within ±10% of an empirical threshold. If it is, calculate the empirical value and binarize the image based on the empirical value to obtain a near-field segmentation map; otherwise, binarize the image based on the empirical threshold to obtain a near-field segmentation map. The empirical value calculation formula is: empirical value = empirical value * 0.8 + average pixel value * 0.

2. After removing holes from the binary image of weeds, a mask is overlaid on the video frame image to obtain the weed segmentation result; at the same time, the height of the fused mask image is determined and then overlaid on the video frame image to obtain the near weed segmentation result. The height determination of the fused mask image is as follows: Get the number of pixels with a value of 255 in each column of the fused mask image, and compare it with half the height of the fused mask image. If it is greater than half, keep the pixels in the current column; otherwise, do not keep them. The height determination is performed in parallel from both the left and right sides of the fused mask image; when 30 consecutive columns do not meet the conditions or reach the center line of the fused mask image, the process is considered complete, and the mask image with the retained results is output. The mask image of the preserved result is superimposed with the video frame image to obtain the segmentation result of the nearby weeds.

2. The method for segmenting roadside weeds based on a vehicle recorder according to claim 1, characterized in that, The target detection model is built on the YOLOv5 model and adopts an anchor-based and one-stage structure, including a BackBone layer, a CSP PAN layer, and a Head layer.

3. The method for segmenting roadside weeds based on a vehicle recorder according to claim 2, characterized in that, The process of locating the weeds and marking them with rectangular boxes to obtain a weed-marked image is as follows: The video frame images are input into the BackBone layer for feature extraction and pooling operations, and then input into the CSP PAN layer for upsampling and downsampling operations to obtain feature maps of three different sizes. The Head layer classifies weeds on feature maps of three different sizes and annotates them with candidate boxes of different sizes to obtain weed-annotated images. At the same time, it uses a weighted non-maximum method to filter candidate boxes and outputs target classification and bounding box regression.

4. The method for segmenting roadside weeds based on a vehicle recorder according to claim 1, characterized in that, The target segmentation model is built based on the SAM model and includes an image encoder, a cue encoder, and a mask decoder.

5. The method for segmenting roadside weeds based on a vehicle recorder according to claim 1, characterized in that, The self-supervised monocular depth estimation model is constructed based on a hybrid CNN and Transformer, including Stage1, Stage2, Stage3, Stage4, average pooling layer, 3 Concat layers, 3 upsampling layers, and 3 prediction heads; The three concat layers are concat1, concat2, and concat3; the three upsampling layers are upsampling layer 1, upsampling layer 2, and upsampling layer 3; and the three prediction heads are prediction head 1, prediction head 2, and prediction head 3. Stage 1 is a Stem Conv convolutional layer, which is connected to Stage 2 via Concat 1 and the average pooling layer. Stage 2, Stage 3 and Stage 4 all include a downsampling layer, a continuously dilated convolutional layer (CDC), and a local global feature interaction layer (LGFI). The downsampling layer, Stage 2, and average pooling layer of Stage 2 are connected to Stage 3 via Concat 2; The downsampling layer, Stage 3, and average pooling layer of Stage 3 are connected to Stage 4 via Concat 3; The three upsampling layers are connected sequentially, and each of them is connected to one of the three prediction heads; Stage 4 and Stage 3 are connected to upsampling layer 1 and output through prediction head 1; Stage 2 and upsampling layer 1 are connected to upsampling layer 2 and output through prediction head 2; Upsampling layer 2 and upsampling layer 3 are connected and output through prediction head 3.

6. The method for segmenting roadside weeds based on a vehicle recorder according to claim 1, characterized in that, The obtained weed morphological diagram is as follows: Construct a two-dimensional matrix of size n*n as the kernel, and assign all values ​​to it the value 255; Use the kernel to slide through each pixel in the weed binary image and align the center of the kernel with the current pixel; if all elements in the kernel match a corresponding image pixel, set the center pixel of the kernel to 255, otherwise set it to 0, to obtain the traversed weed binary image; Set all kernel values ​​to 0, use the kernel to slide through each pixel in the traversed weed binary image again, and align the center of the kernel with the current pixel; if all elements in the kernel match a corresponding image pixel, set the center pixel of the kernel to 0, otherwise set it to 255, and obtain the weed morphology image.

7. A roadside weed segmentation system based on a vehicle-mounted recorder, characterized in that, The method for segmenting roadside weeds based on a vehicle recorder, applicable to any one of claims 1-6, includes a video input module, a weed detection module, a weed segmentation module, an image monocular depth estimation module, an information fusion module, and a nearby weed information output module; The video input module is used to acquire and decode the video captured by the vehicle dashcam to obtain video frame images; The weed detection module is connected to the video input module and is used to input video frame images into the target detection model for detection, locate the weed positions and mark them with rectangular boxes to obtain weed-marked images. The weed segmentation module is connected to the weed detection module and is used to segment the weed-labeled image using the target segmentation model to obtain the accurate location of the weeds. At the same time, redundant information in the rectangular box is removed, and the pixels at the location of the weeds are retained to obtain a binary image of the weeds. The image monocular depth estimation module is connected to the video input module and is used to perform depth estimation on video frame images using a self-supervised monocular depth estimation model to obtain a weed depth map. The information fusion module is connected to both the weed segmentation module and the image monocular depth estimation module. It is used to fuse the weed binary image and the weed depth image using a fusion algorithm to obtain a fused mask image. Specifically: The weed depth map is converted from RGB to grayscale. An empirical threshold is set. After conversion, the pixel values ​​of pixels below the empirical threshold are set to 0, and the values ​​of the remaining pixels are set to 255, thus obtaining the near-field segmentation map. The formula for converting an RGB image to a grayscale image is as follows: Gray = R*0.299 + G*0.587 + B*0.114, Where R, G, and B are the R channel value, G channel value, and B channel value of a pixel in the weed depth map, respectively; Gray is the converted grayscale value, with a value range of 0 to 255. The weed binary image is de-holeed to obtain a weed morphology image. A fusion algorithm is then used to fuse the weed morphology image with the nearby segmentation image, retaining the overlapping information in the two images to obtain a fused mask image. The obtained near-field segmentation map is specifically as follows: After converting the weed depth map to a grayscale image, an empirical threshold value is set; Set up a queue of length m to store images. Insert each image into the queue from the tail of the queue. If the queue is full, discard the image at the head of the queue and insert the new image from the tail of the queue, ensuring that the queue length does not exceed m. Each time an image is inserted into the queue, the average pixel value of the inserted image is calculated using the formula: Average pixel value of the image = (sum of all pixels in the image) / size of the image; Determine whether the average pixel value of the image is within ±10% of an empirical threshold. If it is, calculate the empirical value and binarize the image based on the empirical value to obtain a near-field segmentation map; otherwise, binarize the image based on the empirical threshold to obtain a near-field segmentation map. The empirical value calculation formula is: empirical value = empirical value * 0.8 + average pixel value * 0.

2. The near-field weed information output module is connected to the information fusion module. It is used to perform mask overlay on the weed binary image after hole removal operation and the video frame image to obtain the weed segmentation result; at the same time, the height of the fused mask image is determined and then overlaid on the video frame image to obtain the near-field weed segmentation result. The height determination of the fused mask image is as follows: Get the number of pixels with a value of 255 in each column of the fused mask image, and compare it with half the height of the fused mask image. If it is greater than half, keep the pixels in the current column; otherwise, do not keep them. The height determination is performed in parallel from both the left and right sides of the fused mask image; when 30 consecutive columns do not meet the conditions or reach the center line of the fused mask image, the process is considered complete, and the mask image with the retained results is output. The mask image of the preserved result is superimposed with the video frame image to obtain the segmentation result of the nearby weeds.

8. A computer-readable storage medium storing a program, characterized in that, When the program is executed, it implements the method for separating roadside weeds based on a vehicle recorder as described in any one of claims 1-6.

Images