Road detection method and device

Abstract

This specification provides a road detection method and device, wherein the road detection method includes: determining a road image in a road video collected for a target road, and constructing a multimodal image corresponding to the road image, obtaining rich environmental information, and improving the accuracy of subsequent vehicle detection. The multimodal image is fused in the image channel dimension using a multimodal model, and the target vehicle in the multimodal image is determined based on the fusion result; the road information of the target road is generated based on the target area and the target vehicle in the multimodal image. The multimodal model is used to detect and track the target vehicle in the multimodal image to achieve efficient detection and tracking of the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a road detection method and device. Background Art

[0002] Road condition monitoring plays a vital role in modern traffic management. Its purpose is to provide timely and accurate traffic information to support traffic flow management, accident handling and congestion warning.

[0003] However, traditional road condition monitoring methods usually rely on fixed ground monitoring equipment or manual inspections. These methods not only have limited coverage, but are also easily affected by environmental factors such as weather and light, resulting in low monitoring accuracy. Therefore, a more effective road detection method is urgently needed to solve the above problems. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a road detection method. This specification also relates to a road detection device, a computing device, a computer-readable storage medium and a computer program product to solve the above problems existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a road detection method is provided, including: Determine a road image in a road video collected for a target road, and construct a multimodal image corresponding to the road image; Using a multimodal model to fuse the multimodal images in an image channel dimension, and determining a target vehicle in the multimodal image according to a fusion result; The road information of the target road is generated based on the target area and the target vehicle in the multimodal image.

[0006] Optionally, determining a road image in a road video collected for a target road and constructing a multimodal image corresponding to the road image includes: Using a drone equipped with a binocular image acquisition device and a thermal imaging image acquisition device to collect the road video; Determine a target frame in the road video, and determine an initial thermal imaging image and an initial channel image corresponding to the target frame in the road video; The initial thermal imaging image and the initial channel image are preprocessed respectively to obtain a thermal imaging image and a channel image, and the thermal imaging image and the channel image are used as the road image.

[0007] Optionally, constructing a multimodal image corresponding to the road image includes: Extracting thermal imaging feature points corresponding to the thermal imaging image and channel feature points corresponding to the channel image in the road image based on a feature extraction algorithm; Performing feature point matching on the thermal imaging feature points and the channel feature points to obtain at least one pair of matching feature points; Generate a disparity image based on the disparity information respectively corresponding to the at least one pair of matching feature points, and generate a depth image based on the disparity image; The multimodal image is generated based on the depth image, the thermal imaging image, and the channel image.

[0008] Optionally, generating a depth image based on the disparity image includes: Determining device parameters of the binocular image acquisition device and a disparity value of the disparity image, and calculating a depth value based on the disparity value; The disparity image is converted into the depth image based on the device parameters, the disparity value, and the depth value.

[0009] Optionally, the using a multimodal model to fuse the multimodal images in an image channel dimension includes: Determining the depth module, thermal imaging module and channel module included in the multimodal model; Extracting depth features of the depth image using the depth module, extracting thermal imaging features of the thermal imaging image using the thermal imaging module, and extracting channel features of the channel image using the channel module; The depth feature, the thermal imaging feature and the channel feature are fused in the channel dimension to obtain a fusion result.

[0010] Optionally, determining the target vehicle in the multimodal image according to the fusion result includes: Constructing a multimodal feature map based on the fusion result; The target vehicle included in the multimodal feature map is identified based on a self-attention module of the multimodal model.

[0011] Optionally, the generating the road information of the target road based on the target area and the target vehicle in the multimodal image includes: Determining attribute information of the target vehicle and demarcating the target area in the multimodal image; A vehicle region of the target vehicle is determined based on the attribute information, and road information of the target road is generated based on a positional relationship between the vehicle region and the target region.

[0012] Optionally, determining the vehicle area of ​​the target vehicle based on the attribute information includes: Determine at least two to-be-matched vehicles included in the target vehicle, and determine in the road video an image to be matched that is associated with the road image; Predicting predicted vehicles corresponding to the at least two vehicles to be matched in the image to be matched based on the attribute information, and constructing a vehicle matching pair according to the prediction results; The area corresponding to the detected vehicle in the vehicle matching pair is used as the vehicle area of ​​the target vehicle.

[0013] Optionally, the generating the road information of the target road based on the positional relationship between the vehicle area and the target area includes: Calculating a vehicle density of the target area based on a positional relationship between the vehicle area and the target area; When the vehicle density is greater than a density threshold, the road information of the target road is generated.

[0014] Optionally, after generating the road information of the target road based on the target area in the multimodal image and the target vehicle, the method further includes: The attribute information and the road image are input into a road model to obtain road detection information corresponding to the road information.

[0015] According to a second aspect of an embodiment of this specification, a road detection device is provided, including: A determination module is configured to determine a road image in a road video collected for a target road, and construct a multimodal image corresponding to the road image; A fusion module is configured to fuse the multimodal images in the image channel dimension using a multimodal model, and determine the target vehicle in the multimodal image according to the fusion result; A generating module is configured to generate road information of the target road based on the target area and the target vehicle in the multimodal image.

[0016] According to a third aspect of an embodiment of this specification, a computing device is provided, comprising a memory, a processor, and a computer program or instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the road detection method when executing the computer program or instructions.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program or instructions, and the computer program or instructions implement the steps of the road detection method when executed by a processor.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned road detection method when executed by a processor.

[0019] The road detection method provided in this specification determines the road image in the road video collected for the target road, and constructs a multimodal image corresponding to the road image to obtain rich environmental information and improve the accuracy of subsequent vehicle detection. The multimodal image is fused in the image channel dimension using a multimodal model, and the target vehicle in the multimodal image is determined based on the fusion result; the road information of the target road is generated based on the target area and target vehicle in the multimodal image. The multimodal model is used to detect and track the target vehicle in the multimodal image to achieve efficient detection and tracking of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a road detection method provided in an embodiment of this specification; Figure 2 is a schematic diagram of a multimodal model of a road detection method provided in an embodiment of this specification; Figure 3 It is a processing flow chart of a road detection method applied to highway road condition monitoring provided in an embodiment of this specification; Figure 4 It is a structural schematic diagram of a road detection device provided in an embodiment of this specification; Figure 5 It is a structural block diagram of a computing device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0021] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0022] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0024] First, the terms involved in one or more embodiments of this specification are explained.

[0025] FAST (Features from Accelerated Segment Test): is a fast algorithm for corner detection. The core idea of ​​the FAST algorithm is to identify key points by detecting the local contrast of pixels in the image. Specifically, the algorithm defines a circular area around each pixel and checks whether a series of consecutive pixels in the area meet certain brightness conditions. If the brightness values ​​of these pixels are significantly different from the brightness value of the central pixel, the pixel is considered to be a key point.

[0026] Harris: Corner metric function is an important tool for corner detection in the field of image processing. This function determines whether a point is a corner by calculating the local grayscale change of each pixel in the image. Harris corner detection algorithm extracts corners directly based on grayscale images. Its principle is to use a moving window to calculate the grayscale change value in the image. The characteristic of a corner is that when the window moves in any direction, it will cause a significant change in the grayscale of the image. This change can be quantified by calculating the grayscale difference of the pixels in the window.

[0027] TrafficVision Big Model: It is based on the application of big model technology in the field of transportation, combining the advantages of TrafficVision road traffic flow simulation system. It aims to improve traffic efficiency, reduce traffic congestion and improve traffic safety by analyzing and optimizing historical traffic data and predicting future traffic flow.

[0028] ViLT (Vision-and-Language Transformer): A Transformer model that combines visual and language information, designed to handle cross-modal tasks such as image description generation, visual question answering, etc.

[0029] SORT (Simple Online and Realtime Tracking) algorithm: A classic algorithm for multi-target tracking, the core of which is to combine target detection and motion prediction for efficient target tracking. The design goal of the SORT algorithm is to achieve efficient target tracking with limited computing resources. It detects the position and bounding box of the target object in each frame by using a target detector (such as Faster R-CNN), then uses a Kalman filter to model and predict the target's motion, and finally establishes a target association relationship between the current frame and the previous frame through the Hungarian algorithm to determine the target's identity and trajectory.

[0030] In this specification, a road detection method is provided. This specification also relates to a road detection device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0031] Figure 1 A flow chart of a road detection method provided according to an embodiment of this specification is shown, which specifically includes the following steps: Step 102: determining a road image in a road video collected for a target road, and constructing a multimodal image corresponding to the road image.

[0032] Specifically, the target road may be a highway or other driving road. Road video refers to a video stream collected for the target road. The road video may be collected by a drone equipped with an image acquisition device. When the drone collects the road video, it may hover at a fixed position to continuously collect the video for the target road, or it may collect the road video for the target road as the drone moves. The road image may be an image corresponding to any frame in the road video. The road image includes the target road and vehicles traveling on the target road. A multimodal image refers to an image containing multimodal information such as parallax information, RGB information, and depth information.

[0033] Based on this, a drone equipped with an image acquisition device is used to collect road videos of the target road, and a frame of image is selected from the road video as the road image. Based on the road image, a multimodal image corresponding to the road image is constructed.

[0034] Furthermore, when collecting road videos for target roads, drones equipped with binocular image acquisition equipment and thermal imaging image acquisition equipment can be used for collection, which is specifically implemented as follows: The road video is collected by using a drone equipped with a binocular image acquisition device and a thermal imaging image acquisition device; a target frame is determined in the road video, and an initial thermal imaging image and an initial channel image corresponding to the target frame are determined in the road video; the initial thermal imaging image and the initial channel image are preprocessed respectively to obtain a thermal imaging image and a channel image, and the thermal imaging image and the channel image are used as the road image.

[0035] Specifically, the binocular image acquisition device may be a binocular camera, which uses two cameras to simultaneously capture the same scene and generate two 2D images. The three-dimensional coordinates of the object are calculated by triangulation principle using the baseline distance and parallax between the two cameras. The thermal imaging image acquisition device may be a thermal imaging camera, which captures and measures the heat distribution of the target object by sensing the infrared radiation energy emitted by the target object. The infrared radiation energy is converted into an electrical signal using a thermosensitive element (such as a micro-pile, a thermocouple, etc.), and then the electrical signal is converted into a visual thermal image through signal processing and image generation algorithms. The target frame determined in the road video may be any frame in the road video. The initial thermal imaging image refers to the thermal imaging image corresponding to the target frame, and the initial channel image refers to the RGB image corresponding to the target frame. Preprocessing includes but is not limited to histogram equalization processing and noise reduction (noise suppression) processing. The histogram equalization processing is used to enhance the contrast details of the image, and the noise reduction (Gaussian filtering) is used to enhance the smoothness of the image.

[0036] Based on this, a drone equipped with binocular image acquisition equipment and thermal imaging image acquisition equipment is used to hover above the target road to collect road video. The target frame is determined in the road video, and the target frame is not the last frame in the road video. The initial thermal imaging image and the initial channel image corresponding to the target frame are determined in the road video. The initial thermal imaging image and the initial channel image are preprocessed by histogram equalization and noise reduction, respectively, to obtain a thermal imaging image and a channel image. The thermal imaging image and the channel image are used as road images.

[0037] For example, when conducting road inspection on a target road, a binocular camera and a thermal imaging camera carried by a drone can be used to hover above the road to collect road video data. A binocular visual image stream containing both RGB and thermal imaging information is obtained in each frame to ensure the time synchronization of the two data. For the thermal imaging data, it is first mapped into a pseudo-color image, and then the RGB image and the thermal imaging image are respectively subjected to histogram equalization processing to enhance the contrast and details. On this basis, an adaptive Gaussian filter is applied to suppress noise and improve the smoothness of the image. Using the parallax correction technology of binocular vision, based on the calibrated binocular camera parameters, the coordinates of the two images are aligned and the parallax is adjusted to obtain a road image.

[0038] In summary, using a drone equipped with binocular image acquisition equipment and thermal imaging image acquisition equipment to collect road videos can obtain rich environmental information. By preprocessing the initial thermal imaging image and the initial channel image with histogram equalization and noise reduction, the smoothness and contrast details of the image can be improved.

[0039] Furthermore, considering that the thermal imaging image and the channel image contain different feature points, the parallax image can be generated by matching the feature points. The specific implementation is as follows: Based on a feature extraction algorithm, thermal imaging feature points corresponding to the thermal imaging image and channel feature points corresponding to the channel image are extracted from the road image; feature point matching is performed on the thermal imaging feature points and the channel feature points to obtain at least one pair of matching feature points; a disparity image is generated based on disparity information corresponding to the at least one pair of matching feature points, and a depth image is generated based on the disparity image; and the multimodal image is generated based on the depth image, the thermal imaging image and the channel image.

[0040] Specifically, the feature extraction algorithm may be a FAST algorithm. Thermal imaging feature points may be key feature points represented in the form of feature descriptors in the thermal imaging image, and the key feature points may be determined in the feature points in the thermal imaging image by the Harris corner metric function. Channel feature points may be key feature points represented in the form of feature descriptors in the channel image, and the key feature points in the channel image may be determined in the feature points in the channel image by the Harris corner metric function. A pair of matching feature points may be determined by calculating the Euclidean distance. The disparity information is obtained based on the coordinates of the feature points in a pair of matching feature points. The depth image is an image generated after the depth value is obtained by calculating the depth formula based on the disparity information corresponding to the disparity image.

[0041] Based on this, feature points corresponding to the thermal imaging image in the road image are extracted based on the feature extraction algorithm, and the extracted feature points are calculated using the Harris corner point measurement function, and the thermal imaging feature points with higher response intensity are retained. Feature points corresponding to the channel image are extracted based on the feature extraction algorithm, and the extracted feature points are calculated using the Harris corner point measurement function, and the channel feature points with higher response intensity are retained; in the form of feature descriptors, the thermal imaging feature points and the channel feature points are matched by calculating the Euclidean distance to obtain at least one pair of matching feature points. The disparity information corresponding to at least one pair of matching feature points is calculated, and a disparity image is generated based on the disparity information corresponding to at least one pair of matching feature points, and the disparity image is converted into a depth image. A multimodal image is generated based on the depth image, the thermal imaging image and the channel image.

[0042] Following the above example, for the thermal imaging image and channel image obtained based on the road video, the FAST algorithm is used to find the key feature points, and the feature points with the highest response intensity are retained through the Harris corner metric function; then, the direction of each feature point is calculated, and the BRIEF descriptor is rotated to the main direction of the feature point to obtain a feature descriptor with rotation invariance. The feature descriptors in the thermal imaging image and the channel image are compared, the similarity between each pair of feature descriptors is calculated using the Euclidean distance formula, and the disparity information of the matching points corresponding to each pair of feature descriptors is calculated based on the baseline distance of the binocular camera to generate a disparity map. The depth information is then calculated using the depth formula, and the disparity map is converted into a depth map. A multimodal image is generated based on the depth image, thermal imaging image, and channel image. The calculated depth map is aligned with the thermal imaging image and the RGB image (channel image) so that each pixel contains the thermal imaging temperature, RGB color, and depth information, achieving the initial fusion of multimodal data and obtaining a multimodal image.

[0043] In summary, feature extraction algorithms are used to extract feature points corresponding to thermal imaging images and channel images in road images, and then the extracted feature points are matched to obtain a disparity map. A multimodal image is generated based on the depth map, thermal imaging image, and channel image corresponding to the disparity map, and the road information elements contained in the multimodal image are enhanced.

[0044] Furthermore, after obtaining the disparity map, the disparity map can be converted into a depth map by calculating the image depth information based on the disparity map, which is specifically implemented as follows: Determine device parameters of the binocular image acquisition device and a disparity value of the disparity image, and calculate a depth value based on the disparity value; and convert the disparity image into the depth image based on the device parameters, the disparity value and the depth value.

[0045] Specifically, the device parameters of the binocular image acquisition device may be binocular camera parameters, including but not limited to the focal length and baseline distance of the binocular camera. The disparity value of the disparity image refers to the difference between the horizontal coordinates of each pair of matching feature points in the disparity image in the image coordinate system. The depth value refers to the actual depth value of each pixel in the disparity image calculated based on the depth formula, and the depth value represents the spatial information of the disparity image.

[0046] Based on this, the device parameters such as the focal length and baseline distance of the binocular image acquisition device are determined, the disparity value of the disparity image is determined, and the depth value is calculated based on the disparity value and the depth formula. Based on the device parameters, disparity value and depth value, the disparity image is converted into a three-dimensional scene graph with spatial information, namely a depth image.

[0047] Following the above example, after generating the disparity map, the disparity value of the disparity map is determined. The calculation method of the disparity value of any pair of matching feature points is as follows: Determine the horizontal coordinate X of the feature point corresponding to the thermal imaging image in the matching feature point L The horizontal coordinate X of the feature point corresponding to the channel image R , the disparity value of the matching feature point is d=X L -X R The depth formula is Z=FB / d, where F is the focal length of the binocular camera and B is the baseline distance of the binocular camera. Based on the depth formula, disparity value, and device parameters such as focal length and baseline distance, the depth value can be calculated to generate a depth image corresponding to the disparity image.

[0048] In summary, the disparity image is converted into a three-dimensional scene graph with spatial information, namely, a depth image, based on device parameters, disparity values ​​and depth values, so that the depth image can contain control information and participate in subsequent multimodal image generation as a three-dimensional scene graph.

[0049] Step 104: using a multimodal model to fuse the multimodal images in the image channel dimension, and determining the target vehicle in the multimodal images according to the fusion result.

[0050] Specifically, after determining the road image in the road video collected for the target road and constructing the multimodal image corresponding to the road image, the multimodal model can be used to fuse the multimodal image in the image channel dimension, and the target vehicle in the multimodal image is determined according to the fusion result, wherein the main structure of the multimodal model is a three-branch structure, which is composed of a feature extraction module and a detection module, and the three-branch structure is used to process multimodal information. Fusion in the image channel dimension refers to splicing in units of channels. The fusion result is a feature fusion image obtained by fusing the multimodal image in dimensions such as parallax, RGB, and depth. The target vehicle may be a vehicle contained in the road image.

[0051] Based on this, after determining the road image in the road video collected for the target road and constructing the multimodal image corresponding to the road image, the multimodal image is fused in the image channel dimension using the multimodal model to obtain the fusion result. The multimodal image is identified according to the fusion result, and the target detection is performed on the multimodal image to detect the target vehicle in the multimodal image.

[0052] Furthermore, when the multimodal model is used to process the multimodal image, the depth module, thermal imaging module and channel module included in the multimodal model can be used to process the multimodal image in multiple dimensions. The specific implementation is as follows: Determine the depth module, thermal imaging module and channel module included in the multimodal model; use the depth module to extract the depth features of the depth image, use the thermal imaging module to extract the thermal imaging features of the thermal imaging image, and use the channel module to extract the channel features of the channel image; fuse the depth features, the thermal imaging features and the channel features in the channel dimension to obtain a fusion result.

[0053] Specifically, the depth module, thermal imaging module and channel module are three branch structures of the multimodal model, and each branch structure has the functions of feature extraction and target detection. The depth module, thermal imaging module and channel module all use four parallel residual blocks, and each residual block uses a different convolution kernel size to ensure that each branch responds to features of different scales. The depth features, thermal imaging features and channel features are fused in the channel dimension, which can be the splicing of the depth features, thermal imaging features and channel features in the channel dimension.

[0054] Based on this, the depth module, thermal imaging module and channel module included in the multimodal model are determined. The depth features of the depth image are extracted using the four parallel residual blocks with different convolution kernels included in the depth module, the thermal imaging features of the thermal imaging image are extracted using the four parallel residual blocks with different convolution kernels included in the thermal imaging module, and the channel features of the channel image are extracted using the four parallel residual blocks with different convolution kernels included in the channel module. The depth features, thermal imaging features and channel features are spliced ​​in the channel dimension to obtain the spliced ​​multimodal feature map, that is, the fusion result.

[0055] Using the above example, the multimodal model is a multimodal deep learning model. The main structure of the model is a three-branch structure, which is mainly composed of a feature extraction module and a detection module. The three-branch structure is used to process multimodal information. The three branches extract features from the depth map, thermal image and RGB map respectively. After the extracted features are obtained, they are spliced ​​in the channel dimension. Among them, the feature extraction module is based on the residual block and uses four residual blocks with different convolution kernel sizes to extract features in parallel. Among them, the detection module combines the multi-scale feature pyramid and the self-attention mechanism to integrate the features of each modality to accurately locate and identify the position of the target object. The input of each branch is first adjusted to a resolution of 256×256 and then sent to the corresponding branch. The feature extraction module of each branch uses four parallel residual blocks, and each residual block uses a different convolution kernel size. The convolution kernel sizes of the four parallel residual blocks are 3×3, 5×5, 7×7 and 9×9 respectively to ensure that each branch responds to features of different scales. After four residual blocks are processed in parallel, the output feature map of each branch is resized to 64×64×64, where 64 represents the number of channels. After the three-branch feature maps are concatenated in the channel dimension, the shape of the generated multimodal feature map becomes 64×64×192.

[0056] In summary, the depth module, thermal imaging module and channel module included in the multimodal model are used to extract the depth features of the depth image, the thermal imaging features of the thermal imaging image and the channel features of the channel image respectively, and the extracted features are spliced ​​in the channel dimension to improve the accuracy of subsequent vehicle detection.

[0057] Furthermore, considering the splicing of depth features, thermal imaging features and channel features in the channel dimension to obtain a spliced ​​multimodal feature map, that is, after the fusion results, it is also necessary to detect target vehicles of different scales. Therefore, it is necessary to use the self-attention module of the multimodal model to identify the target vehicle. The specific implementation is as follows: A multimodal feature map is constructed based on the fusion result; and the target vehicle contained in the multimodal feature map is identified based on a self-attention module of the multimodal model.

[0058] Based on this, a multimodal feature map is constructed based on the fusion result, and the multimodal feature map is input into the multi-scale feature pyramid. The target vehicle contained in the multimodal feature map is identified based on the self-attention module of the multimodal model. That is, the self-attention mechanism is applied to each layer of the multi-scale feature pyramid to enhance the focus of the multimodal model on important features and identify the target vehicle in the multimodal feature map.

[0059] In practical applications, the structure of the multimodal model is as follows Figure 2 As shown in the figure, the multimodal model includes three branches: RGB, thermal imaging, and depth map. Based on these three branches, the channel image, thermal imaging image, and depth image can be processed by parallel residual blocks respectively, and feature splicing can be performed. After splicing, they are input into the multi-scale feature pyramid and processed in combination with the convolution layer. Among them, the parallel residual blocks include 3×3 residual blocks, 5×5 residual blocks, 7×7 residual blocks, and 9×9 residual blocks.

[0060] Using the above example, the concatenated multimodal feature map is input into the multi-scale feature pyramid to construct 32×32, 16×16, and 8×8 feature maps to facilitate the detection of objects of different scales. A self-attention mechanism is applied to each layer of the feature pyramid to enhance the model's focus on important features. A self-attention module is used to extract contextual relationships to improve the accuracy of object detection. Finally, the feature map is input into the convolutional layer, and the bounding box coordinates and confidence score of the vehicle are output, and the target vehicle contained in the multimodal feature map is detected.

[0061] In summary, the multimodal feature map is input into the multi-scale feature pyramid, and the self-attention module based on the multimodal model is used to identify the target vehicle contained in the multimodal feature map, thereby improving the accuracy of target vehicle detection.

[0062] Step 106: Generate road information of the target road based on the target area and the target vehicle in the multimodal image.

[0063] Specifically, after the multimodal images are fused in the image channel dimension using the multimodal model and the target vehicle in the multimodal image is determined according to the fusion result, the road information of the target road can be generated based on the target area and the target vehicle in the multimodal image, wherein the target area in the multimodal image can be a detection area in the multimodal image, which is used to detect vehicle congestion in the target area. Road information refers to the road congestion obtained by detecting the target road, and the road information can be information indicating the road state of the target road, such as congestion, unobstructed traffic, or an accident.

[0064] Based on this, after the multimodal images are fused in the image channel dimension using the multimodal model and the target vehicle in the multimodal image is determined according to the fusion result, the target road is detected based on the target area and the target vehicle in the multimodal image to generate road information representing the road condition of the target road.

[0065] Furthermore, based on comparing the target vehicle with the target area in the multimodal image, the road information of the target road can be determined, which is specifically implemented as follows: Determine the attribute information of the target vehicle and delineate the target area in the multimodal image; determine the vehicle area of ​​the target vehicle based on the attribute information, and generate road information of the target road based on the positional relationship between the vehicle area and the target area.

[0066] Specifically, the attribute information of the target vehicle includes but is not limited to the location, speed, direction, size, ID and confidence of the vehicle. The target area can be determined based on the target road. Two virtual rectangles are set in the target road, where two opposite edges are set along the edge of the target road, and the virtual rectangles are parallel to the road surface of the target road. The virtual plane formed by the two virtual rectangles is the target area. The vehicle area is a rectangular frame representing the footprint of the target vehicle. The positional relationship between the vehicle area and the target area can be intersection, inclusion and no association.

[0067] Based on this, the target vehicle's location, speed, direction, size, ID, confidence and other attribute information are determined, and the target area is delineated in the multimodal image by drawing a virtual rectangle. The vehicle area of ​​the target vehicle is determined based on the attribute information, and the positional relationship between the vehicle area and the target area is determined. Based on the positional relationship between the vehicle area and the target area, the road information of the target road is generated, indicating the vehicle density in the target area of ​​the target road. The vehicle density can represent the congestion level of the target road.

[0068] Continuing with the above example, an ID is assigned to the identified target vehicle, and the vehicle's position, speed, direction, size and other status information are initialized. Based on the depth map, two virtual rectangles are set on the road surface, with two opposite edges set along the edge of the road, with a length of 0.5 meters, the rectangle is parallel to the road surface, and the height of the rectangle is 0.5 meters above the road surface. If the vehicle's bounding box intersects with the rectangle and its center point is within the rectangle, it is considered to have passed the virtual plane. The count is increased by one when the vehicle passes through the first rectangle, and the count is reduced by one when the vehicle passes through the second rectangle. The vehicle distribution density in each road section is calculated in real time. When the density is greater than the warning threshold of 0.0335 vehicles per square meter, the congestion cause detection process is triggered.

[0069] In summary, the road information of the target road is generated based on the positional relationship between the vehicle area and the target area, so as to realize the real-time detection of the target road.

[0070] Furthermore, considering that the road video collected for the target road contains multiple video frames, the position of the target vehicle in the next frame can be predicted by multi-frame comparison, and then the vehicle area of ​​the target vehicle can be determined. The specific implementation is as follows: Determine at least two to-be-matched vehicles included in the target vehicle, and determine a to-be-matched image associated with the road image in the road video; predict predicted vehicles corresponding to the at least two to-be-matched vehicles in the to-be-matched image based on the attribute information, and construct a vehicle matching pair according to the prediction results; and use the area corresponding to the detected vehicle in the vehicle matching pair as the vehicle area of ​​the target vehicle.

[0071] Specifically, the to-be-matched vehicles may be all vehicles contained in the road image, and the to-be-matched image may be the next frame image of the road image in the road video. The predicted vehicle refers to the predicted vehicle position obtained by predicting the position of the to-be-matched vehicle in the to-be-matched image based on the attribute information of the target vehicle. The vehicle matching pair refers to the position matching pair of the same vehicle at different times, which is composed of the to-be-matched vehicle in the road image and the predicted vehicle in the to-be-matched image. The detected vehicle in the vehicle matching pair refers to the vehicle in the to-be-matched image that corresponds to the to-be-matched vehicle.

[0072] Based on this, at least two to-be-matched vehicles included in the target vehicle are determined, and the next frame of the to-be-matched image of the associated road image is determined in the road video. Based on the position, speed, direction, size and other attribute information of the to-be-matched vehicles, the predicted vehicles corresponding to at least two to-be-matched vehicles are predicted in the to-be-matched image to obtain a prediction result. A vehicle matching pair consisting of the to-be-matched vehicle and the predicted vehicle is constructed according to the prediction result. The area corresponding to the detected vehicle in the vehicle matching pair is used as the vehicle area of ​​the target vehicle, that is, when the predicted vehicle corresponding to the to-be-matched vehicle is determined in the to-be-matched image, it is determined that the detected vehicle is included in the vehicle matching pair; when the predicted vehicle corresponding to the to-be-matched vehicle is not determined in the to-be-matched image, it is determined that the detected vehicle is not included in the vehicle matching pair.

[0073] Continuing with the above example, based on the position, speed, direction, size and other attribute information of at least two to-be-matched vehicles contained in the target vehicle, predict the position of the to-be-matched vehicle in the next frame and obtain the predicted target. Calculate the intersection-and-union ratio between all predicted targets and the bounding boxes detected in the current frame to form a cost matrix. Apply the Hungarian algorithm to solve the allocation problem and find the optimal matching pair. For successfully associated to-be-matched vehicles and predicted targets, use the Kalman filter to update the state of the target, including position and size. For new detection results that do not match any to-be-matched vehicles, initialize a new tracking target. For to-be-matched vehicles that have not been detected for a long time, mark them as "lost" and consider removing them from the tracking list later.

[0074] In summary, based on the position, speed, direction, size and other attribute information of the vehicle to be matched, at least two predicted vehicles corresponding to the vehicles to be matched are predicted in the image to be matched, so as to achieve vehicle tracking between adjacent frames and improve the accuracy of road information.

[0075] Furthermore, the vehicle density in the target road can affect the road information of the target road, which is specifically implemented as follows: The vehicle density of the target area is calculated based on the positional relationship between the vehicle area and the target area; and when the vehicle density is greater than a density threshold, the road information of the target road is generated.

[0076] Specifically, the vehicle density may represent the degree of congestion in a target area of ​​the target road. The density threshold may be set according to the area of ​​the target area.

[0077] Based on this, the number of vehicles contained in the target area is determined based on the positional relationship between the vehicle area and the target area, and the vehicle density of the target area can be calculated based on the area and number of vehicles in the target area; when the vehicle density is greater than the density threshold, the target road is determined to be congested and the road information of the target road is generated.

[0078] Using the above example, the vehicle density can be determined based on the ratio between the vehicle data in the target area and the area of ​​the target area (L target area length × W target area width). The density threshold is a hyperparameter. The average occupied area A of each vehicle is set to 15 square meters, the maximum acceptable vehicle density is 0.067, and the density threshold formula is D t =αD α , where D t represents the density threshold, D α represents the maximum acceptable vehicle density, α represents the adjustment coefficient, and is taken as 0.5, resulting in a density threshold of 0.0335. When the vehicle density is greater than the density threshold, it is determined that the target road is congested and needs to be cleared in time.

[0079] In summary, when the vehicle density is greater than the density threshold, the target road is determined to be congested, the road information of the target road is generated, and real-time congestion detection of the target road is achieved.

[0080] Furthermore, when it is determined based on the road information that the target road is congested, the road model can be used to analyze the cause of the congestion on the target road, which is specifically implemented as follows: The attribute information and the road image are input into a road model to obtain road detection information corresponding to the road information.

[0081] Based on this, the attribute information and road images are input into the road model to obtain the road detection information corresponding to the road information. The road model can be a large model of TrafficVision, which is based on ViLT and adds a spatial embedding layer to closely combine the position data with the visual features, and weights the multimodal features through the self-attention mechanism to improve the learning ability of complementary information. The road model adopts the Transformer architecture, and enhances the interactive learning between modalities through the multi-head self-attention mechanism to improve the cross-modal perception ability. Position encoding helps to understand the spatial position of the feature, and the feature is optimized through the feedforward neural network, and finally used for target detection or congestion cause analysis.

[0082] Continuing with the above example, the attribute information and road image are input into the road model, and the output content of the road model can be activated: ""Analysis results":{"Is congestion occurring":"Yes","Event type":"Vehicle rollover","Event location":{"Latitude":37.7749,"Longitude":122.4194},"Detected vehicle":{"Type":"Sedan","License plate number":"ABC123","Status":"Flip","Size":{"Width":1.8,"Length":4.5,"Height":1.6},"Thermal imaging features":{"Engine temperature":80,"Body temperature":25}},"Road conditions":{"Traffic density":0.15,"Lane closure":"Two lanes are blocked","Weather conditions":"Clear","Visibility":"High"},"Detection time":"2024-12-24T14:30:00Z"}".

[0083] A road detection method provided by an embodiment of this specification determines a road image in a road video collected for a target road, and constructs a multimodal image corresponding to the road image, thereby obtaining rich environmental information and improving the accuracy of subsequent vehicle detection. The multimodal image is fused in the image channel dimension using a multimodal model, and the target vehicle in the multimodal image is determined based on the fusion result; the road information of the target road is generated based on the target area and the target vehicle in the multimodal image. The multimodal model is used to detect and track the target vehicle in the multimodal image, thereby achieving efficient detection and tracking of the vehicle.

[0084] The following combination Figure 3 , taking the application of the road detection method provided in this specification in highway road condition monitoring as an example, the road detection method is further described. Figure 3 A processing flow chart of a road detection method for highway road condition monitoring provided in an embodiment of this specification is shown, which specifically includes the following steps: Step 302: Use a drone equipped with a binocular camera and a thermal imaging camera to collect a road video, and determine an initial thermal imaging image and an initial channel image corresponding to a target frame in the road video.

[0085] Use binocular cameras and thermal imaging cameras to obtain continuous video streams. Use camera calibration technology to correct binocular images and eliminate distortion caused by lens distortion.

[0086] Step 304: Preprocess the initial thermal imaging image and the initial channel image respectively to obtain a thermal imaging image and a channel image, and extract the thermal imaging feature points corresponding to the thermal imaging image in the road image and the channel feature points corresponding to the channel image based on a feature extraction algorithm.

[0087] Apply histogram equalization to enhance image contrast and details, making features more distinct. Use Gaussian filtering to remove noise from the image to improve the accuracy of subsequent processing. Use feature extraction algorithms to extract feature points in the binocular image.

[0088] Step 306: performing feature point matching on the thermal imaging feature points and the channel feature points to obtain at least one pair of matching feature points, and generating a disparity image based on the disparity information corresponding to the at least one pair of matching feature points, and generating a depth image based on the disparity image.

[0089] Match the feature points in the two images to identify the key points of the same object. Calculate the disparity through the matched feature points to generate a disparity map. Calculate the depth information in the scene based on the disparity map to generate a depth map.

[0090] Step 308: Generate a multimodal image based on the depth image, the thermal imaging image, and the channel image.

[0091] Align the depth map with the thermal image and RGB image so that each pixel position contains three-dimensional depth, temperature and color information to generate a multimodal image. In practical applications, it is necessary to ensure that the resolution of the three modal images is consistent (uniformly adjusted to 256×256). Align at the pixel level (match in space so that the same physical positions in the two images can correspond), calculate the projection matrix of each sensor by calibrating the parameters (intrinsic and extrinsic matrices, which are the parameters of the camera), and map the pixels of the depth map and thermal image to the RGB image coordinate system (complete alignment). Splice RGB (3 channels), thermal image (3 channels) and depth map (1 channel) to generate a multimodal image with a dimension of 256×256×7.

[0092] Step 310: Determine the depth module, thermal imaging module, and channel module included in the multimodal model.

[0093] Step 312: extracting depth features of the depth image using the depth module, extracting thermal imaging features of the thermal imaging image using the thermal imaging module, and extracting channel features of the channel image using the channel module.

[0094] Step 314: Fuse the depth features, thermal imaging features, and channel features in the channel dimension to obtain a multimodal feature map.

[0095] The main structure of the multimodal model is a three-branch structure, which is mainly composed of a feature extraction module and a detection module. The three-branch structure is used to process multimodal information. The three branches extract features from the depth map, thermal image, and RGB image respectively, and then splice the extracted features in the channel dimension. The feature extraction module is based on the residual block, and uses four residual blocks with different convolution kernel sizes to extract features in parallel to obtain a multimodal feature map.

[0096] Step 316: Identify the target vehicle contained in the multimodal feature map based on the self-attention module, determine the attribute information of the target vehicle, and delineate the target area in the multimodal image.

[0097] The spliced ​​multimodal feature map is input into the multi-scale feature pyramid to construct 32×32, 16×16, and 8×8 feature maps to facilitate the detection of objects of different scales. The self-attention mechanism is applied at each layer of the feature pyramid to enhance the model's focus on important features. A self-attention module is used to extract contextual relationships to improve the accuracy of object detection. Finally, the feature map is input into the convolution layer to output the bounding box coordinates and confidence score of the vehicle. A unique ID is assigned to the identified vehicle and the vehicle status information is initialized. Based on the depth map, two virtual rectangles are set on the road surface, two of which are set along the edge of the road opposite to each other, with a length of 0.5 meters, the rectangle is parallel to the road surface, and the height of the rectangle is 0.5 meters above the road surface.

[0098] Step 318: Determine the vehicle area of ​​the target vehicle based on the attribute information, and generate road information of the target road based on the positional relationship between the vehicle area and the target area.

[0099] Use the inter-frame association algorithm SORT to associate the same vehicle target in consecutive frames. Update the status of the tracked object in real time, including information such as position and direction. When the vehicle crosses the first rectangle, the count is increased by one, and when the vehicle crosses the second rectangle, the count is decreased by one. Calculate the vehicle distribution density in each road section in real time and set the warning threshold. When the vehicle density in a specific area exceeds the threshold, the congestion cause judgment process is triggered. Input multimodal data such as thermal imaging, binocular vision depth map, RGB image, location data, etc. into the TrafficVision+ large model, use the TrafficVision+ large model to process multimodal data, analyze vehicle mobility, and determine the dynamic change cause of congestion.

[0100] In summary, by combining binocular vision and thermal imaging technology, rich environmental information can be obtained in real time, and deep learning algorithms can be used for vehicle identification and positioning, so as to accurately determine the vehicle position. Compared with a single visual sensor, the fusion of multimodal data significantly improves the accuracy of target detection, especially in complex weather conditions such as night, rain and fog. Using a multimodal deep learning model for vehicle detection and combining it with the SORT algorithm for target tracking can achieve efficient and stable vehicle detection and tracking. By updating the target state in real time, the system can accurately identify and track the position and dynamic changes of each vehicle, ensuring the accuracy and continuity of vehicle detection. The TrafficVision+ large model is introduced, and by analyzing multimodal data, it can deeply understand the dynamic causes of traffic mobility changes and congestion. This large model not only processes based on visual information, but also takes into account the temporal changes of spatial position and traffic flow, improves the ability to analyze complex traffic conditions, and has higher adaptability and real-time response capabilities.

[0101] Corresponding to the above method embodiment, this specification also provides a road detection device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a road detection device provided in an embodiment of this specification. Figure 4 As shown, the device comprises: A determination module 402 is configured to determine a road image in a road video collected for a target road, and construct a multimodal image corresponding to the road image; A fusion module 404 is configured to fuse the multimodal images in the image channel dimension using a multimodal model, and determine the target vehicle in the multimodal image according to the fusion result; The generating module 406 is configured to generate the road information of the target road based on the target area and the target vehicle in the multimodal image.

[0102] In an optional embodiment, the determining module 402 is further configured to: Using a drone equipped with a binocular image acquisition device and a thermal imaging image acquisition device to collect the road video; Determine a target frame in the road video, and determine an initial thermal imaging image and an initial channel image corresponding to the target frame in the road video; The initial thermal imaging image and the initial channel image are preprocessed respectively to obtain a thermal imaging image and a channel image, and the thermal imaging image and the channel image are used as the road image.

[0103] In an optional embodiment, the determining module 402 is further configured to: Extracting thermal imaging feature points corresponding to the thermal imaging image and channel feature points corresponding to the channel image in the road image based on a feature extraction algorithm; Performing feature point matching on the thermal imaging feature points and the channel feature points to obtain at least one pair of matching feature points; Generate a disparity image based on the disparity information respectively corresponding to the at least one pair of matching feature points, and generate a depth image based on the disparity image; The multimodal image is generated based on the depth image, the thermal imaging image, and the channel image.

[0104] In an optional embodiment, the determining module 402 is further configured to: Determining device parameters of the binocular image acquisition device and a disparity value of the disparity image, and calculating a depth value based on the disparity value; The disparity image is converted into the depth image based on the device parameters, the disparity value, and the depth value.

[0105] In an optional embodiment, the fusion module 404 is further configured to: Determining the depth module, thermal imaging module and channel module included in the multimodal model; Extracting depth features of the depth image using the depth module, extracting thermal imaging features of the thermal imaging image using the thermal imaging module, and extracting channel features of the channel image using the channel module; The depth feature, the thermal imaging feature and the channel feature are fused in the channel dimension to obtain a fusion result.

[0106] In an optional embodiment, the fusion module 404 is further configured to: Constructing a multimodal feature map based on the fusion result; The target vehicle included in the multimodal feature map is identified based on a self-attention module of the multimodal model.

[0107] In an optional embodiment, the generating module 406 is further configured to: Determining attribute information of the target vehicle and demarcating the target area in the multimodal image; A vehicle region of the target vehicle is determined based on the attribute information, and road information of the target road is generated based on a positional relationship between the vehicle region and the target region.

[0108] In an optional embodiment, the generating module 406 is further configured to: Determine at least two to-be-matched vehicles included in the target vehicle, and determine in the road video an image to be matched that is associated with the road image; Predicting predicted vehicles corresponding to the at least two vehicles to be matched in the image to be matched based on the attribute information, and constructing a vehicle matching pair according to the prediction results; The area corresponding to the detected vehicle in the vehicle matching pair is used as the vehicle area of ​​the target vehicle.

[0109] In an optional embodiment, the generating module 406 is further configured to: Calculating a vehicle density of the target area based on a positional relationship between the vehicle area and the target area; When the vehicle density is greater than a density threshold, the road information of the target road is generated.

[0110] In an optional embodiment, the generating module 406 is further configured to: The attribute information and the road image are input into a road model to obtain road detection information corresponding to the road information.

[0111] A road detection device provided in an embodiment of the present specification determines a road image in a road video collected for a target road, and constructs a multimodal image corresponding to the road image, thereby obtaining rich environmental information and improving the accuracy of subsequent vehicle detection. The multimodal image is fused in the image channel dimension using a multimodal model, and the target vehicle in the multimodal image is determined based on the fusion result; the road information of the target road is generated based on the target area and the target vehicle in the multimodal image. The multimodal model is used to detect and track the target vehicle in the multimodal image, thereby achieving efficient detection and tracking of the vehicle.

[0112] The above is a schematic scheme of a road detection device of this embodiment. It should be noted that the technical scheme of the road detection device and the technical scheme of the road detection method described above are of the same concept, and the details not described in detail in the technical scheme of the road detection device can be referred to the description of the technical scheme of the road detection method described above.

[0113] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0114] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0115] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0116] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.

[0117] The processor 520 implements the steps of the road detection method when executing the computer program or instruction.

[0118] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above road detection method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above road detection method.

[0119] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program or instruction. When the computer program or instruction is executed by a processor, the steps of the road detection method described above are implemented.

[0120] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above road detection method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above road detection method.

[0121] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above road detection method when executed by a processor.

[0122] The above is a schematic solution of a computer program product of this embodiment. It should be noted that the technical solution of the computer program product and the technical solution of the above road detection method belong to the same concept, and the details not described in detail in the technical solution of the computer program product can be referred to the description of the technical solution of the above road detection method.

[0123] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] The computer program or instruction includes a computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0125] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0126] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A road detection method, characterized in that: include: Determine a road image in a road video collected for a target road, and construct a multimodal image corresponding to the road image; Using a multimodal model to fuse the multimodal images in an image channel dimension, and determining a target vehicle in the multimodal image according to a fusion result; The road information of the target road is generated based on the target area and the target vehicle in the multimodal image.

2. The road detection method according to claim 1, characterized in that: The step of determining a road image in a road video collected for a target road and constructing a multimodal image corresponding to the road image includes: Using a drone equipped with a binocular image acquisition device and a thermal imaging image acquisition device to collect the road video; Determine a target frame in the road video, and determine an initial thermal imaging image and an initial channel image corresponding to the target frame in the road video; The initial thermal imaging image and the initial channel image are preprocessed respectively to obtain a thermal imaging image and a channel image, and the thermal imaging image and the channel image are used as the road image.

3. The road detection method according to claim 2, characterized in that: The constructing a multimodal image corresponding to the road image includes: Extracting thermal imaging feature points corresponding to the thermal imaging image and channel feature points corresponding to the channel image in the road image based on a feature extraction algorithm; Performing feature point matching on the thermal imaging feature points and the channel feature points to obtain at least one pair of matching feature points; Generate a disparity image based on the disparity information respectively corresponding to the at least one pair of matching feature points, and generate a depth image based on the disparity image; The multimodal image is generated based on the depth image, the thermal imaging image, and the channel image.

4. The road detection method according to claim 3, characterized in that: The generating a depth image based on the disparity image comprises: Determining device parameters of the binocular image acquisition device and a disparity value of the disparity image, and calculating a depth value based on the disparity value; The disparity image is converted into the depth image based on the device parameters, the disparity value, and the depth value.

5. The road detection method according to claim 3, characterized in that: The using a multimodal model to fuse the multimodal images in the image channel dimension includes: Determining the depth module, thermal imaging module and channel module included in the multimodal model; Extracting depth features of the depth image using the depth module, extracting thermal imaging features of the thermal imaging image using the thermal imaging module, and extracting channel features of the channel image using the channel module; The depth feature, the thermal imaging feature and the channel feature are fused in the channel dimension to obtain a fusion result.

6. The road detection method according to claim 1, characterized in that: Determining the target vehicle in the multimodal image according to the fusion result includes: Constructing a multimodal feature map based on the fusion result; The target vehicle included in the multimodal feature map is identified based on a self-attention module of the multimodal model.

7. The road detection method according to claim 1, characterized in that: The generating the road information of the target road based on the target area and the target vehicle in the multimodal image includes: Determining attribute information of the target vehicle and demarcating the target area in the multimodal image; A vehicle region of the target vehicle is determined based on the attribute information, and road information of the target road is generated based on a positional relationship between the vehicle region and the target region.

8. The road detection method according to claim 7, characterized in that: The determining the vehicle area of ​​the target vehicle based on the attribute information includes: Determine at least two to-be-matched vehicles included in the target vehicle, and determine in the road video an image to be matched that is associated with the road image; Predicting predicted vehicles corresponding to the at least two vehicles to be matched in the image to be matched based on the attribute information, and constructing a vehicle matching pair according to the prediction results; The area corresponding to the detected vehicle in the vehicle matching pair is used as the vehicle area of ​​the target vehicle.

9. The road detection method according to claim 7, characterized in that: The generating the road information of the target road based on the positional relationship between the vehicle area and the target area includes: Calculating a vehicle density of the target area based on a positional relationship between the vehicle area and the target area; When the vehicle density is greater than a density threshold, the road information of the target road is generated.

10. The road detection method according to claim 7, characterized in that: After the road information of the target road is generated based on the target area in the multimodal image and the target vehicle, the method further includes: The attribute information and the road image are input into a road model to obtain road detection information corresponding to the road information.

11. A road detection device, characterized in that: include: A determination module is configured to determine a road image in a road video collected for a target road, and construct a multimodal image corresponding to the road image; A fusion module is configured to fuse the multimodal images in the image channel dimension using a multimodal model, and determine the target vehicle in the multimodal image according to the fusion result; A generating module is configured to generate road information of the target road based on the target area and the target vehicle in the multimodal image.

12. A computing device comprising a memory, a processor, and a computer program or instruction stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program or instructions, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing a computer program or instruction, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images