A method and system for detecting and describing autonomous driving risks

Abstract

The present invention discloses a method and system for detecting and describing risks in autonomous driving. The method uses a visual backbone network to obtain low-resolution video features and high-resolution multi-scale feature maps; uses a high- and low-resolution spatiotemporal fusion module to obtain multi-scale spatiotemporal fusion features at different resolutions; and uses a dual-branch abstract module to perform cross-modal learning on the fused features in the visual task domain and the text task domain, respectively, and obtains a target vector and abstract text features, respectively; further uses a multi-task decoding module to decode the target vector into a risk object bounding box, and decodes the abstract text features into a text description of the corresponding scene and risk prompt; the present invention effectively decouples the risk location and description tasks while maintaining object-level correlation, reducing computational complexity, and enhancing the processing capability of fine-grained objects and time information, thereby improving the model's understanding and generalization capabilities in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and autonomous driving technology, and in particular to a method and system for detecting and describing autonomous driving risks. Background Art

[0002] Risk detection aims to identify objects in driving scenarios that pose a threat to the driver, while video description tasks focus on using natural language to describe scenarios and risks and provide driving recommendations. These technologies are widely used in scene understanding and driving decision-making in the field of autonomous driving. Existing methods usually use a single low-resolution input to simultaneously process risk detection and scene description tasks, but there are two major problems: first, these methods can only handle scene description tasks and are difficult to extend to other closely related downstream tasks such as visual question answering and visual localization; second, in complex scenarios, they are insufficient in detecting small targets. Therefore, it is of great practical significance to fuse multi-scale spatiotemporal features of different resolutions and decouple risk detection and video description tasks to improve the accuracy and versatility of the model.

[0003] The key to risk detection and description is to enable the model to learn from both visual and textual modalities and balance the two tasks. Existing methods are mainly divided into three categories: integrating multi-task processing through an end-to-end framework, joint modeling based on natural language generation methods, and utilizing visual language pre-training models. However, there are significant differences in the modal feature requirements of risk detection and scene description. Existing methods usually fail to fully decouple the two tasks or directly use several independent models to complete the tasks. The direct coupling approach may cause the performance of the two to restrict each other, and independent models will prevent the model from learning knowledge in the two task domains. The detection task mainly relies on the extraction of spatial features and object positioning, while the description task focuses more on the integrity of temporal context and semantic information. Due to insufficient feature sharing, the detection task may be interfered with by the redundant information generated by the description, and the description task may generate incoherent natural language content due to the lack of accurate causal information modeling. This modal conflict is particularly prominent in complex scenarios, directly affecting the model's task collaboration performance and actual deployment effect. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the existing technology and provide an autonomous driving risk detection and description method and system. By capturing the characteristics of small and large targets in complex driving scenarios through multi-scale feature fusion, the accuracy of small object risk detection tasks can be enhanced. By decoupling the modal characteristics of the detection and description tasks, the feature sharing mechanism can be optimized, and the language coherence and interpretability of the scene description can be improved to solve the problems raised in the above background technology.

[0005] The technical solution provided by the present invention is as follows: an autonomous driving risk detection and description system, including the following functional modules:

[0006] Visual backbone network module; used to obtain low-resolution video features and high-resolution multi-scale feature maps respectively;

[0007] High- and low-resolution spatiotemporal fusion module; used to capture spatiotemporal features through 3D convolution, and use grouped multi-head attention to fuse multi-scale visual features to obtain features from different resolutions and bidirectional fusion of time and space;

[0008] The dual-branch abstraction module is used to perform cross-modal learning on the fused features in the visual task domain and the text task domain, and obtain the target vector and abstract text features respectively.

[0009] Multi-task decoding module: After obtaining the target vector and abstract text features, it is used to decode the target vector into a risk object bounding box and decode the abstract text features into a text description of the corresponding scene and risk prompt;

[0010] Calculation prediction module; used to calculate the intersection-over-union loss between the real coordinates and the predicted coordinates, and the cross entropy loss between the real text description and the predicted text description;

[0011] Backpropagation algorithm module; used to calculate gradients and update network parameters using stochastic gradient descent method.

[0012] A method for detecting and describing autonomous driving risks, comprising the following steps:

[0013] Step 1: Based on the acquired high-resolution vehicle video data set, a visual backbone network is constructed, which inputs a low-resolution video frame set and a high-resolution key frame, and outputs low-resolution video features and high-resolution multi-scale feature maps;

[0014] Preferably, building a visual backbone network specifically includes the following steps:

[0015] Step 1-1: Given a video Keep last frame As a high-resolution keyframe, c is the number of channels, w is the width, and h is the height;

[0016] Step 1-2: Downsample each frame in the video to obtain The visual features are extracted by pre-trained visual transformer (ViT) in a visual backbone network, and the Where P represents the block extracted by ViT, and D represents the hidden dimension;

[0017] Step 1-3: Concatenate the features along the time dimension to obtain the video feature V. For high-resolution images Use residual neural network for processing, extract the feature maps of the last three layers and mark them as I, Where l represents the depth of the residual neural network, I l Represents the output features of the lth layer residual neural network, c l , w l , h l Respectively represent the number of channels, width and height of the feature map of the lth layer;

[0018] Steps 1-4: Use three independent one-dimensional convolutional layers to map the obtained low-resolution video features and high-resolution multi-scale feature maps from the mapping channel to a unified dimension D, and finally output them as Among them, I' represents a set of multi-scale feature maps of the same dimension D, I' l Represents the output features of the l-th layer residual neural network mapped to D dimensions.

[0019] Step 2: Construct a high- and low-resolution spatiotemporal fusion module, input the obtained low-resolution video features and high-resolution multi-scale feature maps respectively, and output the features of different resolutions, temporal and spatial bidirectional fusion;

[0020] Preferably, the method specifically includes the following steps:

[0021] Step 2-1: Construct a high- and low-resolution spatiotemporal fusion module, input the low-resolution video features and the multi-scale high-resolution feature map into the high- and low-resolution spatiotemporal fusion module, and output the fused feature block;

[0022] Step 2-2: Video features Gridding is performed along the block direction to obtain the feature map V'. The block direction is to split the P dimension into w p ×h p dimension;

[0023] Step 2-3: Obtain the spatiotemporal representation H of the feature map through a 3D convolutional layer l , Among them, w p represents the width of the spatiotemporal representation of the feature map, h p Indicates the length of the spatiotemporal representation of the feature map;

[0024] Step 2-4: First, represent space-time Flatten and evenly split into three parts along the flattened dimension to form a query vector set The dimensions of the three vectors satisfy the following formula: w p×h p =P3+P4+P5, where P3, P4, and P5 represent the number of the 3rd, 4th, and 5th query vector blocks respectively;

[0025] Step 2-5: For each element in the two sets F and H, apply multi-head cross attention in pairs and concatenate the resulting features along the dimension when expanded to produce the final visual feature;

[0026] Step 2-6: The dimension is expanded into Broadcasting along the time dimension, we get Match the size of the video feature V and add it to the original feature map to get an intermediate result, which is then input into the constructed temporal cross attention module for feature interaction enhancement.

[0027] Step 3: Build a dual-branch abstraction module, input the fused features, and output the target vector and abstract text features respectively;

[0028] Preferably, it specifically includes:

[0029] Step 3-1: In the text branch, randomly initialize the learnable Token. As the query vector, where P t Indicates that Q in the learnable Token t For the key and value in the multi-head attention mechanism, the query vector and the input feature token are concatenated together as the key vector and value vector, and then multi-head cross attention is used to fuse and transform cross-modal information;

[0030] Step 3-2: In the visual branch, two sets of randomly initialized learnable tokens are used, denoted as Satisfy P vl +P vc =P v Among them, P vl Indicates that you can learn TokenQ vl The number of blocks, P vc Indicates that you can learn TokenQ vc The number of blocks, P v Represents the total length of the learnable token, by selecting the first P from the input token vl Prior knowledge of the Token injection location;

[0031] Step 3-3: After obtaining all the target features, use MLP to project the text tags to obtain Converted text feature vector exist and Perform matrix multiplication between them to calculate the similarity score M between Token and Token, which is recorded as Along P v Dimensions apply Softmax function, along P t Dimension summation, calculate the normalized score M' of each target feature, Finally, select from M' The object with the highest score as the final target vector.

[0032] Step 4: Construct a multi-task decoding module, input the target vector and abstract text features respectively, and output the risk object bounding box and the corresponding scene and risk warning text description; calculate the intersection-over-union loss between the real coordinates and the predicted coordinates, and the cross-entropy loss between the real text description and the predicted text description, use the backpropagation algorithm to calculate the gradient, and use the stochastic gradient descent method to update the network parameters;

[0033] Preferably, the method specifically includes the following steps:

[0034] Step 4-1: After inputting the target vector and abstract text features, use MLP to decode the target vector to predict the bounding box of the risk object, and use a lightweight language model to decode the abstract text features to generate a description;

[0035] Step 4-2: For the risk detection task, use the intersection-over-union loss function (IoULoss) to measure the degree of overlap between two regions and optimize the model;

[0036] Step 4-3: For the description task, optimize the model using the cross entropy loss function;

[0037] Step 4-4: Complete loss function Intersection-over-union loss With cross entropy loss The sum of is defined as: Where λ is a hyperparameter that balances the two losses;

[0038] Step 4-5: Use the backpropagation algorithm on the complete loss function to calculate the gradient of the parameters in the model and update the model parameters using the stochastic gradient descent algorithm.

[0039] Step 5: Repeat steps 1 to 4 until the model converges or reaches the maximum number of training steps, and then apply it to actual autonomous driving risk detection and description scenarios.

[0040] Compared with the existing technology, the beneficial effects achieved by the present invention are: by introducing a dual-branch abstraction module and a high- and low-resolution spatiotemporal fusion module, the present invention effectively decouples the risk localization and description tasks while maintaining object-level relevance, bridging the gap between the visual and textual fields, reducing computational complexity, and enhancing the processing capabilities of fine-grained objects and temporal information, thereby improving the model's understanding and generalization capabilities in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0042] In the attached figure:

[0043] Figure 1 This is a flowchart of the steps of the autonomous driving risk detection and description method provided by the present invention;

[0044] Figure 2 This is the overall framework diagram of the decoupled dual-branch multi-scale spatiotemporal fusion module provided by the present invention;

[0045] Figure 3 This is a diagram of the operating framework of the high-low resolution spatiotemporal fusion module (HLRST) provided by the present invention;

[0046] Figure 4 This is a diagram of the operating framework of the dual-branch abstract module (DBA) provided by the present invention;

[0047] Figure 5 This is a schematic diagram of the module composition of an autonomous driving risk detection and description system provided by the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The present invention combines Figures 1 to 5 By fusion of multi-scale features, the features of small and large objects in complex driving scenarios are captured, enhancing the accuracy of small object risk detection tasks. By decoupling the modal features of detection and description tasks, the feature sharing mechanism is optimized, improving the language coherence and interpretability of scene descriptions.

[0050] The specific technical solution provided by the present invention is as follows: a method for detecting and describing autonomous driving risks, comprising the following steps:

[0051] Step 1: Based on the acquired high-resolution vehicle video data set, a visual backbone network is constructed, which inputs a low-resolution video frame set and a high-resolution key frame, and outputs low-resolution video features and high-resolution multi-scale feature maps;

[0052] In this embodiment, building a visual backbone network specifically includes the following steps:

[0053] Step 1-1: Given a video Keep last frame As a high-resolution keyframe, c is the number of channels, w is the width, and h is the height;

[0054] Step 1-2: Downsample each frame in the video to obtain The visual features are extracted by pre-trained visual transformer (ViT) in a visual backbone network, and the Where P represents the number of patches extracted by ViT, and D represents the hidden dimension.

[0055] Step 1-3: Concatenate the features along the time dimension to obtain the video feature V. For high-resolution images Use the residual neural network (ResNet) for processing, extract the feature maps of the last three layers and mark them as I, Where l represents the depth of the residual neural network, I l Represents the output features of the lth layer residual neural network, c l , w l , h l Respectively represent the number of channels, width and height of the feature map of the lth layer;

[0056] Steps 1-4: Use three independent one-dimensional convolutional layers to map the obtained low-resolution video features and high-resolution multi-scale feature maps from the mapping channel to a unified dimension D, and finally output them as Among them, I' represents a set of multi-scale feature maps of the same dimension D, I' l Represents the output features of the lth layer residual neural network mapped to D dimensions;

[0057] Step 2: Construct a high- and low-resolution spatiotemporal fusion module, input the obtained low-resolution video features and high-resolution multi-scale feature maps respectively, and output the features of different resolutions, temporal and spatial bidirectional fusion;

[0058] In this embodiment, combined with Figure 3 , Figure 3 Schematic diagram of the high-low resolution spatiotemporal fusion module (HLRST). The specific method of fusing video features and multi-scale features using the constructed high-low resolution spatiotemporal fusion module includes the following steps:

[0059] Step 2-1: Construct a high- and low-resolution spatiotemporal fusion module, input the low-resolution video features and the multi-scale high-resolution feature map into the high- and low-resolution spatiotemporal fusion module, and output the fused feature block;

[0060] For example, for image features at two scales The feature maps of the steps 1-4 layers are converted from (D, w l-1 ,h l-1 ) upsampled to (D,w l ,h l ) matches the space size of the first layer, The dimensions become Same dimension;

[0061] Then the upsampled feature map is compared with I' l Connect and generate a fusion feature map based on the one-dimensional convolution layer Expressed as:

[0062]

[0063] in Represents the output of a one-dimensional convolutional layer, [;] represents a concatenation operation, and upsample represents a nearest neighbor upsampling operation.

[0064] Step 2-2: Video features Gridding is performed along the block direction to obtain the feature map V'. The block direction is to split the P dimension into w p ×h p dimension;

[0065] For example, the feature map is upsampled at the same time to align it with the grid size of the multi-scale feature map, and the upsampled feature map is recorded as V' l , Furthermore, two convolutional layers are used to aggregate spatiotemporal information. The first three-dimensional convolution kernel extracts a spatiotemporal feature. Determines how much information should be retained or discarded, and the second 3D convolution kernel extracts another spatiotemporal feature Injecting spatiotemporal information into multi-scale features, the lth spatiotemporal fusion module is F l , F l The output formula is as follows:

[0066]

[0067] in, Represents the spatiotemporal features extracted by the first three-dimensional convolution kernel, represents the spatiotemporal features extracted by the second 3D convolution kernel, F represents the output feature set of the spatiotemporal fusion module, ⊙ represents element-by-element multiplication, and σ(·) represents the Sigmoid function.

[0068] Step 2-3: Obtain the spatiotemporal representation H of the feature map through a 3D convolutional layer l , Among them, w p represents the width of the spatiotemporal representation of the feature map, h p Indicates the length of the spatiotemporal representation of the feature map;

[0069] Step 2-4: First, represent space-time Flatten and evenly split into three parts along the flattened dimension to form a query vector set The dimensions of the three vectors satisfy the following formula:

[0070] w p ×h p =P3+P4+P5,(6)

[0071] Among them, P3, P4, and P5 represent the number of the 3rd, 4th, and 5th query vector patches respectively.

[0072] Step 2-5: For each element in the two sets F and H, apply multi-head cross attention in pairs and concatenate the resulting features along the dimension when expanded to produce the final visual feature;

[0073] Exemplarily, the process is described as follows:

[0074]

[0075] Among them, C l represents the spatiotemporal fusion features output by the l-th multi-head self-attention module, Represents a multi-head attention module, C represents the complete spatiotemporal features after connecting the 3rd, 4th, and 5th multi-head self-attention results, and [;] represents a connection operation.

[0076] Step 2-6: The dimension is expanded into Broadcasting along the time dimension, we get Match the size of the video feature V and add it to the original feature map to get an intermediate result, which is then input into the constructed temporal cross attention module for feature interaction enhancement.

[0077] Step 3: Build a dual-branch abstraction module, input the fused features, and output the target vector and abstract text features respectively;

[0078] In this embodiment, combined with Figure 4 , Figure 4 This is a schematic diagram of the Dual Branch Abstraction Module (DBA), which specifically includes:

[0079] Step 3-1: In the text branch, first randomly initialize the learnable Token, As the query vector, where P t Indicates that Q in the learnable Token t For the key and value in the multi-head attention mechanism, the query vector and the input feature token are concatenated together as the key vector and value vector, and then multi-head cross attention is used to fuse and transform cross-modal information, which is expressed as follows:

[0080]

[0081] in, Represents the text feature vector after multi-head self-attention operation, represents a multi-head attention module, and [;] represents a connection operation.

[0082] Step 3-2: In the visual branch, two sets of randomly initialized learnable tokens are used, denoted as Satisfy P vl +P vc =P v Among them, P vl Indicates that you can learn TokenQ vl The number of blocks, P vc Indicates that you can learn TokenQ vc The number of blocks, P v Represents the total length of the learnable token, by selecting the first P from the input token vl Prior knowledge of the Token injection location;

[0083] For example, the abstract text features are transformed through a multi-layer perceptron (MLP) and pooling operations, and the content prior knowledge is injected into the learnable token by addition. For the key and value in the multi-head attention mechanism, the query vector and the input feature token are concatenated together as the key vector and value vector. The process is described as follows:

[0084]

[0085] Among them, Q' vlrepresents the learnable parameter used to retrieve position information in the vision branch, Q' vc represents the learnable parameters used to retrieve content information in the vision branch, represents all target feature proposals, represents the multi-head self-attention module, C vl Represents the first P in the input feature Token vl Token, Pooling(·) represents maximum pooling, and MLP(·) represents multi-layer perceptron (MLP).

[0086] Step 3-3: After obtaining all the target features, first use MLP to project the text markup to obtain Converted text feature vector Then, in and Perform matrix multiplication between them to calculate the similarity score M between Token and Token, which is recorded as Along P v Dimensions apply the Softmax function, and then along P t Dimension summation, calculate the normalized score M' of each target feature, Finally, select from M' The object with the highest score As the final target vector, the process is expressed as follows:

[0087]

[0088] in, is the transpose of the matrix, Sum(·) and Softmax(·) are the summation function and Softmax function respectively, the [:] symbol after the tensor indicates selecting values ​​from the given index along a specific dimension, and T(·) represents the Top-K function.

[0089] Step 4: Construct a multi-task decoding module, input the target vector and abstract text features respectively, and output the risk object bounding box and the corresponding scene and risk warning text description; calculate the intersection-over-union loss (IoULoss) between the real coordinates and the predicted coordinates, and the cross-entropy loss (Cross-EntropyLoss) between the real text description and the predicted text description, respectively, use the backpropagation algorithm to calculate the gradient and use the stochastic gradient descent method to update the network parameters;

[0090] In this embodiment, the following steps are specifically included:

[0091] Step 4-1: After inputting the target vector and abstract text features, use MLP to decode the target vector to predict the bounding box of the risk object, and use a lightweight language model to decode the abstract text features to generate a description. The formula is described as follows:

[0092]

[0093] in, and denote the predicted bounding box and word respectively, θ denotes the parameters of the language model, and N w Indicates the number of words in a sentence, N voc Indicates the number of words in the model vocabulary.

[0094] Step 4-2: For the risk detection task, the intersection-over-union loss function (IoULoss) is used to measure the degree of overlap between two regions and optimize the model. The specific expression is as follows:

[0095]

[0096] in, represents the ground-truth bounding box, Represents the predicted bounding box output by the MLP.

[0097] Step 4-3: For the description task, the cross-entropy loss function (Cross-EntropyLoss) is used to optimize the model. The formula is:

[0098]

[0099] in is a one-hot vector representing the i-th word in the true description, N w Indicates the number of words in a sentence. represents the transpose operation, Represents the probability distribution of predicted words.

[0100] Step 4-4: Complete loss function Intersection-over-union loss With cross entropy loss The sum of , which is defined as follows:

[0101]

[0102] where λ is a hyperparameter that balances the two losses.

[0103] Step 4-5: Use the backpropagation algorithm on the complete loss function to calculate the gradient of the parameters in the model, and use the stochastic gradient descent algorithm to update the model parameters.

[0104] Step 5: Repeat steps 1 to 4 until the model converges or reaches the maximum number of training steps, and then apply it to actual autonomous driving risk detection and description scenarios.

[0105] The specific technical solutions provided by the present invention also include: an autonomous driving risk detection and description system, which effectively decouples risk location and description tasks by introducing a dual-branch abstraction module and a high- and low-resolution spatiotemporal fusion module, while maintaining object-level relevance, bridging the gap between visual and textual fields, reducing computational complexity, and enhancing the ability to process fine-grained objects and time information, thereby improving the model's understanding and generalization capabilities in complex scenarios.

[0106] In this embodiment, combined Figure 5 , an autonomous driving risk detection and description system, including the following functional modules:

[0107] A visual backbone network module is used to obtain low-resolution video features and high-resolution multi-scale feature maps using the visual backbone network module;

[0108] High- and low-resolution spatiotemporal fusion module; used to capture spatiotemporal features through 3D convolution, and use grouped multi-head attention to fuse multi-scale visual features to obtain features from different resolutions and bidirectional fusion of time and space;

[0109] The dual-branch abstraction module is used to perform cross-modal learning on the fused features in the visual task domain and the text task domain, and obtain the target vector and abstract text features respectively.

[0110] Multi-task decoding module: After obtaining the target vector and abstract text features, it is used to decode the target vector into a risk object bounding box and decode the abstract text features into a text description of the corresponding scene risk prompt;

[0111] Calculation prediction module; used to calculate the intersection-over-union loss (IoU Loss) between the real coordinates and the predicted coordinates, and the cross-entropy loss (Cross-Entropy Loss) between the real text description and the predicted text description;

[0112] Backpropagation algorithm module; used to calculate gradients using the backpropagation algorithm and update network parameters using the stochastic gradient descent method.

[0113] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0114] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for detecting and describing risks in autonomous driving, characterized by: The following steps are involved: Step 1: Based on the acquired high-resolution vehicle video data set, a visual backbone network is constructed, which inputs a low-resolution video frame set and a high-resolution key frame, and outputs low-resolution video features and high-resolution multi-scale feature maps; Step 2: Construct a high- and low-resolution spatiotemporal fusion module, input the obtained low-resolution video features and high-resolution multi-scale feature maps respectively, and output the features of different resolutions, temporal and spatial bidirectional fusion; Step 3: Build a dual-branch abstraction module, input the fused features, and output the target vector and abstract text features respectively; Step 4: Construct a multi-task decoding module, input the target vector and abstract text features respectively, and output the risk object bounding box and the corresponding scene and risk warning text description; calculate the intersection-over-union loss between the real coordinates and the predicted coordinates, and the cross-entropy loss between the real text description and the predicted text description, use the backpropagation algorithm to calculate the gradient, and use the stochastic gradient descent method to update the network parameters; Step 5: Repeat steps 1 to 4 until the model converges or reaches the maximum number of training steps, and then apply it to actual autonomous driving risk detection and description scenarios; The construction of the visual backbone network specifically includes the following steps: Step 1-1: Given a video Keep last frame As a high-resolution keyframe, c is the number of channels, w is the width, and h is the height; Step 1-2: Downsample each frame in the video to obtain Through a pre-trained visual converter ViT in a visual backbone network, visual features are extracted and obtained Where P represents the block extracted by ViT, and D represents the hidden dimension; Step 1-3: Concatenate the features along the time dimension to obtain the video feature V. For high-resolution keyframes Use residual neural network for processing, extract the feature maps of the last three layers and mark them as I, Where l represents the depth of the residual neural network, I l Represents the output features of the lth layer residual neural network, c l , w l , h l Respectively represent the number of channels, width and height of the feature map of the lth layer; Steps 1-4: Use three independent one-dimensional convolutional layers to map the obtained low-resolution video features and high-resolution multi-scale feature maps from the mapping channel to a unified hidden dimension D, and finally output them as Among them, I' represents the multi-scale feature map set of hidden dimension D, I' l Represents the output features of the l-th layer residual neural network mapped to D dimensions.

2. The method for detecting and describing autonomous driving risks according to claim 1, wherein: The step 4 specifically includes the following steps: Step 4-1: After inputting the target vector and abstract text features, use MLP to decode the target vector to predict the bounding box of the risk object, and use a lightweight language model to decode the abstract text features to generate a description; Step 4-2: For the risk detection task, use the intersection-over-union loss function to measure the degree of overlap between two regions and optimize the model; Step 4-3: For the description task, optimize the model using the cross entropy loss function; Step 4-4: Complete loss function Intersection-over-union loss With cross entropy loss The sum of is defined as: Among them, λ is a hyperparameter that balances the two losses; Step 4-5: Use the backpropagation algorithm on the complete loss function to calculate the gradient of the parameters in the model and update the model parameters using the stochastic gradient descent algorithm.

3. The autonomous driving risk detection and description method according to claim 1 is applied to an autonomous driving risk detection and description system, characterized in that: The system includes the following functional modules: visual backbone network module, high- and low-resolution spatiotemporal fusion module, dual-branch abstraction module, multi-task decoding module, calculation prediction module, and back-propagation algorithm module; The visual backbone network module is used to obtain low-resolution video features and high-resolution multi-scale feature maps respectively; the high-resolution and low-resolution spatiotemporal fusion module is used to capture spatiotemporal features through three-dimensional convolution, and use grouped multi-head attention to fuse multi-scale visual features to obtain features of different resolutions and bidirectional fusion of time and space; The dual-branch abstraction module is used to perform cross-modal learning on the fused features in the visual task domain and the text task domain, respectively, to obtain a target vector and an abstract text feature. The multi-task decoding module is used to decode the target vector into a risk object bounding box after obtaining the target vector and the abstract text feature into a text description of the corresponding scene and risk prompt. The calculation prediction module is used to respectively calculate the intersection-over-union loss between the real coordinates and the predicted coordinates, and the cross entropy loss between the real text description and the predicted text description; the back propagation algorithm module is used to calculate the gradient and update the parameters of the network using the stochastic gradient descent method.

Images