Adverse weather vehicle detection method based on multi-modal large model and related equipment

Abstract

This application provides a method and related equipment for vehicle detection in adverse weather conditions based on a multimodal large model, belonging to the field of intelligent driving technology. The method includes: acquiring forward video data collected by an onboard camera and extracting frame image information; inputting the frame image information into a fusion model, and using the fusion model to identify the forward vehicle in the frame image information according to a recognition strategy, obtaining feature image information labeled with the forward vehicle; inputting the feature image information into a multimodal model, and using the multimodal model to analyze the feature image information to obtain environmental information and a first warning message, and adjusting the recognition strategy of the fusion model according to the weather information in the environmental information; determining the feature distance between the current vehicle and the forward vehicle based on the feature image information, and generating a second warning message when the feature distance is less than a preset distance; and outputting the first warning message and / or the second warning message. This application improves the accuracy of forward vehicle detection and driving safety under adverse weather conditions.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology for vehicles, and in particular to a method and related equipment for vehicle detection in adverse weather conditions based on a multimodal large model. Background Technology

[0002] With the increasing prevalence of automobiles, numerous traffic safety hazards have arisen. Driving safety in adverse weather conditions is of paramount importance.

[0003] Currently, vehicles are generally equipped with detection models to detect whether there are vehicles in front that may collide. However, the models used generally use a combination of sliding windows, handcrafted features and classifiers to achieve detection. Models with this design are difficult to cope with inclement weather. Handcrafted features are sensitive to lighting, occlusion and deformation, and the number of windows increases exponentially with image resolution, which limits the accuracy of vehicle detection under inclement weather conditions and poses safety hazards.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose a method and related equipment for vehicle detection in adverse weather conditions based on a multimodal large model, aiming to improve the accuracy of vehicle detection and driving safety under adverse weather conditions.

[0006] To achieve the above objectives, one aspect of this application proposes a method for vehicle detection in adverse weather conditions based on a multimodal large model, the method comprising:

[0007] Acquire forward video data captured by the vehicle-mounted camera, and extract frame image information from the forward video data;

[0008] The frame image information is input into the fusion model, and the fusion model identifies the vehicle in front of the frame image information according to the recognition strategy to obtain feature image information labeled with the vehicle in front.

[0009] The feature image information is input into a multimodal model, and the feature image information is analyzed by the multimodal model to obtain environmental information and first prompt information. The recognition strategy of the fusion model is adjusted according to the weather information in the environmental information. The first prompt information is driving prompt information generated based on the environmental information.

[0010] Based on the feature image information, the feature distance between the current vehicle and the vehicle in front is determined, and in response to the feature distance being less than a preset distance, a second prompt message is generated;

[0011] Output the first prompt message and / or the second prompt message.

[0012] In some embodiments, adjusting the recognition strategy of the fusion model based on the weather information in the environmental information includes:

[0013] In response to the weather information being normal weather, the identification strategy is set to a performance strategy;

[0014] In response to the weather information indicating severe weather, the identification strategy is set to a precision strategy.

[0015] In some embodiments, the fusion model includes a first recognition model and a second recognition model, wherein the fusion model recognizes the frame image information according to a recognition strategy, including:

[0016] In response to the recognition strategy being the performance strategy, the first thread of the first recognition model is enabled, and the second thread of the second recognition model is temporarily disabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread.

[0017] In some embodiments, the fusion model includes a first recognition model and a second recognition model, wherein the fusion model recognizes the frame image information according to a recognition strategy, including:

[0018] In response to the recognition strategy being the accuracy strategy, the first thread of the first recognition model and the second thread of the second recognition model are enabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread;

[0019] A feature processing interval is determined, which is the cumulative time from the processing of the previous frame image by the second recognition model to the current time. In response to the feature processing interval being greater than or equal to a preset interval, the current processing frame in the first processing queue is copied and input into the second processing queue, and each frame image in the second processing queue is recognized by the second thread.

[0020] In some embodiments, the first recognition model is a YOLOv8 model, and the step of recognizing each frame of image in the first processing queue through the first thread includes:

[0021] Define the current frame to be processed in the first processing queue as the first target image, and perform image preprocessing on the first target image;

[0022] The preprocessed first target image is segmented into multiple grid images of preset sizes, vehicle features in the grid images are identified, and bounding boxes and corresponding confidence levels are determined.

[0023] The vehicle ahead is identified by removing redundant bounding boxes through confidence filtering and IOU filtering.

[0024] In some embodiments, the second recognition model is a Grounding Dino model, and the step of recognizing each frame of image in the second processing queue through the second thread includes:

[0025] The current frame to be processed in the second processing queue is defined as the second target image. The second target image is scaled by bilinear interpolation and then normalized.

[0026] Obtain a preset recognition text, and extract text features based on the preset recognition text, wherein the preset recognition text is used to describe the vehicle ahead of the target recognition;

[0027] The second target image and the text features are fused using a dual encoder and a cross-modal fusion module to determine the bounding box and corresponding confidence level.

[0028] The vehicle ahead is identified by removing redundant bounding boxes through confidence filtering and IOU filtering.

[0029] In some embodiments, determining the feature distance between the current vehicle and the vehicle ahead based on the feature image information includes:

[0030] Determine the target bounding box of the vehicle ahead in the feature image information;

[0031] The pixel width of the vehicle in front in the image is determined based on the target bounding box, and the feature distance is calculated based on the pixel width, the preset standard vehicle width, and the focal length of the vehicle-mounted camera.

[0032] To achieve the above objectives, another aspect of this application proposes a vehicle detection system for severe weather based on a multimodal large model, the system comprising:

[0033] The sensor module is used to acquire forward video data collected by the vehicle-mounted camera and extract frame image information from the forward video data;

[0034] The vehicle recognition module is used to input the frame image information into the fusion model, and use the fusion model to identify the vehicle in front in the frame image information according to the recognition strategy, thereby obtaining feature image information labeled with the vehicle in front.

[0035] An environment recognition module is used to input the feature image information into a multimodal model, analyze the feature image information through the multimodal model to obtain environmental information and first prompt information, and adjust the recognition strategy of the fusion model according to the weather information in the environmental information. The first prompt information is driving prompt information generated based on the environmental information.

[0036] The distance recognition module is used to determine the feature distance between the current vehicle and the vehicle in front based on the feature image information, and generate a second prompt message in response to the feature distance being less than a preset distance;

[0037] The prompting module is used to output the first prompting information and / or the second prompting information.

[0038] To achieve the above objectives, another aspect of this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0039] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0040] The embodiments of this application include at least the following beneficial effects: This application provides a method and related equipment for vehicle detection in adverse weather conditions based on a multimodal large model. This solution acquires forward video data collected by an onboard camera, extracts frame image information from the forward video data, inputs the frame image information into a fusion model, and uses the fusion model to identify the forward vehicle in the frame image information according to a recognition strategy, outputting feature image information annotated with the forward vehicle; based on the feature image information, on the one hand, the feature image information is input into a multimodal model, the multimodal model analyzes the environmental information therein, and generates a first prompt information for driving prompts based on the environmental information. The weather information in the environmental information is also used to determine the recognition strategy of the fusion model, so that the recognition strategy of the fusion model is adapted to the weather conditions; on the other hand, the feature distance between the current vehicle and the vehicle in front is determined based on the feature image information, and it is judged whether the feature distance is less than a preset distance. If it is determined, a second prompt information is generated; the first prompt information about environmental prompts and / or the second prompt information about the distance to the vehicle in front are output to remind the driver to drive safely. Compared with current detection models, this application sets up a fusion model and a multimodal model to respectively undertake the tasks of detecting the vehicle in front and detecting the environment. The recognition strategy of the fusion model can be adjusted according to weather information, so that the recognition of the vehicle in front can minimize the impact of adverse weather conditions and improve detection accuracy. At the same time, the prompts generated based on the distance to the vehicle in front and the prompts based on environmental information provided by the multimodal model can combine to provide multi-dimensional safety reminders for driving, thereby improving driving safety. Attached Figure Description

[0041] Figure 1 This is a flowchart of a vehicle detection method based on a multimodal large model in adverse weather conditions, provided in an embodiment of this application.

[0042] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0043] Figure 3 yes Figure 1 Another flowchart of step S102 in the process;

[0044] Figure 4 This is a schematic diagram illustrating the principle of calculating feature distance in an embodiment of this application;

[0045] Figure 5 This is a schematic diagram of the structure of the severe weather vehicle detection system based on a multimodal large model provided in this application embodiment;

[0046] Figure 6 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0048] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0049] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0051] With the increasing prevalence of automobiles, numerous traffic safety hazards have arisen. Driving safety in adverse weather conditions is of paramount importance.

[0052] Currently, vehicles are generally equipped with detection models to detect whether there are vehicles in front that may collide. However, the models used generally use a combination of sliding windows, handcrafted features and classifiers to achieve detection. Models with this design are difficult to cope with inclement weather. Handcrafted features are sensitive to lighting, occlusion and deformation, and the number of windows increases exponentially with image resolution, which limits the accuracy of vehicle detection under inclement weather conditions and poses safety hazards.

[0053] In view of this, this application provides a method and related equipment for vehicle detection in adverse weather conditions based on a multimodal large model. This method acquires forward video data collected by an onboard camera, extracts frame image information from the forward video data, inputs the frame image information into a fusion model, and uses the fusion model to identify the forward vehicle in the frame image information according to a recognition strategy, outputting feature image information annotated with the forward vehicle. Based on the feature image information, on the one hand, the feature image information is input into a multimodal model, which analyzes the environmental information and generates a first warning message for driving guidance. The weather information in this environmental information is also used to determine the recognition strategy of the fusion model, making the recognition strategy of the fusion model adaptable to the weather conditions. On the other hand, the feature distance between the current vehicle and the vehicle in front is determined based on the feature image information, and it is judged whether the feature distance is less than a preset distance. If determined, a second warning message is generated. The first warning message regarding the environmental conditions and / or the second warning message regarding the distance to the vehicle in front are output to remind the driver to drive safely. Compared with current detection models, this application sets up a fusion model and a multimodal model to respectively undertake the tasks of detecting the vehicle in front and detecting the environment. The recognition strategy of the fusion model can be adjusted according to weather information, so that the recognition of the vehicle in front can minimize the impact of adverse weather conditions and improve detection accuracy. At the same time, the prompts generated based on the distance to the vehicle in front and the prompts based on environmental information provided by the multimodal model can combine to provide multi-dimensional safety reminders for driving, thereby improving driving safety.

[0054] It should be noted that in the various specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information (e.g., video data collected by vehicle cameras), user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the normal operation of the embodiments of this application obtained.

[0055] Figure 1 This is an optional flowchart of the severe weather vehicle detection method based on a multimodal large model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0056] Step S101: Obtain the front video data collected by the vehicle-mounted camera, and extract frame image information from the front video data;

[0057] Specifically, the method in this embodiment is applied to a vehicle in motion. The vehicle can be a private car, such as a sedan, SUV, MPV, or pickup truck. The vehicle can also be a commercial vehicle, such as a van, bus, small truck, or large trailer. The vehicle needs to have an electric motor capable of outputting power or acting as a generator to store mechanical energy. When the vehicle is a new energy vehicle, it can be a hybrid vehicle or a pure electric vehicle.

[0058] The vehicle implementing the method of this embodiment is defined as the current vehicle. By communicating with the intelligent driving system on the vehicle, the forward video data collected by the vehicle-mounted camera is obtained. The vehicle-mounted camera is a camera on the current vehicle specifically used to capture the road environment ahead. Therefore, the captured and recorded forward video data includes the situation in front of the lane in which the current vehicle is traveling, or in front of the adjacent lane, which helps to analyze the vehicle ahead and the driving environment in subsequent steps.

[0059] Based on the video data from the front, the video frames are extracted and their sets are defined as frame image information. Each frame image has a corresponding timestamp recorded when it was captured, so as to facilitate the analysis of the driving process in chronological order.

[0060] Step S102: Input the frame image information into the fusion model, and use the fusion model to identify the vehicle in front in the frame image information according to the recognition strategy to obtain feature image information marked with the vehicle in front.

[0061] Frame image information is input into a fusion model, which comprises at least two different types of models working together to cope with complex situations under severe weather conditions. This embodiment does not limit the specific type or method of collaboration of the collaborating models in the fusion model, but all of them must possess image analysis capabilities to analyze frame image information and identify vehicles ahead in the image.

[0062] In some embodiments, reference Figure 2 Step S102 may include, but is not limited to, steps S201 to S203:

[0063] Step S201: Input the frame image information into the fusion model;

[0064] In step S202, in response to the recognition strategy being a performance strategy, the first thread of the first recognition model is enabled, and the second thread of the second recognition model is temporarily disabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread.

[0065] Step S203: Obtain feature image information marked with vehicles ahead.

[0066] In this embodiment, the fusion model includes a first recognition model and a second recognition model. Each of the first and second recognition models constructs its own processing thread, which is defined as the first thread and the second thread, respectively. The first thread and the second thread are controlled and invoked through a recognition strategy. Simultaneously, a first processing queue is set up as the main task queue for processing frame image information. Optionally, the first and second recognition models can be of different types based on different performance requirements, thereby facilitating the setting of appropriate recognition strategies to invoke the two models and achieving better model collaboration. For example, the first recognition model may have better real-time performance and faster detection speed than the second recognition model, while the second recognition model may be more suitable for handling complex detection tasks and has better general-purpose performance.

[0067] On the other hand, the recognition strategy includes a performance strategy, which is suitable for performance-priority scenarios. These scenarios have relatively low accuracy requirements for recognition, saving performance resources and matching normal weather conditions. Therefore, when the recognition strategy is set to a performance strategy, only the first thread of the first recognition model is enabled, and the second thread of the second recognition model is not enabled. The first recognition model is the one with better real-time performance. Frame image information is input into the first processing queue according to the timestamp of each frame. The first thread is called to complete the analysis of each frame image in the first processing queue. Each frame image in the processing queue is defined as the frame image. The first recognition model then identifies the vehicle in front of the image and outputs structured data including the recognition results. Further annotation tools such as OpenCV and PIL are used to annotate the image based on the structured data, thus obtaining the feature image information annotated with the vehicle in front.

[0068] By setting the performance strategy to the recognition strategy, the vehicle can complete the recognition task of the vehicle in front using only the first recognition model under normal weather conditions, reducing the system performance burden and saving performance resources.

[0069] In some embodiments, reference Figure 3 Step S102 may include, but is not limited to, steps S301 to S304:

[0070] Step S301: Input the frame image information into the fusion model;

[0071] Step S302: In response to the recognition strategy being a precision strategy, the first thread of the first recognition model and the second thread of the second recognition model are enabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread;

[0072] Step S303: Determine the feature processing interval, which is the cumulative time from the second recognition model processing the previous frame image to the current time. In response to the feature processing interval being greater than or equal to the preset interval, copy the current processing frame in the first processing queue into the second processing queue, and recognize each frame image in the second processing queue through the second thread.

[0073] Step S304: Obtain feature image information marked with vehicles ahead.

[0074] In addition to the performance strategies described in the above embodiments, the recognition strategy may also include an accuracy strategy. The accuracy strategy is suitable for scenarios where accuracy is paramount. Compared to the performance strategy, performance resource utilization is a secondary requirement; the primary requirement is to improve detection accuracy. This is a recognition strategy suited for adverse weather conditions. Therefore, when the recognition strategy is an accuracy strategy, the first thread of the first recognition model and the second thread of the second recognition model are activated. The second recognition model is more suitable for handling complex detection tasks than the first recognition model.

[0075] On the other hand, to correspond to the processing threads of the two models, this embodiment also sets up a second processing queue, which corresponds to the second thread and serves as an auxiliary queue to the first processing queue. Specifically, the frame image information is input into the first processing queue in the order of the timestamps of each frame, and the first thread is called to complete the analysis of each frame image in the first processing queue, thereby recognizing the vehicle in front of the image through the first recognition model; at the same time, for the second thread, the time from the processing of the previous frame image by the second recognition model to the current moment is timed, and this time is defined as the feature processing interval. When the feature processing interval is greater than or equal to a preset interval, such as 1 second or 0.8 seconds, it is determined that the time left for the second recognition model to process the previous frame image is sufficient. At this time, the current processing frame in the first processing queue is copied into the second processing queue, and the second thread is called to make the second recognition model start processing the latest copied frame image input into the second processing queue.

[0076] The outputs of both the first and second recognition models are structured data including the recognition results. Since the first and second recognition models will also recognize the same frame of image separately in the above processing flow, the recognition results of the two models for the same frame of image can be compared first to determine whether the recognition results of the two models match, or if they do not match, the recognition result of the second recognition model can be selected first. The recognition results of the two models are then summarized, and then further annotated in the image according to the structured data using annotation tools such as OpenCV and PIL, that is, the feature image information with the vehicle in front is obtained.

[0077] In addition, a timer can be set for the model's recognition results to automatically clear the results after a preset timeout (e.g., 0.5 seconds or 1 second), ensuring that the latest recognition results are output. If the second processing queue is empty, the second thread can be set to perform a short sleep, with a sleep duration of 0.1 seconds or 0.2 seconds, etc., to reduce the CPU usage of the fusion model and further optimize the performance resource pressure of dual-model recognition.

[0078] By setting the performance strategy to an accuracy strategy, the vehicle can perform the task of identifying vehicles ahead even in adverse weather conditions, combining the first and second recognition models as a fusion model. This improves the detection accuracy of vehicles ahead under adverse weather conditions. Furthermore, through the collaboration of the first and second threads and the design of a preset interval, the second recognition model is invoked asynchronously and non-blockingly. This allows the second recognition model sufficient processing time for critical frames, ensuring its recognition performance, while appropriately reducing the performance resource pressure of simultaneously calling both models.

[0079] In some embodiments, the first recognition model is a YOLOv8 model, and step S202 or step S302 involves recognizing each frame of image in the first processing queue via a first thread, including:

[0080] Define the current frame to be processed in the first processing queue as the first target image, and perform image preprocessing on the first target image;

[0081] The preprocessed first target image is segmented into multiple grid images of preset size, vehicle features in the grid images are identified, and bounding boxes and corresponding confidence scores are determined.

[0082] Redundant bounding boxes are removed by confidence level filtering and IOU filtering to identify the vehicle ahead.

[0083] YOLO (You Only Look Once) is a target detection system based on a single neural network, and YOLOv8 is one version of this model that can improve the detection capability of small targets through multi-scale feature fusion.

[0084] For ease of explanation, the current frame that the YOLOv8 model acquires and processes from the first processing queue is defined as the first target image. Regarding the training process of the YOLOv8 model, it first preprocesses the image. The preprocessing steps, in sequence, include image loading, image resizing to match the processing format, channel normalization to normalize image pixel values, format conversion to match the neural network processing format, and optional data augmentation to improve the diversity of training samples. Next, the preprocessed image is segmented into multiple grid images of preset sizes. The YOLOv8 model detects the interior or center point range of each grid image to determine the presence of vehicle features. If present, it determines the bounding box surrounding the vehicle and sets the corresponding confidence level. Then, non-maximum suppression post-processing is performed, followed by confidence level filtering, retaining only those with confidence levels greater than a certain threshold. Boundary boxes with a confidence threshold (e.g., 0.3 or 0.4) are used to reduce the number of candidate boundary boxes and simplify calculations. Next, IOU (Intersection over Union) filtering is performed to determine the overlapping areas of two boundary boxes in the horizontal and vertical directions, and the ratio of their intersection to union is calculated. This process can be achieved by comparing the pixel coordinates of the two boundary boxes to determine if they intersect, and then calculating their overlap length in the horizontal and vertical directions to calculate the intersection-union ratio. If the calculated ratio is greater than the IOU threshold (e.g., 0.5 or 0.6), the two boundary boxes are considered to overlap, and one of them needs to be removed. Through the confidence and IOU filtering steps, boundary boxes that may be repeatedly selected for the same vehicle ahead are removed, thus eliminating redundant boundary boxes and accurately identifying the vehicle ahead. The application of the YOLOv8 model to the first target image can be referenced in the above training process.

[0085] The YOLOv8 model is used to identify vehicles ahead in frame images, meeting the real-time requirements of the first recognition model and improving the user experience of vehicle recognition.

[0086] In some embodiments, the second recognition model is the Grounding Dino model, and step S303, which involves recognizing each frame of image in the second processing queue via a second thread, includes:

[0087] The current frame to be processed in the second processing queue is defined as the second target image. The second target image is scaled by bilinear interpolation and then normalized.

[0088] Obtain the preset recognition text, extract text features based on the preset recognition text, and use the preset recognition text to describe the vehicle in front of the target recognition;

[0089] The second target image and text features are fused using a dual encoder and a cross-modal fusion module, and the bounding box and corresponding confidence score are determined.

[0090] Redundant bounding boxes are removed by confidence level filtering and IOU filtering to identify the vehicle ahead.

[0091] The Grounding Dino model is a text-driven open-set object detection model that supports open text descriptions and is well-suited for complex detection tasks under adverse weather conditions.

[0092] For ease of explanation, the current frame that the Grounding Dino model retrieves and processes from the second processing queue is defined as the second target image. Regarding the training process of the Grounding Dino model, firstly, the Grounding Dino model uses bilinear interpolation to uniformly scale the image to 800. 800 pixels, then perform pixel normalization, refer to the following formula (1):

[0093] , =(0.485,0.456,0.406), =(0.229,0.224,0.225)(1)

[0094] Where I is the input image tensor with size [3] 800 800], This is the mean of the RGB three channels on ImageNet, used to shift pixel values ​​to the zero center; This is the standard deviation of the RGB three channels on ImageNet, used to scale pixel values ​​to unit variance.

[0095] Mapping the original pixels from 0 to 255 to the range of [-2,2] reduces the covariate shift caused by different lighting conditions.

[0096] On the other hand, a preset recognition text is obtained, which is the text describing the vehicle in front that needs to be recognized to drive the Grounding Dino model, such as "red car in front". The tokens are split into token sequences of less than 256 characters according to the WordPiece dictionary by a word segmenter, as shown in equation (2):

[0097] (2)

[0098] in, A token sequence; Let be the integer index of the i-th token; L be the actual length of the sequence, with a maximum of 256; and V be the WordPiece vocabulary, containing 50,525 tokens. Then, by feeding this vocabulary into the BERT-base Chinese pre-trained model, the text feature tensor is obtained. Where v is the hidden dimension of BERT-base, i.e., the vector length of each token; thus, the text features are obtained.

[0099] Subsequently, a dual encoder was introduced, consisting of an image encoder and a text encoder.

[0100] The image encoder uses Swin-Transformer-Large (shift window transformer-large model) as the backbone network, and outputs a four-layer cascaded feature pyramid through four-stage progressive downsampling, as shown in the following equation (3):

[0101] (3)

[0102] in, These represent the number of channels in the feature maps from layer 1 to layer 4; the corresponding number of channels are 192, 384, 768, and 1536, respectively, and the corresponding spatial resolutions are 200×200, 100×100, 50×50, and 25×25, respectively, ensuring that targets of all sizes in the image have a suitable receptive field.

[0103] The text encoder uses the frozen BERT-base (bidirectional encoder representation model) to perform average pooling on the entire text to obtain v-dimensional sentence-level embeddings, thus avoiding the destruction of language priors during subsequent training.

[0104] Then, a cross-modal fusion module is introduced, consisting of 6 layers, each executed sequentially:

[0105] Image self-attention to capture spatial context.

[0106] Cross-modal attention, refer to the following formula (4):

[0107] (4)

[0108] In this context, query Q comes from the image token, and the key K and value V come from the text token, thus achieving a focus from text to image.

[0109] A feedforward network is used to complete the nonlinear mapping; the fused multimodal features are used for language-guided query initialization, first through m learnable queries, as shown in the following equation (5):

[0110] (5)

[0111] As a search agent, it undergoes another cross-attention warm-up with text features, ensuring that each query determines which text segment to match at the beginning of training. Refer to the following formula (6):

[0112] (6)

[0113] Then, m queries enter the cross-modal decoder, which consists of 6 layers, and each layer is executed sequentially:

[0114] Query self-attention to eliminate redundant candidates; image cross-attention to align the query with the image space; text cross-attention to reconfirm the semantics of the text; FFN to output the updated query state.

[0115] After decoding, the output is then processed in parallel by two multilayer sensing heads. The first is the bounding box head, as shown in equation (7) below:

[0116] (7)

[0117] The second is the classification header, as shown in formula (8):

[0118] (8)

[0119] Pair the m predictions with a small number of true boxes, and introduce the Hungarian optimal matching algorithm to construct the matching cost matrix, as shown in equation (9):

[0120] (9)

[0121] in, The matching cost between the i-th prediction and the j-th true value; For comparative cross-entropy loss; This represents the L1 distance between the predicted bounding box and the ground truth bounding box. In a broad sense Loss. There is also an L1 loss penalty for the deviation between the center and the width and height of the bounding box. If the penalty box overlap is insufficient, the classification loss uses the contrastive cross-entropy, as shown in the following formula (10):

[0122] (10)

[0123] The overall training loss is referenced in the following formula (11):

[0124] (11)

[0125] In the inference phase, the sigmoid function is first used to convert the h groups of logits into confidence scores, as shown in the following formula (12):

[0126] (12)

[0127] Finally, the confidence filtering and IOU filtering described in the above embodiment are performed to remove redundant bounding boxes, accurately identifying the vehicle ahead. The application of the Grounding Dino model to the second target image can be referenced in the training process described above.

[0128] The Grounding Dino model is used to identify vehicles ahead in frame images, which meets the requirements of the second recognition model for complex detection tasks and improves the user experience of vehicles recognizing vehicles ahead.

[0129] Step S103: Input the feature image information into the multimodal model, analyze the feature image information through the multimodal model to obtain environmental information and first prompt information, and adjust the recognition strategy of the fusion model according to the weather information in the environmental information. The first prompt information is driving prompt information generated based on the environmental information.

[0130] Optionally, the multimodal model is used to further analyze road conditions and the environment based on the identified feature image information of the vehicle ahead.

[0131] In this embodiment, the multimodal model can be the Qwen model, a visual language model. The Qwen model supports OpenCV NumPy arrays and PIL image objects as input. For images in NumPy array format (usually BGR color space), it converts them to RGB color space to avoid analysis errors caused by color space differences. The Qwen model also defines a prompt output format, allowing the model to perform environmental analysis based on the encoded feature image information and pre-defined task descriptions, and output standardized environmental information. This includes, for example, standardized descriptions of weather, time, road environment recognition (identifying city streets and highways), road sign and obstacle recognition, and lane change detection. The weather information can be extracted and used to adjust the recognition strategy of the fusion model. In addition, the output of the multimodal model also includes a first prompt message. This first prompt message is a driving prompt message output in a fixed format when the model analyzes environmental information and determines that the current environment matches a preset environmental condition. For example, when the environmental information detects that a speed limit sign has been passed, the relevant speed limit requirement is prompted. Other environmental factors such as weather changes, time period changes, road environment changes, and the detection of vehicles changing lanes ahead can also be set with corresponding types of driving prompt messages as the first prompt message, so as to realize the reminder of driving safety from the environmental dimension.

[0132] In step S103 of some embodiments, adjusting the recognition strategy of the fusion model based on weather information in the environmental information includes:

[0133] In response to the weather information indicating normal weather, the recognition strategy is set to a performance strategy.

[0134] In response to severe weather information, the recognition strategy is set to a precision strategy.

[0135] For the two recognition strategies—performance-based and accuracy-based—the system is configured as follows: when multimodal model analysis determines the weather information to be normal, the recognition strategy is set to performance-based, causing the model to begin recognizing vehicles ahead in the next processing cycle. Conversely, when multimodal model analysis determines the weather information to be severe, the recognition strategy is set to accuracy-based, causing the model to begin recognizing vehicles ahead in the next processing cycle. This enables the model used for vehicle recognition to intelligently adjust its recognition strategy according to weather changes, thereby improving recognition performance under adverse weather conditions and enhancing driving safety.

[0136] Step S104: Determine the feature distance between the current vehicle and the vehicle in front based on the feature image information; in response to the feature distance being less than a preset distance, generate a second prompt message.

[0137] In addition to the above-mentioned prompts based on environmental information, prompts can also be made based on the identified vehicles ahead, specifically regarding the safe distance between the current vehicle and the vehicle in front. The feature distance between the current vehicle and the vehicle ahead is determined based on feature image information, and a preset distance is set as the safe distance. When the feature distance is detected to be less than the preset distance, a second prompt message is generated to remind the driver to pay attention to the vehicle ahead.

[0138] Furthermore, it should be noted that there is no fixed order of execution between steps S103 and S104; the two steps can be processed simultaneously. Step S105 also outputs the generated prompt information immediately after the first or second prompt information is generated, without waiting for steps S103 and S104 to complete.

[0139] In step S104 of some embodiments, determining the feature distance between the current vehicle and the vehicle ahead based on feature image information includes:

[0140] Determine the bounding box for the vehicle ahead in the feature image information;

[0141] The pixel width of the vehicle in front in the image is determined based on the target bounding box, and the feature distance is calculated based on the pixel width, the preset standard vehicle width, and the focal length of the vehicle camera.

[0142] Optionally, refer to Figure 4The characteristic distance between the current vehicle and the vehicle in front can be calculated by the principle of similar triangles. Assuming the width of the object is w, its position is equivalent to the object plane. The object is placed at a distance d from the camera, and the position of the camera is equivalent to the pinhole plane. The object is photographed by the camera, and the pixel width of the image is P. The image is formed on the imaging plane. The formula for the camera focal length f is as follows (13):

[0143] (13)

[0144] When the camera is moved closer to or further away from the object, the pixel width of the object in the captured image changes as follows: That is, the distance between the object and the camera can be calculated based on similar triangles. The following formula (14) is given:

[0145] (14)

[0146] Based on the above principle, by substituting the preset standard vehicle width (e.g., the average width of a family sedan is 1.8 meters), the camera focal length (e.g., 12 megapixels), and the target box width (equivalent to the imaging width of an object in an image), the feature distance can be calculated, thereby calculating the distance between the vehicle in front and the current vehicle, to support the detection of whether the vehicle in front and the current vehicle are at a safe distance, thus improving safety.

[0147] Step S105: Output the first prompt message and / or the second prompt message.

[0148] The first and / or second prompt information obtained through the above steps can be sent to the vehicle system and displayed as text or graphics on the vehicle's central control platform, or as voice prompts via microphone, thereby reminding the driver to pay attention to driving environment safety and safe distance from the vehicle in front.

[0149] Steps S101 to S105 as illustrated in the embodiments of this application involve setting a fusion model and a multimodal model to respectively undertake the tasks of detecting the vehicle in front and detecting the environment. The recognition strategy of the fusion model can be adjusted according to weather information, so that the recognition of the vehicle in front can minimize the impact of severe weather and improve detection accuracy. At the same time, the prompts generated based on the distance to the vehicle in front and the prompts based on environmental information provided by the multimodal model can be combined to provide multi-dimensional safety reminders for driving, thereby improving driving safety.

[0150] The following is a detailed description and explanation of the solutions in the embodiments of the present invention, using specific application examples:

[0151] This application provides a method for vehicle detection in adverse weather conditions based on a multimodal large model, which can be applied to vehicles in motion. The method involves acquiring forward video data collected by an onboard camera and extracting frame image information from the forward video data.

[0152] Frame image information is input into the fusion model. In response to a performance-based recognition strategy, the first thread of the first recognition model in the fusion model is activated, while the second thread of the second recognition model is temporarily disabled. Frame image information is input into the first processing queue frame by frame in chronological order, and each frame in the first processing queue is recognized by the first thread. In response to an accuracy-based recognition strategy, both the first and second threads are activated. Frame image information is input into the first processing queue frame by frame in chronological order, and each frame in the first processing queue is recognized by the first thread. Simultaneously, the feature processing interval for the second recognition model, accumulated time from processing the previous frame to the current moment, is determined. When the feature processing interval is greater than or equal to a preset interval, the current processed frame in the first processing queue is copied and input into the second processing queue, and each frame in the second processing queue is recognized by the second thread. Feature image information annotated with vehicles ahead is obtained through fusion model analysis.

[0153] Feature image information is input into a multimodal model, which analyzes the feature image information to obtain environmental information and a first prompt based on that environmental information. On the other hand, the recognition strategy of the fusion model is adjusted based on weather information within the environmental data.

[0154] The feature distance between the current vehicle and the identified vehicle ahead is determined based on the feature image information. When the feature distance is detected to be less than the preset distance, a second prompt message is generated.

[0155] Output the obtained first and / or second prompt information to remind the driver.

[0156] This application embodiment sets up a fusion model and a multimodal model to respectively undertake the tasks of detecting the vehicle in front and detecting the environment. The recognition strategy of the fusion model can be adjusted according to weather information, so that the recognition of the vehicle in front can minimize the impact of severe weather and improve detection accuracy. At the same time, the prompts generated based on the distance to the vehicle in front and the prompts based on environmental information provided by the multimodal model can combine to provide multi-dimensional safety reminders for driving, thereby improving driving safety.

[0157] Please see Figure 5 This application also provides a vehicle detection system for severe weather based on a multimodal large model, which can implement the above-mentioned method. The system includes:

[0158] The sensor module is used to acquire forward video data collected by the vehicle-mounted camera and extract frame image information from the forward video data;

[0159] The vehicle recognition module is used to input frame image information into the fusion model, and the fusion model identifies the vehicles in front in the frame image information according to the recognition strategy to obtain feature image information labeled with the vehicles in front.

[0160] The environment recognition module is used to input feature image information into the multimodal model, analyze the feature image information through the multimodal model to obtain environmental information and first prompt information, and adjust the recognition strategy of the fusion model according to the weather information in the environmental information. The first prompt information is driving prompt information generated based on the environmental information.

[0161] The distance recognition module is used to determine the feature distance between the current vehicle and the vehicle in front based on feature image information, and to generate a second prompt message in response to the feature distance being less than a preset distance.

[0162] The prompt module is used to output the first prompt message and / or the second prompt message.

[0163] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0164] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0165] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0166] Please see Figure 6 , Figure 6 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0167] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0168] The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the methods described in the embodiments of this application.

[0169] The input / output interface 603 is used to implement information input and output;

[0170] The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0171] Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604);

[0172] The processor 601, memory 602, input / output interface 603, and communication interface 604 are interconnected within the device via bus 605.

[0173] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0174] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0175] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0176] The vehicle detection method and related equipment based on a multimodal large model in adverse weather conditions provided in this application embodiment acquires forward video data collected by an onboard camera, extracts frame image information from the forward video data, inputs the frame image information into a fusion model, and uses the fusion model to identify the forward vehicle in the frame image information according to a recognition strategy, outputting feature image information annotated with the forward vehicle; based on the feature image information, on the one hand, the feature image information is input into the multimodal model, which analyzes the environmental information therein and generates a first prompt message for driving prompts based on the environmental information. The weather information in the environmental information is also used to determine the recognition strategy of the fusion model, so that the recognition strategy of the fusion model is adapted to the weather conditions; on the other hand, the feature distance between the current vehicle and the vehicle in front is determined based on the feature image information, and it is judged whether the feature distance is less than a preset distance. If it is determined, a second prompt message is generated; the first prompt message about environmental prompts and / or the second prompt message about the distance to the vehicle in front are output to remind the driver to drive safely. Compared with current detection models, this application sets up a fusion model and a multimodal model to respectively undertake the tasks of detecting the vehicle in front and detecting the environment. The recognition strategy of the fusion model can be adjusted according to weather information, so that the recognition of the vehicle in front can minimize the impact of adverse weather conditions and improve detection accuracy. At the same time, the prompts generated based on the distance to the vehicle in front and the prompts based on environmental information provided by the multimodal model can combine to provide multi-dimensional safety reminders for driving, thereby improving driving safety.

[0177] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0178] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0181] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0182] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0184] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0186] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for vehicle detection in adverse weather conditions based on a multimodal large model, characterized in that, The method includes the following steps: Acquire forward video data captured by the vehicle-mounted camera, and extract frame image information from the forward video data; The frame image information is input into the fusion model, and the fusion model identifies the vehicle in front in the frame image information according to the recognition strategy to obtain feature image information labeled with the vehicle in front; wherein, the fusion model includes a first recognition model and a second recognition model, the first recognition model is the YOLOv8 model, and the second recognition model is the Grounding Dino model; The feature image information is input into a multimodal model, and the feature image information is analyzed by the multimodal model to obtain environmental information and first prompt information. The recognition strategy of the fusion model is adjusted according to the weather information in the environmental information. The first prompt information is driving prompt information generated based on the environmental information. Based on the feature image information, the feature distance between the current vehicle and the vehicle in front is determined, and in response to the feature distance being less than a preset distance, a second prompt message is generated; Output the first prompt message and / or the second prompt message; The step of adjusting the recognition strategy of the fusion model based on the weather information in the environmental information includes: In response to the weather information being normal weather, the identification strategy is set to a performance strategy; In response to the weather information indicating severe weather, the identification strategy is set to a precision strategy. The fusion model identifies the frame image information according to the recognition strategy, including: In response to the recognition strategy being the performance strategy, the first thread of the first recognition model is enabled, and the second thread of the second recognition model is temporarily disabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread. In response to the recognition strategy being the accuracy strategy, the first thread of the first recognition model and the second thread of the second recognition model are enabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread; A feature processing interval is determined, which is the cumulative time from the processing of the previous frame image by the second recognition model to the current time. In response to the feature processing interval being greater than or equal to a preset interval, the current processing frame in the first processing queue is copied and input into the second processing queue, and each frame image in the second processing queue is recognized by the second thread.

2. The method according to claim 1, characterized in that, The step of identifying each frame of image in the first processing queue through the first thread includes: Define the current frame to be processed in the first processing queue as the first target image, and perform image preprocessing on the first target image; The preprocessed first target image is segmented into multiple grid images of preset sizes, vehicle features in the grid images are identified, and bounding boxes and corresponding confidence levels are determined. The vehicle ahead is identified by removing redundant bounding boxes through confidence filtering and IOU filtering.

3. The method according to claim 1, characterized in that, The step of identifying each frame of image in the second processing queue through the second thread includes: The current frame to be processed in the second processing queue is defined as the second target image. The second target image is scaled by bilinear interpolation and then normalized. Obtain a preset recognition text, and extract text features based on the preset recognition text. The preset recognition text is used to describe the vehicle ahead of the target recognition. The second target image and the text features are fused using a dual encoder and a cross-modal fusion module to determine the bounding box and corresponding confidence level. The vehicle ahead is identified by removing redundant bounding boxes through confidence filtering and IOU filtering.

4. The method according to claim 1, characterized in that, Determining the feature distance between the current vehicle and the vehicle ahead based on the feature image information includes: Determine the target bounding box of the vehicle ahead in the feature image information; The pixel width of the vehicle in front in the image is determined based on the target bounding box, and the feature distance is calculated based on the pixel width, the preset standard vehicle width, and the focal length of the vehicle-mounted camera.

5. A vehicle detection system for severe weather based on a multimodal large model, characterized in that, The system includes: The sensor module is used to acquire forward video data collected by the vehicle-mounted camera and extract frame image information from the forward video data; The vehicle recognition module is used to input the frame image information into the fusion model, and use the fusion model to identify the vehicle in front in the frame image information according to the recognition strategy, thereby obtaining feature image information labeled with the vehicle in front; wherein, the fusion model includes a first recognition model and a second recognition model, the first recognition model is a YOLOv8 model, and the second recognition model is a Grounding Dino model; An environment recognition module is used to input the feature image information into a multimodal model, analyze the feature image information through the multimodal model to obtain environmental information and first prompt information, and adjust the recognition strategy of the fusion model according to the weather information in the environmental information. The first prompt information is driving prompt information generated based on the environmental information. The distance recognition module is used to determine the feature distance between the current vehicle and the vehicle in front based on the feature image information, and generate a second prompt message in response to the feature distance being less than a preset distance; The prompting module is used to output the first prompting information and / or the second prompting information; The step of adjusting the recognition strategy of the fusion model based on the weather information in the environmental information includes: In response to the weather information being normal weather, the identification strategy is set to a performance strategy; In response to the weather information indicating severe weather, the identification strategy is set to a precision strategy. The fusion model identifies the frame image information according to the recognition strategy, including: In response to the recognition strategy being the performance strategy, the first thread of the first recognition model is enabled, and the second thread of the second recognition model is temporarily disabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread. In response to the recognition strategy being the accuracy strategy, the first thread of the first recognition model and the second thread of the second recognition model are enabled; the frame image information is input into the first processing queue frame by frame in chronological order, and each frame image in the first processing queue is recognized by the first thread; A feature processing interval is determined, which is the cumulative time from the processing of the previous frame image by the second recognition model to the current time. In response to the feature processing interval being greater than or equal to a preset interval, the current processing frame in the first processing queue is copied and input into the second processing queue, and each frame image in the second processing queue is recognized by the second thread.

6. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images