High-speed emergency lane occupation alarm method based on multi-mode large model region generation

Abstract

The invention discloses a high-speed emergency lane occupation alarm method based on multi-mode large model area generation, and the method comprises the following steps: S10, building a multi-mode large model, building the multi-mode large model based on a Transform deep neural network, and carrying out the multi-mode large model; the multi-modal large model comprises a text information extraction module, a picture feature extraction module and a multi-modal feature fusion module; s20, collecting and processing picture data, collecting the picture data of the expressway, and manually labeling areas and describing picture information; s30, training a multi-modal large model; s40, emergency lane occupation warning is carried out; the method has the beneficial effects that the problems of high marking cost, low generalization performance, inflexibility and non-automation of emergency lane area generation in a high-speed scene are solved by using the multi-modal large model.

Description

Technical Field

[0001] The present invention relates to the technical field of emergency lane control, and more specifically, to a high-speed emergency lane occupancy warning method based on multi-modal large model area generation. Background Art

[0002] In recent years, with the development of social economy, the traffic volume of highways has continued to grow. During holidays, the phenomenon of slow traffic with many vehicles frequently occurs. To solve the actual highway traffic problems, it is necessary to adjust the traffic management control strategies and plans in time to adapt to the ever-changing traffic needs, so as to improve and enhance the operation effect of the highway system and reduce social and economic losses.

[0003] The existing method generally involves manually marking the location of the emergency lane and finding lane lines to divide the highway into lane areas. A vehicle recognition algorithm is then used to find the vehicle in the image. Finally, it is determined whether there is a vehicle in the emergency lane area to determine whether there is a violation and generate warning images and videos.

[0004] Manual area annotation has great limitations, mainly with two fatal shortcomings: 1) When the number of cameras reaches thousands or tens of thousands or more, the labor cost is extremely high; 2) It cannot support changes. When the dome camera (many on highways) or the camera position changes, the annotation becomes invalid and needs to be repeated, which has extremely low adaptability. Although the lane detection algorithm can generate areas through the algorithm without manual annotation, it also has two main problems: 1) The lane lines are often blocked during initialization and cannot be initialized; 2) The generalization performance of the model is uncontrollable, the lane line changes cannot be empty, and the scope of application is too small. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a high-speed emergency lane occupancy warning method based on multi-modal large model area generation.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a high-speed emergency lane occupancy warning method based on multi-modal large model area generation, the improvement of which is that the method includes the following steps:

[0007] Building a multimodal large model, building a multimodal large model based on the Transformer deep neural network, the multimodal large model includes a text information extraction module, an image feature extraction module and a multimodal feature fusion module;

[0008] Collection and processing of image data: Collect image data of highways and manually annotate areas and describe image information;

[0009] The training of the multimodal large model includes inputting the collected image data into the multimodal large model, wherein the image information described manually is input into the text information extraction module to extract text information; the image data of the manually annotated area is input into the image feature extraction module to extract image features; the text information extracted by the text information extraction module and the image features extracted by the image feature extraction module are both input into the multimodal feature fusion module for feature fusion to form fusion features; thereafter, the fusion features are upsampled, the fusion features are mapped back to the original image size, and the loss of each pixel is calculated to train the multimodal large model;

[0010] Emergency lane occupancy alarm is generated by establishing a vehicle recognition model and inputting continuous video decoded frames into the vehicle recognition model to perform vehicle recognition. The multimodal large model obtains decoded frames at set intervals and generates areas to update the emergency lane area. Whether a vehicle occupies the emergency lane is determined based on the vehicle recognition results and areas.

[0011] Furthermore, before the training of the multimodal large model, the following steps are also included:

[0012] The initial version of the base model is trained using mask data and descriptions from the open source dataset coco.

[0013] Furthermore, the training of the multimodal large model includes:

[0014] The loss function BCE LOSS is used to calculate the loss of each pixel.

[0015] Furthermore, the emergency lane occupancy warning further includes the following steps:

[0016] By determining the duration of occupying the emergency lane, possible abnormal points can be filtered out.

[0017] Furthermore, the emergency lane occupancy warning further includes the following steps:

[0018] When a vehicle occupies the emergency lane according to the vehicle recognition result and the area determination, an alarm is issued and a picture and video event are generated.

[0019] Furthermore, the vehicle recognition model adopts the open source model yolov8.

[0020] Furthermore, in the emergency lane occupancy warning, the interval time is 1s.

[0021] The beneficial effects of the present invention are as follows: the present invention proposes a high-speed emergency lane occupancy warning method based on multimodal large model area generation, and uses a multimodal large model to solve the problems of high labeling cost, low generalization performance, inflexibility, and non-automation in the generation of emergency lane areas in high-speed scenarios. In the high-speed monitoring scenario, the cost of labeling the emergency lane area is saved, and the multimodal large model can be used to achieve full automatic deployment without manual participation, and automatically identify and update the emergency lane area. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the high-speed emergency lane occupancy warning method based on multi-modal large model area generation of the present invention.

[0023] Figure 2 Schematic diagram of image data processing by the multimodal large model built for the present invention.

[0024] Figure 3 It is a schematic diagram of the process of emergency lane occupancy warning in the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0026] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0027] Reference Figure 1 As shown, the present invention discloses a high-speed emergency lane occupancy warning method based on multi-modal large model area generation. In this embodiment, the method includes steps S10-S40:

[0028] S10, building a multimodal large model, building a multimodal large model based on the Transformer deep neural network, the multimodal large model includes a text information extraction module, an image feature extraction module and a multimodal feature fusion module;

[0029] With the advent of the 2017-transformer architecture, computer vision and NLP have also ushered in the unification of architectures. Multimodal tasks use transformers more to improve the accuracy and effect of modal fusion. The introduction of the Transformer model is as follows: The Transformer model is a deep learning model that is mainly used for natural language processing (NLP) tasks. It introduces a self-attention mechanism to solve the gradient vanishing and gradient exploding problems encountered by traditional recurrent neural networks (RNNs) when processing long sequences. The core of the Transformer model is the self-attention mechanism, which allows the model to weight information at different positions when processing sequence data, so as to better understand the structure and meaning of the input sequence. The Transformer model contains two multi-head attention layers. The first one uses a masked operation to prevent the current word from knowing the subsequent information in advance during the translation process; the second one uses the encoder's encoding information matrix for calculation. In addition, the decoder of the Transformer model usually contains a self-attention layer, a feedforward layer, and an encoder-decoder attention layer, each with its own independent weights. In the calculation of the Transformer model, PE (Position Encoding) stands for position encoding, which is a way to add position information to the model because the model itself does not contain information to handle the order of the sequence. Position encoding helps the model understand the position of words in the sequence, thereby better understanding the context of the text. As for whether "C" produces a vector, it depends on the specific implementation and context, but in general, the "C" in the Transformer model may refer to the output of the encoder, which is usually a representation in the form of a vector. As for how to implement or understand the Transformer model in MATLAB, this usually involves the use of deep learning frameworks such as TensorFlow or PyTorch, as well as understanding the model architecture and training process. For the input of the decoder, it is usually the information passed from the encoder, which is used by the decoder to generate the output sequence.

[0030] Combination Figure 2 As shown, it is a schematic diagram of the multimodal large model constructed by the present invention to process image data. Among them, the text information extraction module is used to extract the input text information. In this embodiment, the input text information is "find the emergency lane area on the left and right sides"; the image feature extraction module is used to receive the input image data and extract the features of the image data; the specific contents of the text information extraction module, the image feature extraction module and the multimodal feature fusion module will be further explained below.

[0031] S20, collecting and processing image data, collecting image data of the expressway, and manually marking areas and describing image information;

[0032] In this embodiment, a large amount of highway image data is first collected, and the areas and description information are manually annotated.

[0033] Secondly, the mask data and description of the open source dataset coco are used to train the initial version of the base model. Fine-tune is then performed on the collected highway image data. In this embodiment, the amount of labeled data is 50,000, and the iterative training is 120 epochs. The fine-tune learning rate is set to 0.001, and it decreases by 10 times every 50 rounds. The trained model is used to predict the location of the area. It should also be noted that the model output is a map (mask map) showing whether each pixel is a target. The mask map needs to be converted into a polygonal area by fitting the corner points.

[0034] S30, training of multimodal large models, continue to refer to Figure 2 As shown, the collected image data is input into the multimodal large model, wherein the image information through manual description is input into the text information extraction module to extract text information; the image data through manual annotation area is input into the image feature extraction module to extract image features; the text information extracted by the text information extraction module and the image features extracted by the image feature extraction module are both input into the multimodal feature fusion module for feature fusion to form fusion features; thereafter, the fusion features are upsampled (i.e., upsampling operation) to map the fusion features back to the original image size, and the loss of each pixel is calculated to train the multimodal large model; in this embodiment, the loss function BCE LOSS is used to calculate the loss of each pixel.

[0035] In the above-mentioned embodiment, for the vehicle recognition model, the vehicle recognition model in the present invention adopts the open source model yolov8; by annotating the collected 5w high-speed picture bbox, and then fine-tuning a yolov8 model to predict the position of the vehicle in the picture.

[0036] S40, emergency lane occupancy alarm, by establishing a vehicle recognition model, inputting continuous video decoding frames into the vehicle recognition model to perform vehicle recognition; the multimodal large model obtains the decoded frames at a set interval and generates areas, and updates the emergency lane area; determines whether there is a vehicle occupying the emergency lane based on the vehicle recognition result and the area.

[0037] As a preferred embodiment, the emergency lane occupancy warning further includes the following steps:

[0038] By determining the duration of occupying the emergency lane, possible abnormal points can be filtered out.

[0039] Furthermore, the emergency lane occupancy warning further includes the following steps:

[0040] When a vehicle occupies the emergency lane according to the vehicle recognition result and the area determination, an alarm is issued and a picture and video event are generated.

[0041] In the above embodiments, the region generation referred to in the art generally refers to marking region points by technical or manual means to form a region for downstream task judgment, such as illegal intrusion, region counting, etc. Figure 3 As shown, the present invention provides a specific embodiment. In this embodiment, only the processing process of one picture is described in detail, and the processing steps are as follows:

[0042] S401, start;

[0043] S402, the camera obtains a picture;

[0044] S403, the vehicle recognition model recognizes the vehicle, and the multimodal large model obtains a frame at a set interval, generates an area, and updates the emergency lane area;

[0045] S404, judging whether there is a vehicle occupying the emergency lane based on the vehicle recognition result and the area, if so, proceed to the next step, otherwise return to step S402;

[0046] S405, determining whether the duration of the vehicle occupying the emergency vehicle meets the set time, and if so, issuing an alarm;

[0047] S406, end.

[0048] With respect to the image feature extraction module, text information extraction module and multimodal feature fusion module, the present invention provides a specific embodiment, wherein the text information extraction module includes MHA, A&D, FFD and A&D connected in sequence, wherein MHA is a multi-head attention mechanism, A&D is Add and normalization (Norm), and FFD is a feedforward neural network. The image feature extraction module includes MHA, A&D, FFD and A&D connected in sequence. The multimodal feature fusion module includes MHA, A&D, FFD and A&D connected in sequence.

[0049] Based on this, the present invention proposes a high-speed emergency lane occupancy warning method based on multimodal large model area generation. The multimodal large model is used. The model is multimodally trained on a large-scale open source data set to understand language and image information. By inputting to find the emergency lane, the lane can be divided and the area generated. Due to the generalization of the large model, the system of the present invention can be fully automatically deployed, does not require labeling, saves costs, has high generalization performance, and is flexible.

[0050] The present invention saves the cost of marking the emergency lane area in the high-speed monitoring scenario. By using a large multimodal model, it can realize full automatic deployment without manual participation, and automatically identify and update the emergency lane area. The main contributions are as follows: 1. Build and train a large multimodal model to generate the emergency lane area; 2. Based on the warning area combined with vehicle identification, form an alarm system for high-speed emergency lane occupancy; 3. In most high-speed scenarios, eliminate the annotation cost and improve flexible and automated deployment.

[0051] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A high-speed emergency lane occupancy warning method based on multi-modal large model area generation, characterized in that: The method comprises the following steps: Building a multimodal large model, building a multimodal large model based on the Transformer deep neural network, the multimodal large model includes a text information extraction module, an image feature extraction module and a multimodal feature fusion module; Collection and processing of image data: Collect image data of highways and manually annotate areas and describe image information; The training of the multimodal large model includes inputting the collected image data into the multimodal large model, wherein the image information described manually is input into the text information extraction module to extract text information; the image data of the manually annotated area is input into the image feature extraction module to extract image features; the text information extracted by the text information extraction module and the image features extracted by the image feature extraction module are both input into the multimodal feature fusion module for feature fusion to form fusion features; thereafter, the fusion features are upsampled, the fusion features are mapped back to the original image size, and the loss of each pixel is calculated to train the multimodal large model; Emergency lane occupancy alarm is generated by establishing a vehicle recognition model and inputting continuous video decoded frames into the vehicle recognition model to perform vehicle recognition. The multimodal large model obtains decoded frames at set intervals and generates areas to update the emergency lane area. Whether a vehicle occupies the emergency lane is determined based on the vehicle recognition results and areas.

2. The high-speed emergency lane occupancy warning method based on multi-modal large model area generation according to claim 1 is characterized in that: Before the training of the multimodal large model, the following steps are also included: The initial version of the base model is trained using mask data and descriptions from the open source dataset coco.

3. The high-speed emergency lane occupancy warning method based on multi-modal large model area generation according to claim 1 is characterized in that: The training of the multimodal large model includes: The loss function BCE LOSS is used to calculate the loss of each pixel.

4. The high-speed emergency lane occupancy warning method based on multi-modal large model area generation according to claim 1 is characterized in that: The emergency lane occupation warning also includes the following steps: By determining the duration of occupying the emergency lane, possible abnormal points can be filtered out.

5. The high-speed emergency lane occupation warning method based on multi-modal large model area generation according to claim 4 is characterized in that: The emergency lane occupation warning also includes the following steps: When a vehicle occupies the emergency lane according to the vehicle recognition result and the area determination, an alarm is issued and a picture and video event are generated.

6. The high-speed emergency lane occupancy warning method based on multi-modal large model area generation according to claim 1 is characterized in that: The vehicle recognition model adopts the open source model yolov8.

7. The high-speed emergency lane occupation warning method based on multi-modal large model area generation according to claim 1 is characterized in that: In the emergency lane occupancy warning, the interval time is 1s.

Images