Highway reconstruction and expansion traffic safety monitoring method and device based on multi-modal artificial intelligence model, electronic equipment and program product

Abstract

This invention discloses a method, device, electronic equipment, and program product for monitoring traffic safety during highway reconstruction and expansion based on a multimodal artificial intelligence model. The method includes: extracting video frames from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence and a timestamp for each video frame; inputting the video frame sequence and timestamps into a multimodal large-scale model to progressively compare adjacent video frames to obtain multiple inter-frame difference information; generating a video description in chronological order based on the multiple inter-frame difference information; inputting the video description into a large-scale language model for risk assessment to obtain a risk assessment result; and when the risk assessment result indicates the existence of a risk or accident, locating the corresponding video time period and associated key video frames from the highway video data based on the risk assessment result and the timestamps, and generating a risk event record. This highway reconstruction and expansion traffic safety monitoring method can accurately identify risks or accidents and display monitoring results.

Description

Technical Field

[0001] This invention relates to the field of highway transportation, and in particular to a method, device, electronic equipment, and program product for monitoring traffic safety during highway reconstruction and expansion based on a multimodal artificial intelligence model. Background Technology

[0002] With the popularization of video capture equipment and the development of artificial intelligence technology, using artificial intelligence models to analyze highway video data to help identify risks or accidents in highway traffic is gradually becoming a new option for highway traffic safety management.

[0003] However, real-world highway traffic scenarios are complex and diverse, and risk or accident monitoring results need to effectively serve the practical work of traffic safety management. This places higher demands on monitoring methods and the way monitoring results are applied. Existing monitoring schemes still have significant shortcomings in terms of the accuracy, timeliness, and effectiveness of risk or accident identification. These shortcomings are particularly pronounced in mixed scenarios such as road reconstruction and expansion.

[0004] Therefore, it is hoped that a more comprehensive highway traffic safety monitoring solution can be provided to improve the monitoring effect of risks or accidents and provide a basis for traffic management.

[0005] The background description is provided for the purpose of understanding the relevant technologies in this field and is not intended as an admission of prior art. Summary of the Invention

[0006] In this regard, the present invention aims to provide a method, device, electronic device and program product for monitoring traffic safety during highway reconstruction and expansion based on a multimodal artificial intelligence model. It can more accurately grasp the changes in highway traffic conditions over time and identify risks or accidents accordingly, and output and display the monitoring results in a form that is easy to review and retrospectively.

[0007] In a first aspect, embodiments of the present invention provide a method for monitoring traffic safety during highway reconstruction and expansion based on a multimodal artificial intelligence model, which may include:

[0008] Video frames are extracted from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence consisting of multiple extracted video frames and a time stamp of each video frame.

[0009] The video frame sequence, together with the time stamps of each video frame, is input into a multimodal large model. The multimodal large model then performs a stepwise comparison of the differences between adjacent video frames in the video frame sequence to obtain multiple inter-frame difference information. The inter-frame difference information represents the difference between adjacent video frames in at least one of the following: object state, object motion trajectory, and object posture.

[0010] Based on multiple inter-frame difference information, a video description representing changes in highway traffic conditions is generated in chronological order.

[0011] The video description is input into a large language model, and the risk assessment of the video description is performed by the large language model to obtain the risk assessment result;

[0012] If the risk assessment result indicates the existence of a risk or accident, based on the risk assessment result and each of the time markers, the video time period corresponding to the risk or accident and the key video frames associated with the risk or accident are located from the highway video data, and a risk event record corresponding to the risk or accident is generated. The risk event record includes: the risk assessment result, the video description associated with the risk or accident, the key video frames associated with the risk or accident, and the video time information representing the video time period.

[0013] The video description associated with the risk or incident, as well as the key video frames associated with the risk or incident, are visualized.

[0014] In some embodiments, the step of extracting video frames from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence consisting of multiple extracted video frames includes:

[0015] Video frames are extracted sequentially from the highway video data at predetermined intervals, and the video duration corresponding to each extracted video frame is accumulated to obtain the cumulative video duration.

[0016] When the cumulative video duration reaches the predetermined duration, the video frame sequence is composed of multiple extracted video frames, and the time stamp of each video frame is obtained.

[0017] In some embodiments, the method is continuously executed and includes:

[0018] Based on the risk level indicated by the risk assessment result, adjust at least one of the predetermined interval and the predetermined duration used for subsequent video frame extraction.

[0019] Wherein, when the risk level increases, the adjustment includes reducing the predetermined interval and / or shortening the predetermined duration; and / or, when multiple consecutive risk assessment results indicate that there is no risk or accident, the adjustment includes increasing the predetermined interval and / or extending the predetermined duration.

[0020] In some embodiments, the method further includes: obtaining the work area structure information of the road segment corresponding to the highway video data, wherein the work area structure information includes a description of the work area layout of the road segment;

[0021] The step of inputting the video description into a large language model and performing a risk assessment on the video description using the large language model to obtain a risk assessment result includes:

[0022] The video description, along with the work area structure information, is input into the large language model. The large language model then uses the work area structure information to perform a risk assessment on the video description, resulting in a risk assessment result related to work activities within the work area and / or intrusion behavior of non-work objects entering the work area.

[0023] In some embodiments, the step of progressively comparing the differences between adjacent video frames in the video frame sequence using the multimodal large model to obtain multiple inter-frame difference information includes:

[0024] Each pair of adjacent video frames in the video frame sequence is compared one by one, and a text-based difference description is output for each pair of adjacent frame frames as the inter-frame difference information corresponding to that pair of adjacent frame frames.

[0025] The step of generating a video description representing changes in highway traffic conditions in chronological order based on multiple inter-frame difference information includes:

[0026] Arrange the aforementioned differences into a difference description sequence according to the chronological order of their corresponding time markers;

[0027] The video description characterizing changes in highway traffic conditions is generated by inductively summarizing the difference description sequence.

[0028] In some embodiments, the method is continuously executed, and each input video frame sequence of the multimodal large model corresponds to an inference window;

[0029] The method includes:

[0030] Based on the video description corresponding to the prior inference window, previous state information is obtained, which includes the video description corresponding to the prior inference window and / or object state information extracted from the video description.

[0031] The previous state information, together with the video frame sequence corresponding to the current inference window and the time stamp of each video frame, are input into the multimodal large model. The multimodal large model obtains the window difference comparison across inference windows based on the monitoring state information.

[0032] Based on window difference comparisons, additional video descriptions of traffic situation changes across inference windows are generated.

[0033] In some embodiments, generating a risk event record corresponding to the risk or accident includes:

[0034] Based on the risk assessment results, a verification prompt message is generated;

[0035] The verification prompt information, along with the key video frame associated with the risk or accident, or the local area image of the key video frame corresponding to the risk or accident, are input into a multimodal large model for secondary verification to obtain the secondary verification result.

[0036] If the secondary verification result is consistent with the risk assessment result, the risk event record is generated;

[0037] If the secondary verification result is inconsistent with the risk assessment result, the video time period is extended and a new risk assessment is performed for the extended video time period.

[0038] In a second aspect, embodiments of the present invention provide a traffic safety monitoring device for highway reconstruction and expansion based on a multimodal artificial intelligence model, the device comprising:

[0039] The frame extraction unit is configured to extract video frames from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence consisting of multiple extracted video frames and a time stamp of each video frame.

[0040] The comparison unit is configured to input the video frame sequence together with the time stamps of each video frame into a multimodal large model, and to perform stepwise comparison of the differences between adjacent video frames in the video frame sequence through the multimodal large model to obtain multiple inter-frame difference information, wherein the inter-frame difference information represents the difference between adjacent video frames in at least one of object state, object motion trajectory and object posture.

[0041] The description generation unit is configured to generate a video description representing changes in road traffic conditions in chronological order based on multiple inter-frame difference information.

[0042] The risk assessment unit is configured to input the video description into a large language model, perform risk assessment on the video description through the large language model, and obtain a risk assessment result.

[0043] The generation unit is configured to, when the risk assessment result indicates the existence of a risk or accident, locate the video time period corresponding to the risk or accident and the key video frames associated with the risk or accident from the highway video data according to the risk assessment result and each of the time markers, and generate a risk event record corresponding to the risk or accident, wherein the risk event record includes: the risk assessment result, the video description associated with the risk or accident, the key video frames associated with the risk or accident, and video time information characterizing the video time period;

[0044] The display unit is configured to visually display video descriptions associated with the risk or incident, as well as key video frames associated with the risk or incident.

[0045] In a third aspect, embodiments of the present invention provide an electronic device, characterized in that it includes: a processor and a memory storing a computer program, the processor being configured to implement the method as described in the first aspect when executing the computer program.

[0046] In a fourth aspect, embodiments of the present invention provide a program product including a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0047] The method according to embodiments of the present invention extracts video frames at predetermined intervals from continuously acquired highway video data, and inputs the video frame sequence along with the time stamps of each video frame as a whole into a multimodal large model. The multimodal large model progressively compares adjacent video frames rather than identifying each frame independently, thereby reading changes in object state, object trajectory, and object posture that are only apparent in cross-frame situations. Based on this, a video description representing changes in highway traffic conditions is generated in chronological order. Since the video description represents the temporal evolution obtained through combined input and progressive comparison, the large language model can make risk judgments based on the evolution of the situation, rather than isolated moments in the scene. Furthermore, when the risk judgment result indicates the existence of risk or accident, the method of the present invention can backtrack and locate key video frames and video time periods, generating a risk event record that binds the risk judgment result, related video descriptions, key video frames, and video time information together. The located video descriptions and / or time periods and key video frames are then visualized, making the monitoring conclusions interconnected and intuitively presented with their semantic, visual, and temporal basis. The method of this invention overcomes the technical problems of known monitoring methods, which struggle to grasp the evolving traffic situation over time and make it difficult to verify and trace the monitoring results. The method of this invention can more accurately identify gradually evolving risks or accidents, and presents monitoring conclusions in conjunction with their visual and temporal evidence, facilitating timely on-site verification and post-event tracing. This allows monitoring personnel to directly locate and view the time period and images corresponding to risks or accidents without manual review, quickly complete on-site verification, and obtain evidence afterward, thereby significantly improving the accuracy of traffic safety monitoring during highway reconstruction and expansion, as well as the usability of the monitoring results.

[0048] In a further embodiment of the present invention, a text-based difference description is output for each pair of adjacent frames. These difference descriptions are arranged chronologically to form a difference description sequence, which is then used to generate a video description. This allows the multimodal large model to progressively compare the differences between adjacent frames and retain the results in readable text form at each level. Furthermore, the difference description sequence, video description, and risk assessment results form a progressively related and corresponding processing chain. Therefore, the method of this embodiment not only provides sufficient evidence for risk assessment but also provides progressively corresponding intermediate evidence for the review and retrospective analysis of the assessment process.

[0049] In a further embodiment of the present invention, the structural information of the corresponding road section's work area can also be obtained as a priori input to the large language model, so that in complex road sections where construction work and traffic are mixed, the work behavior within the work area, the intrusion behavior of non-work objects entering the work area, and the ordinary traffic behavior outside the work area can be treated differently, reducing the situation of misjudging normal work as risk.

[0050] In a further embodiment of the present invention, highway video data can come from multiple cameras deployed along the highway and perform monitoring separately. After the first camera detects a risk, a second downstream camera can be determined based on the movement direction of the risky object. The description representing the risky object is used as a priori continuation input, and the risk event records of the two cameras are correlated according to video time information to obtain a cross-camera event timeline. Thus, the method according to the embodiments of the present invention can reconstruct the movement trajectory and overall situation evolution of the risky object across the collection range of multiple cameras, avoiding information fragmentation caused by the limited field of view of a single camera, and ensuring the integrity of monitoring the risky object within the monitored road section.

[0051] Other optional features and technical effects of the embodiments of the present invention are partly described below and partly apparent from reading this document. Attached Figure Description

[0052] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the scale shown in the drawings, and the same or similar reference numerals in the drawings denote the same or similar elements, wherein:

[0053] Figure 1 A first exemplary flowchart of a traffic safety monitoring method for highway reconstruction and expansion based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0054] Figure 2 A second exemplary flowchart of a traffic safety monitoring method for highway reconstruction and expansion based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0055] Figure 3 A third exemplary flowchart of a traffic safety monitoring method for highway reconstruction and expansion based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0056] Figure 4 A fourth exemplary flowchart of a traffic safety monitoring method for highway reconstruction and expansion based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0057] Figure 5 A fifth exemplary flowchart of a traffic safety monitoring method for highway reconstruction and expansion based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown;

[0058] Figure 6 A schematic diagram of a monitoring interface for visually displaying video descriptions associated with risks or incidents and key video frames according to a specific embodiment of the present invention is shown.

[0059] Figure 7 An exemplary block diagram of a traffic safety monitoring device for highway reconstruction and expansion based on a multimodal artificial intelligence model according to an embodiment of the present invention is shown; and

[0060] Figure 8 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present invention is shown. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0062] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0063] In this embodiment of the invention, highway video data includes data obtained by video capture of highways and their surrounding areas. The highway is, for example, but not limited to, expressways, and the surrounding area is, for example, but not limited to, areas undergoing highway reconstruction and expansion. By way of explanation and not limitation, highway video data may include real-time video streams captured by surveillance cameras deployed along the highway, such as video streams transmitted using streaming media transmission protocols, or video streams that have undergone preprocessing or standardization. This embodiment of the invention does not limit the source or specific format of the highway video data.

[0064] In this embodiment of the invention, a video frame refers to a single image frame in video data, and a video frame sequence refers to a sequence of multiple video frames arranged chronologically. In this embodiment of the invention, a timestamp refers to information characterizing the temporal position of a video frame.

[0065] In this embodiment of the invention, a multimodal large model refers to a large-scale machine learning model that can simultaneously accept and understand multiple modal inputs such as images and text, and output the understanding results in the form of text, etc., such as, but not limited to, a Vision-Language Model (VLM). By way of explanation and not limitation, in this embodiment of the invention, both the multimodal large model and the large language model can be pre-trained models, which can be invoked through prompts without requiring retraining or fine-tuning for the traffic safety monitoring task of highway reconstruction and expansion; the multimodal large model and the large language model can be implemented by the same or different models, or by the same or different model instances or model routes, and this embodiment of the invention does not impose any restrictions on this.

[0066] As mentioned earlier, using artificial intelligence models to analyze highway video data to help identify risks or accidents has become a new approach to highway traffic safety management. However, there is currently a lack of dedicated AI models for highway traffic safety. Training a dedicated model from scratch faces unbearable costs, long development cycles, and unpredictable model performance leading to a trade-off between benefits. General-purpose AI models often fail to achieve good monitoring results in the face of complex and ever-changing highway traffic environments and high-speed traffic flow, often serving only as a basic auxiliary judgment tool and proving difficult to integrate into actual highway traffic risk or accident monitoring.

[0067] As an explanation, not a limitation, risks or accidents in road traffic are often not static, instantaneous events, but rather processes that unfold gradually over time. For example, in a rainy highway scenario, a vehicle on a slippery surface may first experience slight skidding, then lose control, eventually veering off course or even rolling over; similarly, before a rear-end collision occurs, the distance between the following vehicle and the vehicle in front is often rapidly reduced within seconds. The characteristic of these risks or accidents lies in the evolution of the situation over time, rather than the static content of a single moment, and this non-static content is difficult to accurately understand and identify using general-purpose large-scale models.

[0068] In response, some known monitoring solutions use image recognition models to identify monitored images, analyzing individual image frames to determine if any anomalies exist within them. However, this single-frame identification method, by explanation and not limitation, only represents the composition of the image at a single instant. The model cannot actually understand changes in the situation over a period of time, and therefore is prone to missing or delaying detections of traffic risks that unfold as a process (such as a vehicle gradually losing control or veering off its lane).

[0069] Furthermore, even when other known monitoring schemes detect anomalies, their output is often a general alarm message or a risk score. While such outputs may serve as simple accident alerts (not limiting the scope), they are unsuitable for rapid accident location and analysis, and are also inconvenient for downstream systems to review, retrieve, and trace back. Especially when post-incident evidence collection or tracing is required, alarm information from known monitoring schemes yields almost no usable analytical information or records. It fails to provide a clearly defined monitoring result unit that connects "when it happened, what happened, which footage it was based on, and which video segment it corresponds to," and allows for intuitive viewing. In reality, after receiving a risk alarm notification from a known monitoring scheme, it often requires rapid deployment of personnel for manual processing. This includes manually locating the accident in the surveillance video and conducting manual accident review, followed by manual recording of the incident details. This significantly reduces the efficiency of accident and risk handling. Currently, traffic management departments urgently need an improved, more comprehensive, and practically effective traffic safety monitoring scheme.

[0070] To address this issue, this invention provides a method, device, electronic device, and program product for monitoring traffic safety during highway reconstruction and expansion based on a multimodal artificial intelligence model, which effectively alleviates or overcomes the aforementioned problems. In this invention, a multimodal large model performs stepwise comparisons between adjacent frames of a video frame sequence extracted at predetermined intervals and combined with time stamps to obtain inter-frame difference information and generate a video description characterizing changes in highway traffic conditions. A large language model then performs risk assessment on this video description. If a risk or accident is determined, the risk assessment result triggers the location of the corresponding video time period and key video frames, generating a risk event record that binds the risk assessment result, relevant video description, key video frames, and video time information together. The video description and / or time period, as well as the key video frames, are then visualized, thereby significantly improving the effectiveness of traffic safety monitoring and the efficiency of on-site verification and post-event tracing.

[0071] The traffic safety monitoring method according to embodiments of the present invention can be used for, but is not limited to, traffic safety monitoring in highway reconstruction and expansion construction areas, traffic diversion areas, temporary work areas, and lane borrowing or road closure sections. It can also be used for monitoring highway traffic incidents in scenarios such as ordinary highways, urban expressways, bridges, interchanges, and toll stations, as well as for safety inspections during construction, on-site verification, and post-incident tracing, and other application scenarios requiring real-time perception and risk assessment of highway traffic incidents. The traffic safety monitoring method according to embodiments of the present invention can be executed by, but is not limited to, traffic safety monitoring devices deployed on the roadside, traffic safety monitoring systems, analysis components of video surveillance platforms, monitoring modules of traffic management cloud platforms, and servers associated with video surveillance systems.

[0072] The specific implementation methods of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific examples.

[0073] In some embodiments, reference Figure 1 The method for monitoring traffic safety during highway reconstruction and expansion based on a multimodal artificial intelligence model according to embodiments of the present invention may include the following steps S110, S120, S130, S140, S150, and S160. In some embodiments, at least some, or optionally all, of the above steps are continuously executed.

[0074] S110: Extract video frames from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence consisting of multiple extracted video frames and a time stamp for each video frame.

[0075] In some embodiments, highway video data can be continuously acquired. In some embodiments, highway scenes can be continuously captured by video acquisition devices, thereby generating video data that increases over time.

[0076] In some embodiments, the highway video data is video data collected under mixed traffic scenarios during highway reconstruction and expansion projects. In this embodiment, the mixed traffic scenario includes a scenario where construction vehicles, maintenance machinery, and other vehicles coexist. As an explanation, and not a limitation, highway reconstruction and expansion projects typically involve widening and upgrading existing highways while maintaining traffic flow. Construction vehicles, maintenance machinery, and other vehicles coexist on the same road segment, resulting in complex traffic organization and constantly changing situations. This places high demands on timely risk identification and post-event evidence collection, making the method of this invention particularly suitable for such scenarios.

[0077] In some embodiments, video frames can be extracted from highway video data at predetermined intervals. By way of explanation and not limitation, the extracted video frames can be used for subsequent comparative analysis, such as for difference comparison with adjacent frames. In the above embodiments, the method of extracting video frames at intervals effectively controls the computational load of difference comparison, avoiding a surge in computational overhead and latency.

[0078] In some embodiments, the predetermined interval includes, but is not limited to, an interval in the time dimension or an interval in the frame number dimension. In some embodiments, one frame may be extracted at predetermined intervals. In other embodiments, one frame may be extracted at predetermined number of frames. It is understood that the specific value of the predetermined interval can be pre-configured according to monitoring requirements and computing resource conditions, and the embodiments of the present invention do not limit this.

[0079] In some embodiments, the timestamp of each video frame can be used to characterize the temporal position of the video frame. In some embodiments, the timestamp includes the absolute timestamp of the corresponding video frame in the highway video data, such as, but not limited to, the acquisition time expressed as date and time. In other embodiments, the timestamp may also include a relative time offset relative to the start time of the video frame sequence, which falls within the protection scope of the embodiments of the present invention. As an explanation, by binding timestamps to each video frame, the embodiments of the present invention enable each video frame in the video frame sequence to carry not only the image content but also the time position corresponding to the image content, thereby providing a basis for subsequent time-series comparison and for reversing the location of video time periods based on the judgment results.

[0080] As previously stated, the above steps can be performed continuously. In some embodiments, such as during the continuous execution of step S110, the predetermined interval for extracting video frames and the predetermined duration for determining the video frame sequence can be dynamically adjusted.

[0081] In some embodiments, the scene type corresponding to the highway video data can be identified, and the predetermined interval and / or predetermined duration can be adjusted according to the scene type. In this embodiment, the scene type is, for example, but not limited to, at least one of high-speed driving scenes, low-speed driving scenes, and nighttime scenes. As an explanation and not a limitation, in scenarios where the overall vehicle speed is high, the situation of accidents and risks changes rapidly, and a smaller predetermined interval and / or predetermined duration can be used; conversely, in scenarios of low-speed driving or sparse traffic at night, a larger predetermined interval and / or predetermined duration can be used.

[0082] In other embodiments, the predetermined interval and / or predetermined duration may be adjusted based on subsequent risk assessment results. In some embodiments, the predetermined interval and / or predetermined duration may be adjusted based on the risk level indicated by the risk assessment results. In this embodiment, the risk assessment results and risk levels will be described in detail below in conjunction with step S140. In some embodiments, the predetermined interval and / or predetermined duration may be reduced if the risk level increases; and / or the predetermined interval and / or predetermined duration may be increased if multiple consecutive risk assessment results indicate that there is no risk or accident.

[0083] In some embodiments, a mapping table representing the correspondence between risk levels and predetermined intervals and / or predetermined durations can be maintained in advance. In this embodiment, the mapping table can be queried based on the current risk level to determine the adjusted predetermined interval and / or predetermined duration.

[0084] In some embodiments, reference Figure 2 Step S110 may include the following steps S210 and S220.

[0085] S210: Extract video frames from the highway video data at predetermined intervals, and accumulate the video duration corresponding to each extracted video frame to obtain the cumulative video duration.

[0086] As mentioned earlier, video frames can be extracted sequentially at predetermined intervals. In this embodiment, the total video duration covered by each extracted video frame can be continuously accumulated to obtain the total video duration, rather than not immediately feeding the extracted video frames into subsequent model inference.

[0087] S220: When the cumulative video duration reaches the predetermined duration, a video frame sequence is formed from the multiple extracted video frames, and the time stamp of each video frame is obtained.

[0088] In some embodiments, when the cumulative video duration covered by each extracted video frame reaches a predetermined duration, the extracted video frames can be combined into a single video frame sequence and entered into subsequent processing as a whole. By way of explanation and not limitation, by combining the predetermined interval and predetermined duration, it is possible to control the model invocation overhead while ensuring the time coverage required for timing judgments, thereby ensuring that the multimodal large model can understand the changes in the situation over a sufficient time span.

[0089] S120: Input the video frame sequence along with the time stamps of each video frame into the multimodal large model. The multimodal large model then performs a step-by-step comparison of the differences between adjacent video frames in the video frame sequence to obtain multiple inter-frame difference information.

[0090] In some embodiments, "combined input" means inputting multiple video frames from a video frame sequence, along with their respective timestamps, as a whole into the multimodal large model, rather than inputting each video frame independently. By way of explanation and not limitation, this allows the multimodal large model to simultaneously observe and analyze footage from multiple moments in the same inference process, incorporating temporal information, and to perform cross-frame comparisons.

[0091] In some embodiments, timestamps can be input into the multimodal large model as timestamps superimposed on the corresponding video frame. In one example, the time information of the frame is rendered at a predetermined position on the video frame. In other embodiments, timestamps can also be input into the multimodal large model as accompanying text associated with the corresponding video frame. In one example, the video frames are given sequentially in the input, with the time corresponding to each frame indicated by text.

[0092] In some embodiments, the step of inputting the video frame sequence along with the time stamps of each video frame into a multimodal large model may include: inputting the video frame sequence, the time stamps of each video frame, and preset prompt information into the multimodal large model. In this embodiment, the prompt information can be used to guide the multimodal large model to progressively compare the differences between adjacent video frames.

[0093] In some embodiments, the prompting information may include natural language prompts. In this embodiment, the natural language prompts are used, for example, to instruct the model to compare the position, motion, and pose changes of objects in adjacent frames sequentially over time and output the observed differences. As an explanation, and not a limitation, by inputting preset prompting information, a multimodal large model can be guided to perform verifiable and reproducible stepwise comparisons of adjacent video frames, thereby improving the stability and reproducibility of cross-frame comparisons.

[0094] In some embodiments, unlike independent content identification of each video frame, "stepwise comparison" refers to comparing the differences between adjacent video frames in a chronological sequence. Correspondingly, in some embodiments, the inter-frame difference information obtained from stepwise comparisons of adjacent video frames bound / associated with time markers may also be bound / associated with multiple, for example, two time markers. By way of explanation, and not limitation, this allows any change in situation recorded in the video description to be traced back to its corresponding time marker range after a video description is subsequently generated based on the inter-frame difference information, thereby greatly improving the efficiency of accident or risk location.

[0095] In some embodiments, the inter-frame difference information obtained by stepwise comparison can be used to characterize the differences between adjacent video frames in at least one of object state, object motion trajectory, and object pose.

[0096] In some embodiments, differences in object state include, but are not limited to, changes in the appearance, disappearance, shape, or condition of an object. In a specific example, a difference in object state is a guardrail changing from intact to damaged. In some embodiments, differences in object trajectory include, but are not limited to, changes in object position and direction of change. In a specific example, a difference in object trajectory is, for example, a vehicle deviating from its original lane. In some embodiments, differences in object posture include, but are not limited to, changes in object orientation, tilt, or flipping. In a specific example, a difference in object posture is, for example, a vehicle changing from a normal driving posture to a rollover posture. As an explanation and not a limitation, the present invention recognizes that known solutions cannot monitor events that unfold in a process-like manner, such as a vehicle gradually losing control or gradually deviating from its lane. Embodiments of the present invention specifically compare frame differences step-by-step to obtain inter-frame difference information, thereby enabling accurate monitoring of the aforementioned cross-frame differences.

[0097] S130: Generate a video description representing changes in highway traffic conditions in chronological order based on multiple inter-frame difference information.

[0098] In some embodiments, the video description includes a textual description that represents changes in road traffic conditions in chronological order.

[0099] In one specific embodiment, for a video frame sequence of a vehicle losing control on a flooded road, the video is described as follows: "The video shows a car driving on a highway in the rain, with severe flooding on the right side of the road ahead; a white sedan is coming from the left lane and trying to pass through the flooded area; after entering the flooded area, the car loses control, begins to skid and spin, gradually deviates from the normal lane, and finally rolls over and lands upside down on the road."

[0100] In some embodiments, the video description can be directly generated by the multimodal large model after the stepwise comparison is completed; that is, steps S120 and S130 can be completed successively in the same inference of the multimodal large model, and the embodiments of the present invention do not limit this.

[0101] In some embodiments, reference Figure 3 The steps described above, which involve progressively comparing the differences between adjacent video frames in a video frame sequence using a multimodal large model to obtain multiple inter-frame difference information, may include the following step S310.

[0102] S310: Compare each pair of adjacent video frames in the video frame sequence, and output a text-based description of the difference for each pair of adjacent frames as the inter-frame difference information corresponding to that pair of adjacent frames.

[0103] In some embodiments, every two temporally adjacent video frames in a video frame sequence can be considered as an adjacent frame pair.

[0104] In some embodiments, a multimodal large model can be used to output a text-based difference description for each adjacent frame pair. In this embodiment, the difference description may address changes in one or more of the object's position, velocity, or orientation between the adjacent frame pairs.

[0105] In some embodiments, the difference description can be generated in the format of "start time marker → end time marker: changed content". In one specific embodiment, for an adjacent frame pair "t1→t2" consisting of a video frame corresponding to time t1 and a video frame corresponding to time t2, the difference description is "t1→t2: the white car has shifted laterally to the left by about half a lane". However, it is understood that the above is only an example, and in other embodiments, the difference description may reasonably adopt other formats, which are not limited in this embodiment of the present invention.

[0106] In some embodiments, such as after step S310 described above, the step of generating a video description representing changes in road traffic conditions in chronological order based on multiple inter-frame difference information may include the following steps S320 and S330.

[0107] S320: Arrange the difference descriptions into a difference description sequence according to the order of their corresponding time stamps.

[0108] In some embodiments, the difference descriptions can be arranged chronologically according to their corresponding timestamps to form a sequence of difference descriptions that can serve as intermediate evidence. By way of explanation, and not limitation, this allows each pair of generated difference descriptions to be reconstructed into a complete, sequential temporal description over time. In optional embodiments, the sequence of difference descriptions can also be incorporated into the risk event record described later, which will be described in detail below in conjunction with step S150.

[0109] S330: Video descriptions characterizing changes in highway traffic conditions are generated by inductively summarizing differential description sequences.

[0110] In some embodiments, an artificial intelligence model can be used to summarize the obtained differential description sequence to generate a video description.

[0111] As an explanation, and not a limitation, the above embodiments of the present invention utilize a multimodal large model to progressively compare the differences between adjacent video frames, and then present the structured intermediate result of the difference description in text form. This achieves a processing chain from the original video frame sequence to pairwise difference descriptions, and finally to the summarized video description. The solution of the above embodiments of the present invention, by constraining the multimodal large model to form a processing chain according to a predetermined processing flow, fully leverages the multimodal large model's ability to understand video frames and text. In the above processing flow, the difference description, the difference description sequence, and the video description can all be quickly traced back to determine the corresponding time range and time frame via time stamps. This not only facilitates the use of the obtained video description to provide sufficient evidence for subsequent risk assessment, but also provides intermediate evidence corresponding to each level for the review of the assessment process.

[0112] As mentioned above, in some embodiments, the steps of the above-described method for monitoring traffic safety during highway reconstruction and expansion can be continuously executed. In this embodiment, each input video frame sequence into the multimodal large model corresponds to an inference window. In other words, as highway video data is continuously acquired, new video frame sequences are continuously constructed and input into the multimodal large model in the aforementioned manner. Each input and its corresponding inference constitute an inference window, and each inference window obtains a corresponding video description.

[0113] In some embodiments, the traffic safety monitoring method for highway reconstruction and expansion may further include a cross-inference window comparison step. Accordingly, in some embodiments, reference is made to... Figure 4 The method for monitoring traffic safety during highway reconstruction and expansion may further include the following steps S410, S420 and S430.

[0114] S410: Obtain the previous state information based on the video description corresponding to the previous inference window.

[0115] In some embodiments, prior state information may include a video description corresponding to the previous inference window and / or object state information extracted from that video description. In some embodiments, prior state information may be maintained and updated during continuous monitoring as monitoring state information for use in cross-window comparisons.

[0116] In some embodiments, the video description obtained in the first inference window can be retained and passed to subsequent inference windows, thus serving as a previous state that can be referenced when comparing subsequent inference windows. In some embodiments, the video description corresponding to the first inference window can be used as the previous state information, or object state information such as object identifier, object position, and object motion state can be extracted from the video description corresponding to the first inference window according to preset aspects and used as the previous state information.

[0117] S420: Input the previous state information, along with the video frame sequence corresponding to the current inference window and the time stamp of each video frame, into the multimodal large model. The multimodal large model then obtains the window difference comparison across inference windows based on the monitoring state information.

[0118] In some embodiments, the multimodal large model can not only perform stepwise comparisons between adjacent video frames within the current inference window, but also perform cross-inference window difference comparisons between the object state observed in the current window and the object state recorded in previous windows to obtain window difference comparison results.

[0119] S430: Generate additional video descriptions of traffic situation changes across inference windows based on window difference comparisons.

[0120] In some embodiments, the multimodal large model can generate additional video descriptions of traffic situation changes across inference windows based on previous state information and the video frame sequence of the current inference window. In some embodiments, the additional video descriptions can serve as the video description corresponding to the current inference window. By way of explanation and not limitation, the method according to the above embodiments of the present invention can combine the state continuation of multiple windows to more accurately identify risks, such as identifying risks that behave normally within a single inference window but evolve slowly across windows.

[0121] In an interpretative rather than restrictive manner, the present invention recognizes that for situations such as a vehicle continuously and slightly deviating from its lane, a gradually expanding area of ​​road flooding, or a progressively worsening degree of guardrail damage, the magnitude of change within any single inference window may not be significant, only exhibiting a trend when observed across multiple inference windows. In response, the method of the embodiments of the present invention, by utilizing cross-window continuation carried with previous state information, can monitor such slowly evolving situations and reflect them in supplementary video descriptions, thereby enabling subsequent risk assessments based on these supplementary video descriptions. Furthermore, compared to the unreadable and unrecordable internal inference processes of known solutions, the video descriptions obtained by the embodiments of the present invention can also be included in risk event records for subsequent auditing and further analysis.

[0122] S140: Input the video description into the large language model, and use the large language model to perform risk assessment on the video description to obtain the risk assessment result.

[0123] In some embodiments, the risk assessment results can be used to characterize whether there is a risk or accident in the situation reflected in the video description.

[0124] In some embodiments, the risk assessment result may include an indication of the existence of a risk or incident. In optional embodiments, the risk assessment result may further include at least one of the following: the type of risk or incident identified, the risk level, and explanatory information regarding the risk or incident. In this embodiment, the type of risk or incident may include, but is not limited to, vehicle collision, vehicle loss of control, abnormal stopping, intrusion by a pedestrian or non-operational object, etc. In some embodiments, the risk level in the risk assessment result may be used for the adjustment of the predetermined interval and / or predetermined duration as described above.

[0125] In some embodiments, the video description along with preset judgment prompts can be input into the large language model. In this embodiment, the judgment prompts can be used to explain the task and key points of risk judgment, such as instructing the large language model to identify traffic accidents, dangerous driving behaviors, and other safety hazards from the video description, and output the judgment conclusion in a predetermined form.

[0126] As mentioned above, highway video data can include video data collected in mixed traffic scenarios during highway reconstruction and expansion projects. For explanation, in road sections with work areas, such as those involved in highway reconstruction and expansion projects, the same frame may simultaneously show construction work within the work area and traffic outside the work area. This invention recognizes that in the absence of information on the layout of work areas for that road section, normal operational behaviors such as low-speed driving, lateral movement, reversing of construction vehicles within the work area, and the operational actions of maintenance machinery can easily be misjudged as traffic risks.

[0127] In some embodiments, the method for monitoring traffic safety during highway reconstruction and expansion may further include the following step: acquiring the structural information of the work area corresponding to the road segment in the highway video data. In some embodiments, the work area is the reconstruction and expansion construction area.

[0128] In some embodiments, the work area structure information may include a description of the work area layout of the road segment.

[0129] In some embodiments, the work area structure information may include at least one of the work section layout information and traffic diversion information for the road segment. In some embodiments, the work area structure information includes work area structure information described in text form. In one specific embodiment, the work area structure information is "The left lane of the current road segment is designated as the work area, and the traffic flow is diverted to the right lane via traffic cones."

[0130] Accordingly, step S140 may include: inputting the video description along with the work area structure information into a large language model, and having the large language model, in conjunction with the work area structure information, perform a risk assessment on the video description to obtain risk assessment results related to work behaviors within the work area and / or intrusion behaviors of non-work objects entering the work area. In a specific example, work behaviors within the work area include construction vehicles driving at low speeds within the work area and maintenance machinery performing work actions; intrusion behaviors of non-work objects entering the work area include vehicles crossing the guidance and diversion facilities to enter the work area and pedestrians entering the work area.

[0131] By way of explanation and not limitation, the above embodiments of the present invention provide the structural information of the work area to the artificial intelligence model as prior information for judgment. The artificial intelligence can distinguish between normal work activities within the work area and intrusion behavior by non-work objects entering the work area during risk assessment. This reduces the likelihood of misjudging normal work as risk in complex scenarios where construction work and traffic are mixed, while maintaining the ability to identify genuine intrusion behavior.

[0132] In other embodiments, the structural information of the work area may also be provided to the multimodal large model in step S120 above, so that the multimodal large model knows the layout of the work area when generating inter-frame difference information and video description, thereby distinguishing the positional relationship of objects relative to the work area in the description, which falls within the protection scope of the present invention.

[0133] S150: When the risk assessment result indicates the existence of a risk or accident, based on the risk assessment result and each time marker, locate the video time period corresponding to the risk or accident and the key video frames associated with the risk or accident from the highway video data, and generate a risk event record corresponding to the risk or accident.

[0134] In some embodiments, a risk event record may include a risk assessment result, a video description associated with the risk or accident, key video frames associated with the risk or accident, and video time information characterizing the video time period. As explained above, the inter-frame difference information and the video description can correspond to the time stamps of each video frame. Therefore, for any risk assessment result, the corresponding time stamp range for the accident or risk can be quickly determined based on the video description, thereby enabling rapid location of the video time period corresponding to the risk or accident from highway video data.

[0135] In some embodiments, key video frames may include one or more video frames most relevant to the presentation of a risk or incident within a video time period. In some embodiments, the adjacent frame that corresponds most significantly to the risk or incident, or the video frame that presents the most complete risk or incident pattern, may be identified as the key video frame.

[0136] In some embodiments, video time information can be used to characterize the located video time period. In some embodiments, video time information may include the start and end times of the video time period or a timestamp characterizing the time period. In other embodiments, video time information may also include the time when the video recording was generated; this invention is not limited thereto.

[0137] In some embodiments, risk event records may be generated in a structured data format. In some embodiments, risk event records may include at least: a timestamp field for recording video time information; a risk description field for recording video descriptions associated with the risk or accident and risk assessment results; a key video frame identifier field for identifying key video frames associated with the risk or accident; and an associated video segment identifier field for identifying video segments in highway video data corresponding to video time periods.

[0138] In other embodiments, the risk event record may also include other fields such as a record identifier field and a storage status identifier field. In this embodiment, the record identifier field can be used to uniquely identify the risk event record for merging and updating, and the storage status identifier field can be used to characterize the storage status of the video segment associated with the record. By way of explanation and not limitation, by employing a structured combination of fields, the risk event record can be directly read, indexed, and retrieved as a data unit by the downstream traffic management system, thereby facilitating mechanized review, retrieval, and backtracking by the downstream system.

[0139] By way of explanation and not limitation, the above embodiments of the present invention form a risk event record by associating and binding the risk assessment result, the video description associated with the risk or accident, key video frames, and video time information. Thus, the risk assessment result of a risk or accident is interconnected with its semantic basis (video description), visual basis (key video frames), and temporal basis (video time information) through the risk event record. Furthermore, unlike known solutions that only provide simple reminders, the risk event record of the embodiments of the present invention can characterize a complete accident and risk record unit that represents "when it happened, what happened, which images were used as the basis, and which video segment it corresponds to." This allows traffic management personnel to directly locate the corresponding key video frames and video time periods after obtaining the risk event record, quickly complete on-site verification, and subsequently trace the source of the accident based on the record, without having to search through a large amount of raw video data one by one. This improves the efficiency of on-site verification and post-accident tracing, as well as the traceability and credibility of monitoring results.

[0140] In some embodiments, a secondary verification (review) may be performed when or after the generation of a risk event record corresponding to a risk or incident, but before the output of the risk event record. Accordingly, in some embodiments, reference is made to... Figure 5 The step of generating risk event records corresponding to risks or accidents may include the following steps S510, S520, S530 and S540.

[0141] S510: Generate verification prompt information based on the risk assessment results.

[0142] In some embodiments, the verification prompt information may include the risk type indicated by the risk assessment result and the verification points corresponding to that risk type. In some embodiments, each risk type may be associated with one or more verification points. In a specific embodiment, the verification points associated with the risk type "vehicle collision" may include "whether there are vehicles in contact or deformed in the image," etc.

[0143] S520: Input the verification prompt information, as well as the key video frame associated with the risk or accident, or the local area image of the risk or accident in the key video frame, into the multimodal large model for secondary verification, and obtain the secondary verification result.

[0144] In some embodiments, the input for secondary verification may include the key video frame itself and / or a local region image within the key video frame corresponding to the risk or incident. In some embodiments, the local region image may be obtained, for example, by cropping and enlarging it appropriately outward from the region in the key video frame where the object involved in the risk or incident is located. By way of explanation and not limitation, this allows the multimodal large model to pay more attention to the local region image to focus on the local area of ​​the risk or incident for verification.

[0145] S530: If the secondary verification result is consistent with the risk assessment result, generate a risk event record.

[0146] In some embodiments, if the secondary verification result is consistent with the risk assessment result, it indicates that the risk assessment result based on the video description has a high degree of matching with the actual accident scenario, and a risk event record can be generated at this time.

[0147] S540: If the secondary verification result is inconsistent with the risk assessment result, the video time period is extended and the risk assessment is re-performed for the extended video time period.

[0148] In some embodiments, the inconsistency between the secondary verification result and the risk assessment result may indicate that the risk assessment result based on the video description has a low degree of matching or does not match the actual accident scenario / risk event scenario, and further risk assessment should be carried out in this case.

[0149] In some embodiments, the video time period can be extended forward and / or backward, and the risk assessment can be re-performed for the extended video time period. This allows the large language (artificial intelligence) model to reassess the accident / risk event scenario within a more complete timeframe.

[0150] In some embodiments, if a consensus cannot be reached after reassessment, the risk assessment result can be marked as pending review and the risk event record can be temporarily withheld. As an explanation, and not a limitation, this embodiment of the invention, by reviewing the monitoring results before outputting them, can reduce false alarms caused by insufficient information during video time periods, thereby improving the reliability of the monitoring results.

[0151] As mentioned above, the differential description sequence can also be incorporated into the risk event record described later. In this embodiment, the highway reconstruction and expansion traffic safety monitoring method may further include the following steps A1 and A2.

[0152] A1: Receive the review conclusions for the generated risk event records.

[0153] In some embodiments, the review conclusion may indicate that the risk assessment result is confirmed or determined to be a false alarm.

[0154] A2: If the review conclusion indicates that the risk assessment result is determined to be a false alarm, trace back along the derivation chain of hierarchical association to determine the association level at which the misjudgment occurred.

[0155] In some embodiments, the difference description sequence, the video description, and the risk assessment result can form a hierarchically related derivation chain. As explained above, the inter-frame difference information, the difference description sequence, and the video description can all correspond to the timestamps of each video frame. Therefore, if the assessment result is determined to be a false alarm, the corresponding level of false alarm can be determined according to the timestamps and the derivation chain.

[0156] In some embodiments, if the difference description sequence does not match the video content, the misjudgment can be determined to have occurred in the inter-frame comparison stage; if the difference description sequence is correct but the video description does not match it, the misjudgment can be determined to have occurred in the induction stage; if the video description is correct but the risk assessment result is inappropriate, the misjudgment can be determined to have occurred in the risk assessment stage. As an explanation, and not a limitation, the solutions in the above embodiments can pinpoint the specific stage in the processing chain where the misjudgment occurs, thereby facilitating targeted improvements to the processing of the corresponding stage.

[0157] In other embodiments, the video description and risk assessment result recorded in the risk event that was determined to be a false alarm can also be used as negative example information, serving as judgment prompts for subsequent risk assessments using a large language model. By way of explanation and not limitation, using the review conclusion as an example for the large language model to refer to can reduce the false alarm rate in subsequent assessments.

[0158] As previously stated, the method steps described in the embodiments of the present invention can be continuously executed. In some embodiments, the highway reconstruction and expansion traffic safety monitoring method may further include the following steps B1 and B2.

[0159] B1: Based on the risk assessment results obtained during continuous execution, determine whether the risk assessment result indicates the same risk or accident as the existing risk event, according to the semantic similarity between the video description corresponding to the risk assessment result and the video description corresponding to the existing risk event, and the temporal adjacency between the video time period corresponding to the risk assessment result and the video time period corresponding to the existing risk event.

[0160] In some embodiments, the cosine similarity between the text embedding vectors of each video description can be calculated, and semantic similarity can be determined when the similarity exceeds a preset similarity threshold. However, it is understood that other means can be reasonably used to determine semantic similarity in other embodiments, and the present invention does not limit this.

[0161] B2: When it is determined that the same risk or accident is indicated, the risk event record corresponding to the risk assessment result is merged into the risk event record corresponding to the existing risk event, and the merged risk event record is incrementally updated.

[0162] In some embodiments, incremental updates include at least one of adding key video frames and expanding video time information. By way of explanation and not limitation, the present invention recognizes that a prolonged risk or incident (e.g., persistent traffic congestion, long-term vehicle delays) may span multiple inference windows, each of which may yield a risk assessment result pointing to the same situation. In this case, if risk event records are generated independently window by window, the same event will be split into multiple repetitive and fragmented records. To address this, embodiments of the present invention identify and merge multiple risk assessment results pointing to the same risk or incident based on the semantic similarity between video descriptions and the temporal adjacency between video time segments. The merged risk event record can be incrementally updated as the event continues, for example, by adding newly emerging key video frames or expanding the time range represented by the video time information. Thus, the same risk event corresponds to a continuously improving record, avoiding duplicate alerts and fragmented evidence.

[0163] In some embodiments, the method for monitoring traffic safety during highway reconstruction and expansion may further include: when a risk assessment result indicates the existence of a risk or accident, independently archiving video segments corresponding to video time periods from the highway video data, and using the timestamp of the video frame corresponding to the video segment as the index key for retrieving the video segment. As an explanation, and not a limitation, this invention recognizes that continuously collected highway video data is typically stored on a rolling cycle, potentially overwriting earlier video content. Therefore, the method of this embodiment independently archives video segments corresponding to risks or accidents, using timestamps as index keys, thereby ensuring that the corresponding original video segments can be quickly retrieved based on the video time information in the risk event record during post-event evidence collection.

[0164] S160: Visualize the video description associated with the risk or incident, as well as the key video frames associated with the risk or incident.

[0165] In some embodiments, at or after the generation of a risk event record, the video description associated with the risk or incident and the key video frames associated with the risk or incident may be further visualized.

[0166] In some embodiments, the visualization can be performed on the interface of the monitoring terminal. In some embodiments, the visualization can be divided into sections, such as, but not limited to, real-time warning messages and real-time video streams. In optional embodiments, video snapshots, configuration information, warning message status, video stream status, storage status, etc., can also be visualized in tab format; this invention does not impose limitations on these aspects.

[0167] In some embodiments, visualization can be synchronized with real-time message push. In some embodiments, after a risk event record is generated, the risk event record can be pushed to the monitoring terminal as a real-time warning message and displayed in the aforementioned real-time warning message area, so that monitoring personnel can be informed and view the relevant content as soon as a risk or accident occurs.

[0168] As an explanation and not a limitation, in the above embodiments of the present invention, the monitoring conclusion (risk assessment result / warning content) and its semantic basis (video description), visual basis (key video frames and real-time images), and temporal basis (time information) are presented to the monitoring personnel / traffic management personnel in an intuitive and related manner. Thus, the monitoring personnel / traffic management personnel do not need to manually search through a large number of videos to directly know and view the time, images and descriptions corresponding to the risk or accident, which facilitates timely on-site verification.

[0169] In an optional embodiment, the highway video data may include video data collected by multiple cameras deployed along the highway. In this embodiment, the method steps according to the above embodiments of the present invention can be performed separately for the video data collected by each camera. In other words, for the video data collected by each camera, video frames can be extracted, video frame sequences can be constructed and combined into a multimodal large model, video descriptions can be generated, and risk assessment can be performed.

[0170] As an explanation, and not a limitation, on linearly extending roads such as highways, multiple cameras are typically deployed sequentially along the road's direction, with each camera's acquisition range covering different sections of the road. This invention recognizes that the acquisition range of a single camera is limited, and objects involved in a risk or accident (e.g., out-of-control vehicles) may move out of the acquisition range of one camera and into the acquisition range of a downstream camera during the event. Therefore, if the monitoring of each camera is independent of each other, the evolution of the same risk object across cameras will be fragmented into unrelated segments.

[0171] In an optional embodiment, the method for monitoring traffic safety during highway reconstruction and expansion may further include the following steps C1 to C4.

[0172] C1: If the risk assessment result obtained from the video data collected by the first camera among multiple cameras indicates the existence of a risk or accident, the risk object involved in the risk or accident and the direction of movement of the risk object shall be determined according to the video description corresponding to the risk assessment result.

[0173] In some embodiments, the risk object may include the object involved in the risk or accident indicated by the risk assessment result, such as, but not limited to, an out-of-control vehicle, a vehicle that has entered the work area, etc. In this embodiment, the risk object and the direction of movement of the risk object can be determined from the corresponding video description. In a specific example, the video description is "white sedan, out of control, skidding forward and slightly to the right", from which the risk object can be determined as "white sedan" and the direction of movement is "skidding forward and slightly to the right".

[0174] C2: Determine the second camera located downstream of the direction of motion from among multiple cameras, based on the direction of motion.

[0175] In some embodiments, when or after determining the direction of movement of a risky object, a camera whose acquisition range intersects with the expected travel path of the risky object can be identified as a second camera based on the deployment order of multiple cameras deployed along the highway and the direction of movement of the risky object.

[0176] C3: Use the video description corresponding to the risk assessment result or the description of the risk object in it as prior information, and combine it with the video frame sequence formed by the video frames extracted from the video data collected from the second camera and the time stamp of each video frame, and input it into the multimodal large model.

[0177] In some embodiments, such as in step C3 above, the specific form of the prior information can be referred to the description of the preceding steps. The difference is that in step C3 above, the prior information is obtained from video data collected by the first camera among multiple cameras. As an explanation and not a limitation, by inputting the prior information generated based on the video data collected by the first camera into the multimodal large model, the multimodal large model can perform inference on the video data collected by the second camera based on existing expectations. For example, the multimodal large model can predict what kind of object will enter the frame and in what state. Thus, when the object enters the frame, the multimodal large model can match it with the risk object described in the prior information, thereby achieving continuous monitoring of the same risk object.

[0178] C4: The risk event records generated from the video data collected by the first camera and the risk event records generated from the video data collected by the second camera are correlated with each other according to their respective video time information to obtain the cross-camera event timeline corresponding to the risk object.

[0179] By way of explanation and not limitation, the cross-camera event timeline obtained by the above embodiments of the present invention can be used to reconstruct the movement trajectory and overall situation evolution of the risk object across the acquisition range of multiple cameras. Compared with risk event records based on only a single camera and which are fragmented, it avoids the incomplete accident or risk information caused by the limited field of view of a single camera, and ensures the completeness of monitoring of the risk object in the monitored road section.

[0180] In an optional embodiment, the highway reconstruction and expansion traffic safety monitoring method may further include performing at least one of the following steps D1 and D2 before the risk object enters the acquisition range of the second camera. D1: Reduce the predetermined interval used to extract video frames from the video data acquired by the second camera. D2: Generate expected object prompt information containing a description of the risk object, and combine this expected object prompt information with a video frame sequence composed of video frames extracted from the video data acquired by the second camera, along with the timestamps of each video frame, and input it into a multimodal large model.

[0181] In an interpretive and not restrictive sense, the present invention recognizes that if the second camera monitors at a conventional sampling rhythm, the moment a risky object enters the second camera's acquisition range may fall precisely between two frame samplings, or within an inference window that has not yet been filled, thus causing gaps in the transition period of subsequent monitoring. To address this, embodiments of the present invention pre-decrease the predetermined interval corresponding to the second camera to increase the sampling frequency (sampling density) before the risky object enters the acquisition range, and / or pre-inject expected object prompt information containing the description of the risky object into the model input corresponding to the second camera, so that monitoring for the second camera enters a ready state before the risky object arrives. Therefore, the downstream camera's monitoring can have a certain lead time, so that the risky object can be sampled at a high frequency as soon as it enters the acquisition range, thereby reducing gaps and omissions in cross-camera sequential monitoring.

[0182] In another embodiment, the traffic safety monitoring method for highway reconstruction and expansion projects according to the embodiments of the present invention can be applied not only to mixed traffic scenarios in highway reconstruction and expansion projects, but also to at least one of, for example, urban road scenarios, tunnel scenarios, and interchange hub scenarios. As an explanation and not a limitation, in urban road scenarios, the method according to the embodiments of the present invention can monitor vehicle conflicts and pedestrian crossings at intersections; in tunnel scenarios, the method according to the embodiments of the present invention can monitor abnormal vehicle stops and wrong-way driving; in interchange hub scenarios, the method according to the embodiments of the present invention can monitor vehicle conflicts at ramp merging points. It is understood that in the above scenarios, the features and execution process of each step of the method can be referred to the detailed description of the foregoing embodiments, and will not be repeated here. The main differences lie in the deployment location of video acquisition and the types of events that the risk assessment focuses on, all of which fall within the protection scope of the embodiments of the present invention.

[0183] The following describes the execution process of the method according to embodiments of the present invention in a highway traffic scenario, using specific examples. It is understood that the following specific embodiments are merely examples and do not constitute a limitation of the present invention.

[0184] In one specific embodiment, the highway video data comes from surveillance cameras deployed on a section of the highway. On a rainy day, a large area of ​​water accumulation with rapid current appeared on the right side of the road ahead. The monitoring system continuously extracts video frames from the camera's video stream at predetermined intervals, where the predetermined interval is 2.5 seconds (time stamps in the following records are rounded to the nearest second). After the accumulated video duration reaches the predetermined duration of 10 seconds, a video frame sequence is constructed from the extracted video frames. This video frame sequence approximately corresponds to a section of highway video from 23:27:27 to 23:27:37. This sequence, along with the time stamps of each video frame, is then input into a multimodal large-scale model. Subsequently, the multimodal large model compares adjacent video frames one by one, outputting difference descriptions in the form of the following: "23:27:27→23:27:30: The white sedan is traveling from the left lane towards the waterlogged area at a relatively high speed; 23:27:30→23:27:32: The white sedan enters the waterlogged area, the body skids, and the driving direction begins to deviate from the lane; 23:27:32→23:27:35: The white sedan's skid intensifies and it begins to spin, gradually leaving the normal lane; 23:27:35→23:27:37: The white sedan collides, rolls over, and finally lies upside down on the road with its wheels facing upwards." These difference descriptions are then arranged chronologically to form the difference descriptions. The sequence is used to generate a video description: "Current 10-second monitoring message description: The video shows a car driving on a highway in the rain. There is severe flooding on the right side of the road ahead, with a rapid current. Suddenly, a white sedan comes from the left lane and attempts to pass through the flooded area. When the sedan enters the flood, due to the impact of the water and the slippery road surface, the vehicle loses control, begins to skid and spin. During the spin, the sedan gradually deviates from the normal lane, eventually colliding with the guardrail, rolling over, and landing upside down on the road with its wheels in the air. Throughout the process, the guardrail in the background and the distant power line tower are clearly visible. The weather is overcast, and visibility is low. At the end of the video, the white sedan is still upside down, and no other vehicles or people are seen approaching." In this specific embodiment, unlike the instantaneous conclusions such as "there is flooding on the road" or "there is an upside-down car" obtained by independently identifying only a single video frame, the solution in the above embodiment can understand and completely analyze the loss of control process of the vehicle from normal driving to skidding, spinning, colliding, and rolling over.

[0185] Subsequently, the video description is input into a large language model for risk assessment. Based on the video description, the large language model determines that a vehicle collision has occurred, obtaining a risk assessment result indicating the existence of an accident. This risk assessment result includes, for example, the accident type (vehicle collision), the risk level (high), and a warning message: "Please note that a vehicle collision has occurred, requiring immediate action or notification." Since the risk assessment result indicates the existence of an accident, further based on this risk assessment result and various time markers, the corresponding video time period for the accident is located from the highway video data: 23:27:27 to 23:27:37. The video frame showing the most complete rollover pattern is identified as the key video frame, and a risk event record is generated.

[0186] In this embodiment, the risk event record is generated in JSON format: {"timestamp": "2025-06-28T23:27:42.777496","alert": "Please note that a vehicle collision has occurred and requires immediate attention or notification.", "description": "Current 10-second monitoring message description: The video shows a car driving on a highway in the rain. There is severe flooding on the right side of the road ahead, with a rapid current. Suddenly, a white sedan comes from the left lane and attempts to pass through the flooded area. When the sedan enters the flood, due to the impact of the water flow and the slippery road surface, the vehicle loses control, begins to skid and spin. During the spin, the sedan gradually deviates from the normal lane, eventually colliding with the guardrail, overturning, and landing upside down on the road with its wheels in the air. Throughout the process, the guardrail in the background and the distant power line tower are clearly visible. The weather is overcast and visibility is low. At the end of the video, the white sedan is still upside down, and no other vehicles or people are seen approaching from the surrounding environment.", "video_file_name": The file "warning_2025-06-28-23-27-27.mp4", "picture_file_name": "warning_2025-06-28-23-27-27.jpg", "id": 1751124462779, "stored": true}. The video segment corresponding to this video time period is archived independently. Its video segment file and key video frame image file are named with the timestamp of the video frame at the beginning of the video segment, "warning_2025-06-28-23-27-27", allowing direct retrieval of the corresponding original video segment and key frame from the time information.

[0187] Finally, refer to Figure 6The risk event record is pushed to the monitoring terminal as a real-time warning message. The monitoring terminal displays the record in the real-time warning message area of ​​its interface in chronological order (lower left), including its time information, warning content (alert), video description (description), and record identifier. The real-time video stream area displays the camera's live image (the actual accident video is schematically shown in gray in the figure), and key video frames are obtained from the captured snapshot. Based on this, the monitoring personnel can learn about and review the accident immediately, and can directly retrieve the independently archived original video clips and key video frames for evidence collection afterward based on the time information and file identifier in the record.

[0188] Therefore, the traffic safety monitoring method for highway reconstruction and expansion provided by the embodiments of the present invention can have at least some of the following technical effects:

[0189] The method according to embodiments of the present invention extracts video frames at predetermined intervals from continuously acquired highway video data, and inputs the video frame sequence along with the time stamps of each video frame as a whole into a multimodal large model. The multimodal large model progressively compares adjacent video frames rather than identifying each frame independently, thereby reading changes in object state, object trajectory, and object posture that are only apparent in cross-frame situations. Based on this, a video description representing changes in highway traffic conditions is generated in chronological order. Since the video description represents the temporal evolution obtained through combined input and progressive comparison, the large language model can make risk judgments based on the evolution of the situation, rather than isolated moments in the scene. Furthermore, when the risk judgment result indicates the existence of risk or accident, the method of embodiments of the present invention can backtrack and locate key video frames and video time periods, generating a risk event record that binds the risk judgment result, related video description, key video frames, and video time information together. The located video description and / or time period and key video frames are then visualized, making the monitoring conclusions interconnected and intuitively presented with their semantic, visual, and temporal basis. The method of this invention overcomes the technical problems of known monitoring methods, which struggle to grasp the evolving traffic situation over time and make it difficult to verify and trace the monitoring results. The method of this invention can more accurately identify gradually evolving risks or accidents, and presents monitoring conclusions in conjunction with their visual and temporal evidence, facilitating timely on-site verification and post-event tracing. This allows monitoring personnel to directly locate and view the time period and images corresponding to risks or accidents without manual review, quickly complete on-site verification, and obtain evidence afterward, thereby significantly improving the accuracy of traffic safety monitoring during highway reconstruction and expansion, as well as the usability of the monitoring results.

[0190] In a further embodiment of the present invention, video frames are extracted sequentially at predetermined intervals and their corresponding video durations are accumulated. Once the accumulated video duration reaches the predetermined duration, a video frame sequence is constructed for combined input. This controls the overhead of calling the multimodal large model while ensuring that the video frame sequence entering the inference covers a sufficient time span to support timing judgment.

[0191] In a further embodiment of the present invention, a text-based difference description is output for each pair of adjacent frames. These difference descriptions are arranged chronologically to form a difference description sequence, which is then used to generate a video description. This allows the multimodal large model to progressively compare the differences between adjacent frames and retain the results in readable text form at each level. Furthermore, the difference description sequence, video description, and risk assessment results form a progressively related and corresponding processing chain. Therefore, the method of this embodiment not only provides sufficient evidence for risk assessment but also provides progressively corresponding intermediate evidence for the review and retrospective analysis of the assessment process.

[0192] In a further embodiment of the present invention, the structural information of the corresponding road section's work area can also be obtained as a priori input to the large language model, so that in complex road sections where construction work and traffic are mixed, the work behavior within the work area, the intrusion behavior of non-work objects entering the work area, and the ordinary traffic behavior outside the work area can be treated differently, reducing the situation of misjudging normal work as risk.

[0193] In a further embodiment of the present invention, before generating a risk event record, a verification prompt message can be generated based on the risk assessment result. This message, along with key video frames or images of their local areas, can be input into a multimodal large model for secondary verification. If the two are consistent, a record is issued; if they are inconsistent, the video time period is extended for reassessment. This approach, through the mutual corroboration of verbal judgment and visual verification, further reduces false alarms and improves reliability.

[0194] In a further embodiment of the present invention, multiple risk judgment results pointing to the same risk or accident are merged into the same risk event based on the semantic similarity between video descriptions and the temporal adjacency between video time periods, and the merged record is incrementally updated, so that the same persistent event spanning multiple inference windows corresponds to a dynamically improved record.

[0195] In a further embodiment of the present invention, highway video data can come from multiple cameras deployed along the highway and perform monitoring separately. After the first camera detects a risk, a second downstream camera can be determined based on the movement direction of the risky object. The description representing the risky object is used as a priori continuation input, and the risk event records of the two cameras are correlated according to video time information to obtain a cross-camera event timeline. Thus, the method according to the embodiments of the present invention can reconstruct the movement trajectory and overall situation evolution of the risky object across the collection range of multiple cameras, avoiding information fragmentation caused by the limited field of view of a single camera, and ensuring the integrity of monitoring the risky object within the monitored road section.

[0196] In a further embodiment of the present invention, before the risk object enters the acquisition range of the second camera, the predetermined frame extraction interval of the camera can be reduced in advance and / or expected object prompt information containing the description of the risk object can be injected, so that the downstream monitoring is ready in advance, that is, the monitoring of the downstream camera has a lead time, and then high-frequency sampling can be performed when the risk object enters the acquisition range, reducing gaps and omissions in cross-camera continuous monitoring, and further ensuring the accuracy and effectiveness of monitoring.

[0197] In some embodiments, reference Figure 7 The present invention also provides a highway reconstruction and expansion traffic safety monitoring device 700 based on a multimodal artificial intelligence model. The highway reconstruction and expansion traffic safety monitoring device 700 may include a frame extraction unit 710, a comparison unit 720, a description generation unit 730, a risk judgment unit 740, a generation unit 750, and a display unit 760.

[0198] In some embodiments, the frame extraction unit 710 is configured to extract video frames from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence consisting of a plurality of extracted video frames and a time stamp of each video frame.

[0199] In some embodiments, the comparison unit 720 is configured to input the video frame sequence together with the time stamps of each video frame into a multimodal large model, and to perform stepwise comparison of the differences between adjacent video frames in the video frame sequence through the multimodal large model to obtain multiple inter-frame difference information, wherein the inter-frame difference information represents the difference between adjacent video frames in at least one of object state, object motion trajectory and object posture.

[0200] In some embodiments, the description generation unit 730 is configured to generate a video description characterizing changes in road traffic conditions in chronological order based on a plurality of the inter-frame difference information.

[0201] In some embodiments, the risk assessment unit 740 is configured to input the video description into a large language model, and perform risk assessment on the video description through the large language model to obtain a risk assessment result.

[0202] In some embodiments, the generation unit 750 is configured to, when the risk assessment result indicates the existence of a risk or accident, locate the video time period corresponding to the risk or accident and the key video frames associated with the risk or accident from the highway video data according to the risk assessment result and each of the time markers, and generate a risk event record corresponding to the risk or accident, wherein the risk event record includes: the risk assessment result, the video description associated with the risk or accident, the key video frames associated with the risk or accident, and video time information characterizing the video time period.

[0203] In some embodiments, the display unit 760 is configured to visually display the video description and / or video time period corresponding to the risk or accident, as well as key video frames associated with the risk or accident.

[0204] In this embodiment of the invention, the specific implementation of the above-mentioned highway reconstruction and expansion traffic safety monitoring device 700, as well as the details of each unit performing the corresponding steps in the aforementioned method and its optional implementation methods, can be found in the detailed description of the aforementioned highway reconstruction and expansion traffic safety monitoring method embodiment, and will not be repeated here.

[0205] Accordingly, the highway reconstruction and expansion traffic safety monitoring device and its components, modules, units, and features described in the above embodiments can be incorporated into the highway reconstruction and expansion traffic safety monitoring method of the present invention in a non-contradictory manner, and will not be elaborated here. Conversely, the highway reconstruction and expansion traffic safety monitoring method and its steps, sub-steps, and features described in the embodiments of the present invention can also be incorporated into the highway reconstruction and expansion traffic safety monitoring device of the present invention in a non-contradictory manner.

[0206] It is understood that the division of the units in the highway reconstruction and expansion traffic safety monitoring device 700 is a logical functional division. In actual implementation, there may be other division methods. For example, multiple units may be combined or integrated into one module, and each unit may be implemented by software, hardware or a combination of software and hardware. All of the above fall within the scope of this invention.

[0207] In some embodiments of the present invention, an electronic device is also provided, which includes a processor and a memory storing a computer program, the processor being configured to implement the method of any embodiment of the present invention when running the computer program.

[0208] Figure 8A schematic diagram is shown of a method or electronic device 800 that can be used to implement embodiments of the present invention. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, it can be implemented using a single or multiple electronic devices. It can also be implemented using cloud or distributed electronic devices in some embodiments.

[0209] like Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802. The processor executes programs stored in the memory, which, when executed by a computer, can implement the methods, steps, or functions described in the above embodiments. The processor 801 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 801 and the memory 802 are interconnected via a bus 803. Input / output (I / O) interfaces may also be connected to the bus 803. The systems, devices, modules, or units illustrated in the above embodiments can be implemented by a computer or its associated components.

[0210] Although not shown, in this embodiment of the invention, a program product is provided, including a computer program that, when executed by a processor, implements the method of any embodiment of the invention.

[0211] Although not shown, in an embodiment of the invention, a storage medium is provided storing a computer program configured to be run to implement the method of any embodiment of the invention.

[0212] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, and that can store information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information that can be accessed by a computing device.

[0213] The methods, programs, systems, apparatuses, etc., in embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

[0214] Those skilled in the art will understand that the embodiments described in this specification can be provided as methods, systems, or computer program products. Therefore, those skilled in the art will realize that the functional modules / units or controllers and related method steps described in the above embodiments can be implemented in software, hardware, or a combination of both.

[0215] Unless explicitly stated otherwise, the actions or steps of the methods and procedures described in the embodiments of the present invention do not necessarily have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0216] This document describes several embodiments of the present invention; however, for the sake of brevity, the descriptions of the embodiments are not exhaustive, and identical or similar features or parts between the embodiments may be omitted. In this document, "one embodiment," "some embodiments," "example," "specific example," or "some examples" refers to embodiments applicable to at least one, but not all, of the present invention. The above terms do not necessarily refer to the same embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of the different embodiments or examples.

[0217] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best mode for implementing the systems and methods. Those skilled in the art will understand that various changes can be made to the embodiments of the systems and methods described herein without departing from the spirit and scope of the invention as defined in the appended claims when implementing the systems and / or methods.

Claims

1. A method for monitoring traffic safety during highway reconstruction and expansion based on a multimodal artificial intelligence model, characterized in that, The method includes: Video frames are extracted from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence consisting of multiple extracted video frames and a time stamp of each video frame. The highway video data is video data collected in a mixed traffic scenario of a highway reconstruction and expansion project. The video frame sequence, together with the time stamps of each video frame, is input into a multimodal large model. The multimodal large model then performs a stepwise comparison of the differences between adjacent video frames in the video frame sequence to obtain multiple inter-frame difference information. The inter-frame difference information represents the difference between adjacent video frames in at least one of the following: object state, object motion trajectory, and object posture. Based on multiple inter-frame difference information, a video description representing changes in highway traffic conditions is generated in chronological order. The video description is input into a large language model, and the risk assessment of the video description is performed by the large language model to obtain the risk assessment result; If the risk assessment result indicates the existence of a risk or accident, based on the risk assessment result and each of the time markers, the video time period corresponding to the risk or accident and the key video frames associated with the risk or accident are located from the highway video data, and a risk event record corresponding to the risk or accident is generated. The risk event record includes: the risk assessment result, the video description associated with the risk or accident, the key video frames associated with the risk or accident, and the video time information characterizing the video time period. The video description associated with the risk or incident, as well as the key video frames associated with the risk or incident, are visualized.

2. The method for monitoring traffic safety during highway reconstruction and expansion according to claim 1, characterized in that, The step of extracting video frames from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence composed of multiple extracted video frames includes: Video frames are extracted sequentially from the highway video data at predetermined intervals, and the video duration corresponding to each extracted video frame is accumulated to obtain the cumulative video duration. When the cumulative video duration reaches a predetermined duration, the video frame sequence is composed of multiple extracted video frames, and the time stamp of each video frame is obtained.

3. The method for monitoring traffic safety during highway reconstruction and expansion according to claim 2, characterized in that, The method is continuously executed and includes: Based on the risk level indicated by the risk assessment result, adjust at least one of the predetermined interval and the predetermined duration used for subsequent video frame extraction. Wherein, when the risk level increases, the adjustment includes reducing the predetermined interval and / or shortening the predetermined duration; and / or, when multiple consecutive risk assessment results indicate that there is no risk or accident, the adjustment includes increasing the predetermined interval and / or extending the predetermined duration.

4. The method for monitoring traffic safety during highway reconstruction and expansion according to claim 1, characterized in that, The method further includes: obtaining the work area structure information of the road segment corresponding to the highway video data, wherein the work area structure information includes a description of the work area layout of the road segment; The step of inputting the video description into a large language model and performing a risk assessment on the video description using the large language model to obtain a risk assessment result includes: The video description, along with the work area structure information, is input into the large language model. The large language model then uses the work area structure information to perform a risk assessment on the video description, resulting in a risk assessment result related to work activities within the work area and / or intrusion behavior of non-work objects entering the work area.

5. The method for monitoring traffic safety during highway reconstruction and expansion according to claim 1, characterized in that, The stepwise comparison of differences between adjacent video frames in the video frame sequence using the multimodal large model yields multiple inter-frame difference information, including: Each pair of adjacent video frames in the video frame sequence is compared one by one, and a text-based difference description is output for each pair of adjacent frame frames as the inter-frame difference information corresponding to that pair of adjacent frame frames. The step of generating a video description representing changes in highway traffic conditions in chronological order based on multiple inter-frame difference information includes: Arrange the aforementioned differences into a difference description sequence according to the chronological order of their corresponding time markers; The video description characterizing changes in highway traffic conditions is generated by inductively summarizing the difference description sequence.

6. The method for monitoring traffic safety during highway reconstruction and expansion according to claim 1, characterized in that, The method is executed continuously, and each input video frame sequence of the multimodal large model corresponds to an inference window; The method includes: Based on the video description corresponding to the prior inference window, previous state information is obtained, which includes the video description corresponding to the prior inference window and / or object state information extracted from the video description. The previous state information, together with the video frame sequence corresponding to the current inference window and the time stamp of each video frame, are input into the multimodal large model. The multimodal large model obtains the window difference comparison across inference windows based on the monitoring state information. Based on window difference comparisons, additional video descriptions of traffic situation changes across inference windows are generated.

7. The method for monitoring traffic safety during highway reconstruction and expansion according to claim 1, characterized in that, The generation of risk event records corresponding to the risk or accident includes: Based on the risk assessment results, a verification prompt message is generated; The verification prompt information, along with the key video frame associated with the risk or accident, or the local area image of the key video frame corresponding to the risk or accident, are input into a multimodal large model for secondary verification to obtain the secondary verification result. If the secondary verification result is consistent with the risk assessment result, the risk event record is generated; If the secondary verification result is inconsistent with the risk assessment result, the video time period is extended and a new risk assessment is performed for the extended video time period.

8. A traffic safety monitoring device for highway reconstruction and expansion based on a multimodal artificial intelligence model, characterized in that, The device includes: The frame extraction unit is configured to extract video frames from continuously acquired highway video data at predetermined intervals to obtain a video frame sequence consisting of multiple extracted video frames and a time stamp of each video frame. The comparison unit is configured to input the video frame sequence together with the time stamps of each video frame into a multimodal large model, and to perform stepwise comparison of the differences between adjacent video frames in the video frame sequence through the multimodal large model to obtain multiple inter-frame difference information, wherein the inter-frame difference information represents the difference between adjacent video frames in at least one of object state, object motion trajectory and object posture. The description generation unit is configured to generate a video description representing changes in road traffic conditions in chronological order based on multiple inter-frame difference information. The risk assessment unit is configured to input the video description into a large language model, perform risk assessment on the video description through the large language model, and obtain a risk assessment result. The generation unit is configured to, when the risk assessment result indicates the existence of a risk or accident, locate the video time period corresponding to the risk or accident and the key video frames associated with the risk or accident from the highway video data according to the risk assessment result and each of the time markers, and generate a risk event record corresponding to the risk or accident, wherein the risk event record includes: the risk assessment result, the video description associated with the risk or accident, the key video frames associated with the risk or accident, and video time information characterizing the video time period; The display unit is configured to visually display video descriptions associated with the risk or incident, as well as key video frames associated with the risk or incident.

9. An electronic device, characterized in that, include: A processor and a memory storing a computer program, the processor being configured to implement the method as described in any one of claims 1 to 7 when the computer program is executed.

10. A program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images