A method and related equipment for semantic understanding of long-term traffic videos

Abstract

This application provides a method and related equipment for semantic understanding of long-term traffic videos, relating to the fields of intelligent transportation and computer vision technology. This application reconstructs a target long-term traffic video sequence into a target sparse traffic visual word sequence involving dynamic traffic visual concepts. Then, it performs cross-timescale sparse attention feature learning on the target sparse traffic visual word sequence across multiple traffic state perception dimensions. Based on the assigned desired video semantic understanding task, it fuses the local attention output feature sequences of each of the learned traffic state perception dimensions to obtain the target traffic video semantic feature sequence, minimizing redundant information interference during the traffic video semantic understanding process. Finally, it invokes a large language model to perform cross-modal attention interaction reasoning based on the target traffic video semantic feature sequence and the desired video semantic understanding task, effectively balancing the efficiency and accuracy of video semantic understanding.

Description

Technical Field

[0001] This application relates to the fields of intelligent transportation and computer vision technology, and more specifically, to a method and related equipment for semantic understanding of long-term traffic videos. Background Technology

[0002] With the development of intelligent transportation systems and the acceleration of urban digitalization, video surveillance equipment has been deployed on a large scale at key traffic nodes such as urban roads, intersections, and highways. This has made the traffic video data collected by video surveillance equipment an important source of information on road operation status and traffic behavior. Typically, automated semantic understanding of traffic video data is required to achieve continuous perception of traffic status information such as traffic flow status, abnormal traffic events, and road operation characteristics, which facilitates further optimization of traffic management and intelligent travel services.

[0003] However, it is worth noting that because video surveillance equipment typically operates in all-weather mode, the traffic video data collected often has characteristics such as long time span, complex scene changes, high proportion of background information, and significant heterogeneity in the semantic information distribution of different traffic elements in the spatiotemporal dimension (for example, static traffic scene elements such as traffic signs and road markings usually remain relatively stable in the spatiotemporal dimension, while dynamic traffic participation elements such as vehicles and pedestrians usually exhibit significant dynamic changes in the spatiotemporal dimension). Key traffic events are sparsely distributed and random (for example, most traffic video clips only contain normal traffic flow information or static traffic scene information, and key traffic events such as traffic accidents and violations are often concentrated in a few randomly distributed traffic video clips). As a result, the corresponding long-term traffic video data (i.e., long-time-series traffic video sequences) contains a large amount of redundant information interference that contributes little to video semantic understanding operations. Consequently, the overall dense modeling semantic understanding scheme for long-time-series traffic video sequences cannot actually balance the efficiency and accuracy of video semantic understanding. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a long-time-series traffic video semantic understanding method, computer device, and readable storage medium. This method can achieve a structured and compact representation of the semantic understanding task content of long-time-series traffic video sequences by organically combining sparse semantic reconstruction operations of dynamic traffic participants (i.e., dynamic traffic visual concepts) with a hybrid sparse attention fusion mechanism that considers multiple traffic state perception dimensions (including long-time-series lane perception dimension, key traffic event perception dimension, and / or near-time-series dynamic trajectory perception dimension). This reduces redundant information interference as much as possible, and facilitates further utilization of large language models to improve the efficiency and capability of video semantic understanding of long-time-series traffic videos, thus achieving an effective balance between video semantic understanding efficiency and accuracy.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, this application provides a method for semantic understanding of long-time traffic videos, the method comprising: Dynamic traffic visual concept semantic detection is performed on the acquired target long-time traffic video sequence to obtain a target sparse traffic visual word sequence involving dynamic traffic visual concepts. Sparse attention features are learned across time scales for the target sparse traffic visual word sequence in multiple traffic state perception dimensions to obtain the local attention output feature sequences corresponding to each of the multiple traffic state perception dimensions. Based on the assigned desired video semantic understanding task, feature fusion is performed on the local attention output feature sequences of each of the multiple traffic state perception dimensions to obtain the target traffic video semantic feature sequence. The large language model is invoked to perform cross-modal attention interaction reasoning based on the target traffic video semantic feature sequence and the expected video semantic understanding task, so as to obtain the target semantic understanding result.

[0006] In an optional implementation, the step of performing dynamic traffic visual concept semantic detection on the acquired target long-term traffic video sequence to obtain a target sparse traffic visual lexical sequence involving dynamic traffic visual concepts includes: For each frame of traffic image included in the target long-term traffic video sequence, the original visual feature map of the traffic image is extracted to perform sparse traffic visual concept modeling, and the sparse visual concept semantic matrix of the traffic image is obtained. Based on the sparse visual concept semantic matrix and the original visual feature map of the traffic image frame, calculate the visual concept distribution confidence matrix of the traffic image frame. Based on the visual concept distribution confidence matrix of each traffic image in the target long-term traffic video sequence, calculate the overall spatial position change intensity of each sparse traffic visual concept in the target long-term traffic video sequence, so as to determine all dynamic traffic visual concepts involved in the target long-term traffic video sequence, wherein the overall spatial position change intensity corresponding to each dynamic traffic visual concept exceeds a preset change intensity threshold. For each frame of traffic image, the local sub-matrices related to all dynamic traffic visual concepts in the sparse visual concept semantic matrix and visual concept distribution confidence matrix of the traffic image are extracted and concatenated to obtain the dynamic concept visual lexical features of the traffic image in the target sparse traffic visual lexical sequence.

[0007] In an optional implementation, the step of calculating the visual concept distribution confidence matrix of the traffic image frame based on the sparse visual concept semantic matrix and the original visual feature map of the traffic image frame includes: The original visual feature map of the traffic image frame is subjected to feature-dense representation to obtain the dense feature representation result of the traffic image frame. Calculate the feature cosine similarity between the sparse visual concept semantic matrix and the dense feature representation results of the traffic image frame when projected onto the same shared embedding space; The calculated feature cosine similarity is normalized using the Softmax function with a temperature parameter to obtain the visual concept distribution confidence matrix of the traffic image frame.

[0008] In an optional implementation, the step of calculating the overall spatial position change intensity of each sparse traffic visual concept in the target long-term traffic video sequence based on the visual concept distribution confidence matrix of each traffic image in the target long-term traffic video sequence includes: For each sparse traffic visual concept, calculate the Euclidean norm of the difference matrix between the local confidence submatrices in the visual concept distribution confidence matrix of the sparse traffic visual concept in two adjacent traffic images. The average value of all Euclidean norm values ​​corresponding to this sparse traffic visual concept is calculated to obtain the overall spatial position change intensity of this sparse traffic visual concept at the target long-term traffic video sequence.

[0009] In an optional implementation, the target sparse traffic visual lexical sequence includes dynamic concept visual lexical features corresponding to each traffic image in the target long-term traffic video sequence; when the multiple traffic state perception dimensions include a long-term lane perception dimension, the step of learning sparse attention features across time scales on the target sparse traffic visual lexical sequence in the long-term lane perception dimension to obtain the first local attention output feature sequence corresponding to the long-term lane perception dimension includes: In the target long-time traffic video sequence, at least one frame of target traffic image is determined, wherein each frame of target traffic image corresponds to at least one frame of far-time historical traffic image in the target long-time traffic video sequence, and the number of frames of image interval between each frame of target traffic image and any corresponding frame of far-time historical traffic image is not less than the preset number of far-time division frames. For each frame of target traffic image, determine the mapping of dynamic conceptual visual lexical features of all distant temporal historical traffic images corresponding to that frame of target traffic image in the attention space to traffic semantic memory; Based on the prior information of the target lane road topology to which the target long-time traffic video sequence belongs, the mapping traffic semantic memory of all the far-time historical traffic images is structured, aggregated and compressed to obtain the far-time lane perception structured memory that matches the target traffic image of that frame. The long-term lane perception structured memory matched by the target traffic image of that frame is used as the attention output feature corresponding to the target traffic image of that frame in the first local attention output feature sequence.

[0010] In an optional implementation, where the multiple traffic state perception dimensions also include a key traffic event perception dimension, the step of performing cross-timescale sparse attention feature learning on the target sparse traffic visual word sequence in the key traffic event perception dimension to obtain the second local attention output feature sequence corresponding to the key traffic event perception dimension includes: For each frame of target traffic image, determine the mapping of the dynamic conceptual visual lexical features of the target traffic image in the attention space to the traffic semantic query vector; Based on the mapped traffic semantic query vector of the target traffic image, the long-term lane perception structured memory matched by the target traffic image, and the dynamic concept distribution confidence matrix of each of the long-term historical traffic images corresponding to the target traffic image, the traffic behavior correlation degree between each of the long-term historical traffic images and the target traffic image is calculated. The dynamic concept distribution confidence matrix of any long-term historical traffic image is a local submatrix in the visual concept distribution confidence matrix of the long-term historical traffic image that is related to all dynamic traffic visual concepts. Identify the target long-term historical traffic image whose traffic behavior correlation exceeds a preset correlation threshold, and extract the local structured memory corresponding to all target long-term historical traffic images from the long-term lane perception structured memory of the target traffic image in that frame. The extracted local structured memory is used as the attention output feature corresponding to the target traffic image in the second local attention output feature sequence.

[0011] In an optional implementation, where the multiple traffic state perception dimensions also include a near-temporal dynamic trajectory perception dimension, the step of performing cross-temporal-scale sparse attention feature learning on the target sparse traffic visual word sequence in the near-temporal dynamic trajectory perception dimension to obtain the third local attention output feature sequence corresponding to the near-temporal dynamic trajectory perception dimension includes: For each target traffic image frame, the number of consecutive historical image frames of the target traffic image frame is calculated based on the actual traffic density of the target traffic image frame, wherein the number of consecutive historical image frames of any target traffic image frame is less than the preset number of far-time division frames. Determine the dynamic concept visual lexical features of the target traffic image in the frame and their mapping traffic semantic query vector and mapping traffic semantic memory in the attention space. Also determine the mapping traffic semantic memory of all near-temporal historical traffic images adjacent to the target traffic image in the frame in the attention space. The total number of images of all near-temporal historical traffic images is the number of consecutive historical image frames of the target traffic image in the frame. Standard scaled dot product attention is performed on the mapped traffic semantic query vector and mapped traffic semantic memory of the target traffic image of the frame, as well as the mapped traffic semantic memory of each of the near-temporal historical traffic images, to obtain the near-temporal dynamic trajectory change features of the target traffic image of the frame. The near-temporal dynamic trajectory change features of the target traffic image in this frame are used as the attention output features corresponding to the target traffic image in the third local attention output feature sequence.

[0012] In an optional implementation, the step of fusing the local attention output feature sequences of the various traffic state perception dimensions according to the issued desired video semantic understanding task to obtain the target traffic video semantic feature sequence includes: For each frame of the target traffic image in the target long-time traffic video sequence, global average pooling is performed on the attention output features corresponding to the frame of the target traffic image in multiple local attention output feature sequences to obtain the hidden traffic semantic features of the frame of the target traffic image. The traffic task metadata vector of the desired video semantic understanding task at the target traffic image of the frame is determined, and the traffic perception gating network is invoked to perform feature fusion weight prediction based on the hidden traffic semantic features and the traffic task metadata vector to obtain the desired fusion weights of the multiple attention output features corresponding to the target traffic image of the frame. Based on the expected fusion weights of the multiple attention output features corresponding to the target traffic image of the frame, the multiple attention output features are weighted and fused to obtain the target traffic semantic features corresponding to the target traffic image of the frame in the target traffic video semantic feature sequence.

[0013] Secondly, this application provides a computer device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the long-time traffic video semantic understanding method described in any of the foregoing embodiments.

[0014] Thirdly, this application provides a readable storage medium storing a computer program thereon, which, when executed by a computer device, implements the long-time-series traffic video semantic understanding method described in any of the foregoing embodiments.

[0015] In this case, the beneficial effects of the embodiments of this application may include the following: This application performs dynamic traffic visual concept semantic detection on the acquired target long-term traffic video sequence to obtain a target sparse traffic visual word sequence involving dynamic traffic visual concepts, in order to minimize the redundant information interference brought by static traffic scene elements to the video semantic understanding operation. Then, sparse attention feature learning is performed on the target sparse traffic visual word sequence across multiple traffic state perception dimensions to obtain the local attention output feature sequences corresponding to each of the multiple traffic state perception dimensions. Finally, feature fusion is performed on the local attention output feature sequences of each of the multiple traffic state perception dimensions according to the assigned desired video semantic understanding task to obtain the target traffic video. The semantic feature sequence utilizes a hybrid sparse attention fusion mechanism (which is achieved by superimposing a hybrid sparse attention mechanism and a feature fusion mechanism) to realize a structured and compact representation of traffic video sequences that takes into account the semantic understanding task content. This reduces the redundant information interference caused by irrelevant traffic video segments (e.g., traffic images that do not involve key traffic behaviors or long-term traffic semantic evolution processes) to the video semantic understanding operation. Then, by calling a large language model to perform cross-modal attention interaction inference based on the target traffic video semantic feature sequence and the expected video semantic understanding task, the target semantic understanding result is obtained, thereby achieving an effective balance between the efficiency and accuracy of video semantic understanding.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the composition of a computer device provided in the embodiments of this application; Figure 2 A flowchart illustrating the long-time traffic video semantic understanding method provided in this application embodiment; Figure 3 for Figure 2 A flowchart illustrating the execution process of step S210. Figure 4 for Figure 2 One of the flowcharts illustrating the execution process of step S220; Figure 5 for Figure 2 The second step in the execution flow diagram of step S220; Figure 6 for Figure 2 The third step in the execution flow diagram of step S220; Figure 7 for Figure 2 A schematic diagram of the execution flow of step S230.

[0019] Icons: 10-Computer equipment; 11-Memory; 12-Processor; 13-Communication unit. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] In the description of this application, it should be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are used only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the equipment or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0024] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0025] Furthermore, it is understood in the description of this application that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0026] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0027] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the device composition of the computer device 10 provided in this application embodiment. In this application embodiment, the computer device 10 can communicate with at least one traffic video surveillance device (deployed at key traffic nodes such as urban roads, intersections, or highway sections), and acquire long-term traffic video sequences (composed of multiple frames of traffic images continuously distributed in the time dimension) collected by each of the at least one traffic video surveillance device within a certain monitoring period. This allows the computer device to perform a traffic video semantic understanding task issued by an external user on the acquired long-term traffic video sequences, and generate corresponding matching video semantic understanding results (which typically include information such as traffic event text descriptions, traffic behavior recognition results, and dynamic target motion trajectory prediction results). The computer device 10 can be, but is not limited to, a server, a personal computer, a laptop computer, etc.

[0028] In this embodiment of the application, the computer device 10 may include a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0029] In this embodiment, the memory 11 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 11 is used to store computer programs, and the processor 12 can execute the computer programs accordingly after receiving execution instructions.

[0030] In this embodiment, the processor 12 can be an integrated circuit chip with signal processing capabilities. The processor 12 can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment.

[0031] In this embodiment, the communication unit 13 is used to establish a communication connection between the computer device 10 and other electronic devices via a network, and to send and receive data via the network, wherein the network includes wired communication networks and wireless communication networks. For example, the computer device 10 can obtain a long-term traffic video sequence uploaded by any traffic video surveillance device through the communication unit 13.

[0032] In this embodiment, the computer device 10 may pre-store a specific computer program related to the semantic understanding function of long-time traffic video in the memory 11. By driving the processor 12 to execute the specific computer program, a structured and compact representation of the semantic understanding task content of long-time traffic video sequences is achieved through the organic combination of sparse semantic reconstruction operation for dynamic traffic visual concepts (i.e., dynamic traffic participation elements with significant temporal motion characteristics) and a hybrid sparse attention fusion mechanism that considers multiple traffic state perception dimensions (including long-time lane perception dimension, key traffic event perception dimension and / or near-time dynamic trajectory perception dimension). This minimizes redundant information interference and facilitates further use of large language models to improve the efficiency and capability of video semantic understanding of long-time traffic videos, achieving an effective balance between video semantic understanding efficiency and accuracy.

[0033] Understandable, Figure 1 The block diagram shown is only a schematic diagram of one configuration of the computer device 10. The computer device 10 may also include components such as... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0034] In this application, to ensure that the computer device 10 can effectively balance video semantic understanding efficiency and accuracy during long-term traffic video semantic understanding, this application provides a long-term traffic video semantic understanding method to achieve the aforementioned objective. The long-term traffic video semantic understanding method provided in this application will be described in detail below.

[0035] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the long-time-series traffic video semantic understanding method provided in this application embodiment. In this application embodiment, the long-time-series traffic video semantic understanding method may include steps S210 to S240.

[0036] Step S210: Perform dynamic traffic visual concept semantic detection on the acquired target long-time traffic video sequence to obtain a target sparse traffic visual word sequence involving dynamic traffic visual concepts.

[0037] In this embodiment, the target long-time traffic video sequence is a long-time traffic video sequence that needs to perform a desired video semantic understanding task (which is issued by an external user); the target long-time traffic video sequence consists of multiple consecutive frames of traffic images taken at preset time intervals; the computer device 10 can perform image preprocessing such as time synchronization, inter-frame alignment, and image scale normalization on the acquired target long-time traffic video sequence to eliminate the noise effects caused by factors such as camera shake, resolution differences, and inconsistent frame rates.

[0038] After acquiring the target long-term traffic video sequence, the computer device 10 can perform dynamic traffic visual concept semantic detection on the target long-term traffic video sequence. This is achieved by utilizing sparse semantic reconstruction operations targeting dynamic traffic visual concepts to eliminate the influence of static traffic scene elements in the target long-term traffic video sequence. The resulting target sparse traffic visual lexical sequence is then considered key visual feature information that plays a decisive role in traffic video semantic understanding operations (including but not limited to: traffic behavior analysis, abnormal traffic event identification, and dynamic target trajectory prediction), thus minimizing redundant information interference from static traffic scene elements. Specifically, the target sparse traffic visual lexical sequence is time-aligned with the target long-term traffic video sequence. Each frame of traffic image in the target long-term traffic video sequence corresponds to a unique dynamic concept visual lexical feature at the target sparse traffic visual lexical sequence (which describes the semantic embedding vector of each dynamic traffic visual concept at that frame of traffic image, and the distribution confidence status of each dynamic traffic visual concept at each image position in that frame of traffic image).

[0039] Alternatively, please refer to Figure 3 , Figure 3 yes Figure 2 A schematic diagram of the execution flow of step S210. In this embodiment of the application, step S210 may include sub-steps S211 to S214 to ensure that the constructed target sparse traffic visual word sequence can eliminate the influence of static traffic scene elements, thereby minimizing the redundant information interference brought by static traffic scene elements to the video semantic understanding operation.

[0040] Sub-step S211: For each frame of traffic image included in the target long-term traffic video sequence, extract the original visual feature map of the traffic image of that frame and perform sparse traffic visual concept modeling to obtain the sparse visual concept semantic matrix of the traffic image of that frame.

[0041] In this embodiment, for each frame of traffic image in the target long-term traffic video sequence, a pre-trained visual feature encoder (which can be implemented using a visual Transformer network) is invoked to extract high-dimensional semantic features from the frame, obtaining the original visual feature map of the frame. Then, a sparse concept encoder, which incorporates multiple learnable latent query vectors (each latent query vector corresponds to a sparse traffic visual concept), is invoked to jointly model the original visual feature map of the frame. The forward propagation mechanism of the sparse concept encoder is then used to directly output the sparse visual concept semantic matrix of the frame. The sparse concept encoder involves multiple sparse traffic visual concepts, which may include common static traffic scene elements (e.g., traffic signs, road markings) and dynamic traffic participation elements (e.g., vehicles, pedestrians). The sparse visual concept semantic matrix of any frame of traffic image records the semantic embedding vectors of each of the multiple sparse traffic visual concepts at that traffic image location.

[0042] Sub-step S212: Based on the sparse visual concept semantic matrix and the original visual feature map of the traffic image frame, calculate the visual concept distribution confidence matrix of the traffic image frame.

[0043] In this embodiment, after determining the sparse visual concept semantic matrix and the original visual feature map of any traffic image frame, the original visual feature map of the traffic image frame can be subjected to feature densification representation to obtain the dense feature representation result of the traffic image frame. Then, the dense feature representation result and the sparse visual concept semantic matrix of the traffic image frame are projected into the same shared embedding space, and the feature cosine similarity between the sparse visual concept semantic matrix and the dense feature representation result of the traffic image frame under the same shared embedding space is calculated. At this time, the calculated feature cosine similarity can be normalized by the Softmax function with temperature parameter to obtain the visual concept distribution confidence matrix of the traffic image frame. Among them, the visual concept distribution confidence matrix of any traffic image frame records the distribution confidence values ​​of the various sparse traffic visual concepts at each image position in the traffic image.

[0044] Sub-step S213: Based on the visual concept distribution confidence matrix of each traffic image in the target long-term traffic video sequence, calculate the overall spatial position change intensity of each sparse traffic visual concept in the target long-term traffic video sequence, so as to determine all dynamic traffic visual concepts involved in the target long-term traffic video sequence.

[0045] In this embodiment, after determining the visual concept distribution confidence matrix of each traffic image in the target long-term traffic video sequence, the visual concept distribution confidence matrix of each adjacent two traffic images in all traffic images can be modeled temporally based on the target lane road topology prior information of the target long-term traffic video sequence (i.e., the real road topology prior information of the traffic key nodes observed in the target long-term traffic video sequence, which usually describes the lane line geometry information, lane connection relationship information and allowed driving direction constraint information of the traffic key node, so as to effectively limit the legal movement space range of various dynamic traffic participation elements at the traffic key node). In order to use the change amplitude of the visual concept distribution confidence matrix between adjacent two traffic images, the stability of the positional distribution of the above-mentioned sparse traffic visual concepts within the time period represented by the adjacent two traffic images can be characterized. Therefore, by applying the aforementioned method for calculating the change amplitude of the confidence matrix of visual concept distribution to the entire target long-term traffic video sequence, the overall spatial position change intensity of each of the various sparse traffic visual concepts within the overall time period represented by the entire target long-term traffic video sequence can be determined. This allows the overall spatial position change intensity to characterize the dynamic change characteristics of the corresponding sparse traffic visual concept within the aforementioned overall time period, thereby achieving decoupling between dynamic traffic visual concepts (corresponding to dynamic traffic participation elements) and static background visual concepts (corresponding to static traffic scene elements). Specifically, the dynamic traffic visual concept is a sparse traffic visual concept whose overall spatial position change intensity exceeds a preset change intensity threshold, while the static background visual concept is a sparse traffic visual concept whose overall spatial position change intensity is less than or equal to the preset change intensity threshold.

[0046] In this process, the step "calculating the overall spatial location change intensity of each of the sparse traffic visual concepts at the target long-term traffic video sequence" in sub-step S213 above may include: For each sparse traffic visual concept, calculate the Euclidean norm of the difference matrix between the local confidence submatrices at the visual concept distribution confidence matrix of two adjacent traffic images in all traffic images for that sparse traffic visual concept. The average value of all Euclidean norm values ​​corresponding to this sparse traffic visual concept is calculated to obtain the overall spatial position change intensity of this sparse traffic visual concept at the target long-term traffic video sequence.

[0047] The visual concept distribution confidence matrix of any traffic image frame can be regarded as the result of splicing together the local confidence submatrices of all the above sparse traffic visual concepts. The local confidence submatrices of each sparse traffic visual concept record the distribution confidence value of the sparse traffic visual concept at each image position in the corresponding traffic image.

[0048] Sub-step S214: For each frame of traffic image, extract the local sub-matrices related to all dynamic traffic visual concepts in the sparse visual concept semantic matrix and visual concept distribution confidence matrix of the traffic image of that frame, and perform matrix concatenation to obtain the dynamic concept visual word features of the traffic image of that frame in the target sparse traffic visual word sequence.

[0049] In this embodiment, after determining all dynamic traffic visual concepts involved in the target long-term traffic video sequence, for each frame of traffic image, a first local sub-matrix related to all dynamic traffic visual concepts can be extracted from the sparse visual concept semantic matrix of that frame of traffic image, and a second local sub-matrix related to all dynamic traffic visual concepts can be extracted from the visual concept distribution confidence matrix of that frame of traffic image. Then, the second local sub-matrix is ​​transposed, and the first local sub-matrix and the transposed second local sub-matrix are concatenated to obtain the dynamic concept visual lexical features corresponding to that frame of traffic image in the target sparse traffic visual lexical sequence.

[0050] Therefore, by executing the above sub-steps S211 to S214, this application utilizes the sparse semantic reconstruction operation of dynamic traffic visual concepts to effectively eliminate the influence of static traffic scene elements in the corresponding constructed target sparse traffic visual word sequence, thereby minimizing the redundant information interference brought by static traffic scene elements to the video semantic understanding operation.

[0051] Step S220: Perform sparse attention feature learning across time scales on the target sparse traffic visual word sequence in multiple traffic state perception dimensions to obtain the local attention output feature sequences corresponding to each of the multiple traffic state perception dimensions.

[0052] In this embodiment, the multiple traffic state perception dimensions may include a long-term lane perception dimension, a key traffic event perception dimension, and / or a near-term dynamic trajectory perception dimension. Each of these multiple traffic state perception dimensions corresponds to a different time scale, enabling collaborative modeling of dynamic conceptual traffic semantic information at different time scales using a hybrid sparse attention mechanism to obtain the local attention output feature sequences corresponding to each of the multiple traffic state perception dimensions. Specifically, the long-term lane perception dimension is used to model lane direction perception details within a long-term time scale; the key traffic event perception dimension is used to achieve accurate localization and modeling of key traffic events within a long-term time scale; and the near-term dynamic trajectory perception dimension is used to perform high-resolution modeling of the local motion trajectory changes of dynamic traffic participants within a near-term time scale.

[0053] Alternatively, please refer to Figure 4 , Figure 4 yes Figure 2 One of the step execution flowcharts for step S220. In this embodiment of the application, for the far-time lane perception dimension included in the above-mentioned multiple traffic state perception dimensions, the step corresponding to the far-time lane perception dimension in step S220, "performing cross-timescale sparse attention feature learning on the target sparse traffic visual word sequence in the far-time lane perception dimension to obtain the first local attention output feature sequence corresponding to the far-time lane perception dimension", may include sub-steps S221 to S224, so as to achieve accurate modeling effect of lane direction perception details based on the target sparse traffic visual word sequence within the far-time scale.

[0054] Sub-step S221: Determine at least one frame of target traffic image in the target long-time traffic video sequence.

[0055] In this embodiment, each frame of the target traffic image in the target long-time traffic video sequence corresponds to at least one frame of far-time historical traffic image, and the number of frames between each frame of the target traffic image and any corresponding frame of far-time historical traffic image is not less than a preset number of far-time division frames (e.g., 6). There is an intersection between the far-time historical traffic image sets of different target traffic images.

[0056] Sub-step S222: For each frame of target traffic image, determine the mapping of the dynamic concept visual lexical features of all distant temporal historical traffic images corresponding to that frame of target traffic image in the attention space as traffic semantic memory.

[0057] In this embodiment, for each frame of traffic image in the target long-term traffic video sequence, the dynamic concept visual lexical features corresponding to the frame of traffic image can be mapped to the attention space using a linear mapping function to obtain the mapped traffic semantic query vector, mapped traffic semantic key vector, and mapped traffic semantic value vector of the frame of traffic image in the attention space. The mapped traffic semantic memory of the frame of traffic image in the attention space can be characterized by combining the mapped traffic semantic key vector and the mapped traffic semantic value vector.

[0058] Sub-step S223: Based on the prior information of the target lane road topology to which the target long-time traffic video sequence belongs, the mapping traffic semantic memory of all far-time historical traffic images is structurally aggregated and compressed to obtain the far-time lane perception structured memory that matches the target traffic image of that frame.

[0059] In this embodiment, for each frame of far-time historical traffic image corresponding to any target traffic image, the target lane road topology prior information can be used to perform lane assignment mapping on each visual feature word in the corresponding original visual feature map (where each visual feature word corresponds to an image position in the original visual feature map), so as to project each visual feature word onto the corresponding lane coordinate system. Then, a one-dimensional pooling compression operation is performed on the mapped traffic semantic memory of the far-time historical traffic image along the lane line direction recorded by the target lane road topology prior information, while maintaining a high spatial resolution in the vertical direction of the lane line direction, so as to construct an adapted asymmetric spatial pooling function for lane direction perception capability, thereby realizing the memory structured aggregation and compression function of the far-time historical traffic image, and obtaining the local structured memory of the far-time historical traffic image considering lane direction perception details. At this time, the far-time lane perception structured memory of the target traffic image is composed of the local structured memories of all far-time historical traffic images corresponding to the target traffic image.

[0060] Sub-step S224: The far-temporal lane perception structured memory matched by the target traffic image of the frame is used as the attention output feature corresponding to the target traffic image of the frame in the first local attention output feature sequence.

[0061] Therefore, by executing the above sub-steps S221 to S224, this application can achieve accurate modeling of lane direction perception details based on the target sparse traffic visual lexical sequence within a long temporal scale.

[0062] Alternatively, please refer to Figure 5 , Figure 5 yes Figure 2The second step in the execution flow diagram of step S220 is shown below. In this embodiment of the application, for the key traffic event perception dimension included in the above-mentioned multiple traffic state perception dimensions, the step corresponding to the key traffic event perception dimension in step S220, "performing cross-timescale sparse attention feature learning on the target sparse traffic visual word sequence in the key traffic event perception dimension to obtain the second local attention output feature sequence corresponding to the key traffic event perception dimension", may include sub-steps S225 to S228, so as to achieve accurate localization and modeling of key traffic events based on the target sparse traffic visual word sequence within a long-term time scale.

[0063] Sub-step S225: For each frame of target traffic image, determine the mapping of the dynamic concept visual lexical features of the target traffic image in the attention space to the traffic semantic query vector.

[0064] Sub-step S226: Based on the mapped traffic semantic query vector of the target traffic image frame, the long-term lane perception structured memory matched by the target traffic image frame, and the dynamic concept distribution confidence matrix of each of the long-term historical traffic images corresponding to the target traffic image frame, calculate the traffic behavior correlation degree between each of the long-term historical traffic images and the target traffic image frame.

[0065] In this embodiment, the dynamic concept distribution confidence matrix of any frame of far-time historical traffic image is a local sub-matrix in the visual concept distribution confidence matrix of that frame of far-time historical traffic image that is related to all dynamic traffic visual concepts. That is, the dynamic concept distribution confidence matrix of any frame of far-time historical traffic image is the second local sub-matrix corresponding to that frame of far-time historical traffic image.

[0066] Specifically, for each frame of long-term historical traffic image, the position change velocity vector and position change acceleration vector of all dynamic traffic visual concepts at that frame are determined by comparing the second local sub-matrix of that frame with the second local sub-matrix of the previous frame. Then, a pre-constructed motion saliency scoring function is called to calculate the motion saliency score based on the mapped traffic semantic query vector of the corresponding target traffic image, the mapped traffic semantic key vector of that frame, and the position change velocity vector and position change acceleration vector of all dynamic traffic visual concepts at that frame. This yields the traffic behavior correlation degree between that frame and the corresponding target traffic image. A higher motion saliency score indicates a higher traffic behavior correlation between the corresponding long-term historical traffic image and the target traffic image.

[0067] Sub-step S227: Determine the target long-term historical traffic image whose traffic behavior correlation exceeds the preset correlation threshold, and extract the local structured memory corresponding to all target long-term historical traffic images from the long-term lane perception structured memory of the target traffic image.

[0068] Sub-step S228: The extracted local structured memory is used as the attention output feature corresponding to the target traffic image in the second local attention output feature sequence.

[0069] Therefore, by executing the above sub-steps S225 to S228, this application can achieve accurate localization and modeling of key traffic events based on the target sparse traffic visual word sequence within a long time scale.

[0070] Alternatively, please refer to Figure 6 , Figure 6 yes Figure 2 The third step in the execution flow diagram of step S220. In this embodiment of the application, for the near-temporal dynamic trajectory perception dimension included in the above-mentioned multiple traffic state perception dimensions, the step corresponding to the near-temporal dynamic trajectory perception dimension in step S220, "performing cross-timescale sparse attention feature learning on the target sparse traffic visual word sequence in the near-temporal dynamic trajectory perception dimension to obtain the third local attention output feature sequence corresponding to the near-temporal dynamic trajectory perception dimension", may include sub-steps S229 to S2212, so as to achieve high-resolution modeling of the local motion trajectory change details of dynamic traffic participation elements based on the target sparse traffic visual word sequence within the near-temporal time scale.

[0071] Sub-step S229: For each target traffic image frame, calculate the number of consecutive historical image frames of the target traffic image frame based on the actual traffic density of the target traffic image frame.

[0072] In this embodiment, the actual traffic density of the target traffic image can be determined by statistically analyzing the number of dynamic traffic participants or traffic flow information within the image acquisition period corresponding to the target traffic image frame. Then, a pre-trained near-time series image frame number prediction function is used to predict the number of consecutive historical image frames of the target traffic image frame based on the actual traffic density, thereby obtaining the number of consecutive historical image frames of the target traffic image frame. The number of consecutive historical image frames for any given target traffic image frame is less than the preset far-time series frame number, and the number of consecutive historical image frames is positively correlated with the corresponding actual traffic density.

[0073] Sub-step S2210: Determine the dynamic concept visual lexical features of the target traffic image in the frame and their mapping traffic semantic query vector and mapping traffic semantic memory in the attention space, and determine the mapping traffic semantic memory of all near-temporal historical traffic images adjacent to the target traffic image in the frame in the attention space.

[0074] The total number of frames in all near-time-series historical traffic images is the number of consecutive historical image frames of the target traffic image in that frame.

[0075] Sub-step S2211: Perform standard scaled dot product attention calculation on the mapped traffic semantic query vector and mapped traffic semantic memory of the target traffic image of the frame, as well as the mapped traffic semantic memory of each of the near-temporal historical traffic images, to obtain the near-temporal dynamic trajectory change features of the target traffic image of the frame.

[0076] Sub-step S2212: The near-temporal dynamic trajectory change features of the target traffic image of the frame are used as the attention output features of the target traffic image of the frame in the third local attention output feature sequence.

[0077] Therefore, by executing the above sub-steps S229 to S2212, this application can achieve high-resolution modeling of the local motion trajectory changes of dynamic traffic participants based on the target sparse traffic visual lexical sequence within a near-temporal time scale.

[0078] Step S230: Based on the given desired video semantic understanding task, feature fusion is performed on the local attention output feature sequences of various traffic state perception dimensions to obtain the target traffic video semantic feature sequence.

[0079] In this embodiment, after obtaining the local attention output feature sequences of each of the multiple traffic state perception dimensions, these multiple local attention output feature sequences can be uniformly mapped to the same feature dimension and token representation space to obtain multiple local attention output feature sequences aligned with the feature dimensions. Then, using the specific task content of the desired video semantic understanding task, the dynamic feature fusion weights of each of these multiple local attention output feature sequences at different target traffic images can be predicted. This ensures that the final target traffic video semantic feature sequence is essentially a structured and compact representation of the traffic video sequence considering the semantic understanding task content, thereby reducing the redundant information interference caused by irrelevant traffic video segments (e.g., traffic images that do not involve key traffic behaviors or long-term traffic semantic evolution processes) to the video semantic understanding operation.

[0080] Alternatively, please refer to Figure 7 , Figure 7 yes Figure 2A schematic diagram of the execution flow of step S230. In this embodiment of the application, step S230 may include sub-steps S231 to S233 to obtain a structured and compact representation of the traffic video sequence that takes into account the semantic understanding task content using a feature fusion mechanism, thereby reducing the redundant information interference caused by irrelevant traffic video segments to the video semantic understanding operation.

[0081] Sub-step S231: For each frame of the target traffic image in the target long-time traffic video sequence, perform global average pooling on the attention output features corresponding to the target traffic image in multiple local attention output feature sequences to obtain the hidden traffic semantic features of the target traffic image.

[0082] The hidden traffic semantic features of any frame of target traffic image are obtained by global average pooling of multiple attention output features corresponding to that frame of target traffic image (each attention output feature belongs to a local attention output feature sequence).

[0083] Sub-step S232: Determine the traffic task metadata vector of the desired video semantic understanding task at the target traffic image of the frame, and call the traffic perception gating network to perform feature fusion weight prediction based on the hidden traffic semantic features and the traffic task metadata vector to obtain the desired fusion weights of the multiple attention output features corresponding to the target traffic image of the frame.

[0084] In this embodiment, the desired video semantic understanding task can be decomposed to obtain a traffic task metadata vector sequence that is time-aligned with any local attention output feature sequence (it consists of traffic task metadata vectors corresponding to all target traffic images in the target long-time traffic video sequence, wherein the traffic task metadata vector of each frame of the target traffic image records scene type information of the corresponding traffic scene, task type identifier of the semantic understanding decomposition subtask, real-time traffic status statistics, context summary information of the queried historical traffic events, etc.); the sum of the desired fusion weights of the multiple attention output features corresponding to any frame of the target traffic image is 1.

[0085] Sub-step S233: Based on the expected fusion weights of the multiple attention output features corresponding to the target traffic image of the frame, perform weighted fusion processing on the multiple attention output features to obtain the target traffic semantic features corresponding to the target traffic image of the frame in the target traffic video semantic feature sequence.

[0086] The target traffic video semantic feature sequence is composed of the target traffic semantic features of each target traffic image in the target long-time-series traffic video sequence.

[0087] Therefore, by executing the above sub-steps S231 to S233, this application can obtain a structured and compact representation of traffic video sequences that takes into account the semantic understanding task content through feature fusion mechanism, thereby reducing the redundant information interference caused by irrelevant traffic video segments to video semantic understanding operations.

[0088] Step S240: Call the large language model to perform cross-modal attention interaction reasoning based on the semantic feature sequence of the target traffic video and the expected video semantic understanding task to obtain the target semantic understanding result.

[0089] In this embodiment, after determining the target traffic video semantic feature sequence, the desired video semantic understanding task can be processed by text instruction decomposition to obtain a task text instruction decomposition sequence that is time-aligned with the target traffic video semantic feature sequence (which records the semantic understanding instruction text information to be executed configured for each of the target traffic images). At this time, the task text instruction decomposition sequence and the target traffic video semantic feature sequence can be directly input into the large language model. The large language model uses a cross-modal attention interaction mechanism to realize joint reasoning between traffic visual semantics and text semantics to obtain a structured target semantic understanding result (which includes information such as traffic event text description, traffic behavior recognition results, and dynamic target motion trajectory prediction results that match the desired video semantic understanding task).

[0090] Therefore, by executing the above steps S210 to S240, this application can achieve a structured and compact representation of the semantic understanding task content of long-term traffic video sequences by organically combining the sparse semantic reconstruction operation for dynamic traffic visual concepts with the hybrid sparse attention fusion mechanism that considers multiple traffic state perception dimensions. This reduces redundant information interference as much as possible, and facilitates further use of large language models to improve the efficiency and capability of video semantic understanding of long-term traffic videos, thus achieving an effective balance between video semantic understanding efficiency and accuracy.

[0091] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of the apparatus, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0093] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for semantic understanding of long-term traffic videos, characterized in that, The method includes: Dynamic traffic visual concept semantic detection is performed on the acquired target long-time traffic video sequence to obtain a target sparse traffic visual word sequence involving dynamic traffic visual concepts. Sparse attention features are learned across time scales for the target sparse traffic visual word sequence in multiple traffic state perception dimensions to obtain the local attention output feature sequences corresponding to each of the multiple traffic state perception dimensions. Based on the assigned desired video semantic understanding task, feature fusion is performed on the local attention output feature sequences of each of the multiple traffic state perception dimensions to obtain the target traffic video semantic feature sequence. The large language model is invoked to perform cross-modal attention interaction reasoning based on the target traffic video semantic feature sequence and the expected video semantic understanding task, so as to obtain the target semantic understanding result.

2. The method according to claim 1, characterized in that, The step of performing dynamic traffic visual concept semantic detection on the acquired target long-term traffic video sequence to obtain a target sparse traffic visual lexical sequence involving dynamic traffic visual concepts includes: For each frame of traffic image included in the target long-term traffic video sequence, the original visual feature map of the traffic image is extracted to perform sparse traffic visual concept modeling, and the sparse visual concept semantic matrix of the traffic image is obtained. Based on the sparse visual concept semantic matrix and the original visual feature map of the traffic image frame, calculate the visual concept distribution confidence matrix of the traffic image frame. Based on the visual concept distribution confidence matrix of each traffic image in the target long-term traffic video sequence, calculate the overall spatial position change intensity of each sparse traffic visual concept in the target long-term traffic video sequence, so as to determine all dynamic traffic visual concepts involved in the target long-term traffic video sequence, wherein the overall spatial position change intensity corresponding to each dynamic traffic visual concept exceeds a preset change intensity threshold. For each frame of traffic image, the local sub-matrices related to all dynamic traffic visual concepts in the sparse visual concept semantic matrix and visual concept distribution confidence matrix of the traffic image are extracted and concatenated to obtain the dynamic concept visual lexical features of the traffic image in the target sparse traffic visual lexical sequence.

3. The method according to claim 2, characterized in that, The step of calculating the visual concept distribution confidence matrix of the traffic image frame based on the sparse visual concept semantic matrix and the original visual feature map of the traffic image frame includes: The original visual feature map of the traffic image frame is subjected to feature-dense representation to obtain the dense feature representation result of the traffic image frame. Calculate the feature cosine similarity between the sparse visual concept semantic matrix and the dense feature representation results of the traffic image frame when projected onto the same shared embedding space; The calculated feature cosine similarity was normalized using the Softmax function with a temperature parameter to obtain the visual concept distribution confidence matrix of the traffic image frame.

4. The method according to claim 2, characterized in that, The step of calculating the overall spatial position change intensity of each sparse traffic visual concept in the target long-term traffic video sequence based on the visual concept distribution confidence matrix of each traffic image in the target long-term traffic video sequence includes: For each sparse traffic visual concept, calculate the Euclidean norm of the difference matrix between the local confidence submatrices in the visual concept distribution confidence matrix of the sparse traffic visual concept in two adjacent traffic images. The average value of all Euclidean norm values ​​corresponding to this sparse traffic visual concept is calculated to obtain the overall spatial position change intensity of this sparse traffic visual concept at the target long-term traffic video sequence.

5. The method according to any one of claims 1-4, characterized in that, The target sparse traffic visual lexical sequence includes the dynamic concept visual lexical features corresponding to each traffic image in the target long-time traffic video sequence. When the multiple traffic state perception dimensions include a long-term lane perception dimension, the step of learning sparse attention features across time scales on the target sparse traffic visual word sequence in the long-term lane perception dimension to obtain the first local attention output feature sequence corresponding to the long-term lane perception dimension includes: In the target long-time traffic video sequence, at least one frame of target traffic image is determined, wherein each frame of target traffic image corresponds to at least one frame of far-time historical traffic image in the target long-time traffic video sequence, and the number of frames of image interval between each frame of target traffic image and any corresponding frame of far-time historical traffic image is not less than the preset number of far-time division frames. For each frame of target traffic image, determine the mapping of dynamic conceptual visual lexical features of all distant temporal historical traffic images corresponding to that frame of target traffic image in the attention space to traffic semantic memory; Based on the prior information of the target lane road topology to which the target long-time traffic video sequence belongs, the mapping traffic semantic memory of all the far-time historical traffic images is structured, aggregated and compressed to obtain the far-time lane perception structured memory that matches the target traffic image of that frame. The long-term lane perception structured memory matched by the target traffic image of that frame is used as the attention output feature corresponding to the target traffic image of that frame in the first local attention output feature sequence.

6. The method according to claim 5, characterized in that, When the multiple traffic state perception dimensions also include a key traffic event perception dimension, the step of performing cross-timescale sparse attention feature learning on the target sparse traffic visual word sequence in the key traffic event perception dimension to obtain the second local attention output feature sequence corresponding to the key traffic event perception dimension includes: For each frame of target traffic image, determine the mapping of the dynamic conceptual visual lexical features of the target traffic image in the attention space to the traffic semantic query vector; Based on the mapped traffic semantic query vector of the target traffic image, the long-term lane perception structured memory matched by the target traffic image, and the dynamic concept distribution confidence matrix of each of the long-term historical traffic images corresponding to the target traffic image, the traffic behavior correlation degree between each of the long-term historical traffic images and the target traffic image is calculated. The dynamic concept distribution confidence matrix of any long-term historical traffic image is a local submatrix in the visual concept distribution confidence matrix of the long-term historical traffic image that is related to all dynamic traffic visual concepts. Identify the target long-term historical traffic image whose traffic behavior correlation exceeds a preset correlation threshold, and extract the local structured memory corresponding to all target long-term historical traffic images from the long-term lane perception structured memory of the target traffic image in that frame. The extracted local structured memory is used as the attention output feature corresponding to the target traffic image in the second local attention output feature sequence.

7. The method according to claim 5, characterized in that, When the multiple traffic state perception dimensions also include a near-temporal dynamic trajectory perception dimension, sparse attention features are learned across time scales on the target sparse traffic visual word sequence in the near-temporal dynamic trajectory perception dimension. The steps for obtaining the third local attention output feature sequence corresponding to the near-temporal dynamic trajectory perception dimension include: For each target traffic image frame, the number of consecutive historical image frames of the target traffic image frame is calculated based on the actual traffic density of the target traffic image frame, wherein the number of consecutive historical image frames of any target traffic image frame is less than the preset number of far-time division frames. Determine the dynamic concept visual lexical features of the target traffic image in the frame and their mapping traffic semantic query vector and mapping traffic semantic memory in the attention space. Also determine the mapping traffic semantic memory of all near-temporal historical traffic images adjacent to the target traffic image in the frame in the attention space. The total number of images of all near-temporal historical traffic images is the number of consecutive historical image frames of the target traffic image in the frame. Standard scaled dot product attention is performed on the mapped traffic semantic query vector and mapped traffic semantic memory of the target traffic image of the frame, as well as the mapped traffic semantic memory of each of the near-temporal historical traffic images, to obtain the near-temporal dynamic trajectory change features of the target traffic image of the frame. The near-temporal dynamic trajectory change features of the target traffic image in this frame are used as the attention output features corresponding to the target traffic image in the third local attention output feature sequence.

8. The method according to any one of claims 1-4, characterized in that, The step of fusing the local attention output feature sequences of the various traffic state perception dimensions according to the issued expected video semantic understanding task to obtain the target traffic video semantic feature sequence includes: For each frame of the target traffic image in the target long-time traffic video sequence, global average pooling is performed on the attention output features corresponding to the frame of the target traffic image in multiple local attention output feature sequences to obtain the hidden traffic semantic features of the frame of the target traffic image. The traffic task metadata vector of the desired video semantic understanding task at the target traffic image of the frame is determined, and the traffic perception gating network is invoked to perform feature fusion weight prediction based on the hidden traffic semantic features and the traffic task metadata vector to obtain the desired fusion weights of the multiple attention output features corresponding to the target traffic image of the frame. Based on the expected fusion weights of the multiple attention output features corresponding to the target traffic image of the frame, the multiple attention output features are weighted and fused to obtain the target traffic semantic features corresponding to the target traffic image of the frame in the target traffic video semantic feature sequence.

9. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor, the processor being able to execute the computer program to implement the long-time traffic video semantic understanding method according to any one of claims 1-8.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a computer device, it implements the long-time-series traffic video semantic understanding method according to any one of claims 1-8.

Images