Traffic video processing analysis method and system

Abstract

The invention discloses a traffic video processing analysis method and system. The method comprises the following steps: acquiring collected video data; obtaining a coding feature sequence aligned with a text mode based on the video data; sampling based on the coding feature sequence to obtain a sampling sequence, and obtaining a video feature vector aligned with a text mode based on the sampling sequence; and analyzing and processing the video data based on the video feature vector. According to the scheme provided by the invention, the video and text information can be more efficiently associated, and the feature vector is generated by combining the static information and the dynamic information, so that the accuracy of subsequent application is further improved.

Description

Technical Field

[0001] This invention relates to the field of video processing, and more specifically to a traffic video processing and analysis method and system. Background Technology

[0002] Roadside cameras play a crucial role in traffic applications. They monitor road conditions 24 / 7, and effectively utilizing this video information has significant social and economic value. For example, natural language processing (NLP) can be used to retrieve traffic management videos, enabling case studies in traffic control and improving training efficiency; NLP can be used to retrieve accident-prone scenarios, building scenario datasets to aid in the development of intelligent vehicle planning and control algorithms; and NLP can be used to retrieve congestion videos, assisting in the analysis of congestion causes.

[0003] However, current methods for detecting video data often focus on extracting raw visual information, such as the color, texture, and shape of frame images. These features lack a direct alignment with text modalities, leading to inaccurate associations and low matching efficiency in applications involving video and text association, such as retrieving relevant video clips based on text descriptions. Furthermore, in terms of information extraction, existing methods struggle to efficiently filter out key information from complex video data, often including a large amount of redundant and irrelevant data. This not only increases the burden of data processing and reduces the efficiency of information extraction but also fails to accurately describe the video due to interference from irrelevant information. Summary of the Invention

[0004] In view of this, in order to overcome at least one aspect of the above problems, embodiments of the present invention propose a traffic video processing and analysis method, comprising the following steps: Acquire the collected video data; Based on the video data, an encoded feature sequence aligned with the text modality is obtained; A sampling sequence is obtained by sampling based on the encoded feature sequence, and a video feature vector aligned with the text modality is obtained based on the sampling sequence; The video data is analyzed and processed based on the video feature vector.

[0005] In some embodiments, analyzing and processing the video data based on the video feature vector further includes: In response to receiving a retrieval command, multiple image-coded features aligned with the text modality are obtained based on the video data; The similarity between the video data and the search text in the search instruction is calculated based on the multiple image encoding features and the video feature vector. Based on the similarity, video data corresponding to the searched text is determined.

[0006] In some embodiments, calculating the similarity between the video data and the search text in the search instruction based on the plurality of image coding features and the video feature vector further includes: Based on the retrieved text, a first text feature and a second text feature are obtained; A first similarity is obtained based on the first text feature and the video feature vector; a similarity with the second text feature is obtained based on each of the image encoding features; and a second similarity is obtained based on multiple similarities. Based on the first similarity and the second similarity, video data corresponding to the searched text is determined.

[0007] In some embodiments, obtaining a first text feature and a second text feature based on the retrieved text further includes: Static description text is obtained based on the retrieved text; The first text feature is obtained based on the retrieval text encoding, and the second text feature is obtained based on the static description text encoding.

[0008] In some embodiments, analyzing and processing the video data based on the video feature vector further includes: In response to receiving a description instruction, the feature vector is decoded based on the first decoder to obtain a text description of the video data.

[0009] In some embodiments, it also includes: A training set is constructed based on the feature vectors and the actual text descriptions; A loss function is constructed based on the predicted text description obtained from the first decoder and the corresponding actual text description. The first decoder is trained and optimized based on the loss function.

[0010] In some embodiments, sampling is performed based on the encoded feature sequence to obtain a sampling sequence, and a video feature vector aligned with the text modality is obtained based on the sampling sequence, further including: Based on the coded feature sequence, a first sub-sequence and a second sub-sequence are obtained, and a video feature vector aligned with the text modality is obtained based on the first sub-sequence and the second sub-sequence.

[0011] In some embodiments, a first sub-sequence and a second sub-sequence are obtained based on the encoded feature sequence, and a video feature vector aligned with the text modality is obtained based on the first sub-sequence and the second sub-sequence, further comprising: The first sub-sequence is obtained by sampling the encoded feature sequence based on the first adapter; The second sub-sequence is obtained by sampling the encoded feature sequence based on the second adapter, wherein the convolution kernel in the second adapter is smaller than the convolution kernel in the first adapter, and the number of convolutional layers in the second adapter is smaller than the number of convolutional layers in the first adapter; The first subsequence and the second subsequence are encoded by the encoder to obtain a video feature vector aligned with the text modality.

[0012] In some embodiments, it also includes: A training set is constructed based on video data and actual text descriptions. An encoded feature sequence aligned with the text modality is obtained based on the video data, and text features are obtained based on the actual text. The feature vector is obtained based on the first adapter, the second adapter, the encoder, and the encoded feature sequence; Calculate the first loss function based on the feature vector and the text features; The first adapter, the second adapter, and the encoder are trained and optimized based on the first loss function.

[0013] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a traffic video processing and analysis system, comprising: The acquisition module is configured to acquire collected video data; The encoding module is configured to obtain an encoded feature sequence aligned with the text modality based on the video data; The sampling module is configured to sample based on the encoded feature sequence to obtain a sampling sequence, and to obtain a video feature vector aligned with the text modality based on the sampling sequence; The analysis module is configured to analyze and process the video data based on the video feature vector.

[0014] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer device, comprising: At least one processor; and The memory stores a computer program that can run on the processor, which, when executing the program, performs the steps of any of the traffic video processing and analysis methods described above.

[0015] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the traffic video processing and analysis methods described above.

[0016] This invention offers one of the following beneficial technical effects: The proposed solution processes video data to obtain an encoded feature sequence aligned with the text modality. Because the encoded feature sequence is directly aligned with the text features, it can more efficiently associate video and text information in subsequent applications such as accident analysis and data retrieval compared to traditional methods. Furthermore, this invention samples based on the encoded feature sequence, separating static and dynamic information within the sequence, thereby combining static and dynamic information to generate feature vectors, further improving the accuracy of subsequent applications. This enables the solution to accurately describe videos even when faced with complex backgrounds and numerous interfering targets. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the traffic video processing and analysis method provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the similarity calculation process between video and text provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating the similarity calculation process between images and static text provided for embodiments of the present invention; Figure 4 A schematic diagram of the video-text training process provided for embodiments of the present invention; Figure 5 A schematic diagram of the traffic video processing and analysis system provided in an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a computer device provided for an embodiment of the present invention; Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0020] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0021] According to one aspect of the present invention, embodiments of the present invention provide a traffic video processing and analysis method, such as... Figure 1 As shown, it may include the following steps: S1, acquire the collected video data; S2, Based on the video data, obtain an encoded feature sequence aligned with the text modality; S3, sampling is performed based on the encoded feature sequence to obtain a sampling sequence, and a video feature vector aligned with the text modality is obtained based on the sampling sequence; S4, Analyze and process the video data based on the video feature vector.

[0022] Specifically, raw video data consists of pixel information from consecutive frames, containing a large amount of redundant information. Directly sampling the video would retain a significant amount of non-critical information, requiring subsequent analysis to process this redundant data and increasing computational load. This is especially problematic in high-resolution video scenarios, where the introduction of irrelevant information can reduce processing efficiency. The proposed solution in this embodiment obtains an coded feature sequence aligned with the text modality from the video data, and then samples based on this sequence. Since the coded feature sequence is directly aligned with the text features, it can more efficiently correlate video and text information in subsequent applications such as accident analysis and data retrieval compared to traditional methods. Furthermore, sampling only targets the condensed features, significantly reducing the amount of data and substantially decreasing computational resource consumption in the S4 analysis stage, thus improving processing speed. For example, it can accurately and quickly describe the video when faced with complex backgrounds and numerous targets.

[0023] In some embodiments, analyzing and processing the video data based on the video feature vector further includes: In response to receiving a retrieval command, multiple image-coded features aligned with the text modality are obtained based on the video data; The similarity between the video data and the search text in the search instruction is calculated based on the multiple image encoding features and the video feature vector. Based on the similarity, video data corresponding to the searched text is determined.

[0024] Specifically, upon receiving a search command, multiple image-coded features aligned with the text modality are obtained for each frame of the video data. These image-coded features, along with the previously obtained video feature vector, are then used to calculate the similarity between the video data and the search text in the search command. The video feature vector provides global semantic information aligning the video to the text modality, while the multiple image-coded features associate specific content related to the text at the frame level. This combination ensures that the similarity calculation encompasses both the overall semantics of the video and the detailed features of each frame, thus more accurately measuring the degree of matching between the video and the search text. Finally, based on the calculated similarity results, the video data that best matches the search text is selected, completing the search process. This combination of multi-dimensional features (video feature vector and multiple image-coded features) improves search accuracy, enabling the rapid location of videos matching the search text content from a large amount of video data.

[0025] In some embodiments, calculating the similarity between the video data and the search text in the search instruction based on the plurality of image coding features and the video feature vector further includes: Based on the retrieved text, a first text feature and a second text feature are obtained; A first similarity is obtained based on the first text feature and the video feature vector; a similarity with the second text feature is obtained based on each of the image encoding features; and a second similarity is obtained based on multiple similarities. Based on the first similarity and the second similarity, video data corresponding to the searched text is determined.

[0026] Specifically, the image-text pre-training model CLIP (Contrastive Language-Image Pre-training) can initially unify images and text. In CLIP, images and their corresponding text descriptions are treated as positive sample pairs, while different images and text descriptions are treated as negative sample pairs. The model's goal is to make the feature representations of positive sample pairs as close as possible and the feature representations of negative sample pairs as far apart as possible in the feature space, thereby learning the semantic relationships between images and text. This allows for image-text alignment in CLIP.

[0027] When a search instruction including the search text is received for video retrieval, dynamic similarity and static similarity can be combined for comparison. Therefore, the process can be divided into two branches, calculating the two similarities separately. When calculating dynamic similarity, such as... Figure 2As shown, the CLIP Text Encoder can be used directly to encode the input search text to obtain the first text feature, and the CLIP Image Encoder can be used to encode the input video data to obtain the encoded feature sequence. Then, Video Fast and Video Slow are used for sampling to obtain a first subsequence including dynamic information and a second subsequence including static information, respectively. Finally, the Feature Encoder fuses the first and second subsequences to obtain the video feature vector. Since the feature vector obtained from the video data is also based on the CLIP Image Encoder, after obtaining the first text feature and the video feature vector, the similarity S between the first text feature and the video feature can be directly calculated. Video This yields the dynamic similarity between the text and the video, also known as the first similarity.

[0028] When calculating static similarity, such as Figure 3 As shown, an Action Text Filter is used to remove action-related content from the input search text, retaining only the static attribute descriptions. This static attribute description-only text is then input into a CLIP Text Encoder to obtain the second text feature. Each frame of the input video is encoded using a CLIP Image Encoder to obtain individual encoded features for each frame. Since a video dataset can include N frames, calculating the static similarity yields N similarities between the second text feature and each of the N frames. The average or maximum of these N similarities can be taken as the final static similarity. Thus, the static similarity S between the static text and the image can be obtained using the second text feature and the individual encoded features of each frame. Frame This is also known as the second similarity.

[0029] Finally, the overall similarity is obtained based on dynamic similarity and static similarity: S Fusion =S Video +W Frame *S Frame Among them, W Frame This indicates the fusion weight.

[0030] In some embodiments, obtaining a first text feature and a second text feature based on the retrieved text further includes: Static description text is obtained based on the retrieved text; The first text feature is obtained based on the retrieval text encoding, and the second text feature is obtained based on the static description text encoding.

[0031] Specifically, large language models (such as Tongyi Qianwen, DeepSeek, etc.) can be used to remove action-related content from the searched text. For example, "a blue car is driving on the road" becomes "a blue car is on the road" after removing action content, retaining only static information. This eliminates the influence of dynamic content when calculating static similarity, making the static similarity more accurate.

[0032] In some embodiments, analyzing and processing the video data based on the video feature vector further includes: In response to receiving a description instruction, the feature vector is decoded based on the first decoder to obtain a text description of the video data.

[0033] Specifically, when a description instruction is received for video description, since the feature vector obtained based on the video data has been aligned with the text features, the feature vector can be converted into a text description by decoding the feature vector through the first decoder, thereby obtaining a text description of the video data.

[0034] For example, when a roadside camera captures a video of a traffic accident, the video is processed according to the embodiment proposed in this invention to obtain a feature vector aligned with the text modality. Then, the feature vector is decoded using a first decoder to obtain a text description of the traffic accident video.

[0035] In some embodiments, it also includes: A training set is constructed based on the feature vectors and the actual text descriptions; A loss function is constructed based on the predicted text description obtained from the first decoder and the corresponding actual text description. The first decoder is trained and optimized based on the loss function.

[0036] Specifically, the first decoder, Feature Decoder, can be continuously trained and optimized by constructing a training set. When constructing the training set, multiple feature vectors and their corresponding text descriptions can be used as samples, or video data and their corresponding text descriptions can be used. If video data and their corresponding text descriptions are used as samples, during Feature Decoder training, the video data can first be converted into feature vectors aligned with the text modality. Then, the Feature Decoder decodes the feature vectors to obtain the predicted text descriptions. The loss function is calculated using the predicted text descriptions and the actual text descriptions, and finally, the Feature Decoder is optimized based on the loss function. The loss function can be cross-entropy loss, for example: T D =Decoder Feature (F V ) L T Reconstruct =Loss(T D ,T) Among them, F V It is an eigenvector, T D T is the predicted text description, and T is the truth value (the actual text description).

[0037] In some embodiments, S2, in obtaining the encoded feature sequence aligned with the text modality based on the video data, specifically, the alignment of images and text can be achieved in CLIP, and the image encoder (CLIPImage Encoder) in the CLIP model can be used to encode the video data to obtain the encoded feature sequence aligned with the text modality.

[0038] For example, if the input video is represented as V=[I1,I2,…,I…] N ], where N represents the number of video frames. The CLIP ImageEncoder is used to process each frame of image I. i Encoding is performed to obtain an encoded feature F. i =Encoder CLIPImage (I i ), where F i For a feature map that includes location information, all encoded features F i The set can be used to obtain the encoded feature sequence [F1, F2, ..., F]. N ].

[0039] In some embodiments, sampling is performed based on the encoded feature sequence to obtain a sampling sequence, and a video feature vector aligned with the text modality is obtained based on the sampling sequence, further including: Based on the coded feature sequence, a first sub-sequence and a second sub-sequence are obtained, and a video feature vector aligned with the text modality is obtained based on the first sub-sequence and the second sub-sequence.

[0040] Specifically, in the process of sampling based on the encoded feature sequence to obtain the sampling sequence, and further obtaining the video feature vector aligned with the text modality, a first sub-sequence and a second sub-sequence are first sampled from the encoded feature sequence aligned with the text modality. Since the encoded feature sequence itself is already aligned with the text features and condenses the key information related to the text in the video, both the first and second sub-sequences are core content highly related to the text modality, rather than irrelevant redundant data from the original video. When obtaining the video feature vector aligned with the text modality based on the first and second sub-sequences, the two sub-sequences can capture key information from the encoded feature sequence from different dimensions, avoiding important content that might be missed by sampling a single sub-sequence. This more comprehensively preserves the feature details related to the text, and the final video feature vector not only inherits the characteristic of the encoded feature sequence being aligned with the text modality, but also more accurately covers the core information related to the text in the video because the two sub-sequences can cover it more precisely. In scenarios such as accident analysis and data retrieval, when facing complex backgrounds and interference from multiple targets, the association between video and text can be achieved more efficiently, improving processing accuracy and speed.

[0041] In some embodiments, a first sub-sequence and a second sub-sequence are obtained based on the encoded feature sequence, and a video feature vector aligned with the text modality is obtained based on the first sub-sequence and the second sub-sequence, further comprising: The first sub-sequence is obtained by sampling the encoded feature sequence based on the first adapter; The second sub-sequence is obtained by sampling the encoded feature sequence based on the second adapter, wherein the convolution kernel in the second adapter is smaller than the convolution kernel in the first adapter, and the number of convolutional layers in the second adapter is smaller than the number of convolutional layers in the first adapter; The first subsequence and the second subsequence are encoded by the encoder to obtain a video feature vector aligned with the text modality.

[0042] Specifically, such as Figure 2 As shown, the input video is processed by CLIP Image Encoder to obtain an encoded feature sequence. In order to better separate the static and dynamic information in the encoded feature sequence and extract time-sensitive features, the encoded feature sequence can be sampled through two branches, thereby separating the static and dynamic information in the encoded feature sequence.

[0043] One branch is the slow stream, which uses Video Slow (the first adapter) for sampling. This low sampling rate captures global spatial information. It employs larger spatiotemporal convolutional kernels and more convolutional layers to extract more spatial details, thus obtaining the first subsequence: VS=[VS1,VS2,…,VS…]. N ]=Video Slow ([F1,F2,…,F N ]).

[0044] Another branch is the Fast Stream, which uses Video Fast (second adapter) for sampling. It has a high sampling rate and can capture short-term motion information. It uses smaller spatiotemporal convolutional kernels and fewer convolutional layers to reduce computational cost, thus obtaining a second subsequence: VF=[VF1,VF2,…,VF…]. N ]=Video Fast ([F1,F2,…,F N ]).

[0045] Both Video Slow and Video Fast can include structures such as convolution and fully connected layers, and the dimensions of the first and second subsequences output by both are the same.

[0046] Finally, the two types of features obtained are input into the Feature Encoder for encoding to obtain the video feature vector aligned with the text modality.

[0047] In some embodiments, it also includes: A training set is constructed based on video data and actual text descriptions. An encoded feature sequence aligned with the text modality is obtained based on the video data, and text features are obtained based on the actual text. The feature vector is obtained based on the first adapter, the second adapter, the encoder, and the encoded feature sequence; Calculate the first loss function based on the feature vector and the text features; The first adapter, the second adapter, and the encoder are trained and optimized based on the first loss function.

[0048] Specifically, such as Figure 4 As shown, a training set can be constructed based on video data and actual text descriptions, and then the training set can be used to continuously train and optimize the first adapter, the second adapter, and the encoder, as follows. Figure 4As shown, CLIP Text Encoder is used to encode the input text, and CLIP Image Encoder is used to obtain the encoded feature sequence based on the input video data. Therefore, the existing CLIP model encoder can be used as CLIP Text Encoder and CLIP Image Encoder without training. Only three modules need to be trained: Video Fast, Video Slow, and Feature Encoder.

[0049] Among them, CLIP Text Encoder can obtain a text description feature: F based on the input text T. T =Encoder CLIPText (T); CLIP Image Encoder can be based on the input video V=[I1,I2,…,I…] N The encoded feature sequence [F1, F2, ..., F] is obtained. N ]; Encode the feature sequence [F1, F2, ..., F N After passing through Video Fast and Video Slow, we obtain the first and second subsequences: VS=[VS1,VS2,…,VS N ]=Video Slow ([F1,F2,…,F N ]) VF=[VF1,VF2,…,VF N ]=Video Fast ([F1,F2,…,F N ]) Finally, the Feature Encoder is used to fuse the VF and VS obtained from N frames of video data to output a feature vector F. V Its dimension is the same as F T same: F V =Encoder Feature ([VS 1:N VF 1:N ]) Thus, based on the principle that the feature distances corresponding to video and text are close and the feature distances corresponding to video and text are far apart, using F... V and F T Calculate the first loss function L Contrastive : L Contrastive :F V (V i )·FT (T i )=1, F V (V i )·F T (T j )=0, i≠j Where V i T represents the i-th video. i and T j This represents the i-th and j-th texts.

[0050] The proposed solution processes video data to obtain a text modality-aligned encoded feature sequence. Because this sequence is directly aligned with text features, it more efficiently associates video and text information in subsequent applications such as accident analysis and data retrieval compared to traditional methods. Furthermore, this invention samples the encoded feature sequence to separate static and dynamic information, thereby combining these two elements to generate feature vectors, further improving the accuracy of subsequent applications. This allows the solution to accurately describe videos even when faced with complex backgrounds and numerous interfering targets.

[0051] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a traffic video processing and analysis system 400, such as... Figure 5 As shown, it includes: Module 401 is configured to acquire captured video data; Encoding module 402 is configured to obtain an encoded feature sequence aligned with the text modality based on the video data; The sampling module 403 is configured to sample based on the encoded feature sequence to obtain a sampling sequence, and to obtain a video feature vector aligned with the text modality based on the sampling sequence; The analysis module 404 is configured to analyze and process the video data based on the video feature vector.

[0052] In some embodiments, analyzing and processing the video data based on the video feature vector further includes: In some embodiments, analyzing and processing the video data based on the video feature vector further includes: In response to receiving a retrieval command, multiple image-coded features aligned with the text modality are obtained based on the video data; The similarity between the video data and the search text in the search instruction is calculated based on the multiple image encoding features and the video feature vector. Based on the similarity, video data corresponding to the searched text is determined.

[0053] In some embodiments, calculating the similarity between the video data and the search text in the search instruction based on the plurality of image coding features and the video feature vector further includes: Based on the retrieved text, a first text feature and a second text feature are obtained; A first similarity is obtained based on the first text feature and the video feature vector; a similarity with the second text feature is obtained based on each of the image encoding features; and a second similarity is obtained based on multiple similarities. Based on the first similarity and the second similarity, video data corresponding to the searched text is determined.

[0054] In some embodiments, obtaining a first text feature and a second text feature based on the retrieved text further includes: Static description text is obtained based on the retrieved text; The first text feature is obtained based on the retrieval text encoding, and the second text feature is obtained based on the static description text encoding.

[0055] In some embodiments, analyzing and processing the video data based on the video feature vector further includes: The text description of the video data is obtained by decoding the feature vector based on the first decoder.

[0056] In some embodiments, it also includes: A training set is constructed based on the feature vectors and the actual text descriptions; A loss function is constructed based on the predicted text description obtained from the first decoder and the corresponding actual text description. The first decoder is trained and optimized based on the loss function.

[0057] In some embodiments, a first sub-sequence and a second sub-sequence are obtained based on the encoded feature sequence, and a video feature vector aligned with the text modality is obtained based on the first sub-sequence and the second sub-sequence, further comprising: The first sub-sequence is obtained by sampling the encoded feature sequence based on the first adapter; The second sub-sequence is obtained by sampling the encoded feature sequence based on the second adapter, wherein the convolution kernel in the second adapter is smaller than the convolution kernel in the first adapter, and the number of convolutional layers in the second adapter is smaller than the number of convolutional layers in the first adapter; The first subsequence and the second subsequence are encoded by the encoder to obtain a video feature vector aligned with the text modality.

[0058] In some embodiments, it also includes: A training set is constructed based on video data and actual text descriptions. An encoded feature sequence aligned with the text modality is obtained based on the video data, and text features are obtained based on the actual text. The feature vector is obtained based on the first adapter, the second adapter, the encoder, and the encoded feature sequence; Calculate the first loss function based on the feature vector and the text features; The first adapter, the second adapter, and the encoder are trained and optimized based on the first loss function.

[0059] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 6 As shown, embodiments of the present invention also provide a computer device 501, comprising: At least one processor 520; and The memory 510 stores a computer program 511 that can run on a processor. When the processor 520 executes the program, it performs the steps of any of the traffic video processing and analysis methods described above.

[0060] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 7 As shown, embodiments of the present invention also provide a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, it performs the steps of any of the traffic video processing and analysis methods described above.

[0061] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods.

[0062] Furthermore, it should be understood that the computer-readable storage medium (e.g., memory) described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0063] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0064] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0065] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0066] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0067] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0068] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A traffic video processing and analysis method, characterized in that, Includes the following steps: Acquire the collected video data; Based on the video data, an encoded feature sequence aligned with the text modality is obtained; A sampling sequence is obtained by sampling based on the encoded feature sequence, and a video feature vector aligned with the text modality is obtained based on the sampling sequence; The video data is analyzed and processed based on the video feature vector.

2. The method as described in claim 1, characterized in that, The analysis and processing of the video data based on the video feature vector further includes: In response to receiving a retrieval command, multiple image-coded features aligned with the text modality are obtained based on the video data; The similarity between the video data and the search text in the search instruction is calculated based on the multiple image encoding features and the video feature vector. Based on the similarity, video data corresponding to the searched text is determined.

3. The method as described in claim 2, characterized in that, Calculating the similarity between the video data and the search text in the search instruction based on the multiple image coding features and the video feature vector, further including: Based on the retrieved text, a first text feature and a second text feature are obtained; A first similarity is obtained based on the first text feature and the video feature vector; a similarity with the second text feature is obtained based on each of the image encoding features; and a second similarity is obtained based on multiple similarities. Based on the first similarity and the second similarity, video data corresponding to the searched text is determined.

4. The method as described in claim 3, characterized in that, Based on the retrieved text, the first text feature and the second text feature are obtained, further including: Static description text is obtained based on the retrieved text; The first text feature is obtained based on the retrieval text encoding, and the second text feature is obtained based on the static description text encoding.

5. The method as described in claim 1, characterized in that, The analysis and processing of the video data based on the video feature vector further includes: In response to receiving a description instruction, the feature vector is decoded based on the first decoder to obtain a text description of the video data.

6. The method as described in claim 5, characterized in that, Also includes: A training set is constructed based on the feature vectors and the actual text descriptions; A loss function is constructed based on the predicted text description obtained from the first decoder and the corresponding actual text description. The first decoder is trained and optimized based on the loss function.

7. The method as described in claim 1, characterized in that, The sampling sequence is obtained by sampling based on the encoded feature sequence, and a video feature vector aligned with the text modality is obtained based on the sampling sequence, further including: Based on the coded feature sequence, a first sub-sequence and a second sub-sequence are obtained, and a video feature vector aligned with the text modality is obtained based on the first sub-sequence and the second sub-sequence.

8. The method as described in claim 7, characterized in that, Based on the coded feature sequence, a first sub-sequence and a second sub-sequence are obtained, and based on the first sub-sequence and the second sub-sequence, a video feature vector aligned with the text modality is obtained, further comprising: The first sub-sequence is obtained by sampling the encoded feature sequence based on the first adapter; The second sub-sequence is obtained by sampling the encoded feature sequence based on the second adapter, wherein the convolution kernel in the second adapter is smaller than the convolution kernel in the first adapter, and the number of convolutional layers in the second adapter is smaller than the number of convolutional layers in the first adapter; The first subsequence and the second subsequence are encoded by the encoder to obtain a video feature vector aligned with the text modality.

9. The method as described in claim 8, characterized in that, Also includes: A training set is constructed based on video data and actual text descriptions. An encoded feature sequence aligned with the text modality is obtained based on the video data, and text features are obtained based on the actual text. The feature vector is obtained based on the first adapter, the second adapter, the encoder, and the encoded feature sequence; Calculate the first loss function based on the feature vector and the text features; The first adapter, the second adapter, and the encoder are trained and optimized based on the first loss function.

10. A traffic video processing and analysis system, characterized in that, include: The acquisition module is configured to acquire captured video data. The encoding module is configured to obtain an encoded feature sequence aligned with the text modality based on the video data; The sampling module is configured to sample based on the encoded feature sequence to obtain a sampling sequence, and to obtain a video feature vector aligned with the text modality based on the sampling sequence; The analysis module is configured to analyze and process the video data based on the video feature vector.

Images