Automatic driving scene recognition method and device, computer equipment, medium and product

Abstract

The invention relates to an automatic driving scene recognition method and device, computer equipment, a medium and a product. The method comprises the following steps: segmenting a video frame in a to-be-processed video into a plurality of image patches, pre-training a model according to the video frame and corresponding text data, generating a video proxy token and a patch token according to the to-be-processed video under a preset attention mechanism, and encoding the to-be-processed video frame to obtain video features, and coding the text data through the video-text model obtained through training to obtain text features, taking an information noise comparison estimation function as a loss function, and optimizing the video-text model according to the similarity between the video features and the text features to obtain a video-text model. And analyzing key information in the to-be-processed video through the optimized video-text model to obtain an automatic driving scene corresponding to the to-be-processed video. By adopting the method, a more accurate automatic driving scene can be obtained.

Description

Technical Field

[0001] The present application relates to the technical field of visual language models, and in particular to an autonomous driving scene recognition method, apparatus, computer equipment, medium and product. Background Art

[0002] In the field of autonomous driving, the scenario and quality of training data play a vital role in model learning. In order to obtain interesting driving scenarios that meet people's needs, the retrieval method for mining training data needs to have strong retrieval capabilities. However, the two-dimensional image features extracted by the algorithm model usually have inherent limitations, resulting in poor representation of global features. In the process of autonomous driving, accurate global feature representation is crucial for the vehicle to accurately perceive the surrounding environment and make correct decisions. Due to this limitation of two-dimensional image features, it is difficult for autonomous driving systems to fully and accurately understand environmental information in complex scenarios, thus affecting their safety and reliability. Summary of the invention

[0003] Based on this, it is necessary to provide an autonomous driving scene recognition method, device, computer equipment, medium and product that can quickly and accurately screen valid data in response to the above-mentioned technical problems.

[0004] In a first aspect, the present application provides an autonomous driving scene recognition method, comprising:

[0005] Segment the video frames in the video to be processed into multiple image patches, and add feature information to each image patch; the feature information includes spatial information and temporal information;

[0006] Acquire text data corresponding to the video frame, and pre-train the image-text model according to the video frame and the text data to obtain a video-text model; the text data includes real subtitles corresponding to the video frame and auxiliary subtitles corresponding to the intermediate frame of the video to be processed;

[0007] Under the preset attention mechanism, video proxy tokens and patch tokens are generated according to the video to be processed, and the video frame to be processed is encoded based on the video proxy tokens and patch tokens to obtain video features;

[0008] Encode text data through the video-text model to obtain text features, and calculate the similarity between video features and text features;

[0009] The information-noise contrast estimation function is used as the loss function, and the video-text model is optimized according to the calculated similarity. The key information in the video to be processed is analyzed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

[0010] In one embodiment, the step of generating a video proxy token and a patch token according to the video to be processed includes:

[0011] Generate a preset number of video proxy tokens according to the video to be processed; the video proxy tokens are used to represent the time information of the video level of the video frame to be processed;

[0012] Mapping the image patch of the video frame to a one-dimensional token space to obtain a patch token; the patch token is used to represent the feature information of the image patch;

[0013] Among them, the video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

[0014] In one embodiment, the step of optimizing the video-text model according to the calculated similarity using the information-noise contrast estimation function as the loss function includes:

[0015] Determining the degree of association between the video frame and the subtitles of the video frame according to the calculated similarity;

[0016] Positive samples and negative samples are obtained according to video frames and text data corresponding to the video frames; positive samples include video-subtitle pairs consisting of video frames and matching real subtitles, and auxiliary frame-subtitle pairs consisting of intermediate frames and matching auxiliary subtitles; negative samples include video-language pairs consisting of video frames and unmatched real subtitles;

[0017] In the process of using the information-noise contrast estimation function as the loss function and performing comparative learning on positive samples and negative samples through the video-text model, the model parameters of the video-text model are adjusted until the loss function converges.

[0018] In one embodiment, the method further comprises:

[0019] Extracting the global BEV features of the video to be processed and the comprehensive semantic features of the text data respectively, the global BEV features include spatial features and temporal features;

[0020] Establish a cross-modal interaction component between the global BEV feature and the comprehensive semantic feature, and fuse the global BEV feature and the comprehensive semantic feature through the cross-modal interaction component to obtain the fused feature;

[0021] The video to be processed is identified according to the fusion features to obtain the autonomous driving scene corresponding to the video to be processed.

[0022] In one embodiment, the method further comprises:

[0023] The global BEV features and comprehensive semantic features are transformed respectively to obtain the corresponding graph embedding sequence and text embedding sequence, and the graph embedding sequence and text embedding sequence are fused to obtain a synthetic representation sequence;

[0024] The similarity between the synthetic representation sequence and the database object is obtained, and the target object is queried from the database according to the similarity; the target object is an image or text that has an association relationship with the video to be processed.

[0025] In one of the embodiments, the auxiliary subtitles are generated by an image subtitle model; the method further comprises:

[0026] Obtaining an intermediate real subtitle corresponding to the intermediate frame, and obtaining a real synthetic representation and an auxiliary synthetic representation corresponding to the intermediate real subtitle and the auxiliary subtitle respectively;

[0027] Adjust model parameters of the image captioning model based on the degree of difference between the true synthetic representation and the auxiliary synthetic representation.

[0028] In a second aspect, the present application also provides an autonomous driving scene recognition device, comprising:

[0029] A video segmentation module is used to segment a video frame in a to-be-processed video into multiple image patches and add feature information to each image patch; the feature information includes spatial information and temporal information;

[0030] A model training module is used to obtain text data corresponding to video frames, and pre-train an image-text model based on the video frames and text data to obtain a video-text model; the text data includes real subtitles corresponding to video frames and auxiliary subtitles corresponding to intermediate frames of the video to be processed;

[0031] A video encoding module is used to generate video proxy tokens and patch tokens according to the video to be processed under a preset attention mechanism, and encode the video frame to be processed based on the video proxy tokens and patch tokens to obtain video features;

[0032] A text encoding module is used to encode text data through a video-text model to obtain text features and perform similarity calculation on video features and text features;

[0033] The scene recognition module is used to optimize the video-text model based on the calculated similarity using the information-noise contrast estimation function as the loss function, and to parse the key information in the video to be processed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method steps of any one of the first aspects are implemented.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements any one of the method steps in the first aspect when the computer program is executed by a processor.

[0036] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any one of the method steps in the first aspect when executed by a processor.

[0037] The above-mentioned autonomous driving scene recognition method, device, computer equipment, medium and product, by dividing the video frame into multiple image patches and adding feature information, can enable the model to learn the local features of the video more carefully, make full use of the spatial information of the video, and improve the model's ability to understand the video content. The video proxy token is used as an agent to pass the video-level time information to the local image patch, so that the model can consider the changes in the time dimension when processing the video, and the patch token can focus on the local spatial features. At the same time, with the help of the time information introduced by the video proxy token, the position and role of the current frame in the video can be better understood. In addition, by calculating the similarity of video features and text features, and optimizing the model with the information noise contrast estimation function as the loss function, the model can learn a more accurate video-text alignment relationship. By pulling the aligned video-subtitle pairs closer and pushing the unaligned pairs away, the model can better understand the correspondence between the video and the text, and improve the performance on the video-text task, so that the optimized video-text model can parse the key information in the processed video to obtain a more accurate autonomous driving scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] FIG1 is a diagram of an application environment of an autonomous driving scene recognition method according to an embodiment;

[0040] FIG2 is a schematic diagram of a flow chart of an autonomous driving scene recognition method in one embodiment;

[0041] FIG3 is a schematic diagram of a flow chart of an autonomous driving scene recognition method in another embodiment;

[0042] FIG4 is a structural block diagram of an automatic driving scene recognition device in one embodiment;

[0043] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The automatic driving scene recognition method provided in the embodiment of the present application can be applied in the application environment shown in FIG1. ​​The terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0046] Among them, the terminal 102 is used to obtain a video to be processed from the server 104, divide the video frame in the video to be processed into multiple image patches, add feature information to each image patch, obtain text data corresponding to the video frame, pre-train the image-text model according to the video frame and the text data to obtain a video-text model, generate video proxy tokens and patch tokens according to the video to be processed under a preset attention mechanism, encode the video frame to be processed based on the video proxy tokens and the patch tokens to obtain video features, encode the text data through the video-text model to obtain text features, calculate the similarity between the video features and the text features, use the information noise contrast estimation function as the loss function, optimize the video-text model according to the calculated similarity, and parse the key information in the video to be processed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

[0047] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0048] In an exemplary embodiment, as shown in FIG2 , a method for identifying an autonomous driving scene is provided, which is described by taking the method applied to the terminal 102 in FIG1 as an example, and includes the following steps 202 to 210. Among them:

[0049] S202: Segment a video frame in the video to be processed into a plurality of image patches, and add feature information to each image patch; the feature information includes spatial information and temporal information.

[0050] Optionally, the purpose of segmenting a video frame into multiple image patches is to decompose a complex video frame into smaller, easier-to-process units, allowing the model to learn the local features of the video more carefully and improve the model's ability to understand the video content. For example, in an autonomous driving scenario, image patches in different areas may contain different objects or environmental features, such as vehicles, pedestrians, traffic signs, etc. By segmenting into image patches, the model can focus on the features of each small area and better identify and understand these objects. Among them, the video frames are obtained by uniformly sampling the video to be processed, and are all scaled to a fixed size.

[0051] Optionally, spatial information refers to the position and spatial relationship of an image patch in a video frame. Adding spatial information to each image patch allows the model to understand the position of the patch in the entire video frame, thereby better understanding the spatial structure of the image. For example, two-dimensional coordinate information can be added to each image patch to represent the horizontal and vertical coordinates of the patch in the video frame. When processing image patches, the model can determine the relative position relationship between different patches based on the spatial information and better identify the shape, size, and position of objects. Temporal information refers to the temporal position and temporal changes of an image patch in a video sequence. Since a video is composed of a series of continuous frames, adding temporal information to an image patch allows the model to perceive the temporal dynamic changes of the video.

[0052] S204: Acquire text data corresponding to the video frame, and pre-train the image-text model according to the video frame and the text data to obtain a video-text model; the text data includes real subtitles corresponding to the video frame and auxiliary subtitles corresponding to the intermediate frame of the video to be processed.

[0053] Optionally, the real subtitles corresponding to the video frames are usually obtained by manual annotation or automatic extraction through specific video processing tools. The real subtitles are usually used to describe the content in the video frames. For example, in the autonomous driving scenario, the real subtitles may include the name of the traffic sign, the description of the road conditions, etc. The auxiliary subtitles corresponding to the middle frame are generated by a specific image subtitle model. For example, an image subtitle model such as OFA-Caption can be used to generate a subtitle description for the middle frame of each video in the self-collected data to reduce the language domain gap between the pre-training and downstream data. Among them, the maximum length of the auxiliary subtitles is usually 16 words to control the length and complexity of the subtitles.

[0054] Furthermore, the acquired video frames and corresponding text data are used as input to pre-train the image-text model, where the image-text model is usually a deep learning model that can process both image and text data, such as the CLIP (Contrastive Language-Image Pretraining) model. The goal of pre-training is to enable the model to learn common image and text feature representations, as well as the correspondence between them. The resulting video-text model can be used to process new video data, extract key information from the video, and associate and analyze it with the corresponding text description. In autonomous driving scenarios, this model can help vehicles better understand the surrounding environment, identify traffic signs, road conditions, and other vehicle behaviors, and provide more accurate information for decision-making and control.

[0055] S206: Under the preset attention mechanism, a video proxy token and a patch token are generated according to the video to be processed, and the video frame to be processed is encoded based on the video proxy token and the patch token to obtain video features.

[0056] Optionally, a preset attention mechanism enables the model to selectively focus on different parts of the video, thereby extracting features more effectively. It can be designed according to different task requirements and data characteristics to improve the model's understanding and encoding capabilities of the video content. For example, a cross-attention mechanism can be used, which can be used to fuse information from different modalities, such as video and text.

[0057] Optionally, the video proxy token can be used as a proxy to help each local frame-level image information introduce video-level temporal information. Since the video is composed of a series of continuous frames, a single video frame may be difficult to capture the temporal dynamic changes of the video, and the video proxy token can serve as a bridge to pass the temporal information to the local image information. Patch tokens are tokens obtained by segmenting a video frame into multiple patches. Each patch can be regarded as a local image region. Patch tokens represent the feature information of these local regions. By segmenting the video frame into patches and generating patch tokens, the model can better learn the local features of the video and improve the ability to represent the video content.

[0058] Furthermore, the video proxy token and the patch token interact with each other during the encoding process, where the video proxy token can interact with all patches to integrate global temporal information, while the patch token only interacts with the video proxy token and patch token in the same frame, focusing on local spatial information while taking advantage of the temporal information introduced by the video proxy token. In the Transformer architecture, a specific attention mechanism can be designed to allow the video proxy token and the patch token to interact in different attention heads, thereby achieving video encoding processing.

[0059] S208: Encode the text data through the video-text model to obtain text features, and calculate the similarity between the video features and the text features.

[0060] Optionally, the video-text model is a pre-trained model that can process data in both video and text modes. The text data is encoded through the video-text model to convert the text into a feature representation. Afterwards, the similarity between the video features and the text features is calculated. In scenarios such as autonomous driving, by comparing the image content in the video and the corresponding text description, it is determined whether they match, so that the meaning and scene of the video can be better understood. Through similarity calculation, the performance of the video-text model can be evaluated to determine whether the model can accurately associate the video and text. If the similarity is high, it means that the feature representations of the video and text are close, and the model has established a good correspondence between video understanding and text description.

[0061] S210: Using the information noise contrast estimation function as the loss function and optimizing the video-text model according to the calculated similarity, and parsing the key information in the video to be processed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

[0062] Optionally, during the model optimization process, a loss function is used to measure the difference between the model's predictions and the true results. By minimizing the loss function, the model can adjust its own parameters to improve the accuracy of the prediction. Among them, the information noise contrast estimation (Info-NCE) function is used in contrastive learning tasks, and its purpose is to bring positive sample pairs closer together while pushing negative sample pairs away. In the video-text model, the positive sample pair can be a matching video-subtitle pair, and the negative sample pair is an unmatched video and subtitle combination.

[0063] Optionally, based on the calculated similarity between the video features and the text features, and the Info-NCE loss function, an optimization algorithm (such as stochastic gradient descent, etc.) is used to update the parameters of the video-text model. In each iteration, the value of the loss function is calculated, and the model parameters are adjusted according to the loss function, so that the loss function gradually decreases, so that the model continuously learns how to better associate the video and text, and improves the recognition ability of the positive sample pairs.

[0064] Furthermore, the optimized video-text model has better performance and accuracy. When the video to be processed is input into this model, the model can analyze and process the video and extract key information from it. In the autonomous driving scenario, key information may include traffic signs, vehicles, pedestrians, road conditions, etc. Through the model's analysis of the video, these key elements can be identified and associated with the corresponding text description, which can improve the performance of the model in scenarios such as autonomous driving and help the system better understand and cope with complex environments.

[0065] In the above-mentioned autonomous driving scene recognition method, by dividing the video frame into multiple image patches and adding feature information, the model can learn the local features of the video more carefully, make full use of the spatial information of the video, and improve the model's ability to understand the video content. The video proxy token is used as an agent to pass the video-level time information to the local image patch, so that the model can consider the changes in the time dimension when processing the video, and the patch token can focus on the local spatial features. At the same time, with the help of the time information introduced by the video proxy token, the position and role of the current frame in the video can be better understood. In addition, by calculating the similarity of video features and text features, and optimizing the model with the information noise contrast estimation function as the loss function, the model can learn a more accurate video-text alignment relationship. By pulling the aligned video-subtitle pairs closer and pushing the unaligned pairs away, the model can better understand the correspondence between the video and the text, and improve the performance on the video-text task, so that the optimized video-text model can parse the key information in the processed video to obtain a more accurate autonomous driving scene.

[0066] In an exemplary embodiment, the steps of generating video proxy tokens and patch tokens according to the video to be processed include: generating a preset number of video proxy tokens according to the video to be processed; the video proxy tokens are used to represent the time information of the video level of the video frame to be processed; the image patch of the video frame is mapped to a one-dimensional token space to obtain a patch token; the patch token is used to represent the feature information of the image patch; wherein the video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

[0067] Optionally, in order to introduce temporal information at the video level, the model can better understand the temporal dynamic changes of the video. Since the video is composed of a series of continuous frames, a single image patch or frame itself may be difficult to capture the temporal characteristics of the entire video, and the video proxy token can be used as a proxy to pass the temporal information to the local image patch. Among them, the preset number can be determined according to the specific task requirements and model design. Generally, this number may affect the model's ability to represent temporal information and the computational complexity. If the number is too small, the temporal changes of the video may not be fully captured; if the number is too large, the computational burden may increase.

[0068] Optionally, after segmenting the video frame into image patches, these patches need to be converted into a form suitable for model processing. By mapping them to a one-dimensional token space, patch tokens are obtained, and the features of the image patch can be represented as a vector of fixed length, which is convenient for the model to calculate and analyze. Patch tokens are used to represent the feature information of image patches, including spatial information and other related features.

[0069] Furthermore, the video proxy token integrates global temporal information by interacting with all patch tokens. It can learn the temporal relationship between different frames in the video and the overall temporal trend of the video. The patch token can learn local and global feature information by interacting with other patch tokens in the same video frame and with the video proxy token. The video proxy token can pay attention to all patch tokens in each attention calculation block, so that the patch token can perceive the temporal dynamic changes of the video.

[0070] In this embodiment, by generating video proxy tokens and patch tokens according to the video to be processed, the video proxy tokens interact with all patch tokens, and the patch tokens interact with other patch tokens in the same video frame, so that the video proxy tokens can integrate global time information and pass it to all patch tokens. At the same time, the interaction between patch tokens can help the model learn local spatial features and contextual information, so that the model can better understand the spatiotemporal characteristics of the video and provide support for subsequent video processing and analysis tasks.

[0071] In an exemplary embodiment, the step of optimizing the video-text model according to the calculated similarity using the information noise contrast estimation function as the loss function includes: determining the degree of association between the video frame and the subtitles of the video frame according to the calculated similarity; obtaining positive samples and negative samples according to the video frame and the text data corresponding to the video frame; the positive samples include video-subtitle pairs consisting of video frames and matching real subtitles, and auxiliary frame-subtitle pairs consisting of intermediate frames and matching auxiliary subtitles; the negative samples include video frames and unmatched real subtitles. The video-language pair; in the process of using the information noise contrast estimation function as the loss function and performing comparative learning on the positive and negative samples through the video-text model, the model parameters of the video-text model are adjusted until the loss function converges.

[0072] Optionally, based on the calculated similarity between the video features and the text features, the degree of association between the video frame and the subtitles of the video frame can be determined. The higher the similarity, the stronger the association between the video frame and the subtitles. This degree of association reflects the degree of match between the video content and the text description.

[0073] Optionally, positive samples include video-subtitle pairs consisting of video frames and matching real subtitles, and auxiliary frame-subtitle pairs consisting of intermediate frames and matching auxiliary subtitles. These positive samples represent the correct video and text correspondence. The model should learn the features and associations in these samples so that it can accurately identify matching videos and subtitles in future predictions. Negative samples include video-language pairs consisting of video frames and unmatched real subtitles. The purpose of negative samples is to allow the model to learn to distinguish irrelevant video and text combinations, thereby improving the ability to recognize correct associations.

[0074] Furthermore, the information noise contrast estimation function is used as the loss function. The purpose of the Info-NCE loss function is to bring the distance between positive sample pairs closer and push away negative sample pairs during contrastive learning. In this way, the model can learn the ability to distinguish different samples and improve the recognition of the correct association between video and text. In contrastive learning, the model processes positive and negative samples at the same time. For positive samples, the model strives to make their representations in the feature space closer, and for negative samples, the model tries to make their representations farther away.

[0075] Optionally, during contrastive learning, the parameters of the video-text model are adjusted according to the value of the loss function through an optimization algorithm (such as stochastic gradient descent). In each iteration, the gradient of the loss function to the model parameters is calculated, and the parameters are updated according to the gradient direction so that the loss function gradually decreases. This process is continued until the loss function converges. At this point, the parameters of the model have been adjusted to be able to better process positive and negative samples and accurately identify the association between video and text.

[0076] In this embodiment, by using the information-noise contrast estimation function as the loss function, determining the degree of association based on the calculated similarity, obtaining positive and negative samples for comparative learning, and adjusting the model parameters until convergence, this process can effectively optimize the video-text model and improve the model's ability to recognize the association between video and text in scenarios such as autonomous driving.

[0077] In an exemplary embodiment, the method also includes: respectively extracting global BEV features of the video to be processed and comprehensive semantic features of the text data, the global BEV features including spatial features and temporal features; establishing a cross-modal interaction component between the global BEV features and the comprehensive semantic features, and fusing the global BEV features with the comprehensive semantic features through the cross-modal interaction component to obtain a fused feature; identifying the video to be processed according to the fused feature to obtain an autonomous driving scene corresponding to the video to be processed.

[0078] Optionally, the global BEV (Bird's Eye View) features include spatial features and temporal features of the video to be processed. Spatial features are used to describe the position, shape, size and other information of objects in the video. Temporal features reflect the changes of objects in the video over time, such as the speed and direction of the objects. By analyzing continuous video frames, temporal features can be extracted to help the model understand the dynamic changes of the scene. The comprehensive semantic features of text data are extracted from the text description related to the video. The comprehensive semantic features are designed to capture the semantic information in the text, such as keywords describing the scene, the grammatical structure of the sentence, semantic relations, etc. These features can help the model understand the content of the video and associate with the BEV features.

[0079] Furthermore, the cross-modal interaction component between the global BEV feature and the comprehensive semantic feature is established to achieve information fusion between different modal data. Since the BEV feature and the text feature come from different data sources and have different representation forms and semantic connotations, a special component is needed to establish the connection between them. The cross-modal interaction component can be implemented in many ways, such as using a cross-attention mechanism, a fusion network, etc. These methods allow the model to perform targeted information transfer and fusion between features of different modalities.

[0080] Furthermore, through the cross-modal interaction component, the global BEV features are fused with the comprehensive semantic features to obtain fused features. The fused features can contain a comprehensive expression of spatial, temporal and semantic information, enabling the model to better understand the scenes and objects in the video, as well as the relationship between them and the text description. The video to be processed is identified based on the fused features to obtain the corresponding autonomous driving scene of the video to be processed. At this stage, the model can use the fused features to perform various tasks, such as target detection, scene classification, path planning, etc.

[0081] In this embodiment, by respectively extracting the global BEV features of the video to be processed and the comprehensive semantic features of the text data, establishing a cross-modal interaction component for fusion, and then performing video recognition based on the fused features, the autonomous driving system's ability to understand and respond to complex scenes can be improved.

[0082] In an exemplary embodiment, the method also includes: converting the global BEV features and the comprehensive semantic features respectively to obtain corresponding graph embedding sequences and text embedding sequences, fusing the graph embedding sequences and the text embedding sequences to obtain a synthetic representation sequence; obtaining the similarity between the synthetic representation sequence and the database object, and querying the target object from the database based on the similarity; the target object is an image or text that has an association relationship with the video to be processed.

[0083] Optionally, the global BEV features mainly describe the spatial and temporal information in the autonomous driving scenario, and the purpose of converting them into a sequence of graph embeddings is to transform these complex feature representations into a form that is easier to process and compare. For example, a specific neural network layer or algorithm can be used to process the global BEV features and map them to a low-dimensional vector space to obtain a series of graph embeddings, each of which can be regarded as an abstract representation of a local area or a specific object in the global BEV features.

[0084] Alternatively, the comprehensive semantic features come from text data related to the video, which contains rich semantic information. Converting it into a text embedding sequence can make the text features more compact and easier to calculate. Similar to the word embedding method, natural language processing technology can be used to convert words or phrases in the text into vector representations, and then the text embedding sequence is obtained by processing the entire text. Each text embedding can represent a specific concept or semantic unit in the text.

[0085] Furthermore, fusing the image embedding sequence and the text embedding sequence can make comprehensive use of the information of global BEV features and comprehensive semantic features. Through fusion, the association between images and texts can be established, providing richer feature representations for subsequent queries and analysis. The synthetic representation sequence is the result of the fusion, which contains the comprehensive information of images and texts. This sequence can be used as a whole to describe the characteristics of the video to be processed, and can be used for comparison and query with objects in the database.

[0086] Furthermore, when used for query tasks, by calculating the similarity between the synthetic representation sequence and the database object, the similarity between the synthetic representation sequence and each object in the database can be measured. According to the calculated similarity, the target object that has an association relationship with the video to be processed is queried from the database. The target object can be an image or text, which has a certain correlation with the video to be processed in terms of content.

[0087] In this embodiment, by converting and fusing the global BEV features and the comprehensive semantic features to obtain a synthetic representation sequence, and querying the database for target objects that have an association with the video to be processed based on the similarity, information retrieval and analysis in scenarios such as autonomous driving can be achieved quickly and efficiently.

[0088] In an exemplary embodiment, auxiliary subtitles are generated by an image subtitle model; the method also includes: obtaining intermediate real subtitles corresponding to intermediate frames, and respectively obtaining real synthetic representations and auxiliary synthetic representations corresponding to the intermediate real subtitles and the auxiliary subtitles; and adjusting model parameters of the image subtitle model according to the degree of difference between the real synthetic representation and the auxiliary synthetic representation.

[0089] Optionally, for the intermediate real subtitles and auxiliary subtitles, they are fused with corresponding video features (such as global BEV features or other relevant features) respectively to obtain real synthetic representations and auxiliary synthetic representations. The synthetic representation can be obtained by fusing text embedding and image features, such as using an attention mechanism or other fusion strategies to fuse the features of text and image into a unified representation vector.

[0090] Furthermore, the degree of difference between the real synthetic representation and the auxiliary synthetic representation is calculated. The degree of difference reflects the gap between the auxiliary subtitles and the real subtitles. The greater the difference, the lower the accuracy and quality of the auxiliary subtitles. According to the degree of difference between the real synthetic representation and the auxiliary synthetic representation, the model parameters of the image subtitle model are adjusted so that the auxiliary subtitles generated by the model are closer to the real subtitles. By continuously adjusting the parameters, the model gradually improves the accuracy and quality of the generated auxiliary subtitles.

[0091] In this embodiment, by adjusting the model parameters according to the degree of difference between the synthetic representations corresponding to the intermediate real subtitles and the auxiliary subtitles, the performance of the image subtitle model can be improved, and more accurate text information can be provided for video processing and analysis.

[0092] In an exemplary embodiment, as shown in FIG3 , a method for identifying an autonomous driving scene is provided, the method comprising the following steps:

[0093] (1) Video frame segmentation: The video frames in the video to be processed are segmented into multiple image patches, and feature information is added to each image patch; the feature information includes spatial information and temporal information.

[0094] (2) Training the image-text model: Obtain the text data corresponding to the video frames, and pre-train the image-text model based on the video frames and text data to obtain a video-text model; the text data includes the real subtitles corresponding to the video frames and the auxiliary subtitles corresponding to the intermediate frames of the video to be processed.

[0095] (3) Generate video proxy tokens and patch tokens: Under the preset attention mechanism, a preset number of video proxy tokens are generated according to the video to be processed; the video proxy tokens are used to represent the temporal information of the video level of the video frame to be processed; the image patch of the video frame is mapped to the one-dimensional token space to obtain the patch token; the patch token is used to represent the feature information of the image patch; among them, the video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

[0096] (4) Encoding to obtain video features: Encode the video frames to be processed based on the video proxy tokens and patch tokens to obtain video features.

[0097] (5) Encoding to obtain text features: Encode the text data through the video-text model to obtain text features, and calculate the similarity between the video features and the text features.

[0098] (6) Optimizing the video-text model through the similarity between video features and text features: Determine the degree of association between the video frame and the subtitles of the video frame based on the calculated similarity; Obtain positive samples and negative samples based on the video frame and the text data corresponding to the video frame; Positive samples include video-subtitle pairs consisting of video frames and matching real subtitles, and auxiliary frame-subtitle pairs consisting of intermediate frames and matching auxiliary subtitles; Negative samples include video-language pairs consisting of video frames and unmatched real subtitles; In the process of comparative learning of positive and negative samples through the video-text model using the information-noise contrast estimation function as the loss function, adjust the model parameters of the video-text model until the loss function converges.

[0099] (7) Identify autonomous driving scenarios: Use the optimized video-text model to parse the key information in the video to be processed and obtain the autonomous driving scenario corresponding to the video to be processed.

[0100] (8) Fusion of multimodal features: Extract the global BEV features of the video to be processed and the comprehensive semantic features of the text data respectively, where the global BEV features include spatial features and temporal features; establish a cross-modal interaction component between the global BEV features and the comprehensive semantic features, and fuse the global BEV features with the comprehensive semantic features through the cross-modal interaction component to obtain a fused feature; identify the video to be processed based on the fused features to obtain the autonomous driving scene corresponding to the video to be processed.

[0101] (9) Fusion of video representation and semantic representation: The global BEV features and comprehensive semantic features are converted respectively to obtain the corresponding graph embedding sequence and text embedding sequence, and the graph embedding sequence and the text embedding sequence are fused to obtain a synthetic representation sequence; the similarity between the synthetic representation sequence and the database object is obtained, and the target object is queried from the database based on the similarity; the target object is an image or text that has an association relationship with the video to be processed.

[0102] (10) Optimizing the image subtitle model: obtaining the intermediate real subtitles corresponding to the intermediate frames, and obtaining the real synthetic representation and the auxiliary synthetic representation corresponding to the intermediate real subtitles and the auxiliary subtitles respectively; adjusting the model parameters of the image subtitle model according to the degree of difference between the real synthetic representation and the auxiliary synthetic representation.

[0103] In this embodiment, by dividing the video frame into multiple image patches and adding feature information, the model can learn the local features of the video more carefully, make full use of the spatial information of the video, and improve the model's ability to understand the video content. The video proxy token is used as an agent to pass the video-level time information to the local image patch, so that the model can take into account the changes in the time dimension when processing the video, and the patch token can focus on the local spatial features. At the same time, with the help of the time information introduced by the video proxy token, the position and role of the current frame in the video can be better understood. In addition, by calculating the similarity of video features and text features, and optimizing the model with the information noise contrast estimation function as the loss function, the model can learn a more accurate video-text alignment relationship. By pulling the aligned video-subtitle pairs closer and pushing away the unaligned pairs, the model can better understand the correspondence between the video and the text, and improve the performance on the video-text task, so that the optimized video-text model can parse the key information in the video to be processed, and a more accurate autonomous driving scene can be obtained.

[0104] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0105] Based on the same inventive concept, the embodiment of the present application also provides an autonomous driving scene recognition device for implementing the autonomous driving scene recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more autonomous driving scene recognition device embodiments provided below can refer to the limitations of the autonomous driving scene recognition method above, and will not be repeated here.

[0106] In an exemplary embodiment, as shown in FIG4 , an autonomous driving scene recognition device is provided, comprising: a video segmentation module 10, a model training module 20, a video encoding module 30, a text encoding module 40 and a scene recognition module 50, wherein:

[0107] The video segmentation module 10 is used to segment the video frame in the video to be processed into multiple image patches and add feature information to each image patch; the feature information includes spatial information and temporal information.

[0108] The model training module 20 is used to obtain text data corresponding to the video frame, and pre-train the image-text model according to the video frame and the text data to obtain a video-text model; the text data includes the real subtitles corresponding to the video frame and the auxiliary subtitles corresponding to the intermediate frames of the video to be processed.

[0109] The video encoding module 30 is used to generate video proxy tokens and patch tokens according to the video to be processed under a preset attention mechanism, and encode the video frame to be processed based on the video proxy tokens and patch tokens to obtain video features.

[0110] The text encoding module 40 is used to encode text data through a video-text model to obtain text features and perform similarity calculation on the video features and the text features.

[0111] The scene recognition module 50 is used to optimize the video-text model based on the calculated similarity using the information-noise contrast estimation function as the loss function, and to parse the key information in the video to be processed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

[0112] In an exemplary embodiment, the video encoding module 30 is also used to generate a preset number of video proxy tokens based on the video to be processed; the video proxy tokens are used to represent the time information of the video level of the video frame to be processed; the image patch of the video frame is mapped to a one-dimensional token space to obtain a patch token; the patch token is used to represent the feature information of the image patch; wherein, the video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

[0113] In an exemplary embodiment, the scene recognition module 50 is also used to determine the degree of association between the video frame and the subtitles of the video frame based on the calculated similarity; obtain positive samples and negative samples based on the video frame and the text data corresponding to the video frame; the positive samples include video-subtitle pairs consisting of video frames and matching real subtitles, and auxiliary frame-subtitle pairs consisting of intermediate frames and matching auxiliary subtitles; negative samples include video-language pairs consisting of video frames and unmatched real subtitles; in the process of comparative learning of positive samples and negative samples through the video-text model using the information noise contrast estimation function as the loss function, adjust the model parameters of the video-text model until the loss function converges.

[0114] In an exemplary embodiment, the scene recognition module 50 is also used to respectively extract the global BEV features of the video to be processed and the comprehensive semantic features of the text data, wherein the global BEV features include spatial features and temporal features; establish a cross-modal interaction component between the global BEV features and the comprehensive semantic features, and fuse the global BEV features with the comprehensive semantic features through the cross-modal interaction component to obtain a fused feature; identify the video to be processed according to the fused features to obtain the autonomous driving scene corresponding to the video to be processed.

[0115] In an exemplary embodiment, the scene recognition module 50 is also used to convert the global BEV features and the comprehensive semantic features respectively to obtain corresponding graph embedding sequences and text embedding sequences, fuse the graph embedding sequences and the text embedding sequences to obtain a synthetic representation sequence; obtain the similarity between the synthetic representation sequence and the database object, and query the target object from the database based on the similarity; the target object is an image or text that has an association with the video to be processed.

[0116] In an exemplary embodiment, auxiliary subtitles are generated by an image subtitle model; the scene recognition module 50 is also used to obtain intermediate real subtitles corresponding to intermediate frames, and to obtain real synthetic representations and auxiliary synthetic representations corresponding to the intermediate real subtitles and auxiliary subtitles, respectively; and adjust the model parameters of the image subtitle model according to the degree of difference between the real synthetic representation and the auxiliary synthetic representation.

[0117] Each module in the above-mentioned automatic driving scene recognition device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0118] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for recognizing an autonomous driving scene is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0119] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: dividing a video frame in a video to be processed into multiple image patches, and adding feature information to each image patch; the feature information includes spatial information and temporal information; obtaining text data corresponding to the video frame, and pre-training an image-text model according to the video frame and the text data to obtain a video-text model; the text data includes real subtitles corresponding to the video frame and auxiliary subtitles corresponding to the intermediate frame of the video to be processed; under a preset attention mechanism, generating video proxy tokens and patch tokens according to the video to be processed, encoding the video frame to be processed based on the video proxy tokens and the patch tokens to obtain video features; encoding the text data through the video-text model to obtain text features, and calculating the similarity between the video features and the text features; using an information noise contrast estimation function as a loss function, optimizing the video-text model according to the calculated similarity, and parsing key information in the video to be processed through the optimized video-text model to obtain an autonomous driving scene corresponding to the video to be processed.

[0121] In one embodiment, the generation of video proxy tokens and patch tokens according to the video to be processed involved in the execution of a computer program by a processor includes: generating a preset number of video proxy tokens according to the video to be processed; the video proxy tokens are used to represent the time information of the video level of the video frame to be processed; the image patch of the video frame is mapped to a one-dimensional token space to obtain a patch token; the patch token is used to represent the feature information of the image patch; wherein the video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

[0122] In one embodiment, when a processor executes a computer program, the information noise contrast estimation function is used as a loss function, and the video-text model is optimized according to the calculated similarity, including: determining the degree of association between the video frame and the subtitles of the video frame according to the calculated similarity; obtaining positive samples and negative samples according to the video frame and the text data corresponding to the video frame; the positive samples include video-subtitle pairs consisting of video frames and matching real subtitles, and auxiliary frame-subtitle pairs consisting of intermediate frames and matching auxiliary subtitles; negative samples include video frames and unmatched real subtitles. The video-language pair; in the process of using the information noise contrast estimation function as the loss function and performing comparative learning on the positive and negative samples through the video-text model, the model parameters of the video-text model are adjusted until the loss function converges.

[0123] In one embodiment, when the processor executes the computer program, the following steps are also implemented: respectively extracting the global BEV features of the video to be processed and the comprehensive semantic features of the text data, the global BEV features including spatial features and temporal features; establishing a cross-modal interaction component between the global BEV features and the comprehensive semantic features, and fusing the global BEV features with the comprehensive semantic features through the cross-modal interaction component to obtain a fused feature; identifying the video to be processed according to the fused feature to obtain the autonomous driving scene corresponding to the video to be processed.

[0124] In one embodiment, when the processor executes the computer program, the following steps are also implemented: respectively converting the global BEV features and the comprehensive semantic features to obtain corresponding graph embedding sequences and text embedding sequences, fusing the graph embedding sequences and the text embedding sequences to obtain a synthetic representation sequence; obtaining the similarity between the synthetic representation sequence and the database object, and querying the target object from the database based on the similarity; the target object is an image or text that has an association relationship with the video to be processed.

[0125] In one embodiment, auxiliary subtitles are generated by an image subtitle model; when the processor executes the computer program, the following steps are also implemented: obtaining intermediate real subtitles corresponding to the intermediate frames, obtaining real synthetic representations and auxiliary synthetic representations corresponding to the intermediate real subtitles and the auxiliary subtitles, respectively; and adjusting the model parameters of the image subtitle model according to the degree of difference between the real synthetic representation and the auxiliary synthetic representation.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: dividing a video frame in a video to be processed into multiple image patches, and adding feature information to each image patch; the feature information includes spatial information and temporal information; obtaining text data corresponding to the video frame, and pre-training an image-text model according to the video frame and the text data to obtain a video-text model; the text data includes real subtitles corresponding to the video frame and auxiliary subtitles corresponding to the intermediate frame of the video to be processed; under a preset attention mechanism, generating video proxy tokens and patch tokens according to the video to be processed, encoding the video frame to be processed based on the video proxy tokens and the patch tokens to obtain video features; encoding the text data through the video-text model to obtain text features, and calculating the similarity between the video features and the text features; using the information noise contrast estimation function as the loss function, optimizing the video-text model according to the calculated similarity, and parsing the key information in the video to be processed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

[0127] In one embodiment, the computer program, when executed by a processor, involves generating video proxy tokens and patch tokens based on a video to be processed, including: generating a preset number of video proxy tokens based on the video to be processed; the video proxy tokens are used to represent the time information of the video level of the video frame to be processed; the image patch of the video frame is mapped to a one-dimensional token space to obtain a patch token; the patch token is used to represent the feature information of the image patch; wherein the video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

[0128] In one embodiment, when a computer program is executed by a processor, the method involves optimizing a video-text model based on a calculated similarity using an information-noise contrast estimation function as a loss function, including: determining the degree of association between a video frame and its subtitles based on the calculated similarity; obtaining positive samples and negative samples based on the video frame and the text data corresponding to the video frame; the positive samples include video-subtitle pairs consisting of a video frame and matching real subtitles, and auxiliary frame-subtitle pairs consisting of an intermediate frame and matching auxiliary subtitles; the negative samples include video-language pairs consisting of a video frame and unmatched real subtitles; in the process of using the information-noise contrast estimation function as a loss function and performing comparative learning on positive and negative samples through a video-text model, adjusting the model parameters of the video-text model until the loss function converges.

[0129] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: respectively extracting the global BEV features of the video to be processed and the comprehensive semantic features of the text data, the global BEV features including spatial features and temporal features; establishing a cross-modal interaction component between the global BEV features and the comprehensive semantic features, and fusing the global BEV features with the comprehensive semantic features through the cross-modal interaction component to obtain a fused feature; identifying the video to be processed according to the fused feature to obtain the autonomous driving scene corresponding to the video to be processed.

[0130] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: the global BEV features and the comprehensive semantic features are converted respectively to obtain corresponding graph embedding sequences and text embedding sequences, and the graph embedding sequences and the text embedding sequences are merged to obtain a synthetic representation sequence; the similarity between the synthetic representation sequence and the database object is obtained, and the target object is queried from the database based on the similarity; the target object is an image or text that has an association relationship with the video to be processed.

[0131] In one embodiment, auxiliary subtitles are generated by an image subtitle model; when the computer program is executed by a processor, the following steps are also implemented: obtaining intermediate real subtitles corresponding to intermediate frames, obtaining real synthetic representations and auxiliary synthetic representations corresponding to the intermediate real subtitles and auxiliary subtitles, respectively; and adjusting model parameters of the image subtitle model according to the degree of difference between the real synthetic representation and the auxiliary synthetic representation.

[0132] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: dividing a video frame in a video to be processed into multiple image patches, and adding feature information to each image patch; the feature information includes spatial information and temporal information; obtaining text data corresponding to the video frame, and pre-training an image-text model according to the video frame and the text data to obtain a video-text model; the text data includes real subtitles corresponding to the video frame and auxiliary subtitles corresponding to the intermediate frame of the video to be processed; under a preset attention mechanism, generating video proxy tokens and patch tokens according to the video to be processed, encoding the video frame to be processed based on the video proxy tokens and the patch tokens to obtain video features; encoding the text data through the video-text model to obtain text features, and calculating the similarity between the video features and the text features; using the information noise contrast estimation function as the loss function, optimizing the video-text model according to the calculated similarity, and parsing the key information in the video to be processed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

[0133] In one embodiment, the computer program, when executed by a processor, involves generating video proxy tokens and patch tokens based on a video to be processed, including: generating a preset number of video proxy tokens based on the video to be processed; the video proxy tokens are used to represent the time information of the video level of the video frame to be processed; the image patch of the video frame is mapped to a one-dimensional token space to obtain a patch token; the patch token is used to represent the feature information of the image patch; wherein the video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

[0134] In one embodiment, when a computer program is executed by a processor, the method involves optimizing a video-text model based on a calculated similarity using an information-noise contrast estimation function as a loss function, including: determining the degree of association between a video frame and its subtitles based on the calculated similarity; obtaining positive samples and negative samples based on the video frame and the text data corresponding to the video frame; the positive samples include video-subtitle pairs consisting of a video frame and matching real subtitles, and auxiliary frame-subtitle pairs consisting of an intermediate frame and matching auxiliary subtitles; the negative samples include video-language pairs consisting of a video frame and unmatched real subtitles; in the process of using the information-noise contrast estimation function as a loss function and performing comparative learning on positive and negative samples through a video-text model, adjusting the model parameters of the video-text model until the loss function converges.

[0135] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: respectively extracting the global BEV features of the video to be processed and the comprehensive semantic features of the text data, the global BEV features including spatial features and temporal features; establishing a cross-modal interaction component between the global BEV features and the comprehensive semantic features, and fusing the global BEV features with the comprehensive semantic features through the cross-modal interaction component to obtain a fused feature; identifying the video to be processed according to the fused feature to obtain the autonomous driving scene corresponding to the video to be processed.

[0136] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: the global BEV features and the comprehensive semantic features are converted respectively to obtain corresponding graph embedding sequences and text embedding sequences, and the graph embedding sequences and the text embedding sequences are merged to obtain a synthetic representation sequence; the similarity between the synthetic representation sequence and the database object is obtained, and the target object is queried from the database based on the similarity; the target object is an image or text that has an association relationship with the video to be processed.

[0137] In one embodiment, auxiliary subtitles are generated by an image subtitle model; when the computer program is executed by a processor, the following steps are also implemented: obtaining intermediate real subtitles corresponding to intermediate frames, obtaining real synthetic representations and auxiliary synthetic representations corresponding to the intermediate real subtitles and auxiliary subtitles, respectively; and adjusting model parameters of the image subtitle model according to the degree of difference between the real synthetic representation and the auxiliary synthetic representation.

[0138] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying an autonomous driving scene, characterized in that: The method comprises: Segmenting a video frame in a to-be-processed video into a plurality of image patches, and adding feature information to each image patch; the feature information includes spatial information and temporal information; Acquire text data corresponding to the video frame, and pre-train an image-text model according to the video frame and the text data to obtain a video-text model; the text data includes real subtitles corresponding to the video frame and auxiliary subtitles corresponding to the intermediate frame of the video to be processed; Under a preset attention mechanism, a video proxy token and a patch token are generated according to the video to be processed, and the video frame to be processed is encoded based on the video proxy token and the patch token to obtain a video feature; Encoding the text data through the video-text model to obtain text features, and calculating similarity between the video features and the text features; The information noise contrast estimation function is used as the loss function, the video-text model is optimized according to the calculated similarity, and the key information in the video to be processed is analyzed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

2. The method according to claim 1, characterized in that The step of generating a video proxy token and a patch token according to the video to be processed includes: Generate a preset number of video proxy tokens according to the video to be processed; the video proxy tokens are used to represent the time information of the video level of the video frame to be processed; Mapping the image patch of the video frame to a one-dimensional token space to obtain a patch token; the patch token is used to represent feature information of the image patch; The video proxy token interacts with all patch tokens, and the patch token interacts with other patch tokens of the same video frame.

3. The method according to claim 1, characterized in that: The method of optimizing the video-text model according to the calculated similarity by using the information noise contrast estimation function as the loss function includes: Determining the degree of association between the video frame and the subtitles of the video frame according to the calculated similarity; Acquire positive samples and negative samples according to the video frame and the text data corresponding to the video frame; the positive samples include video-subtitle pairs consisting of the video frame and the matching real subtitles, and auxiliary frame-subtitle pairs consisting of the intermediate frame and the matching auxiliary subtitles; the negative samples include video-language pairs consisting of the video frame and the unmatched real subtitles; In the process of performing comparative learning on the positive sample and the negative sample through the video-text model using the information-noise contrast estimation function as the loss function, the model parameters of the video-text model are adjusted until the loss function reaches convergence.

4. The method according to claim 1, characterized in that: The method further comprises: Respectively extracting the global BEV features of the video to be processed and the comprehensive semantic features of the text data, wherein the global BEV features include spatial features and temporal features; Establishing a cross-modal interaction component between the global BEV feature and the comprehensive semantic feature, and fusing the global BEV feature with the comprehensive semantic feature through the cross-modal interaction component to obtain a fused feature; The video to be processed is identified according to the fusion features to obtain the automatic driving scene corresponding to the video to be processed.

5. The method according to claim 4, characterized in that The method further comprises: The global BEV feature and the comprehensive semantic feature are respectively converted to obtain a corresponding graph embedding sequence and a text embedding sequence, and the graph embedding sequence and the text embedding sequence are fused to obtain a synthetic representation sequence; The similarity between the synthetic representation sequence and the database object is obtained, and a target object is queried from the database according to the similarity; the target object is an image or text that has an associated relationship with the video to be processed.

6. The method according to claim 5, characterized in that The auxiliary subtitles are generated by an image subtitle model; the method further comprises: Acquire an intermediate real subtitle corresponding to the intermediate frame, and respectively acquire a real synthetic representation and an auxiliary synthetic representation corresponding to the intermediate real subtitle and the auxiliary subtitle; According to the degree of difference between the true synthetic representation and the auxiliary synthetic representation, the model parameters of the image caption model are adjusted.

7. An automatic driving scene recognition device, characterized in that: The device comprises: A video segmentation module is used to segment a video frame in a video to be processed into multiple image patches and add feature information to each image patch; the feature information includes spatial information and temporal information; A model training module is used to obtain text data corresponding to the video frame, and pre-train an image-text model according to the video frame and the text data to obtain a video-text model; the text data includes real subtitles corresponding to the video frame and auxiliary subtitles corresponding to the intermediate frame of the video to be processed; A video encoding module, used for generating a video proxy token and a patch token according to the video to be processed under a preset attention mechanism, and encoding the video frame to be processed based on the video proxy token and the patch token to obtain a video feature; A text encoding module, used for encoding the text data through the video-text model to obtain text features, and performing similarity calculation on the video features and the text features; The scene recognition module is used to optimize the video-text model according to the calculated similarity using the information-noise contrast estimation function as the loss function, and to parse the key information in the video to be processed through the optimized video-text model to obtain the autonomous driving scene corresponding to the video to be processed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images