Vehicle driving environment identification method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a vehicle driving environment identification method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a driving image of a vehicle in real time; inputting the driving image into a pre-trained target scene attribute recognition model, extracting image features of the driving image through the target scene attribute recognition model, fusing the image features with preset question text features to obtain target fusion features, and obtaining a driving image based on the target fusion features. Identifying a scene attribute of the driving image; and determining the driving environment of the vehicle according to the recognition result of the scene attribute in the driving image. According to the invention, the surrounding environment during vehicle driving can be accurately identified, and the safety of vehicle driving is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a method, device, electronic device, and storage medium for identifying a vehicle driving environment. Background Art

[0002] Today, with the annual increase in the number of automobiles and the continuous progress of autonomous driving technology, people's attention to vehicle driving safety has been increasing day by day. Traffic accidents caused by bad weather conditions such as fog and low visibility are common, becoming problems that need to be solved urgently in daily life. When a vehicle is driving, accurate and effective identification of the surrounding environment is crucial because it directly relates to driving safety.

[0003] However, the vehicle driving environment is complex and changeable. How to accurately identify the surrounding environment when a vehicle is driving and effectively improve the driving safety of the vehicle is a key technical problem that needs to be solved urgently. Summary of the Invention

[0004] Embodiments of this application provide a method, device, electronic device, and storage medium for identifying a vehicle driving environment, which can accurately identify the surrounding environment when a vehicle is driving and effectively improve the driving safety of the vehicle.

[0005] In a first aspect, embodiments of this application provide a method for identifying a vehicle driving environment, including:

[0006] Obtaining real-time driving images of the vehicle;

[0007] Inputting the driving image into a pre-trained target scene attribute recognition model, extracting image features of the driving image through the target scene attribute recognition model, fusing the image features with preset question text features to obtain target fusion features, and identifying the scene attributes of the driving image based on the target fusion features;

[0008] Determining the driving environment of the vehicle according to the recognition result of the scene attributes in the driving image.

[0009] In a possible implementation manner of the first aspect, before the step of inputting the driving image into the pre-trained target scene attribute recognition model, it further includes:

[0010] Obtaining a sample driving image set, where the sample driving images in the sample driving image set include scene attribute annotations;

[0011] Generating question-and-answer texts corresponding to the sample driving images according to the scene attribute annotations;

[0012] Constructing a scene attribute recognition model;

[0013] Use the sample driving images in the sample driving image set, the sample driving images and the questions in their corresponding Q&A texts as the input for model training, and use the scene attribute annotations of the sample driving images as the target output to train the scene attribute recognition model;

[0014] When the training of the scene attribute recognition model reaches the preset training conditions, obtain the target scene attribute recognition model.

[0015] In a possible implementation manner of the first aspect, the scene attribute recognition model includes an image feature extraction layer, a text feature extraction layer, a feature fusion layer, and a feature decoding layer; the steps of training the scene attribute recognition model include:

[0016] Extract the sample driving image features in the sample driving images through the image feature extraction layer;

[0017] Extract the sample text features of the questions in the Q&A text through the text feature extraction layer;

[0018] Fuse the sample driving image features and the sample text features through the feature fusion layer to obtain sample fusion features;

[0019] Predict the scene attributes in the sample driving images according to the sample fusion features through the feature decoding layer;

[0020] Calculate the training loss between the predicted scene attributes and the scene attribute annotations, and optimize the model parameters of the scene attribute recognition model according to the training loss.

[0021] In a possible implementation manner of the first aspect, the step of fusing the sample driving image features and the sample text features through the feature fusion layer to obtain sample fusion features includes:

[0022] Fuse the sample driving image features and the sample text features by using a cross-attention mechanism to obtain sample fusion features.

[0023] In a possible implementation manner of the first aspect, the step of obtaining the sample driving image set includes:

[0024] Collect historical driving videos;

[0025] Perform an interception process on the historical driving videos to obtain a plurality of video segments, where the historical driving images in each video segment have the same scene attributes;

[0026] Perform scene attribute annotation on each video segment;

[0027] Extract frames from the video segments after annotating the scene attributes;

[0028] Use the historical driving images obtained by frame extraction as sample video images to construct a sample image set.

[0029] In a possible implementation manner of the first aspect, the step of determining the driving environment of the vehicle according to the recognition result of the scene attribute in the driving image includes:

[0030] Obtain a driving environment framework;

[0031] Fill the recognition result of the scene attribute into the driving environment framework to obtain the driving environment of the vehicle.

[0032] In a possible implementation manner of the first aspect, after the step of determining the driving environment of the vehicle according to the recognition result of the scene attribute in the driving image, it further includes:

[0033] Generate a driving strategy according to the recognized scene attribute and the weight of the scene attribute in the driving environment;

[0034] Generate driving warning information based on the driving strategy.

[0035] In a second aspect, an embodiment of the present application provides a vehicle driving environment recognition device, including:

[0036] An image acquisition unit for real-time acquisition of driving images of a vehicle;

[0037] An attribute recognition unit for inputting the driving image into a pre-trained target scene attribute recognition model, extracting image features of the driving image through the target scene attribute recognition model, fusing the image features with preset question text features to obtain target fusion features, and based on the target fusion features, recognizing the scene attribute of the driving image;

[0038] A driving environment determination unit for determining the driving environment of the vehicle according to the recognition result of the scene attribute in the driving image.

[0039] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the vehicle driving environment recognition method as described in the first aspect above.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the vehicle driving environment recognition method as described in the first aspect above.

[0041] Fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, it causes the electronic device to execute the vehicle driving environment recognition method as described in the first aspect above.

[0042] In an embodiment of the present application, by obtaining the driving image of the vehicle in real time, inputting the driving image into a pre-trained target scene attribute recognition model, extracting the image features of the driving image through the target scene attribute recognition model, fusing the image features with the preset question text features to obtain target fusion features, and then based on the target fusion features, recognizing the scene attributes of the driving image, making full use of multi-modal information for scene attribute recognition, the scene attributes can be recognized more comprehensively and accurately. Then, according to the scene attributes of the driving image, the driving environment of the vehicle is accurately recognized, assisting the vehicle to make reasonable driving decisions, thereby effectively reducing the possibility of accidents during vehicle driving and improving driving safety. Description of the Drawings

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 It is a flowchart of the implementation of the vehicle driving environment recognition method provided by the embodiment of the present application;

[0045] Figure 2 It is a specific implementation flowchart of obtaining the target scene attribute recognition model in the vehicle driving environment recognition method provided by the embodiment of the present application;

[0046] Figure 3 It is a specific implementation flowchart of obtaining the sample driving image set in the vehicle driving environment recognition method provided by the embodiment of the present application;

[0047] Figure 4 It is a specific implementation flowchart of training the scene attribute recognition model in the vehicle driving environment recognition method provided by the embodiment of the present application;

[0048] Figure 5 It is a specific implementation flowchart of step S103 in the vehicle driving environment recognition method provided by the embodiment of the present application;

[0049] Figure 6 It is a block diagram of the structure of the vehicle driving environment recognition device provided by the embodiment of the present application;

[0050] Figure 7It is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0051] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0052] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0053] It should also be understood that the term "and / or" used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0054] As used in the specification and appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0055] In addition, in the description of the specification and appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0056] Referring to "one embodiment" or "some embodiments" described in the specification of the present application means that specific features, structures, or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0057] By way of example and not limitation, the vehicle driving environment recognition method provided by the embodiments of the present application can be applied to various types of electronic devices, specifically including in-vehicle terminals, mobile phones, tablet computers, laptop computers, ultra-mobile personal computers (UMPCs), desktop computers, etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0058] Figure 1 The implementation process of the vehicle driving environment recognition method provided by the embodiments of the present application is shown. This method process includes steps S101 to S103. The specific implementation principles of each step are as follows:

[0059] Step S101: Obtain the driving image of the vehicle in real time.

[0060] In the embodiments of the present application, image acquisition devices such as in-vehicle cameras acquire the driving images of the vehicle at regular time intervals (for example, every 1 second) to ensure continuous and timely acquisition of image data during the vehicle's driving process. The driving image is the picture taken and recorded by image acquisition devices such as in-vehicle cameras during the vehicle's driving process.

[0061] The picture recorded by the driving image includes road conditions, surrounding environment, weather conditions, and traffic conditions. Road conditions, surrounding environment, weather conditions, and traffic conditions are also the scene attributes to be recognized in the driving image.

[0062] Road conditions include the appearance characteristics of the road where the vehicle is driving. For example, whether the road is a smooth asphalt road or a potholed gravel road, the width of the road, whether it is a straight avenue or a small road with many curves, etc. At the same time, it can also present the signs and markings on the road, such as lane dividers, stop lines, and guiding arrows. Road conditions help to determine the type of road where the vehicle is located (such as urban streets, highways, etc.).

[0063] The surrounding environment includes the natural landscapes around the road, such as mountains, trees in mountainous areas, high-rise buildings and street shops in cities; the surrounding environment can also reflect the existence of nearby man-made facilities, such as bus stops, gas stations, parking lots, etc. The surrounding environment helps to further determine the specific driving environment of the vehicle.

[0064] Weather conditions can reflect the weather conditions at that time during the driving process. If the picture is relatively blurred and white, it may be foggy or snowy; if there are raindrops falling in the picture or there is water reflection on the ground, it can be judged as rainy; a picture with bright sunlight and clear colors probably corresponds to sunny days, etc.

[0065] Traffic conditions include the number and type of other vehicles on the road where the vehicle is traveling (such as cars, trucks, motorcycles, etc.), their driving states (such as normal driving, overtaking, or parking, etc.), and the presence of pedestrians (whether there are pedestrians waiting to cross the road by the roadside, or pedestrians are crossing the road, etc.). Traffic conditions are of great significance for evaluating the degree of traffic congestion and potential traffic risks.

[0066] In some possible implementation manners, the in-vehicle camera is communicatively connected to the in-vehicle terminal. More than one in-vehicle camera is installed on the vehicle, and the installation position and shooting angle of each in-vehicle camera are different. For example, the in-vehicle camera includes a front camera, a rear camera, and a side camera. The front camera mainly captures the road conditions in front of the vehicle; the rear camera records the situation behind the vehicle to assist in reversing and observing oncoming vehicles from behind, etc.; the side camera captures the traffic conditions and surrounding environment on both sides of the vehicle. The images captured by the in-vehicle cameras with different perspectives cooperate with each other to provide a basis for comprehensive travel images.

[0067] In the embodiments of the present application, as the vehicle continues to travel, the driving images are in a continuously updated state, which reflects in real time the changes in the surrounding environment over time and with the movement of the vehicle's position. For example, the images gradually switch from passing through a section of urban streets to entering a highway, or the originally sunny image changes to a rainy scene as the weather changes, etc.

[0068] Step S102: Input the driving image into a pre-trained target scene attribute recognition model, extract the image features of the driving image through the target scene attribute recognition model, fuse the image features with the preset problem text features to obtain target fusion features, and based on the target fusion features, identify the scene attributes of the driving image.

[0069] The target scene attribute recognition model is a computational model for identifying the scene attributes of driving images. In the embodiments of the present application, the target scene attribute recognition model is based on a complex neural network architecture and mainly consists of multiple functional layers such as a feature extraction layer, a feature fusion layer, and a feature decoding layer. These functional layers cooperate with each other, extract image features from the input driving image, fuse them with the preset text features, and then analyze based on the target fusion features to finally output the recognition result of the scene attributes.

[0070] Traditional scene attribute recognition relies on single-modal image data, and there are often problems of insufficient recognition accuracy in complex and changing actual environments. In the embodiments of the present application, the travel images obtained in real time are input into the target scene attribute recognition model, the image features are fused with the preset problem text features, and a question-and-answer form is used to specifically identify whether a certain scene attribute exists in the driving image from the target fusion features, which can effectively improve the accuracy of scene attribute recognition.

[0071] The preset question text feature is a form of representation of text information related to the vehicle driving environment. It is a text feature vector corresponding to a series of pre-designed preset questions, which are centered around the scene attributes in the driving image and aim to guide the model to focus on specific environmental details during the recognition process. For example, "Is the current scene rainy?" "Is there a bus stop on the road?" etc. are all preset questions.

[0072] Preset questions are usually simple and clear interrogative sentences, covering all aspects of the vehicle driving environment, such as road conditions ("Is this a highway?"), surrounding environment ("Is there a gas station nearby?"), weather conditions ("Are there dark clouds in the sky?"), and traffic conditions ("Is the road congested?"), etc. Preset questions are set according to the actual needs of vehicle driving environment recognition and are used to obtain information closely related to driving safety and environmental understanding.

[0073] During the scene attribute recognition process, the preset question text feature acts like a guide, prompting the model to analyze the driving image in a targeted manner. For example, when the preset question is "Is there a traffic sign on the road?", the model will focus more attention on the positions in the image where traffic signs may appear, such as areas along the roadside and above intersections, thereby improving the recognition accuracy of the traffic sign scene attribute.

[0074] Although the driving image contains rich visual information, some scene attributes may be difficult to accurately judge only through the image. The preset question text feature can provide additional clues. For example, in some cases, the weather condition in the image may be relatively blurred. By using the question text "Is it snowing currently?" and its feature vector, combined with some blurred clues in the image (such as white objects that may exist in the picture), it can help the model more accurately judge that the weather is a snowy day. The target scene attribute recognition model fuses the preset question text feature with the image feature of the driving image, integrates the information of two different modalities, gives play to their respective advantages, and realizes more comprehensive and accurate scene attribute recognition. By comprehensively considering language knowledge and visual information, it provides stronger support for the accurate understanding of the vehicle driving environment.

[0075] As a possible implementation manner of this application, Figure 2 shows a specific implementation process for obtaining the target scene attribute recognition model in the vehicle driving environment recognition method provided by the embodiments of this application, which is described in detail as follows:

[0076] A1: Obtain a sample driving image set, and the sample driving images in the sample driving image set include scene attribute annotations.

[0077] In some possible implementation manners, Figure 3A specific implementation process for obtaining the above-mentioned sample driving image set is shown as follows:

[0078] A11: Collect historical driving videos.

[0079] The historical driving videos are driving videos in various driving environments. On the one hand, the collected historical driving videos cover roads in different cities, towns, and suburbs, specifically including bustling streets, main roads in the city center, as well as rural roads, mountain paths, etc., to ensure the diversity of vehicle driving environments. On the other hand, the collected historical driving videos cover different seasons, time periods, and weather conditions. For example, snowy days in winter, rainy days in summer, sunny days in spring and autumn, such as day, night, morning and evening rush hours. Collecting historical driving videos in various different scenarios can provide rich, diverse, and comprehensive samples for model training.

[0080] A12: Perform interception processing on the historical driving videos to obtain multiple video segments, where the historical driving images in each video segment have the same scene attribute.

[0081] A13: Label the scene attributes of each video segment. For example, the scene attributes that can be labeled include road type (city, countryside, highway, tunnel, underground garage), weather condition (day, night, whether foggy, whether rainy), surrounding environment (whether there is a bus stop, whether there is a gas station); generally, there is at least one different scene attribute label for different video segments.

[0082] A14: Extract frames from the video segments after labeling the scene attributes.

[0083] A15: Use the historical driving images obtained by frame extraction as sample video images to construct a sample image set.

[0084] The methods of frame extraction include extraction at fixed time intervals and random extraction, and a sample image set is constructed based on the sample video images obtained by frame extraction from multiple video segments.

[0085] In some possible implementation manners, at least one video image can be extracted from the video frame sequence of the video segment at a fixed interval as a sample video image. The fixed interval can be a frame number interval. For example, a fixed frame number interval is set, and one video image is extracted every 10 frames as a sample video image. The fixed interval can also be a time interval. For example, one frame is extracted every 1 second. If the frame rate of the video is 30 frames per second, then that is, one frame is extracted every 30 frames as a sample video image. Among them, the fixed interval can be determined according to the duration of the video segment, that is, for video segments of different durations, the fixed interval of frame extraction can be different.

[0086] In some possible embodiments, a video image is randomly selected from the video frame sequence of a video segment as a sample video image. The random selection method can be random frame selection. For example, an integer is randomly generated within the total number of frames of the video segment, and then the video image corresponding to this integer frame is selected from the video frame sequence of the video segment. The random selection method can also adopt a random sampling method to randomly select a certain number of video images from the video frame sequence of the video segment as sample video images, which can ensure the sample quantity.

[0087] In the embodiments of this application, the extracted frames are saved in a required format, such as PNG, JPEG, etc., to retain the details and quality of the images.

[0088] A2: Generate the Q&A text corresponding to the sample driving image according to the scene attribute annotation. Generate a unified and fixed question for each sample driving image, and use the annotated scene attribute as the answer to the question.

[0089] In some possible embodiments, the Q&A text is in English (or in a specific language according to requirements), and the answers in the Q&A text can be neutral answers. It should be noted that the Q&A text uses standard English and needs to follow the grammar rules of this language, avoiding overly casual expressions such as abbreviations and slang.

[0090] Exemplarily, for "What street scene does it belong to", the answer options are: city, country, highway, tunnel, underground parking, difficult to judge; for "Is it day or night", the answer options are: day, night, difficult to judge; for "Is it dense fog", the answer options are: yes, no, difficult to judge; for "Is it rainy", the answer options are: yes, no, difficult to judge; for "Is there a bus stop", the answer options are: yes, no, difficult to judge; for "Is there a gas station", the answer options are: yes, no, difficult to judge.

[0091] The questions in the Q&A text revolve around the characteristics and impacts of scene attributes. In some possible implementation manners, a question template is designed according to scene attributes, and based on the designed question template and scene attribute annotation, automated generation of Q&A text is achieved through programming. Manual sampling inspection is performed on the automatically generated Q&A text to correct possible errors or unreasonable parts. For example, some questions may not fit well with the actual scene due to limitations in template design, and during manual review, the questions can be adjusted to make them more accurate; or if some language expressions are found to be not natural enough, they are optimized to ensure the high quality of the Q&A text.

[0092] Through automated text generation and manual sampling detection, Q&A text corresponding to the sample driving images can be efficiently generated according to scene attribute annotation, providing high-quality text data for subsequent model training.

[0093] A3: Build a scene attribute recognition model.

[0094] In the embodiments of this application, the scene attribute recognition model includes an image feature extraction layer, a text feature extraction layer, a feature fusion layer, and a feature decoding layer.

[0095] The image feature extraction layer is used to extract valuable image features from the driving images. The image feature extraction layer can adopt a convolutional neural network (CNN), such as pre-trained ResNet (Residual Network), VGG (Visual Geometry Group Network), etc. Taking ResNet as an example, it automatically extracts features such as the texture, shape, and color of the image through a series of convolutional layers, pooling layers, and residual connections. For example, for the recognition of road types, it can extract the texture features of the road to distinguish between urban asphalt roads and rural dirt roads; for weather conditions, it can capture features such as the shape of raindrops and the degree of fogginess, which provide basic information for subsequent scene attribute recognition.

[0096] The text feature extraction layer is used to extract the text features of the questions in the Q&A text. The text features may contain some hint information related to scene attributes, such as "Is it a rainy day?" "Is the road an urban road or a rural road?" etc. Pre-trained natural language processing models can be used, such as BERT (Bidirectional Encoder Representations from Transformers, bidirectional transformer model) or CLIP (Contrastive Language-Image Pretraining, contrastive language-image pre-training) model. Taking CLIP as an example, it can convert text into appropriate feature vectors, and the feature vectors can be fused with image features in the same feature space, enabling the model to better understand the image by combining text information.

[0097] The feature fusion layer is used to fuse the image features obtained by the image feature extraction layer and the text features obtained by the text feature extraction layer.

[0098] In some possible implementations, a cross-attention mechanism is used to fuse the sample driving image features with the sample text features to obtain sample fusion features. The cross-attention mechanism allows the model to dynamically focus on the related parts between the image and the text. For example, when the text question is "Is there a bus stop?", the model uses a cross-attention mechanism to focus on the image features of areas in the image where bus stops may appear (such as open areas on the roadside), and then fuses these related image features and text features to obtain more targeted fusion features, enabling the model to comprehensively utilize information from both modalities to identify scene attributes.

[0099] The formula of the cross attention mechanism is as follows (1):

[0100]

[0101] Among them, Attention(Q Text ,K Img ,V Img ) is the fused attention, Q Text The query matrix representing text features, K Img Key matrix representing image features, V Img A matrix of values representing image features, Represents the dimension of the key matrix, and T represents the transpose operation. Multiply the image feature with the fused attention to obtain the fused feature.

[0102] The feature decoding layer decodes the fused features and outputs the scene attribute recognition results. This layer can be based on a Transformer architecture or a fully connected neural network. The Transformer architecture's self-attention mechanism effectively processes sequential data and outputs reasonable scene attribute recognition results based on the different information weights in the fused features, such as "daytime, city street, no bus stop." The fully connected neural network uses linear and nonlinear transformations across multiple neurons to map the fused features into the category space of scene attributes for classification.

[0103] A4: The sample driving images in the sample driving image set and the questions in the corresponding question-and-answer texts are used as inputs for model training, and the scene attribute annotations of the sample driving images are used as target outputs to train the scene attribute recognition model.

[0104] In the embodiments of the present application, the scene attribute recognition model is trained according to the sample driving images with scene attribute annotations in the sample driving image set and the questions in the corresponding Q&A texts, where the scene attribute annotation is also the answer to the question in the corresponding Q&A text. End-to-end training can be performed using the backpropagation algorithm and gradient descent.

[0105] The sample driving image set can be divided into a training set, a validation set, and a test set to scientifically and systematically evaluate and optimize the model performance. The data in the training set is used to train the model, and the data in the validation set is used to evaluate the model at certain stages of model training (such as every several iteration cycles). By comparing the predicted results of the model on the validation set with the true scene attribute annotations, performance metrics such as accuracy and recall are calculated to determine whether the model has overfitting or underfitting phenomena. The data in the test set is used to conduct an "ultimate assessment" of the model after the model is completed training and optimized based on the validation set, so as to evaluate the performance of the model on data that is completely independent and not involved in the training process.

[0106] As a possible implementation manner of the present application, as Figure 4 shown, the steps of training the scene attribute recognition model include:

[0107] B1: Extract the sample driving image features in the sample driving image through the image feature extraction layer.

[0108] B2: Extract the sample text features of the question in the Q&A text through the text feature extraction layer.

[0109] B3: Fuse the sample driving image features and the sample text features through the feature fusion layer to obtain sample fusion features.

[0110] B4: Predict the scene attributes in the sample driving image according to the sample fusion features through the feature decoding layer.

[0111] B5: Calculate the training loss between the predicted scene attributes and the scene attribute annotations, and optimize the model parameters of the scene attribute recognition model according to the training loss.

[0112] The training loss can be calculated according to a preset loss function. The preset loss function can be a cross-entropy loss function, a mean squared error loss function, etc. In the embodiments of the present application, the preset loss function can be determined according to actual needs, and the type of the preset loss function is not limited.

[0113] A5: When the training of the scene attribute recognition model reaches the preset training conditions, a target scene attribute recognition model is obtained. The preset training conditions include the maximum number of iterations (e.g., 500 times) for the preset model training, or the preset training conditions include a preset loss threshold (e.g., 0.01), or the preset training conditions include a preset accuracy target (e.g., 90%). When the training of the scene attribute recognition model reaches the maximum number of iterations, or the training loss is less than or equal to the preset loss threshold, or the output accuracy of the model reaches the preset accuracy target, the training ends and a target scene attribute recognition model is obtained. By setting the accuracy target, it can be ensured that the model reaches a certain performance level during the training process.

[0114] The trained target scene attribute recognition model can more accurately identify the scene attributes in the driving images by integrating image and text information. Compared with the traditional method that only relies on image analysis, it can use text information to make up for the ambiguity or uncertainty of image information. For example, when it is difficult to distinguish between a misty weather and a cloudy day in the image, the text information can provide clues to help the model make a more accurate judgment. This multi-modal fusion method can effectively improve the accuracy of vehicle driving environment recognition and provide a reliable basis for subsequent driving strategy generation and driving safety guarantee.

[0115] Exemplarily, a sample driving image M of a driving scenario is obtained, and scene attributes are labeled for the sample driving image M. Text questions and answers are created for each scene attribute. Question A is "What street scene does it belong to", and the answer A is "highway"; Question B is "Is it day or night", and the answer B is "day"; Question C is "Is it dense fog", and the answer C is "no"; Question D is "Is it rainy", and the answer D is "yes"; Question E is "Is there a bus stop", and the answer E is "no"; Question F is "Is there a gas station", and the answer F is "no"; The sample image M is input into the image feature extraction layer to extract image features; the text questions are input into the text feature extraction layer to extract text features, the image features and text features are fused based on the cross-attention mechanism, the obtained attention is multiplied by the image features to obtain the weighted fused features, and the feature decoding layer decodes the fused features to obtain the output of the scene attribute recognition model. The output is compared with the label of the sample driving image M to calculate the loss, and the entire model is iteratively trained based on the loss. The training parameters include a learning rate of 0.00001, a batch size of 4, an optimization function of AdamW, and a weight decay of 0.1. After training is completed, a test image is selected, a scene attribute question (such as: Is it day or night) is selected, input into the model, and the question answer (day or night) is obtained to evaluate the performance of the model through testing.

[0116] Step S103: Determine the driving environment of the vehicle according to the recognition result of the scene attributes in the driving image.

[0117] In the embodiment of the present application, after various scene attributes of the driving image are recognized by the target scene attribute recognition model, these scattered scene attributes are integrated to depict the complete driving environment where the vehicle is located.

[0118] As a possible implementation manner of the present application, Figure 5 Fig. shows a specific implementation process of step S103 in the vehicle driving environment recognition method provided by the embodiment of the present application, which is described in detail as follows:

[0119] C1: Obtain a driving environment framework.

[0120] C2: Fill the recognition result of the scene attribute into the driving environment framework to obtain the driving environment of the vehicle.

[0121] The driving environment framework includes various scenario attributes, such as road conditions, surrounding environment, weather conditions, and traffic conditions, etc., and also includes tags corresponding to the scenario attributes. In the embodiments of the present application, a tag system is established, and the tags are used to map the specific attribute values of various scenario attributes. For example, for the scenario attribute of road conditions, its tags include urban streets, asphalt roads, highways, etc.; for the attribute of the surrounding environment, its tags include mountains, trees, high-rise buildings, etc.; for the scenario attribute of weather conditions, its tags include foggy days, rainy days, snowy days, etc.; for the attribute of traffic conditions, its tags include the driving state of the vehicle ahead, the number of pedestrians, etc. By matching the recognition results of the scenario attributes with the tags and mapping them to the corresponding positions in the driving environment framework, the driving environment of the vehicle is integrated.

[0122] To ensure the accuracy of the mapping match between the recognition results of the scenario attributes and the tags, the tags of each scenario attribute constructed need to have clear and accurate semantic definitions, and the boundaries between different tags should be clear.

[0123] In some possible implementation manners, the mapping match situation between the recognition results of the scenario attributes and the tags is monitored in real time. If it is found that the probability of incorrect mapping is higher than the preset incorrect probability threshold, an alarm reminder is triggered in a timely manner. Additionally, the mapping rules between the scenario attributes and the tags can be dynamically updated according to the actual application scenarios.

[0124] In some possible implementation manners, a driving strategy is generated according to the recognized scenario attributes and the weights of the scenario attributes in the driving environment; and driving warning information is generated based on the driving strategy.

[0125] The weights of various scenario attributes can be determined according to the experience and professional knowledge of industry experts and senior drivers, or can be determined according to the statistical analysis results of historical trip data. For example, in a traffic congestion scenario, the weight of the road type may be relatively high because different road types have different impacts on traffic flow; in the weather conditions of different seasons, snowy days in winter may have a greater impact on driving safety, and the weight of the snowy day scenario attribute is relatively high.

[0126] The weights of various scenario attributes can also be determined according to the correlation between the recognized scenario attributes and between the scenario attributes and the vehicle driving state. For example, when driving on a highway, there may be a certain correlation between the weather conditions and the road conditions. The weights of the weather conditions and the road conditions are determined by calculating the correlation between the weather conditions and the road conditions.

[0127] In the embodiments of the present application, a comprehensive driving strategy is formulated according to the weights and combination situations of different scenario attributes. And warning information is generated based on the driving strategy, and the warning information is used to prompt the driver to adjust the driving strategy.

[0128] As can be seen from the above, in the embodiments of the present application, by obtaining the driving image of the vehicle in real time, inputting the driving image into the pre-trained target scene attribute recognition model, extracting the image features of the driving image through the target scene attribute recognition model, and fusing the image features with the preset question text features to obtain the target fusion features, and then based on the target fusion features, identifying the scene attributes of the driving image, making full use of multi-modal information for scene attribute recognition, the scene attributes can be identified more comprehensively and accurately. Then, according to the scene attributes of the driving image, the driving environment of the vehicle is accurately identified, assisting the vehicle to make reasonable driving decisions, thereby effectively reducing the possibility of accidents during vehicle driving and improving driving safety.

[0129] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0130] Corresponding to the vehicle driving environment recognition method described in the above embodiments, Figure 6 The block diagram of the vehicle driving environment recognition device provided by the embodiments of the present application is shown. For the convenience of description, only the parts related to the embodiments of the present application are shown.

[0131] Referring to Figure 6 , the vehicle driving environment recognition device includes: an image acquisition unit 61, an attribute recognition unit 62, and a driving environment determination unit 63, where:

[0132] The image acquisition unit 61 is configured to obtain the driving image of the vehicle in real time;

[0133] The attribute recognition unit 62 is configured to input the driving image into the pre-trained target scene attribute recognition model, extract the image features of the driving image through the target scene attribute recognition model, fuse the image features with the preset question text features to obtain the target fusion features, and based on the target fusion features, identify the scene attributes of the driving image;

[0134] The driving environment determination unit 63 is configured to determine the driving environment of the vehicle according to the recognition result of the scene attributes in the driving image.

[0135] As a possible implementation manner of the present application, the vehicle driving environment recognition device further includes a model training unit, configured to:

[0136] Obtain a sample driving image set, where the sample driving images in the sample driving image set include scene attribute annotations;

[0137] Generate the question-and-answer text corresponding to the sample driving image according to the scene attribute annotation;

[0138] Build a scene attribute recognition model;

[0139] Use the sample driving images in the sample driving image set and the questions in their corresponding Q&A texts as the input for model training, and use the scene attribute annotations of the sample driving images as the target output to train the scene attribute recognition model;

[0140] When the training of the scene attribute recognition model reaches the preset training conditions, obtain the target scene attribute recognition model.

[0141] As a possible implementation manner of this application, the scene attribute recognition model includes an image feature extraction layer, a text feature extraction layer, a feature fusion layer, and a feature decoding layer; the steps of training the scene attribute recognition model include:

[0142] Extract the sample driving image features in the sample driving images through the image feature extraction layer;

[0143] Extract the sample text features of the questions in the Q&A text through the text feature extraction layer;

[0144] Fuse the sample driving image features and the sample text features through the feature fusion layer to obtain sample fusion features;

[0145] Predict the scene attributes in the sample driving images according to the sample fusion features through the feature decoding layer;

[0146] Calculate the training loss between the predicted scene attributes and the scene attribute annotations, and optimize the model parameters of the scene attribute recognition model according to the training loss.

[0147] As a possible implementation manner of this application, the step of fusing the sample driving image features and the sample text features through the feature fusion layer to obtain sample fusion features includes:

[0148] Use the cross-attention mechanism to fuse the sample driving image features and the sample text features to obtain sample fusion features.

[0149] As a possible implementation manner of this application, the step of obtaining the sample driving image set includes:

[0150] Collect historical driving videos;

[0151] Perform interception processing on the historical driving videos to obtain multiple video segments, where the historical driving images in each video segment have the same scene attributes;

[0152] Perform scene attribute annotation on each video segment;

[0153] Extract frames from the segmented videos after annotating the scene attributes;

[0154] Use the historical driving images obtained by frame extraction as sample video images to construct a sample image set.

[0155] As a possible implementation manner of this application, the driving environment determination unit 63 includes:

[0156] An environment framework acquisition module, configured to acquire a driving environment framework;

[0157] A driving environment determination module, configured to fill the recognition result of the scene attributes into the driving environment framework to obtain the driving environment of the vehicle.

[0158] As a possible implementation manner of this application, the vehicle driving environment recognition device further includes:

[0159] A policy generation unit, configured to generate a driving policy according to the recognized scene attributes and the weights of the scene attributes in the driving environment;

[0160] A driving warning unit, configured to generate driving warning information based on the driving policy.

[0161] As can be seen from the above, in the embodiments of this application, by acquiring the driving images of the vehicle in real time, inputting the driving images into a pre-trained target scene attribute recognition model, extracting the image features of the driving images through the target scene attribute recognition model, fusing the image features with the preset problem text features to obtain target fusion features, and then based on the target fusion features, recognizing the scene attributes of the driving images, making full use of multi-modal information for scene attribute recognition, the scene attributes can be recognized more comprehensively and accurately. Then, according to the scene attributes of the driving images, the driving environment of the vehicle is accurately recognized, assisting the vehicle to make reasonable driving decisions, thereby effectively reducing the possibility of accidents occurring during vehicle driving and improving driving safety.

[0162] It should be noted that for the information interaction, execution process, etc. between the above-mentioned device / units, since they are based on the same concept as the method embodiments of this application, their specific functions and the technical effects brought are specifically described in the method embodiment part, and will not be elaborated here.

[0163] The embodiments of this application also provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the steps of any one of the vehicle driving environment recognition methods as Figures 1 to 5 represented.

[0164] The embodiments of the present application further provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the vehicle driving environment recognition methods as Figures 1 to 5 shown are implemented.

[0165] The embodiments of the present application further provide a computer program product. When the computer program product runs on an electronic device, the electronic device is caused to execute the steps of any one of the vehicle driving environment recognition methods as Figures 1 to 5 shown.

[0166] Figure 7 FIG. is a schematic diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned various embodiments of the vehicle driving environment recognition method are implemented, such as Figure 1 the steps S101 to S103 shown. Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are implemented, such as Figure 6 the functions of the units 61 to 63 shown.

[0167] Exemplarily, the computer program 72 may be divided into one or more modules / units. The one or more modules / units are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 72 in the electronic device 7.

[0168] The electronic device 7 may be an in-vehicle terminal. The electronic device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art can understand that Figure 7 merely an example of the electronic device 7, which does not constitute a limitation on the electronic device 7. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device 7 may further include input / output devices, network access devices, buses, etc.

[0169] The processor 70 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0170] The memory 71 may be an internal storage unit of the electronic device 7, such as the hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk equipped on the electronic device 7, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 71 may also include both the internal storage unit and the external storage device of the electronic device 7. The memory 71 is used to store the computer program and other programs and data required by the electronic device. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0171] It should be noted that for the information interaction, execution process, etc. between the above-mentioned device / units, since they are based on the same concept as the method embodiment of the present application, for their specific functions and the technical effects brought, reference may be specifically made to the method embodiment part, and details are not described herein again.

[0172] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used for illustration. In actual applications, the above functions may be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0174] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0175] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for identifying a vehicle driving environment, characterized in that, Including: Obtaining the driving image of the vehicle in real time; Inputting the driving image into a pre-trained target scene attribute recognition model, extracting the image features of the driving image through the target scene attribute recognition model, fusing the image features with preset question text features to obtain target fusion features, and identifying the scene attributes of the driving image based on the target fusion features; Determining the driving environment of the vehicle according to the recognition result of the scene attributes in the driving image.

2. The method according to claim 1, characterized in that, Before the step of inputting the driving image into a pre-trained target scene attribute recognition model, it further includes: Obtaining a sample driving image set, where the sample driving images in the sample driving image set include scene attribute annotations; Generating question-and-answer texts corresponding to the sample driving images according to the scene attribute annotations; Constructing a scene attribute recognition model; Using the sample driving images and the questions in their corresponding question-and-answer texts in the sample driving image set as the input for model training, and using the scene attribute annotations of the sample driving images as the target output to train the scene attribute recognition model; When the training of the scene attribute recognition model reaches the preset training conditions, obtaining a target scene attribute recognition model.

3. The method according to claim 2, wherein The scene attribute recognition model includes an image feature extraction layer, a text feature extraction layer, a feature fusion layer, and a feature decoding layer; The step of training the scene attribute recognition model includes: Extracting sample driving image features in the sample driving images through the image feature extraction layer; Extracting sample text features of the questions in the question-and-answer texts through the text feature extraction layer; Fusing the sample driving image features and the sample text features through the feature fusion layer to obtain sample fusion features; Predicting the scene attributes in the sample driving images according to the sample fusion features through the feature decoding layer; Calculating the training loss between the predicted scene attributes and the scene attribute annotations, and optimizing the model parameters of the scene attribute recognition model according to the training loss.

4. The method according to claim 3, wherein The step of fusing the sample driving image features and the sample text features through the feature fusion layer to obtain sample fusion features includes: Using a cross-attention mechanism to fuse the sample driving image features and the sample text features to obtain sample fusion features.

5. The method according to claim 2, characterized in that, The step of obtaining the sample driving image set includes: Collecting historical driving videos; Performing an interception process on the historical driving videos to obtain multiple video segments, where the historical driving images in each video segment have the same scene attributes; Performing scene attribute annotation on each video segment; Performing frame extraction on the video segments after scene attribute annotation; Using the historical driving images obtained by frame extraction as sample video images to construct a sample image set.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the driving environment of the vehicle according to the recognition result of the scene attributes in the driving image includes: Obtaining a driving environment framework; Filling the recognition result of the scene attributes into the driving environment framework to obtain the driving environment of the vehicle.

7. The method according to any one of claims 1 to 5, characterized in that After the step of determining the driving environment of the vehicle according to the recognition result of the scene attribute in the driving image, the method further includes: Generating a driving strategy according to the recognized scene attribute and the weight of the scene attribute in the driving environment; Generating driving warning information based on the driving strategy.

8. A vehicle driving environment recognition device, characterized in that The method includes: An image acquisition unit, configured to acquire a driving image of the vehicle in real time; An attribute recognition unit, configured to input the driving image into a pre-trained target scene attribute recognition model, extract the image features of the driving image through the target scene attribute recognition model, fuse the image features with preset question text features to obtain target fusion features, and recognize the scene attribute of the driving image based on the target fusion features; A driving environment determination unit, configured to determine the driving environment of the vehicle according to the recognition result of the scene attribute in the driving image.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the vehicle driving environment recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the vehicle driving environment recognition method according to any one of claims 1 to 7 is implemented.

Images