Vehicle behavior recognition method, intelligent device and readable storage medium

Abstract

The application relates to the technical field of automatic driving, in particular to a vehicle behavior identification method, an intelligent device and a readable storage medium, and aims to solve the technical problem of how to accurately identify the behavior of other vehicles related to a self vehicle. To this end, the application obtains a scene video collected by the self vehicle and an identification task text prompt of a deep learning neural network model, performs a vehicle behavior identification task on other vehicles related to the self vehicle in the scene video based on the deep learning neural network model, performs true value labeling by using a tree thinking chain, realizes disassembly of the vehicle behavior identification task, makes the process of the vehicle behavior identification task more flexible, and covers more extensive scenes. The deep learning neural network model is trained based on the tree thinking chain, the deep learning neural network model is guided to perform part disassembly and logical reasoning according to the tree thinking chain during the vehicle behavior identification task, and the accuracy and reliability of the vehicle behavior identification task are improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a vehicle behavior recognition method, an intelligent device, and a readable storage medium. Background Technology

[0002] With the widespread adoption of autonomous driving applications, accurately identifying stationary vehicles on the road, especially those that are considered "dead cars" and require detours, is crucial for improving the efficiency of autonomous driving traffic and ensuring driving safety. However, dead car identification requires logical reasoning and comprehensive analysis, taking into account various factors such as vehicle status, road conditions, and traffic conditions.

[0003] Traditional methods based on perception models and post-processing logic can only handle a few typical scenarios of vehicle crashes. In practical applications, they often result in missed or incorrect judgments, which seriously affects the traffic efficiency of autonomous vehicles and also brings a poor autonomous driving experience to users.

[0004] Accordingly, there is a need in this field for a new vehicle behavior recognition scheme to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned deficiencies, this application is made to solve, or at least partially solve, the technical problem of how to accurately identify the behavior of other vehicles related to one's own vehicle.

[0006] In a first aspect, a vehicle behavior recognition method is provided, the method comprising:

[0007] The system acquires scene videos collected from the vehicle and text prompts for the recognition task of a preset deep learning neural network model; the text prompts for the recognition task are text prompts describing the preset vehicle behavior recognition task performed by the deep learning neural network model.

[0008] Based on the deep learning neural network model, and according to the scene video and the recognition task text prompts, the vehicle behavior recognition task is performed on other vehicles related to the vehicle in the scene video to obtain the behavior recognition results of other vehicles related to the vehicle in the scene video.

[0009] The deep learning neural network model is obtained based on supervised training. The ground truth data for supervised training is obtained by labeling other vehicles related to the vehicle in the training scene video according to a preset tree-like thinking chain. The ground truth data is used to supervise the behavior recognition results output by the deep learning neural network model in each iteration of the supervised training, so as to realize the training of the deep learning neural network model. The tree-like thinking chain is used to label the ground truth based on at least one of the vehicle position, vehicle state and external environment of the other vehicles related to the vehicle.

[0010] In one technical solution of the above vehicle behavior recognition method, the method further includes obtaining the ground truth data according to the following steps:

[0011] The scene video used for training is labeled with key clues to obtain key clue labeling results; wherein, the key clues include at least one of the vehicle position, vehicle status and external environment of other vehicles related to the vehicle.

[0012] Based on the key clue annotation results, key clue analysis is performed to obtain key clue analysis results;

[0013] Based on the analysis results of the key clues and the tree-like thinking chain, the scene videos used for training are labeled with ground truth to obtain the ground truth data of the scene videos used for training.

[0014] In one technical solution of the above vehicle behavior recognition method, the step of annotating the scene video used for training with key clues and obtaining the key clue annotation results includes:

[0015] Text annotations are performed on other vehicles related to the vehicle in the scene video used for training, including vehicle location, vehicle status, and at least one key clue in the external environment, to obtain the key clue annotation results.

[0016] In one technical solution of the above vehicle behavior recognition method, the step of performing key clue analysis based on the key clue annotation results to obtain key clue analysis results includes:

[0017] Based on the key clue annotation results, key clue analysis is performed to obtain at least one of the traffic rule information and driving standard information corresponding to the scene video, which is used as the key clue analysis result.

[0018] In one technical solution of the above vehicle behavior recognition method, the method further includes supervised training of the deep learning neural network model according to the following steps:

[0019] For each iteration of the supervised training, the scene video used for training and the text prompts for the recognition task are used as input data for the deep learning neural network model to obtain the behavior recognition result of the current iteration of the deep learning neural network model;

[0020] Based on the behavior recognition results of the current iteration and the ground truth data, obtain the loss of the current iteration;

[0021] Based on the loss, the parameters of the deep learning neural network model are updated, and the next iteration is performed to achieve supervised training of the deep learning neural network model.

[0022] In one technical solution of the above vehicle behavior recognition method, the vehicle behavior recognition task is to determine whether the vehicle in front of the vehicle in the scene video is a dead vehicle.

[0023] The term "dead car" refers to a stationary vehicle that does not currently have any intention to move forward.

[0024] In one technical solution of the above vehicle behavior recognition method, the method further includes constructing the tree-like thought chain according to the following steps:

[0025] Construct multiple node problems based on at least one of the following in the scene video: the vehicle position and status of the vehicle in front of the vehicle, and the external environment.

[0026] Based on the relationships between the node problems, the tree-like thinking chain is constructed.

[0027] In one technical solution of the above vehicle behavior recognition method, the step of constructing the tree-like thought chain based on the relationship between the node problems includes:

[0028] S1: Determine whether the hazard lights of the vehicle in front of the vehicle in the scene video are on; if yes, proceed to step S8; if no, proceed to step S2.

[0029] S2: Determine whether there is passable space on both sides of the vehicle in front of the vehicle; if yes, proceed to step S3; if no, proceed to step S10.

[0030] S3: Determine if there are vehicles lined up with the vehicles in front of the vehicle; if yes or if it cannot be determined, proceed to step S4; if no, proceed to step S8.

[0031] S4: Determine whether the deviation angle of the vehicle in front of the vehicle from the direction of its lane is greater than a preset angle threshold; if yes, proceed to step S5; if no, proceed to step S6.

[0032] S5: Determine whether the brake lights of the vehicle in front of the vehicle are on; if yes or cannot be determined, proceed to step S6; if no, proceed to step S8.

[0033] S6: Determine whether the position of the vehicle in front of the vehicle is to the right of the center line of the lane and the distance from the center line of the lane is greater than a preset distance threshold; if yes, proceed to step S7; if no, proceed to step S10.

[0034] S7: Determine whether there is passable space on the left side of the vehicle in front of the vehicle; if yes, proceed to step S9; if no, proceed to step S8.

[0035] S8: Determine that the vehicle in front of the vehicle is a dead vehicle;

[0036] S9: Unable to determine whether the vehicle in front of the vehicle is a dead vehicle;

[0037] S10: Determine that the vehicle in front of the vehicle is not a dead vehicle.

[0038] In a second aspect, a smart device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above-described technical solutions for vehicle behavior recognition.

[0039] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described technical solutions for vehicle behavior recognition.

[0040] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0041] In implementing the vehicle behavior recognition method provided in this application, this application acquires scene videos collected by the vehicle itself and recognition task text prompts from a deep learning neural network model. Based on the deep learning neural network model, according to the scene video and recognition task text prompts, vehicle behavior recognition tasks are performed on other vehicles related to the vehicle in the scene video, obtaining the behavior recognition results of other vehicles related to the vehicle in the scene video. The deep learning neural network model is obtained based on supervised training, and the ground truth data of supervised training is obtained by ground truth annotation according to a preset tree-like thinking chain. The tree-like thinking chain is ground truth annotated based on at least one of the vehicle position, vehicle state, and external environment of other vehicles related to the vehicle. Through the above configuration method, this application, by applying tree-like thinking chain for ground truth annotation, can decompose the vehicle behavior recognition task, so that the problem nodes after decomposition are interconnected, making the process of performing the vehicle behavior recognition task more flexible, thereby covering a wider range of scenarios. Meanwhile, training a deep learning neural network model based on a tree-like thinking chain can guide the deep learning neural network model to decompose and logically reason about the vehicle behavior recognition task in the form of a tree-like thinking chain during the process of recognizing the vehicle behavior of other vehicles related to the vehicle itself. This improves the accuracy and reliability of the deep learning neural network model's vehicle behavior recognition task, thereby effectively improving the traffic efficiency and safety of the vehicle during driving. Attached Figure Description

[0042] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0043] Figure 1 This is a schematic flowchart of the main steps of a vehicle behavior recognition method according to an embodiment of this application;

[0044] Figure 2 This is a schematic flowchart of the main steps of a vehicle behavior recognition method according to one embodiment of the present application.

[0045] Figure 3 This is a flowchart illustrating the main steps of constructing a tree-like thought chain for a vehicle behavior recognition task in a scene video to identify whether the vehicle in front of the vehicle is a dead car, according to one embodiment of this application.

[0046] Figure 4 This is a flowchart illustrating the main steps of a scene video truth annotation process according to one embodiment of this application.

[0047] Figure 5 This is a schematic diagram of the main steps of supervised training of a deep learning neural network model according to one embodiment of the present application.

[0048] Figure 6 This is a schematic flowchart of the main steps of a vehicle behavior recognition method according to one embodiment of the present application.

[0049] Figure 7 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.

[0050] Figure label:

[0051] 11: Memory; 12: Processor. Detailed Implementation

[0052] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0053] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0054] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0055] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0056] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0057] Here we will first explain some of the terms used in this application.

[0058] A tree-like thought chain refers to the hierarchical logical structure formed by a deep learning neural network model when generating results. This structure helps the model understand the input content, thereby generating more logically sound results based on learned knowledge and patterns.

[0059] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a vehicle behavior recognition method according to an embodiment of this application. Figure 1As shown, the vehicle behavior recognition method in this application embodiment mainly includes the following steps S101 to S102.

[0060] Step S101: Acquire the scene video collected by the vehicle and the recognition task text prompts of the preset deep learning neural network model; the recognition task text prompts are text prompts describing the preset vehicle behavior recognition task performed by the deep learning neural network model.

[0061] In this embodiment, scene videos collected by the vehicle and text prompts for recognition tasks from a deep learning neural network model can be obtained.

[0062] In one implementation, the deep learning neural network model can be a Transformer-based model, such as a large language model or a large visual language model.

[0063] In one implementation, the deep learning neural network model can be a Convolutional Neural Network (CNN) model, a Recurrent Neural Network (RNN) model, a Generative Adversarial Network (GAN) model, a Deep Neural Network (DNN) model, or the like.

[0064] In one implementation, the vehicle behavior recognition task can be a task of recognizing whether the vehicle in front of the vehicle in the scene video is a stationary vehicle. Here, a stationary vehicle is a vehicle that does not have a subjective intention to move forward at the current moment, such as a damaged vehicle, a parked vehicle, or a vehicle temporarily picking up or dropping off passengers.

[0065] In one implementation, the vehicle behavior recognition task can be a task of recognizing whether the vehicles in front of the vehicle in the scene video are not stationary. Here, a non-stationary vehicle is a vehicle that is currently stationary, but its stationary state is caused by external environmental factors, and the vehicle itself has a subjective intention to move forward, such as vehicles in a queue or waiting at a red light.

[0066] In a specific example, taking the vehicle behavior recognition task as an example of identifying whether a vehicle in front of your own vehicle in a scene video is a "dead car," the recognition task text prompt could be: "A dead car refers to a stationary vehicle that, at the current moment, has no subjective intention to move forward due to internal reasons, such as a damaged vehicle, a parked vehicle, or a vehicle temporarily picking up or dropping off passengers. Vehicles that have the intention to move but cannot move forward due to external environmental reasons are not considered dead cars, such as vehicles queuing or waiting at a red light. Given that the vehicle in front is stationary, please analyze whether the vehicle is a dead car based on the front view image of your own vehicle. Note that you should construct a complete and logical thought process for reasoning and output the reasoning process and judgment result." Those skilled in the art can adaptively configure the recognition task text prompt according to the needs of actual applications.

[0067] Step S102: Based on the deep learning neural network model, and according to the scene video and recognition task text prompts, perform vehicle behavior recognition tasks on other vehicles related to the vehicle in the scene video to obtain the behavior recognition results of other vehicles related to the vehicle in the scene video; wherein, the deep learning neural network model is obtained based on supervised training, and the ground value data for supervised training is obtained by ground value annotation of other vehicles related to the vehicle in the scene video used for training according to a preset tree-like thinking chain; the ground value data is used to supervise the behavior recognition results output by the deep learning neural network model in each iteration of supervised training, so as to realize the training of the deep learning neural network model; the tree-like thinking chain is used to perform ground value annotation based on at least one of the vehicle position, vehicle state, and external environment of other vehicles related to the vehicle.

[0068] In this embodiment, the scene video collected by the vehicle and the recognition task text prompts can be used as input data for the deep learning neural network model. Based on the deep learning neural network model, the vehicle behavior recognition task of other vehicles related to the vehicle in the scene video is performed, thereby obtaining the behavior recognition results of other vehicles related to the vehicle in the scene video. A tree-like thought chain can be constructed based on at least one of the vehicle positions, vehicle states, and external environment of other vehicles related to the vehicle to obtain ground truth data for supervised training. Based on this tree-like thought chain, ground truth annotation is performed on the scene video used for training to obtain ground truth data for supervised training. The training process of the deep learning neural network model is supervised based on the ground truth data.

[0069] In one implementation, other vehicles associated with the vehicle are those that the vehicle needs to avoid, detour around, or queue behind during its journey.

[0070] In one implementation, the behavior recognition result can be a text-based recognition result, that is, a text-based description of the behavior recognition results of other vehicles related to the vehicle in the scene video.

[0071] In one implementation, the text-based recognition result can be a behavior recognition result that includes the logical reasoning process of the vehicle behavior recognition task. That is, the text-based recognition result of the deep learning neural network model can include both the logical reasoning process of the vehicle behavior recognition task and the behavior recognition result.

[0072] In one implementation, the truth data labeled according to the tree-like thought chain can be truth data in text form.

[0073] In one implementation, other vehicles related to the vehicle can be vehicles in front of the vehicle. Specifically, the vehicle in front of the vehicle is the vehicle closest to the vehicle in front of it. For example, a vehicle behavior recognition task can be performed to determine whether the vehicle in front of the vehicle in the scene video is a stopped vehicle, thereby determining whether the vehicle needs to detour around the stopped vehicle, thus improving the efficiency of autonomous driving and ensuring the driving safety of the vehicle.

[0074] Based on the methods described in steps S101 to S102 above, this embodiment of the application obtains scene video collected by the vehicle and recognition task text prompts from a deep learning neural network model. Based on the deep learning neural network model, according to the scene video and recognition task text prompts, a vehicle behavior recognition task is performed on other vehicles related to the vehicle in the scene video, obtaining the behavior recognition results of other vehicles related to the vehicle in the scene video. The deep learning neural network model is obtained based on supervised training, and the ground truth data of the supervised training is obtained by ground truth annotation according to a preset tree-like thinking chain. The tree-like thinking chain is used to annotate ground truth based on at least one of the vehicle position, vehicle state, and external environment of other vehicles related to the vehicle. Through the above configuration, this embodiment of the application, by applying a tree-like thinking chain for ground truth annotation, can decompose the vehicle behavior recognition task, making the decomposed problem nodes interconnected, thus making the process of performing the vehicle behavior recognition task more flexible and able to cover a wider range of scenarios. Meanwhile, training a deep learning neural network model based on a tree-like thinking chain can guide the deep learning neural network model to decompose and logically reason about the vehicle behavior recognition task in the form of a tree-like thinking chain during the process of recognizing the vehicle behavior of other vehicles related to the vehicle itself. This improves the accuracy and reliability of the deep learning neural network model's vehicle behavior recognition task, thereby effectively improving the traffic efficiency and safety of the vehicle during driving.

[0075] The following sections will further explain the process of obtaining ground truth data for supervised training, the training process of deep learning neural network models, and the process of constructing a tree-like thought chain.

[0076] In one embodiment of this application, ground truth data for supervised training can be obtained according to steps S201 to S203:

[0077] Step S201: Label the scene video used for training with key clues and obtain the key clue labeling results; wherein, the key clues include at least one of the vehicle position, vehicle status and external environment of other vehicles related to the vehicle.

[0078] In this embodiment, text annotations can be performed on other vehicles related to the user vehicle in the scene video used for training, identifying at least one key clue regarding vehicle location, vehicle status, and external environment, and obtaining key clue annotation results. That is, key clues such as vehicle location, vehicle status, and external environment of other vehicles related to the user vehicle in the scene video can be identified, and these key clues can be annotated with text.

[0079] Step S202: Based on the key clue annotation results, perform key clue analysis to obtain key clue analysis results.

[0080] In this embodiment, key clue analysis can be performed based on the key clue annotation results to obtain at least one of the traffic rule information and driving standard information corresponding to the scene video as the key clue analysis result.

[0081] Step S203: Based on the key clue analysis results and the tree-like thinking chain, perform ground truth annotation on the scene videos used for training to obtain ground truth data for the scene videos used for training.

[0082] In this embodiment, ground truth annotation can be performed on the scene videos used for training based on the key clue analysis results and the tree-like thinking chain to obtain the ground truth data of the scene videos used for training.

[0083] In one embodiment of this application, the supervised training of the deep learning neural network model can be performed according to the following steps S301 to S303:

[0084] Step S301: For each iteration of supervised training, the scene video and recognition task text prompts used for training are used as input data for the deep learning neural network model to obtain the behavior recognition result of the current iteration of the deep learning neural network model.

[0085] Step S302: Based on the behavior recognition results and ground truth data of the current iteration, obtain the loss of the current iteration.

[0086] Step S303: Update the parameters of the deep learning neural network model based on the loss and perform the next iteration to achieve supervised training of the deep learning neural network model.

[0087] In this embodiment, in each iteration of supervised training, the scene video and recognition task text prompts used for training can be input into the deep learning neural network model to obtain the behavior recognition result of the current iteration; the loss of the current iteration is calculated based on the behavior recognition result and ground truth data; the parameters of the deep learning neural network model are updated based on the loss of the current iteration, and the next iteration is performed, thereby realizing supervised training of the deep learning neural network.

[0088] In one implementation, taking the example where both the ground truth data and the behavior recognition results are in text form, the ground truth data and behavior recognition results of each iteration can be converted into a vocabulary, and word vectors can be constructed respectively. The distance between the word vectors of the ground truth data and the word count of the behavior recognition results can be calculated. The loss of the current iteration is obtained based on the distance, and the parameters of the deep learning neural network model are updated based on the loss (e.g., the parameters of the deep learning neural network model can be updated based on the gradient descent method), thereby realizing supervised training of the deep learning neural network model.

[0089] The following example uses vehicle recognition as an example, focusing on identifying whether a vehicle in front of the user in a video scene is a stopped car. A deep learning neural network model is used as the basis for this large-scale visual language model. Figures 2 to 6 The vehicle behavior recognition method of the embodiments of this application will be described in detail.

[0090] See appendix Figure 2 , Figure 2 This is a schematic flowchart illustrating the main steps of a vehicle behavior recognition method according to one embodiment of this application. Figure 2 As shown, the vehicle behavior recognition method may include the following steps S401 to S404:

[0091] Step S401: Design of a tree-like thought chain for the vehicle behavior recognition task of a stopped car.

[0092] Step S402: Perform ground truth annotation on the scene videos used for training based on the tree-like thought chain.

[0093] Step S403: Supervised training of the deep learning neural network model based on the tree-like thinking chain of the vehicle behavior recognition task for dead cars.

[0094] Step S404: Based on the trained deep learning neural network model, perform the vehicle behavior recognition task for a stopped car and obtain the behavior recognition results.

[0095] See appendix Figure 3 , Figure 3This is a schematic diagram illustrating the main steps of constructing a tree-like thought process for a vehicle behavior recognition task, based on one embodiment of this application, to identify whether a vehicle in front of a vehicle in a scene video is a stopped car. (See attached diagram.) Figure 3 As shown, a tree-like thought chain can be constructed according to the following steps S501 to S510:

[0096] Step S501: Determine whether the hazard lights of the vehicle in front of your car are on in the scene video; if yes, proceed to step S508; if no, proceed to step S502.

[0097] Step S502: Determine whether there is passable space on both sides of the vehicle in front of the vehicle; if yes, proceed to step S503; if no, proceed to step S510.

[0098] Step S503: Determine if there are vehicles lined up with the vehicles in front of the vehicle; if yes or if it cannot be determined, proceed to step S504; if no, proceed to step S508.

[0099] Step S504: Determine whether the deviation angle of the vehicle in front of the vehicle from the direction of the lane is greater than a preset angle threshold; if yes, proceed to step S505; if no, proceed to step S506.

[0100] Step S505: Determine whether the brake lights of the vehicle in front of your vehicle are on; if yes or cannot be determined, proceed to step S506; if no, proceed to step S508.

[0101] Step S506: Determine whether the position of the vehicle in front of the vehicle is to the right of the center line of the lane and the distance from the center line of the lane is greater than a preset distance threshold; if yes, proceed to step S507; if no, proceed to step S510.

[0102] Step S507: Determine whether there is passable space on the left side of the vehicle in front of the vehicle; if yes, proceed to step S509; if no, proceed to step S508.

[0103] Step S508: Determine that the vehicle in front of your vehicle is a dead vehicle.

[0104] Step S509: Unable to determine whether the vehicle in front of the vehicle is a dead vehicle.

[0105] Step S510: Determine that the vehicle in front of your vehicle is not a dead vehicle.

[0106] See appendix Figure 4 , Figure 4 This is a schematic flowchart illustrating the main steps of a scene video truth annotation process according to one embodiment of this application. Figure 4As shown, key clues regarding the vehicle's position, status, and external environment can be labeled for vehicles in front of the user in the scene video to obtain key clue labeling results; key clue analysis can be performed based on the key clue labeling results to obtain key clue analysis results; based on the key clue analysis results and the tree-like thinking chain for judging whether the vehicle in front of the user is a dead car, ground truth labeling of whether the vehicle in front of the user in the scene video used for training is a dead car can be performed to obtain ground truth data for the scene video used for training.

[0107] For example, see Table 1, which is an example table of the process of ground truth annotation of scene videos used for training based on a tree-like thought chain. It includes key clue annotation results, key clue analysis results, and ground truth data of scene videos used for training.

[0108] Table 1. Example of ground truth labeling process for scene videos used for training based on tree-like thinking chain.

[0109] Note that "the vehicle in front" in Table 1 refers to the vehicle in front of your vehicle, such as the vehicle closest to your vehicle in front of you.

[0110] See appendix Figure 5 , Figure 5 This is a schematic diagram illustrating the main steps of supervised training of a deep learning neural network model according to one embodiment of this application, as shown below. Figure 5 As shown, the training scene videos can be annotated using a tree-like thought chain to obtain ground truth data for the thought chain of dead car recognition. A text prompt for the dead car recognition task is then set: "A dead car is a stationary vehicle that, at the current moment, has no subjective intention to move forward due to internal reasons, such as a damaged vehicle, a parked vehicle, or a vehicle temporarily picking up or dropping off passengers. Vehicles that have the intention to move but cannot move forward due to external environmental reasons are not considered dead cars, such as vehicles queuing or waiting at a red light. Given that the vehicle in front is stationary, please analyze whether the vehicle is a dead car based on your own front view image. Note that you should construct a complete and logical thought chain for reasoning and output the reasoning process and judgment result." The training scene videos and the text prompt for the dead car recognition task are input into the visual language model to obtain the dead car recognition result text output as the behavior recognition result of the current iteration. By using the truth data of the thought chain as the supervisory data for the text output of the dead vehicle recognition result, the visual language big model is trained under supervision. This enables the visual language big model to automatically perform step-by-step analysis and logical reasoning when performing dead vehicle recognition, and to give accurate and reliable dead vehicle recognition results.

[0111] See appendix Figure 6 , Figure 6This is a schematic flowchart illustrating the main steps of a vehicle behavior recognition method according to one embodiment of this application. Figure 6 As shown, the scene video collected by the vehicle and the text prompt for the dead vehicle recognition task are output to the visual language model to obtain the behavior recognition result of the vehicle in front of the vehicle in the form of a text output of the dead vehicle recognition result. The text prompt for the dead vehicle recognition task is: "A dead vehicle is a stationary vehicle that, at the current moment, has no subjective intention to move forward due to internal reasons, such as a damaged vehicle, a parked vehicle, or a vehicle temporarily picking up or dropping off passengers. Vehicles that have the intention to move but cannot move forward due to external environmental reasons are not considered dead vehicles, such as vehicles queuing or waiting at a red light. Given that the vehicle in front is stationary, please analyze whether the vehicle is a dead vehicle based on the front view image of the vehicle. Note that you should construct a complete and logical thought process for reasoning and output the reasoning process and judgment result." An example is shown below. Figure 6 As shown, the text output of the dead vehicle identification result can be: "The vehicle in front is located in the leftmost lane, and is approaching an intersection, indicating that the leftmost lane is likely a left-turn lane. Although it's impossible to determine if there are queuing vehicles ahead due to obstruction, the vehicle is within that lane and has driving conditions, suggesting it may be waiting to turn left. Therefore, the vehicle in front may be in a state of 'having a subjective intention to move but unable to move forward due to external reasons.' Furthermore, the vehicle in front is a construction vehicle, but no construction-related clues are visible nearby, indicating no obvious intention to stop due to 'subjective reasons' such as emergency breakdown, temporary work, or picking up / dropping off passengers. Based on the above analysis, the vehicle in front is not a dead vehicle."

[0112] Note, Figure 5 and Figure 6 The "vehicle in front" in this context refers to the vehicle in front of your vehicle, such as the vehicle closest to your vehicle in front of you.

[0113] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0114] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0115] Another aspect of this application provides a computer-readable storage medium.

[0116] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the vehicle behavior recognition method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described vehicle behavior recognition method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0117] Another aspect of this application provides a smart device.

[0118] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The smart device described in this application may include driving equipment, smart vehicles, robots, and other devices. See appendix. Figure 7 , Figure 7 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.

[0119] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments.

[0120] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A vehicle behavior recognition method, characterized in that, The method includes: The system acquires scene videos collected from the vehicle and text prompts for the recognition task of a preset deep learning neural network model; the text prompts for the recognition task are text prompts describing the preset vehicle behavior recognition task performed by the deep learning neural network model. Based on the deep learning neural network model, and according to the scene video and the recognition task text prompts, the vehicle behavior recognition task is performed on other vehicles related to the vehicle in the scene video to obtain the behavior recognition results of other vehicles related to the vehicle in the scene video. The deep learning neural network model is obtained based on supervised training. The ground truth data for supervised training is obtained by labeling other vehicles related to the vehicle in the training scene video according to a preset tree-like thinking chain. The ground truth data is used to supervise the behavior recognition results output by the deep learning neural network model in each iteration of the supervised training, so as to realize the training of the deep learning neural network model. The tree-like thinking chain is used to label the ground truth based on at least one of the vehicle position, vehicle state and external environment of the other vehicles related to the vehicle.

2. The vehicle behavior recognition method according to claim 1, characterized in that, The method further includes obtaining the truth data according to the following steps: The scene video used for training is labeled with key clues to obtain key clue labeling results; wherein, the key clues include at least one of the vehicle position, vehicle status and external environment of other vehicles related to the vehicle. Based on the key clue annotation results, key clue analysis is performed to obtain key clue analysis results; Based on the analysis results of the key clues and the tree-like thinking chain, the scene videos used for training are labeled with ground truth to obtain the ground truth data of the scene videos used for training.

3. The vehicle behavior recognition method according to claim 2, characterized in that, The step of annotating key clues in the scene video used for training and obtaining the key clue annotation results includes: Text annotations are performed on other vehicles related to the vehicle in the scene video used for training, including vehicle location, vehicle status, and at least one key clue in the external environment, to obtain the key clue annotation results.

4. The vehicle behavior recognition method according to claim 2, characterized in that, The step of performing key clue analysis based on the key clue annotation results to obtain key clue analysis results includes: Based on the key clue annotation results, key clue analysis is performed to obtain at least one of the traffic rule information and driving standard information corresponding to the scene video, which is used as the key clue analysis result.

5. The vehicle behavior recognition method according to claim 1, characterized in that, The method further includes performing supervised training on the deep learning neural network model according to the following steps: For each iteration of the supervised training, the scene video used for training and the text prompts for the recognition task are used as input data for the deep learning neural network model to obtain the behavior recognition result of the current iteration of the deep learning neural network model; Based on the behavior recognition results of the current iteration and the ground truth data, obtain the loss of the current iteration; Based on the loss, the parameters of the deep learning neural network model are updated, and the next iteration is performed to achieve supervised training of the deep learning neural network model.

6. The vehicle behavior recognition method according to any one of claims 1 to 5, characterized in that, The vehicle behavior recognition task is to identify whether the vehicle in front of the vehicle in the scene video is a dead vehicle. The term "dead car" refers to a stationary vehicle that does not currently have any intention to move forward.

7. The vehicle behavior recognition method according to claim 6, characterized in that, The method also includes constructing the tree-like thought chain according to the following steps: Construct multiple node problems based on at least one of the following in the scene video: the vehicle position and status of the vehicle in front of the vehicle, and the external environment. Based on the relationships between the node problems, the tree-like thinking chain is constructed.

8. The vehicle behavior recognition method according to claim 7, characterized in that, The step of constructing the tree-like thought chain based on the node problems and the relationships between them includes: S1: Determine whether the hazard lights of the vehicle in front of the vehicle in the scene video are on; if yes, proceed to step S8; if no, proceed to step S2. S2: Determine whether there is passable space on both sides of the vehicle in front of the vehicle; if yes, proceed to step S3; if no, proceed to step S10. S3: Determine if there are vehicles lined up with the vehicles in front of the vehicle; if yes or if it cannot be determined, proceed to step S4; if no, proceed to step S8. S4: Determine whether the deviation angle of the vehicle in front of the vehicle from the direction of its lane is greater than a preset angle threshold; if yes, proceed to step S5; if no, proceed to step S6. S5: Determine whether the brake lights of the vehicle in front of the vehicle are on; if yes or cannot be determined, proceed to step S6; if no, proceed to step S8. S6: Determine whether the position of the vehicle in front of the vehicle is to the right of the center line of the lane and the distance from the center line of the lane is greater than a preset distance threshold; if yes, proceed to step S7; if no, proceed to step S10. S7: Determine whether there is passable space on the left side of the vehicle in front of the vehicle; if yes, proceed to step S9; if no, proceed to step S8. S8: Determine that the vehicle in front of the vehicle is a dead vehicle; S9: Unable to determine whether the vehicle in front of the vehicle is a dead vehicle; S10: Determine that the vehicle in front of the vehicle is not a dead vehicle.

9. A smart device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the vehicle behavior recognition method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the vehicle behavior recognition method according to any one of claims 1 to 8.

Images