A method, system, device and medium for identifying interactive behavior of multiple traffic participants

The graph neural network model is used to model and identify the interaction relationship between traffic participants, which solves the problem of low recognition accuracy of interaction behavior among multiple traffic participants in the prior art, and achieves higher recognition accuracy and reliability.

CN113989782BActive Publication Date: 2024-11-01BEIJING INST OF TECH +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111310948.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-08
Publication Date
2024-11-01
Estimated Expiration
2041-11-08

AI Technical Summary

Technical Problem

Existing traffic participant behavior recognition methods usually only focus on the interaction between a certain type of traffic participant and the main car, ignoring the interaction between multiple traffic participants, resulting in low recognition accuracy.

Method used

Using graph neural network model, traffic participants are modeled as nodes and hypernodes through the instance layer and the category layer, and interactive relationships between traffic participants are modeled using time and space to perform interaction behaviors among multiple traffic participants. Identification.

Benefits of technology

It significantly improves the recognition accuracy of the interaction behavior of multiple traffic participants in complex traffic scenarios, and obtains more accurate and reliable identification results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989782B_ABST
    Figure CN113989782B_ABST
Patent Text Reader

Abstract

The present invention relates to a method, system, device and medium for identifying interactive behaviors of multiple traffic participants, belonging to the field of traffic participant behavior identification. First, a data set is obtained; the data set is divided to obtain a training set and a verification set; a graph neural network model is established; the graph neural network model includes an instance layer and a category layer; the training set and the verification set are preprocessed to obtain a preprocessed training set and a verification set; the preprocessed training set is input into the graph neural network model for training to obtain a trained graph neural network model; finally, the preprocessed verification set is input into the trained graph neural network model to obtain the identification results of the interactive behaviors of multiple traffic participants. By comprehensively considering the interactive behaviors between all traffic participants, the recognition accuracy of traffic interactive behaviors can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of traffic participant behavior recognition, and in particular to a method, system, device and medium for recognizing interactive behavior of multiple traffic participants. Background Art

[0002] At present, the development of intelligent transportation is in full swing. The construction and development of intelligent transportation can effectively improve road mobility. Autonomous driving is a major development trend in intelligent transportation. If you want to achieve safe autonomous driving in complex dynamic scenes, then the recognition of the behavior of traffic participants is particularly important.

[0003] However, existing traffic participant behavior recognition methods usually only focus on the interaction between a certain type of traffic participant and the main vehicle, such as only considering the interaction between electric vehicles and the main vehicle, or the interaction between pedestrians and the main vehicle. Since the traffic environment is usually complex, the interaction between the main vehicle in the traffic environment generally involves multiple traffic participants. Therefore, if only the interaction between a certain type of traffic participant and the main vehicle is considered, while ignoring the interaction between multiple traffic participants, the final recognition result will be inaccurate. Therefore, existing traffic interaction behavior recognition methods generally have the problem of low recognition accuracy due to not considering the interaction between multiple traffic participants. Summary of the invention

[0004] The purpose of the present invention is to provide a method, system, device and medium for identifying interactive behaviors of multiple traffic participants, which can improve the recognition accuracy of interactive behaviors of multiple traffic participants and solve the problem of low recognition accuracy caused by not considering the recognition of interactive behaviors between multiple traffic participants in the prior art.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] In one aspect, the present invention provides a method for identifying interactive behaviors of multiple traffic participants, comprising:

[0007] Acquire a data set; the data set includes multiple traffic scenes and interactive behaviors of multiple traffic participants;

[0008] Dividing the data set into a training set and a validation set;

[0009] A graph neural network model is established; the graph neural network model includes an instance layer and a category layer; in the instance layer, each traffic participant at different times is defined as a node, and the nodes are connected through time edges and space edges; in the category layer, different types of traffic participants at the same time are defined as super nodes, and the super nodes are connected to the nodes through information transfer edges;

[0010] Preprocessing the training set and the validation set according to the graph neural network model to obtain a preprocessed training set and a preprocessed validation set;

[0011] Inputting the preprocessed training set into the graph neural network model for training to obtain a trained graph neural network model;

[0012] The preprocessed validation set is input into the trained graph neural network model to obtain recognition results of the interactive behaviors of multiple traffic participants.

[0013] On the other hand, the present invention also provides a system for identifying interactive behaviors of multiple traffic participants, comprising:

[0014] A data set acquisition module, used to acquire a data set; the data set includes a variety of traffic scenes and interactive behaviors of a variety of traffic participants;

[0015] A data set partitioning module, used to partition the data set into a training set and a validation set;

[0016] A graph neural network model building module, used to build a graph neural network model; the graph neural network model includes an instance layer and a category layer; in the instance layer, each traffic participant at different times is defined as a node, and the nodes are connected through time edges and space edges; in the category layer, different types of traffic participants at the same time are defined as super nodes, and the super nodes are connected to the nodes through information transfer edges;

[0017] A data set preprocessing module, used to preprocess the training set and the validation set according to the graph neural network model to obtain a preprocessed training set and a preprocessed validation set;

[0018] A graph neural network model training module, used to input the preprocessed training set into the graph neural network model for training to obtain a trained graph neural network model;

[0019] The interactive behavior recognition result output module is used to input the preprocessed verification set into the trained graph neural network model to obtain the recognition results of the interactive behaviors of multiple traffic participants.

[0020] On the other hand, the present invention also provides a device for identifying interactive behaviors of multiple traffic participants, comprising: a memory, a processor, and a computer program stored in the memory and running on the processor, and when the computer program is executed by the processor, the various steps of the method for identifying interactive behaviors of multiple traffic participants are implemented.

[0021] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the various steps of the method for identifying interactive behaviors of multiple traffic participants are implemented.

[0022] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0023] The present invention proposes a method for identifying interactive behaviors of multiple traffic participants based on graph neural networks. The method takes a data set with interactive behaviors of multiple traffic scenes and multiple traffic participants as the data basis, focuses on the interaction between each traffic participant in the traffic scene, takes into account the interactive behaviors between multiple types of traffic participants, and adopts a graph neural network model as an identification model for interactive behaviors between traffic participants. The data set is combined with the graph neural network model, and the graph neural network model is trained and verified using data samples in the data set, so that the characteristics of multiple traffic scenes and multiple traffic participants in the data set are reflected in the graph neural network model, so that the trained graph neural network model can have higher recognition accuracy for interactive behaviors between multiple types of traffic participants, and can significantly improve the recognition accuracy of interactive behaviors of multiple traffic participants in complex traffic scenes, and obtain more accurate and reliable recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The following drawings are not deliberately scaled according to the actual size, and the focus is on illustrating the main purpose of the present invention.

[0025] Figure 1 A flowchart of a method for identifying interactive behaviors of multiple traffic participants provided in Example 1 of the present invention;

[0026] Figure 2 A schematic diagram of the structure of the graph neural network model provided in Example 1 of the present invention;

[0027] Figure 3 A structural block diagram of a multi-traffic participant interactive behavior recognition system provided in Example 2 of the present invention;

[0028] Figure 4 A structural block diagram of a device for identifying interactive behaviors of multiple traffic participants provided in Embodiment 3 of the present invention;

[0029] Figure 5 This is a structural block diagram of a computer-readable storage medium provided for Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] As shown in the present invention and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0032] Although the present invention has made various references to certain modules in the system according to an embodiment of the present invention, any number of different modules can be used and run on a user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0033] The present invention uses a flow chart to illustrate the operations performed by the system according to an embodiment of the present invention. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0034] The purpose of the present invention is to provide a method, system, device and medium for identifying interactive behaviors of multiple traffic participants, which comprehensively considers all traffic participants in the traffic environment and simultaneously identifies the interactive behaviors between multiple traffic participants and the vehicle, so as to improve the recognition accuracy of traffic interactive behaviors and solve the problem of low recognition accuracy caused by not considering the recognition of interactive behaviors between multiple traffic participants in the prior art.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment proposes a method for identifying interactive behaviors of multiple traffic participants, which specifically includes the following steps:

[0038] Step S1, obtaining a data set; the data set includes a variety of traffic scenes and a variety of interactive behaviors of traffic participants. Specifically including:

[0039] Step S1.1, selecting a data set in the form of video images containing a variety of traffic scenes and a variety of interactive behaviors of traffic participants as the data set;

[0040] Step S1.2, marking the types and coordinates of all traffic participants in each frame of each video in the data set, and setting a unique ID for each of the traffic participants; the types of traffic participants include pedestrians, non-motor vehicles and motor vehicles;

[0041] Step S1.3: labeling the interaction behavior category between the main vehicle and other traffic participants around the main vehicle in each frame image of each video segment in the data set.

[0042] The BLVD dataset is used in this embodiment, and the BLVD dataset is used as the data basis for interactive behavior recognition between multiple traffic participants. Unlike previous datasets for static detection, semantic or instance segmentation tasks, the BLVD dataset aims to provide a unified verification platform for tasks such as dynamic 4D tracking, 5D interactive event recognition, and intention prediction. Among them, 4D refers to adding "time" to the traditional 3D space, and 5D refers to adding "interactive behavior" to 4D. Compared with previous three-dimensional target tracking, behavior understanding and analysis datasets in driving scenes, the BLVD dataset has richer traffic scenes, including different driving scenes (cities and highways), multiple lighting conditions (daytime and nighttime), and multiple individual densities. Therefore, based on the above advantages of the BLVD dataset, the BLVD dataset is very suitable for interactive behavior recognition scenarios of multiple traffic participants in traffic scenes.

[0043] This embodiment selects the BLVD dataset as the dataset for training and verification of the graph neural network model. The BLVD dataset is a dataset containing a variety of traffic scenes. The dataset includes 654 calibrated video images with a sampling frequency of 10 Hz. Each frame image in the video is a moment. For each frame image, the BLVD dataset marks the type and coordinate data of the surrounding traffic participants at the current moment, and marks a unique ID for each traffic participant.

[0044] The types of traffic participants include pedestrians, non-motor vehicles and motor vehicles. In the BLVD dataset, the code 1 represents non-motor vehicles, 2 represents motor vehicles, and 3 represents pedestrians. Since non-motor vehicles and motor vehicles basically represent all types of vehicles in traffic scenes, combined with pedestrians, these three types of traffic participants actually include all types of traffic participants that can appear in traffic scenes.

[0045] In addition, each frame of each video in the BLVD dataset also marks the interactive behavior category between the main vehicle and other traffic participants around the main vehicle at each moment. There are 13 interactive behavior categories, namely overtaking from the left side of the main vehicle, overtaking from the right side of the main vehicle, accelerating in a straight line in front of the main vehicle, decelerating in a straight line in front of the main vehicle, driving at a constant speed in front of the main vehicle, merging into the main vehicle lane from the right lane, merging into the main vehicle lane from the left lane, changing lanes to the right in front of the main vehicle, changing lanes to the left in front of the main vehicle, parallel driving on the left side of the main vehicle, parallel driving on the right side of the main vehicle, stopping in front of the main vehicle and meeting the main vehicle. The above 13 interactive behavior categories are marked with numbers 1 to 13.

[0046] In this embodiment, behavior recognition refers to the identification of the above 13 types of interactive behaviors. The labels 1 to 13 corresponding to these 13 types of interactive behaviors will be used as the true values ​​during the training of the graph neural network model to train the graph neural network model.

[0047] The present invention takes the BLVD dataset with multiple traffic scenes and interactive behaviors of multiple traffic participants as the data basis, focuses on the interaction between various traffic participants in the traffic scenes, takes into account the interactive behaviors between various types of traffic participants, and uses the BLVD dataset as the training and verification dataset of the graph neural network model, thereby realizing the recognition of multiple interactive behaviors between multiple traffic participants, thereby improving the recognition accuracy of the graph neural network model for traffic interactive behaviors.

[0048] It should be noted that this embodiment does not limit the specific data set, and the BLVD data set is only a preferred solution and should not be used as a limitation on the scope of protection of the present invention. In addition to the preferred BLVD data set, other data sets in the field of intelligent vehicles can also be used, which can be determined according to actual needs.

[0049] Step S2: Divide the data set into a training set and a validation set. Specifically, it includes:

[0050] All data in the data set are divided according to a preset ratio to obtain the training set and the validation set. The preset division ratio is not fixed, and the ratio of the training set to the sample set can be 8:2 or 9:1, etc., which can be set according to actual conditions.

[0051] It should also be noted that in this embodiment, in order to test the training effect of the graph neural network model, a test set can also be divided. This embodiment preferably divides the BLVD data set into a training set, a validation set, and a test set in a ratio of 8:1:1. Alternatively, the sample data can also be divided in a ratio of 6:2:2 or 7:1.5:1.5. Generally, the more sample data in the training set, the higher the accuracy of the trained model. Therefore, this embodiment preferably obtains a training set, a validation set, and a test set in a ratio of 8:1:1.

[0052] It is easy to understand that the ratio of 8:1:1 used in this embodiment is only a preferred value, but this value is not fixed or unique, and should not be used as a limitation on the scope of protection of the present invention. The specific ratio value can be set according to the actual situation. In addition, whether to divide the test set can be determined according to the actual situation. At least the training set and the verification set need to be divided. On the basis of ensuring model training and verification, the recognition accuracy of the model can also be tested using the test set to ensure the recognition effect of the model.

[0053] Step S3: Establish a graph neural network model.

[0054] like Figure 2 As shown, the graph neural network model includes an instance layer and a category layer. In the instance layer, each traffic participant at different times is defined as a node, and the nodes are connected through time edges and space edges; in the category layer, different types of traffic participants at the same time are defined as super nodes, and the super nodes are connected to the nodes through information transfer edges.

[0055] The process of establishing the graph neural network model in the present invention specifically includes:

[0056] Step S3.1: In the instance layer, define each traffic participant at each moment as a node , using LSTM neural network to process the characteristic information of each traffic participant ; Use LSTM neural network to process the relative position information of the two nodes in each spatial edge ; Use LSTM neural network to process the historical movement information of the two nodes in each time edge in the time series The spatial edge is bidirectional, indicating an edge connecting two different nodes at the same time; the temporal edge is unidirectional, indicating an edge connecting the same node at two different times.

[0057] Since the number of traffic participants in the traffic scene at each moment is not constant, a traffic participant may still be interacting with other traffic participants in the traffic scene at the current moment, but may have left the traffic scene in the next moment, or new traffic participants may enter the traffic scene. Therefore, the graph neural network model is a dynamic model, which is reflected in the disappearance of old nodes and the establishment of new nodes.

[0058] Step S3.2: In the category layer, define different types of traffic participants at each moment as a supernode There is an information transmission edge between each super node and each node in the instance layer of all corresponding types at the current moment. The information transmission edge is used to aggregate the interactive behavior information of the same type of traffic participants to the super node, and then the super node uses the LSTM neural network to process the interactive behavior information and return the interactive behavior information to the corresponding node in the instance layer to obtain the recognition result of the interactive behavior information; there is also a time edge between the same super node at two different moments, and the LSTM neural network is used to process the historical information of the two super nodes in the time series in each time edge.

[0059] Step S3.3: The instance layer and the category layer are connected through the information transfer edge between the node and the super node of the corresponding category.

[0060] The present invention uses the instance layer of the graph neural network model to model the traffic participants in the scene as nodes, and models the spatial and temporal influence relationships as edges, namely temporal edges and spatial edges, and considers the movement patterns of different types of traffic participants through the category layer of the graph neural network model to identify the interactive behaviors between traffic participants and the main vehicle. Taking all types of traffic participants in the traffic scene, such as pedestrians, motor vehicles and non-motor vehicles, into account in the graph neural network model, and showing the interactive relationship of information transmission between various traffic participants in the graph neural network model can effectively improve the recognition accuracy of the graph neural network model for the interactive behaviors of multiple traffic participants, thereby making the recognition results of interactive behaviors more accurate and reliable.

[0061] Step S4, preprocessing the training set and the validation set according to the graph neural network model to obtain a preprocessed training set and a preprocessed validation set. Specifically comprising:

[0062] According to the data structure required in the graph neural network model, the data in the training set and the validation set are preprocessed to obtain the preprocessed training set and the preprocessed validation set that can be input into the graph neural network model.

[0063] In this embodiment, since a test set is also provided, the data in the training set, the validation set and the test set are preprocessed to obtain the preprocessed training set, validation set and test set respectively, which can be input into the graph neural network model.

[0064] Step S4 specifically includes the following steps:

[0065] Step S4.1: Set the characteristics of the traffic participant to ,in Indicates the x, y coordinate information of the traffic participant, Indicates the type of traffic participants. It represents the characteristic information of traffic participants input when the LSTM neural network is used to process the nodes in the graph neural network model. The long short-term memory neural network (LSTM) processing process is required when constructing the graph neural network model.

[0066] Step S4.2: Since the spatial edge is bidirectional, Figure 2 As shown, when considering When Represents a node Nodes in space The impact is characterized by ;in, Respectively represent nodes Relative to the node The x, y coordinate position, Indicates the interaction behavior between two nodes, obtained by coding the types of traffic participants of the two nodes in the BLVD dataset 1-3; where 1 represents non-motor vehicles, 2 represents motor vehicles, and 3 represents pedestrians;

[0067] When considering When Represents a node Nodes in space The impact is characterized by ;in, Respectively represent nodes Relative to the node The x, y coordinate position, It is obtained by coding the types of traffic participants at two nodes in the BLVD data set.

[0068] in, They respectively represent the characteristic information of traffic participants input when the LSTM neural network is used to process the spatial edges in the graph neural network model.

[0069] Step S4.3: Different from spatial edges, temporal edges are unidirectional, indicating the transmission of information in the temporal direction. A i exist t and t When both appear in two adjacent frames at time +1, a time edge will be defined between them, and the time edge is expressed as , which is characterized by ,in, Represent the same node A i exist t +1 time relative to t The x, y coordinate position at the moment, Obtained by coding the traffic participant type of this node in the BLVD data set; Represents the characteristic information of traffic participants that is input when the LSTM neural network processes the time edges in the graph neural network.

[0070] Step S5: input the preprocessed training set into the graph neural network model for training to obtain a trained graph neural network model. Specifically including:

[0071] Step S5.1: After the structure of the data set is preprocessed according to step S4, the preprocessed training set is arranged into a tensor form and input into the graph neural network model for training. When performing LSTM neural network processing, the tensor form of the input nodes, the spatial edges and the temporal edges is M × N ×3, where M Indicates the number of samples input at a time. In this example M The value is 16; N Indicates the length of observation time. In this embodiment N The value is 1.6 seconds, i.e. 16 frames; 3 is the dimension of the input feature information;

[0072] The forward data transmission process in the graph neural network model includes:

[0073] For each node and each edge in the instance layer, a linear mapping function is set to preprocess the input information, and then enter the LSTM neural network after preprocessing:

[0074]

[0075] in, embed (·) represents a linear mapping function, which encodes the input into a vector of a specific length. The specific length here is 128, that is, it is mapped into a 128-dimensional vector; represents the output of the spatial edge, represents the output of the time edge, and the output of the spatial edge and the time edge are the output hidden states of the corresponding LSTM neural network; They represent the weight parameters of the linear mapping layer during input information preprocessing, They respectively represent the weight parameters of the neurons in the LSTM neural network.

[0076] Step S5.2: For the same node A i , after the information of all the edges around it is processed by the LSTM neural network, a weight is assigned through an attention mechanism. The weight represents the degree of influence of the edge on the node, which is expressed as:

[0077]

[0078] in, Represents the corresponding spatial edge output The weight of W ii and W ij They represent the weight parameters of the linear mapping layer when the attention mechanism assigns weights, Dot (·) represents the dot multiplication operation, represents the scale factor; Calculated by weighted average , Represents a node A i The surrounding traffic participants are at all times t Pair Node A i Impact on space;

[0079] and Contains A i All the information of the connected edges is combined and processed by linear mapping to obtain a fixed-length vector ; At the same time, the characteristic information of traffic participants is input when the node is processed by LSTM neural network It is also linearly mapped to , then:

[0080]

[0081] in, and Represent the linear mapping layer weights respectively; and Respectively represent the linear mapping results;

[0082] The above linear mapping results and As the input of the LSTM neural network processing node, the output hidden state is obtained , output hidden state It is a node A i The first stage output is:

[0083]

[0084] in, Represents a node A i exist t The output hidden state of the first stage output at time t; Represents a node A i exist t The output hidden state of the final output at time -1;

[0085] The category layer aggregates the output hidden states from nodes of the same category and neuronal states , extract the characteristics of the same type of traffic participants and feed them back to the nodes, then we have:

[0086]

[0087] in, n Represents a supernode The corresponding number of nodes, Indicates the extracted t The characteristic information of the same type of traffic participants at the same time; comprehensive consideration of the time t The movement characteristics of a certain type of traffic participants, As part of the input to the LSTM neural network processing of the node, the temporal edge between the same supernode in two consecutive frames is similar to the instance layer. The temporal edge is expressed as the intermediate quantity As input, get the hidden state containing time series information ,in, Indicates the extracted t- The characteristic information of the same type of traffic participants at time 1 is:

[0088]

[0089] in, and They represent the weights of the linear mapping layer and the weights of the neurons of the LSTM neural network when obtaining the hidden state containing the time series information; As another part of the input, Together they are input into the LSTM neural network processing of the supernode:

[0090]

[0091] in, and They respectively represent the weight of the linear mapping layer and the weight of the LSTM neural network when the super node is processed by the LSTM neural network; Represents feedback information that is fed back to the instance layer along the message passing edge from supernode to node.

[0092] Step S5.3: When feedback information After returning to the node, the super node Connected m Nodes A m Receive feedback , the node A m exist t The first stage output at the moment Feedback Connect and get the final output after linear mapping :

[0093]

[0094] in, Represents the weight of the linear mapping layer during linear mapping processing after feedback information is returned; Represents a node A m exist t The output hidden state of the first stage output at time t, Represents the node obtained after linear mapping processing A m exist t The output hidden state of the final output at time t;

[0095] Each node is processed by the LSTM neural network and outputs Indicates the interactive behavior recognition result between the traffic participant and the main vehicle corresponding to the node B .

[0096] Step S5.4: The loss function used in the training of the graph neural network model is a multi-classification cross entropy function. The weight parameters of the entire neural network are updated through the gradient descent algorithm of data reverse transmission to realize the training of the graph neural network model and obtain the trained graph neural network model.

[0097] In this embodiment, since a test set is set in advance and preprocessed, in step S5, the preprocessed training set is input into the graph neural network model for training. After the trained graph neural network model is obtained, the following steps may also be included:

[0098] The trained graph neural network model is tested using the preprocessed test set to obtain a tested graph neural network model.

[0099] After the graph neural network training is completed, this embodiment uses a test set to test the recognition accuracy of the graph neural network model, and the recognition accuracy result is 96%, which proves that the trained graph neural network model has high recognition accuracy and can meet the recognition of interactive behaviors of multiple traffic participants in different traffic scenarios, and can obtain accurate and reliable recognition results.

[0100] Step S6: input the preprocessed verification set into the trained graph neural network model to obtain the recognition results of the interactive behaviors of multiple traffic participants.

[0101] After determining the data set and preprocessing the data set, this embodiment allows the data in the data set to be directly input into the graph neural network model for training and testing, and finally uses the verification set to verify the trained graph neural network model. The verification set here is the video data to be identified. After inputting the video data to be identified, the graph neural network model can output the recognition results of the interactive behaviors of multiple traffic participants. Since the various interactive behaviors between multiple types of traffic participants in various scenarios are taken into account when training the graph neural network model using the data set, the interactive behaviors between all types of traffic participants are fully considered. Therefore, this recognition result is more accurate and true, and can better reflect the actual traffic environment and the interactive behaviors between multiple traffic participants.

[0102] The present invention proposes a method for identifying interactive behaviors of multiple traffic participants based on a graph neural network. The method focuses on the interactions between traffic participants in a traffic environment, comprehensively considers the explicit definition of interactive behaviors between all types of traffic participants, and uses the instance layer of the graph neural network model to model the participants in the scene as nodes, and model the spatial and temporal influence relationships as edges, namely temporal edges and spatial edges. The movement patterns of different types of traffic participants are considered through the category layer of the graph neural network model to identify the interactive behaviors between traffic participants and the host vehicle. The present invention takes into account all types of traffic participants in the traffic scene, such as pedestrians, motor vehicles and non-motor vehicles, in the graph neural network model, and represents the interactive relationship of information transmission between each traffic participant in the graph neural network model, which can effectively improve the recognition accuracy of the graph neural network model for the interactive behaviors of multiple traffic participants, thereby making the recognition results of the interactive behaviors more accurate and reliable.

[0103] Existing behavior recognition methods often only consider the interaction between a certain type of traffic participant and the main vehicle, such as only considering the impact of vehicles or pedestrians. The present invention takes into account all types of traffic participants in the traffic scene in the graph neural network model, including all types of traffic participants such as pedestrians, motor vehicles and non-motor vehicles. It has certain innovative significance, not only improves the recognition accuracy, but also can be widely used in various complex traffic scenes. It has universality and is of great significance for identifying the movement trajectory of traffic participants, planning routes, and automatic driving.

[0104] Example 2

[0105] like Figure 3 As shown, this embodiment provides a system for identifying interactive behaviors of multiple traffic participants, which specifically includes:

[0106] The data set acquisition module M1 is used to acquire a data set; the data set includes a variety of traffic scenes and a variety of interactive behaviors of traffic participants;

[0107] The data set division module M2 is used to divide the data set into a training set and a validation set;

[0108] A graph neural network model building module M3 is used to build a graph neural network model; the graph neural network model includes an instance layer and a category layer; in the instance layer, each traffic participant at different times is defined as a node, and the nodes are connected through time edges and space edges; in the category layer, different types of traffic participants at the same time are defined as super nodes, and the super nodes are connected to the nodes through information transfer edges;

[0109] A data set preprocessing module M4 is used to preprocess the training set and the verification set according to the graph neural network model to obtain a preprocessed training set and a preprocessed verification set;

[0110] A graph neural network model training module M5, used to input the preprocessed training set into the graph neural network model for training to obtain a trained graph neural network model;

[0111] The interactive behavior recognition result output module M6 is used to input the preprocessed verification set into the trained graph neural network model to obtain the recognition results of the interactive behaviors of multiple traffic participants.

[0112] Example 3

[0113] like Figure 4 As shown, this embodiment provides a multi-traffic participant interactive behavior recognition device, specifically comprising:

[0114] A memory N1, a processor N2, and a computer program N3 stored in the memory and running on the processor.

[0115] When the computer program N3 is executed by the processor N2, the steps of the identification method in Embodiment 1 are implemented.

[0116] In addition, the method according to the embodiment of the present invention can also be used Figure 4 The architecture of the identification device shown is implemented. Figure 4 The structure of the identification device is shown in FIG. Figure 4 As shown, the recognition device may include a memory N1, a processor N2, and a computer program N3 stored in the memory and running on the processor, etc., and may also include other hardware devices. The memory N1 in the recognition device may be a ROM, a USB flash drive, a mechanical hard disk, a solid-state hard disk, or a mobile hard disk. It may store various data or files used for processing and / or communication of the method for identifying interactive behaviors of multiple traffic participants provided by the present invention, as well as program instructions executed by the processor N2. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or at least two components of the identification device shown, or in Figure 4 On the basis of the identification device, one or at least two components are added, and the specific types of the components include but are not limited to the hardware described above.

[0117] Example 4

[0118] like Figure 5 As shown, this embodiment provides a computer-readable storage medium on which a computer program P1 is stored.

[0119] When the computer program is executed by a processor, the steps of the identification method in Embodiment 1 are implemented.

[0120] Similarly, the storage medium may be a ROM, a USB flash drive, a mechanical hard disk, a solid-state hard disk, or a mobile hard disk. Its readable state may be read and executed by any one or more mobile terminals such as a computer, a mobile phone, a tablet computer, etc. Of course, this embodiment only provides a basic architecture. Figure 5 On the basis of the storage medium, one or at least two components are added, and the specific types of the components include but are not limited to the hardware described above. Any hardware or software that can implement one or more steps in the method in Example 1 is within the protection scope of the present invention.

[0121] The program part of the technology can be considered as a "product" or "manufactured product" in the form of executable code and / or related data, which is participated in or realized by computer-readable media. Tangible and permanent storage media can include any memory or storage used by any computer, processor, or similar device or related module. For example, various semiconductor memories, tape drives, disk drives or any similar devices that can provide storage functions for software.

[0122] All or part of the software may sometimes communicate over a network, such as the Internet or other communications network. Such communications can load software from one computer device or processor to another. For example: from a server or host computer of a video target detection device to a hardware platform of a computer environment, or other computer environment that implements the system, or a system with similar functions related to providing information required for target detection. Therefore, another medium capable of transmitting software elements can also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., transmitted through cables, optical cables or air. Physical media used to carry carriers, such as cables, wireless connections or optical cables and the like, can also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, other terms referring to computer or machine "readable media" refer to media involved in the process of the processor executing any instructions.

[0123] The present invention uses specific words to describe the embodiments of the present invention. For example, "first / second embodiment", "one embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic associated with at least one embodiment of the present invention. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of the present invention may be appropriately combined.

[0124] In addition, it will be understood by those skilled in the art that various aspects of the present invention may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present invention may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of the present invention may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0125] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0126] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A method for identifying interactive behaviors of multiple traffic participants. It is characterized in that include: Get the dataset; The data set includes a variety of traffic scenarios and interactive behaviors of a variety of traffic participants; Dividing the data set into a training set and a validation set; A graph neural network model is established; the graph neural network model includes an instance layer and a category layer; in the instance layer, each traffic participant at different times is defined as a node, and the nodes are connected through time edges and space edges; in the category layer, different types of traffic participants at the same time are defined as super nodes, and the super nodes are connected to the nodes through information transfer edges; Preprocessing the training set and the validation set according to the graph neural network model to obtain a preprocessed training set and a preprocessed validation set; Inputting the preprocessed training set into the graph neural network model for training to obtain a trained graph neural network model; The preprocessed validation set is input into the trained graph neural network model to obtain recognition results of the interactive behaviors of multiple traffic participants.

2. The method for identifying interactive behaviors of multiple traffic participants according to claim 1, It is characterized in that The obtaining of the data set specifically includes: Selecting a data set in the form of video images containing multiple traffic scenes and multiple interactive behaviors of traffic participants as the data set; The types and coordinates of all traffic participants in each frame of each video in the data set are marked, and a unique ID is set for each traffic participant; the types of traffic participants include pedestrians, non-motor vehicles and motor vehicles; The interactive behavior categories between the main vehicle and other traffic participants around the main vehicle in each frame image of each video in the data set are labeled.

3. The method for identifying interactive behaviors of multiple traffic participants according to claim 2, It is characterized in that The data set is the BLVD data set.

4. The method for identifying interactive behaviors of multiple traffic participants according to claim 3, It is characterized in that The establishing of the graph neural network model specifically includes: In the instance layer, each traffic participant at each moment is defined as a node, and the LSTM neural network is used to process the characteristic information of each traffic participant; the LSTM neural network is used to process the relative position information of the two nodes in each spatial edge; the LSTM neural network is used to process the historical movement information of the two nodes in each temporal edge in the time series; wherein the spatial edge is bidirectional, indicating an edge connecting two different nodes at the same moment; the temporal edge is unidirectional, indicating an edge connecting the same node at two different moments; In the category layer, different types of traffic participants at each moment are defined as a supernode, and there is an information transfer edge between each supernode and each node in the instance layer of all corresponding types at the current moment. The information transfer edge is used to aggregate the interactive behavior information of the same type of traffic participants to the supernode, and then the supernode processes the interactive behavior information using the LSTM neural network and returns the interactive behavior information to the corresponding node in the instance layer to obtain the recognition result of the interactive behavior information; the time edge also exists between the same supernode at two different moments, and the LSTM neural network is used to process the historical information of the two supernodes in the time series in each time edge; The instance layer and the category layer are connected by the information transfer edges between nodes and super nodes of corresponding categories.

5. The method for identifying interactive behaviors of multiple traffic participants according to claim 4, It is characterized in that Preprocessing the training set and the validation set according to the graph neural network model to obtain a preprocessed training set and a preprocessed validation set specifically includes: According to the data structure required in the graph neural network model, the data in the training set and the validation set are preprocessed to obtain the preprocessed training set and the preprocessed validation set that can be input into the graph neural network model, including: The characteristics of the traffic participant are set as in Indicates the x, y coordinate information of the traffic participant, Indicates the type of traffic participants, f i t Represents the characteristic information of traffic participants input when the LSTM neural network is used to process the nodes in the graph neural network model; Since the space edge is bidirectional, when considering When Represents a node Nodes in space The impact is characterized by in, Respectively represent nodes Relative to the node The x, y coordinate position, Obtained by encoding the types of traffic participants at two nodes in the BLVD data set; When considering When Represents a node Nodes in space The impact is characterized by in, Respectively represent nodes Relative to the node The x, y coordinate position, Obtained by encoding the types of traffic participants at two nodes in the BLVD data set; in, and They respectively represent the characteristic information of traffic participants input when the LSTM neural network is used to process the spatial edges in the graph neural network model; Since the time edge is unidirectional, when the same node A i When it appears in two adjacent frames at time t and t+1, the time edge is expressed as Its characteristics are in, Respectively represent the same node A i The x, y coordinate position at time t+1 relative to time t, Obtained by coding the traffic participant type of this node in the BLVD data set; Represents the characteristic information of traffic participants that is input when the LSTM neural network processes the time edges in the graph neural network.

6. The method for identifying interactive behaviors of multiple traffic participants according to claim 5, It is characterized in that The step of inputting the preprocessed training set into the graph neural network model for training to obtain a trained graph neural network model specifically includes: Arranging the preprocessed training set into a tensor form and inputting it into the graph neural network model for training. When performing LSTM neural network processing, the tensor form of the input nodes, the spatial edges, and the temporal edges is M×N×3, where M represents the number of input samples at a time, N represents the length of observation time, and 3 represents the dimension of the input feature information; The forward data transmission process in the graph neural network model includes: For each node and each edge in the instance layer, a linear mapping function is set to preprocess the input information, and then enter the LSTM neural network after preprocessing: Where embed(·) represents a linear mapping function that encodes the input into a vector of a specific length; represents the output of the spatial edge, represents the output of the time edge, and the output of the spatial edge and the time edge are the output hidden states of the corresponding LSTM neural network; They represent the weight parameters of the linear mapping layer during input information preprocessing, They represent the weight parameters of the neurons of the LSTM neural network respectively; For the same node A i , after the information of all the edges around it is processed by the LSTM neural network, a weight is assigned through an attention mechanism. The weight represents the degree of influence of the edge on the node, which is expressed as: in, Represents the corresponding spatial edge output The weight of ii and W ij They represent the weight parameters of the linear mapping layer when the attention mechanism assigns weights, Dot(·) represents the dot multiplication operation, represents the scale factor; Calculated by weighted average Represents node A i The surrounding traffic participants at time t i Impact on space; and Contains A i All the information of the connected edges is combined and processed by linear mapping to obtain a fixed-length vector At the same time, the characteristic information of traffic participants f is input when the node is processed by LSTM neural network i t It is also linearly mapped to Then we have: in, and Represent the linear mapping layer weights respectively; and Respectively represent the linear mapping results; and As the input of the LSTM neural network processing node, the output hidden state is obtained Output hidden state It is node A i The first stage output is: in, Represents node A i The output hidden state of the first stage output at time t; Represents node A i The output hidden state of the final output at time t-1; The category layer aggregates the output hidden states from nodes of the same category and neuronal states Extract the features of the same type of traffic participants and feed them back to the nodes, then we have: Where n represents the super node The corresponding number of nodes, Represents the extracted characteristic information of the same type of traffic participants at time t; Comprehensively consider the movement characteristics of a certain type of traffic participants at time t, As part of the input to the LSTM neural network processing of the node, the temporal edge between the same supernode in two consecutive frames is expressed as the intermediate quantity As input, get the hidden state containing time series information in, Represents the extracted characteristic information of the same type of traffic participants at time t-1, then: in, and They represent the weights of the linear mapping layer and the weights of the neurons of the LSTM neural network when obtaining the hidden state containing the time series information; As another part of the input, Together they are input into the LSTM neural network processing of the supernode: in, and They respectively represent the weight of the linear mapping layer and the weight of the LSTM neural network when the super node is processed by the LSTM neural network; Represents feedback information that is fed back to the instance layer along the information transfer edge from the supernode to the node; When feedback information After returning to the node, the super node The mth node A connected m Receive feedback Node A m The output of the first stage at time t Feedback Connect and get the final output after linear mapping in, Represents the weight of the linear mapping layer during linear mapping processing after feedback information is returned; Represents node A m The output hidden state of the first stage output at time t, Represents the node A obtained after linear mapping m The output hidden state of the final output at time t; Each node is processed by the LSTM neural network and outputs Indicates the interactive behavior recognition result between the traffic participant and the main vehicle corresponding to the node.

7. The method for identifying interactive behaviors of multiple traffic participants according to claim 6, It is characterized in that The loss function used in training the graph neural network model is a multi-classification cross entropy function, and the weight parameters of the entire neural network are updated through a gradient descent algorithm with reverse data transmission.

8. A multi-traffic participant interactive behavior recognition system, It is characterized in that include: A data set acquisition module, used to acquire a data set; the data set includes a variety of traffic scenes and interactive behaviors of a variety of traffic participants; A data set partitioning module, used to partition the data set into a training set and a validation set; A graph neural network model building module, used to build a graph neural network model; the graph neural network model includes an instance layer and a category layer; in the instance layer, each traffic participant at different times is defined as a node, and the nodes are connected through time edges and space edges; in the category layer, different types of traffic participants at the same time are defined as super nodes, and the super nodes are connected to the nodes through information transfer edges; A data set preprocessing module, used to preprocess the training set and the validation set according to the graph neural network model to obtain a preprocessed training set and a preprocessed validation set; A graph neural network model training module, used to input the preprocessed training set into the graph neural network model for training to obtain a trained graph neural network model; The interactive behavior recognition result output module is used to input the preprocessed verification set into the trained graph neural network model to obtain the recognition results of the interactive behaviors of multiple traffic participants.

9. A device for identifying interactive behaviors of multiple traffic participants. include: A memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that the steps of the method according to any one of claims 1 to 7 are implemented when the computer program is executed by the processor.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.