Pedestrian intention recognition method and device, electronic equipment and storage medium

Abstract

The invention relates to a pedestrian intention recognition method and device, electronic equipment and a storage medium. Acquiring an image frame sequence acquired for a vehicle driving environment; determining a multi-source information feature sequence of a pedestrian target and an interaction object associated with the pedestrian target based on the image frame sequence; inputting the multi-source information feature sequence into a space-time attention graph convolution model to obtain space-time correlation features; and determining the passing intention of the pedestrian target based on the space-time correlation characteristics. According to the embodiment of the invention, the interaction relationship between the pedestrian and the traffic environment is fully mined through the space-time attention graph convolution model, the pedestrian passing intention is analyzed according to the pedestrian, the influence of the traffic environment on the pedestrian passing intention is fully considered, the pedestrian passing intention is comprehensively identified from more dimensions, and the pedestrian passing intention is accurately identified. The recognition of the pedestrian passing intention is more accurate, and the generalization is higher.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a method, device, electronic device and storage medium for pedestrian intention recognition. Background Art

[0002] With the rapid development of autonomous driving technology, the safe and efficient driving of vehicles in complex urban environments has become an urgent problem to be solved. Pedestrians are an important part of road users, and their unpredictable behavior poses a huge challenge to autonomous driving systems.

[0003] In order to improve the safety and driving efficiency of autonomous vehicles, pedestrian intention recognition technology has emerged. Pedestrian intention recognition predicts pedestrians' possible future actions, such as whether they are ready to cross the road or suddenly change direction, by analyzing their current behavior and status. Its accuracy directly affects the decision-making and planning of the autonomous driving system, and has broad application prospects in the field of autonomous driving. First, this technology can significantly improve safety and reliability in complex traffic environments and reduce traffic accidents. Second, it can optimize driving paths and behavioral decisions, improve driving efficiency and ride comfort. In addition, pedestrian intention recognition can also be applied to intelligent transportation systems, smart cities, and security monitoring, promoting the intelligent development of traffic management and public safety. Therefore, pedestrian intention recognition is a key part of high-level autonomous driving technology. Through advanced technologies such as deep learning and time series modeling, the ability to perceive the environment and make behavioral decisions can be significantly improved, paving the way for safer and more efficient autonomous driving.

[0004] However, the pedestrian intention recognition methods in current autonomous driving technology are not accurate enough to meet the needs of autonomous vehicles. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a pedestrian intention recognition method, device, electronic device and storage medium.

[0006] In a first aspect, the present application provides a method for pedestrian intention recognition, comprising:

[0007] Acquire a sequence of image frames collected from the vehicle driving environment;

[0008] Determine a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence;

[0009] Inputting the multi-source information feature sequence into a spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, wherein the spatiotemporal correlation features are used to characterize the spatiotemporal correlation relationship between the pedestrian target and the interactive object associated with it;

[0010] The pedestrian's crossing intention is determined based on the spatiotemporal correlation features.

[0011] Optionally, determining a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence includes:

[0012] Determine a target image sequence and a target frame sequence based on the image frame sequence;

[0013] A multi-source information feature sequence of a pedestrian target and an interactive object associated therewith is determined based on the target image sequence and the target frame sequence.

[0014] Optionally, determining a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the target image sequence and the target frame sequence includes:

[0015] Determining a maximum number of pedestrian targets and interactive objects in each image frame of the image frame sequence;

[0016] Determine a visual feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target image sequence;

[0017] Determine a dynamic feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target bounding box sequence;

[0018] The multi-source information feature sequence is determined based on the visual feature sequence and the dynamic feature sequence.

[0019] Optionally, determining a visual feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target image sequence includes:

[0020] For each target image in the target image sequence, extracting visual features of pedestrian targets in the target image to obtain pedestrian target features;

[0021] Extracting visual features of the interactive objects in the target image to obtain interactive object features;

[0022] Based on the maximum number, the pedestrian target feature and the interactive object feature are spliced ​​to obtain a spliced ​​feature;

[0023] The splicing features corresponding to each target image in the target image sequence are cascaded to obtain the visual feature sequence.

[0024] Optionally, determining a dynamic feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target bounding box sequence includes:

[0025] For each target frame in the target frame sequence, determining first position information of a pedestrian target detection frame and second position information of the interactive object detection frame;

[0026] splicing the first position information of the pedestrian target and the second position information of the interactive object based on the maximum number to obtain spliced ​​position information;

[0027] The splicing position information corresponding to each target frame in the target frame sequence is cascaded to obtain the dynamic feature sequence.

[0028] Optionally, determining the multi-source information feature sequence based on the visual feature sequence and the dynamic feature sequence includes:

[0029] Get vehicle speed data;

[0030] Determining vehicle speed features corresponding to different moments based on the vehicle speed data as a vehicle feature sequence;

[0031] The vehicle speed feature sequence, the visual feature sequence and the dynamic feature sequence are determined as the multi-source information feature sequence.

[0032] Optionally, the multi-source information feature sequence is input into a spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, including:

[0033] Determine the semantic relationship feature and the first temporal relationship feature between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset first spatiotemporal attention graph convolution module;

[0034] Determine the spatial relationship feature and the second temporal relationship feature between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset second spatiotemporal attention graph convolution module;

[0035] The semantic relationship feature, the spatial relationship feature, the first temporal relationship feature and the second temporal relationship feature are fused to obtain the spatiotemporal correlation feature.

[0036] Optionally, determining the semantic relationship features between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and a preset first spatiotemporal attention graph convolution module includes:

[0037] Inputting the visual feature sequence in the multi-source information feature sequence into a first spatiotemporal attention graph convolution module to obtain semantic relationship features between pedestrian targets and interactive objects associated therewith;

[0038] or,

[0039] The visual feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the first spatiotemporal attention graph convolution module to obtain the semantic relationship features between the pedestrian target and the interactive object associated with it.

[0040] Optionally, determining the spatial relationship features between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and a preset second spatiotemporal attention graph convolution module includes:

[0041] Inputting the dynamic feature sequence in the multi-source information feature sequence into the second spatiotemporal attention graph convolution module to obtain the spatial relationship feature between the pedestrian target and the interactive object associated therewith;

[0042] or,

[0043] The dynamic feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the second spatiotemporal attention graph convolution module to obtain the spatial relationship features between the pedestrian target and the interactive objects associated with it.

[0044] Optionally, the first spatiotemporal attention graph convolution module includes: a first graph convolution unit, a first attention unit and a first temporal convolution unit;

[0045] The first graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0046] The first attention unit is used to determine a node visual feature sequence of each node in the star graph based on the visual feature sequence, and determine a semantic relationship feature between the pedestrian target and the interactive object according to each of the node visual feature sequences using an attention mechanism;

[0047] The first temporal convolution unit is used to determine a first temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target image in the target image sequence.

[0048] Optionally, the second spatiotemporal attention graph convolution module includes: a second graph convolution unit, a second attention unit and a second temporal convolution unit;

[0049] The second graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0050] The second attention unit is used to determine a node spatial feature sequence of each node in the star graph based on the dynamic feature sequence, and determine a spatial relationship feature between the pedestrian target and the interactive object according to each node spatial feature sequence using an attention mechanism;

[0051] The second temporal convolution unit is used to determine a second temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target frame in the target frame sequence.

[0052] Optionally, determining the pedestrian target's crossing intention based on the spatiotemporal correlation feature includes:

[0053] Inputting the multi-source information features into a long short-term memory network to obtain decoding features corresponding to the pedestrian target;

[0054] The decoded features are input into a fully connected network to obtain the pedestrian target's crossing intention.

[0055] In a second aspect, the present application provides a pedestrian intention recognition device, comprising:

[0056] An acquisition module, used to acquire a sequence of image frames collected from a vehicle driving environment;

[0057] A first determination module, configured to determine a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence;

[0058] An input module, used for inputting the multi-source information feature sequence into a spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, wherein the spatiotemporal correlation features are used to characterize the spatiotemporal correlation relationship between the pedestrian target and the interactive object associated with it;

[0059] The second determination module is used to determine the pedestrian target's crossing intention based on the spatiotemporal correlation characteristics.

[0060] Optionally, the first determining module includes:

[0061] A first determination module, configured to determine a target image sequence and a target frame sequence based on the image frame sequence;

[0062] The second determination module is used to determine the multi-source information feature sequence of the pedestrian target and the interactive objects associated therewith based on the target image sequence and the target frame sequence.

[0063] Optionally, the second determining module includes:

[0064] A first determination submodule, configured to determine the maximum number of pedestrian targets and interactive objects in each image frame of the image frame sequence;

[0065] A second determination submodule, configured to determine a visual feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target image sequence;

[0066] A third determination submodule, configured to determine a dynamic feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target frame sequence;

[0067] The fourth determination submodule is used to determine the multi-source information feature sequence based on the visual feature sequence and the dynamic feature sequence.

[0068] Optionally, the second determining submodule includes:

[0069] A first extraction unit is used to extract visual features of pedestrian targets in each target image in the target image sequence to obtain pedestrian target features;

[0070] A second extraction unit is used to extract visual features of the interactive object in the target image to obtain interactive object features;

[0071] A first splicing unit, configured to splice the pedestrian target feature and the interactive object feature based on the maximum number to obtain a spliced ​​feature;

[0072] The first cascading unit is used to cascade the splicing features corresponding to each target image in the target image sequence to obtain the visual feature sequence.

[0073] Optionally, the third determining submodule includes:

[0074] A first determining unit is used to determine, for each target frame in the target frame sequence, first position information of a pedestrian target detection frame and second position information of the interactive object detection frame;

[0075] A second splicing unit is used to splice the first position information of the pedestrian target and the second position information of the interactive object based on the maximum number to obtain spliced ​​position information;

[0076] The second cascading unit is used to cascade the splicing position information corresponding to each target frame in the target frame sequence to obtain the dynamic feature sequence.

[0077] Optionally, the fourth determining submodule includes:

[0078] An acquisition unit, used for acquiring vehicle speed data;

[0079] A second determining unit, configured to determine vehicle speed features corresponding to different moments based on the vehicle speed data as a vehicle feature sequence;

[0080] The third determining unit is used to determine the vehicle speed feature sequence, the visual feature sequence and the dynamic feature sequence as the multi-source information feature sequence.

[0081] Optionally, the input module includes:

[0082] A fifth determination submodule, configured to determine a semantic relationship feature and a first temporal relationship feature between a pedestrian target and an interactive object associated therewith based on the multi-source information feature sequence and a preset first spatiotemporal attention graph convolution module;

[0083] A sixth determination submodule, configured to determine a spatial relationship feature and a second temporal relationship feature between a pedestrian target and an interactive object associated therewith based on the multi-source information feature sequence and a preset second spatiotemporal attention graph convolution module;

[0084] A fusion submodule is used to fuse the semantic relationship feature, the spatial relationship feature, the first temporal relationship feature and the second temporal relationship feature to obtain the spatiotemporal correlation feature.

[0085] Optionally, the fifth determining submodule is further used for:

[0086] Inputting the visual feature sequence in the multi-source information feature sequence into a first spatiotemporal attention graph convolution module to obtain semantic relationship features between pedestrian targets and interactive objects associated therewith;

[0087] or,

[0088] The visual feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the first spatiotemporal attention graph convolution module to obtain the semantic relationship features between the pedestrian target and the interactive object associated with it.

[0089] Optionally, the sixth determining submodule is further used for:

[0090] Inputting the dynamic feature sequence in the multi-source information feature sequence into the second spatiotemporal attention graph convolution module to obtain the spatial relationship feature between the pedestrian target and the interactive object associated therewith;

[0091] or,

[0092] The dynamic feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the second spatiotemporal attention graph convolution module to obtain the spatial relationship features between the pedestrian target and the interactive objects associated with it.

[0093] Optionally, the first spatiotemporal attention graph convolution module includes: a first graph convolution unit, a first attention unit and a first temporal convolution unit;

[0094] The first graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0095] The first attention unit is used to determine a node visual feature sequence of each node in the star graph based on the visual feature sequence, and determine a semantic relationship feature between the pedestrian target and the interactive object according to each of the node visual feature sequences using an attention mechanism;

[0096] The first temporal convolution unit is used to determine a first temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target image in the target image sequence.

[0097] Optionally, the second spatiotemporal attention graph convolution module includes: a second graph convolution unit, a second attention unit and a second temporal convolution unit;

[0098] The second graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0099] The second attention unit is used to determine a node spatial feature sequence of each node in the star graph based on the dynamic feature sequence, and determine a spatial relationship feature between the pedestrian target and the interactive object according to each node spatial feature sequence using an attention mechanism;

[0100] The second temporal convolution unit is used to determine a second temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target frame in the target frame sequence.

[0101] Optionally, the second determining module includes:

[0102] A first input submodule is used to input the multi-source information features into a long short-term memory network to obtain a decoding feature corresponding to the pedestrian target;

[0103] The second input submodule is used to input the decoded features into a fully connected network to obtain the pedestrian's crossing intention.

[0104] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0105] Memory, used to store computer programs;

[0106] The processor is used to implement any pedestrian intention recognition method described in the first aspect when executing the program stored in the memory.

[0107] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program of a pedestrian intention recognition method is stored. When the program of the pedestrian intention recognition method is executed by a processor, the steps of the pedestrian intention recognition method described in any one of the first aspects are implemented.

[0108] Beneficial effects of the present invention: The embodiment of the present application extracts a multi-source information feature sequence of pedestrian targets and interactive objects associated with them in an image frame sequence, uses the multi-source information feature sequence as the input of a spatiotemporal attention graph convolution model, uses the spatiotemporal attention graph convolution model to extract the spatiotemporal correlation features between pedestrians and interactive objects associated with them, and finally uses the spatiotemporal correlation features to determine the pedestrian's crossing intention. The present application fully explores the interactive relationship between pedestrians and the traffic environment through the spatiotemporal attention graph convolution model, and while analyzing the pedestrian's crossing intention based on the pedestrian himself, fully considers the impact of the traffic environment on the pedestrian's crossing intention, and comprehensively identifies the pedestrian's crossing intention from more dimensions, making the identification of the pedestrian's crossing intention more accurate and more generalizable. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0110] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0111] Figure 1 A flowchart of a method for pedestrian intention recognition provided in an embodiment of the present application;

[0112] Figure 2 A schematic diagram of the principle of a pedestrian intention recognition method provided in an embodiment of the present application;

[0113] Figure 3 for Figure 1 Flow chart of step S102;

[0114] Figure 4 for Figure 3 Flow chart of step S202;

[0115] Figure 5 for Figure 4 Flowchart of step S302;

[0116] Figure 6 for Figure 4 Flow chart of step S303;

[0117] Figure 7 for Figure 4 Flow chart of step S304;

[0118] Figure 8 for Figure 1 Flow chart of step S103;

[0119] Fig. 9 A schematic diagram of the structure of an attention unit provided in an embodiment of the present application;

[0120] Fig.10 A structural diagram of a temporal convolutional network provided in an embodiment of the present application;

[0121] Fig.11 for Figure 1 Flow chart of step S104;

[0122] Fig.12 A structural diagram of a pedestrian intention recognition device provided in an embodiment of the present application;

[0123] Fig.13 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0124] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0125] In order to solve the technical problem that the pedestrian intention recognition method in the current autonomous driving technology is not accurate enough and cannot meet the needs of autonomous driving vehicles, the present application provides a pedestrian intention recognition method, device, electronic device and storage medium, which can achieve a more accurate and generalized recognition of pedestrian crossing intentions.

[0126] The present application embodiment provides a method for pedestrian intention recognition, such as Figure 1 As shown, the following steps may be included:

[0127] Step S101, obtaining a sequence of image frames collected from a vehicle driving environment;

[0128] The vehicle driving environment may refer to the environment available for vehicle driving, including but not limited to road conditions (such as road surface flatness, whether there are potholes or obstacles), traffic flow, weather conditions (such as sunny, rainy, snowy, etc.), visibility, the influence of surrounding buildings or natural terrain, etc. The present application focuses on pedestrian targets in the vehicle driving environment and interactive objects that are interactively associated with pedestrian targets. Therefore, each image frame in the image frame sequence contains pedestrian targets and interactive objects that are interactively associated with pedestrian targets. Interactive objects that are interactively associated with pedestrian targets may refer to traffic factors that affect the pedestrian target's intention to cross, such as: crosswalks, vehicles, traffic lights, traffic signs, etc.

[0129] At least one image acquisition device may be provided on the vehicle to acquire images or videos of the vehicle driving environment. The image acquisition device may output a video stream, which may be directly used as an image frame sequence, or multiple image frames may be extracted from the video stream, such as image frames containing pedestrian targets and interactive objects associated with pedestrian targets, to obtain an image frame sequence. Multiple image frames containing pedestrian targets and interactive objects associated with pedestrian targets outputted successively by the image acquisition device may also be combined as an image frame sequence. For example, an image frame sequence such as Figure 2 shown.

[0130] The pedestrian intention recognition method of the present application can be applied to the reasoning stage. When using the spatiotemporal attention graph convolution model for pedestrian intention recognition, an image frame sequence output by an image acquisition device on the vehicle can be obtained, and each image frame in the image frame sequence corresponds to a different acquisition time.

[0131] The pedestrian intention recognition method of the present application can also be applied to the training stage, so the image frame sequence can also be obtained from the dataset used for model training, such as the Pedestrian Intention Estimate (PIE) dataset. The PIE dataset is obtained based on the image acquisition devices on multiple vehicles to collect image frame sequences. The PIE dataset used in the training stage of the present application contains 909,000 frames of urban road video frame data in front of the vehicle during the day, 30 frames per second and a resolution of 1920x1080. The dataset contains 293,000 annotated frames, where the label of each frame not only includes the action labels of the pedestrian "walking" and "not walking", but also includes the speed, direction and GPS coordinates of the vehicle, and contains 1,842 pedestrians and 739,000 pedestrian frames. The 1,842 pedestrian trajectories in the dataset are divided into 512 trajectories in which pedestrians intended to cross and executed the crossing results, 898 trajectories in which pedestrians intended to cross but did not execute the crossing results, 2 trajectories in which pedestrians did not intend to cross but executed the crossing results, and 430 trajectories in which pedestrians did not intend to cross and did not execute the crossing results. The data division ratios for the training set, test set, and validation set are 48%, 39%, and 13%, respectively.

[0132] After the image frame sequence is acquired, the image frame sequence may be preprocessed to determine the type and quantity of interactive objects that affect pedestrians' crossing intention in each image frame for subsequent use.

[0133] Step S102, determining a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence;

[0134] In this step, after acquiring the image frame sequence, a multi-source information feature sequence including pedestrian targets and interactive objects associated with the pedestrian targets can be determined in the image frame sequence. Each image frame in the image frame sequence can include at least one pedestrian target and an interactive object associated with the pedestrian target. The multi-source information feature sequence can include multi-source information features corresponding to multiple different acquisition moments. The multi-source information features are obtained by utilizing a feature extraction model to perform feature extraction from multiple angles on pedestrian targets and interactive objects associated with pedestrian targets.

[0135] In the embodiment of the present application, the feature extraction of pedestrian targets and their associated interactive objects mainly includes: visual feature extraction and dynamic feature extraction. The visual feature extraction obtains the following Figure 2 The visual feature sequence shown in FIG. 1 contains the semantic relationship and the first-time relationship between the pedestrian target and the interactive object associated with it. The dynamic feature extraction is as follows: Figure 2 The dynamic feature sequence shown contains the spatial relationship and the second temporal relationship between the pedestrian target and the interactive objects associated with it.

[0136] Step S103, inputting the multi-source information feature sequence into the spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features;

[0137] In this step, the multi-source information feature sequence can be input into the spatiotemporal attention graph convolution model so that the spatiotemporal attention graph convolution model outputs spatiotemporal correlation features, and the spatiotemporal correlation features are used to characterize the spatiotemporal correlation relationship between the pedestrian target and the interactive objects associated with it.

[0138] When the present application is applied to the reasoning stage, the spatiotemporal attention graph convolution model may be a pre-trained model. When the present application is applied to the training stage, the spatiotemporal attention graph convolution model may be an untrained or incompletely trained model.

[0139] like Figure 2 As shown, the spatiotemporal attention graph convolution model may include one or more spatiotemporal attention graph convolution modules, and different spatiotemporal attention graph convolution modules may be used to obtain semantic relationship features, spatial relationship features, and temporal relationship features between pedestrian targets and their associated interactive objects, and finally obtain the spatiotemporal correlation features between pedestrian targets and their associated interactive objects.

[0140] Step S104: determining the pedestrian target's crossing intention based on the spatiotemporal correlation features.

[0141] In this step, the spatiotemporal correlation features can be input into the decoding classification network to finally obtain the pedestrian's crossing intention, that is, whether to cross the road.

[0142] The embodiment of the present application essentially provides a pedestrian intention recognition model, such as Figure 2 As shown in the figure, the pedestrian intention recognition model includes: feature extraction model, spatiotemporal attention graph convolution model, long short-term memory network and fully connected network, etc. The pedestrian intention recognition model takes image frame sequence as input, and uses feature extraction model to extract visual features, dynamic features and time features, etc. to obtain multi-source information feature sequence, and uses the output multi-source information feature sequence of feature extraction model as input of spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, and then uses spatiotemporal correlation features as input of decoding classification network to finally obtain the pedestrian target's crossing intention.

[0143] In the training phase, the application can use the PIE (Pedestrian Intention Estimate) image data of the pedestrian intention recognition model to train the network model and obtain the network weight parameters. The application is a supervised learning model, and the loss function used is a two-term cross entropy function with a sigmoid activation function, which is specifically expressed as follows:

[0144]

[0145] Among them, y represents the pedestrian’s actual crossing intention category, Represents the pedestrian crossing intention predicted by the model.

[0146] The embodiment of the present application extracts a multi-source information feature sequence of pedestrian targets and interactive objects associated with them from an image frame sequence, uses the multi-source information feature sequence as the input of a spatiotemporal attention graph convolution model, uses the spatiotemporal attention graph convolution model to extract the spatiotemporal correlation features between pedestrians and interactive objects associated with them, and finally uses the spatiotemporal correlation features to determine the pedestrian's crossing intention. The present application fully explores the interactive relationship between pedestrians and the traffic environment through the spatiotemporal attention graph convolution model, and while analyzing the pedestrian's crossing intention based on the pedestrian himself, fully considers the impact of the traffic environment on the pedestrian's crossing intention, and comprehensively identifies the pedestrian's crossing intention from more dimensions, making the identification of the pedestrian's crossing intention more accurate and more generalizable.

[0147] In another embodiment of the present application, Figure 3 As shown, step S102 determines a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence, including:

[0148] Step S201, determining a target image sequence and a target frame sequence based on the image frame sequence;

[0149] In the present application embodiment, Figure 2 As shown, the target image sequence includes: pedestrian target images in pedestrian target detection frames and interactive object images in interactive object detection frames in multiple image frames, the interactive objects are associated with the pedestrian targets, and the target frame sequence includes the pedestrian target detection frames and the interactive object detection frames;

[0150] like Figure 2 As shown, a target image sequence refers to an image sequence including multiple target images, and the multiple target images are sorted in the order of the acquisition time of the image frames to which they belong. The target images include pedestrian target images and interactive object images, etc. A target frame sequence refers to a frame sequence including multiple target frames, and the multiple target frames are sorted in the order of the acquisition time of the image frames to which they belong. The target frames include pedestrian target detection frames and interactive object detection frames, etc.

[0151] Since the target image sequence includes pedestrian target images and interactive object images, the target image sequence can reflect the semantic relationship characteristics of the pedestrian target and the interactive objects associated with it; since the target border sequence includes pedestrian target detection frames and interactive object detection frames, the target border sequence can reflect the spatial relationship characteristics of the pedestrian target and the interactive objects associated with it; since the target image sequence has a time attribute, the target image sequence can reflect the first time relationship characteristics of the pedestrian target and the interactive objects associated with it. Similarly, the target border sequence can also reflect the second time relationship characteristics of the pedestrian target and the interactive objects associated with it. The first time relationship characteristics can be the same or different.

[0152] In this step, target detection can be performed on each image frame in the image frame sequence to detect pedestrian targets and interactive objects, and obtain pedestrian target detection frames and interactive object detection frames. The image in the pedestrian target detection frame is used as the pedestrian target image, the image in the interactive object detection frame is used as the interactive object image, the pedestrian target image and the interactive object image are used as target images to obtain a target image sequence, and the pedestrian target detection frame and the interactive object detection frame are used as target detection frames to obtain a target frame sequence.

[0153] Step S202 : determining a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the target image sequence and the target frame sequence.

[0154] In this step, the multi-source information features of the pedestrian target and the interactive objects associated therewith can be determined in each target image in the target image sequence and each target frame in the target frame sequence, thereby obtaining a multi-source information feature sequence.

[0155] The embodiment of the present application determines a multi-source information feature sequence based on a target image sequence and a target border sequence, so that the final multi-source information feature sequence includes semantic relationship features, first temporal relationship features, spatial relationship features, and second temporal relationship features of pedestrian targets and their associated interactive objects, thereby fully extracting the multi-source information association between pedestrian targets and their associated interactive objects to facilitate subsequent feature fusion and utilization.

[0156] In another embodiment of the present application, Figure 4 As shown, step S202 determines a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the target image sequence and the target frame sequence, including:

[0157] Step S301, determining the maximum number of pedestrian targets and interactive objects in each image frame of the image frame sequence;

[0158] Since the image frame sequence has been preprocessed after being acquired to determine the types and numbers of interactive objects that affect pedestrians' intention to cross in each image frame, the maximum number can be calculated based on the number of pedestrian targets and the number of interactive objects that affect pedestrians' intention to cross in each image frame.

[0159] Step S302, determining a visual feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target image sequence;

[0160] Since the number of interactive objects in different image frames in the image frame sequence is different, the final visual feature sequence and dynamic feature sequence cannot be aligned. Therefore, before obtaining the visual feature sequence and dynamic feature sequence, the maximum number of pedestrian targets and interactive objects can be determined and the maximum value can be recorded. This value is used as the feature number of pedestrian targets and interactive objects in each frame.

[0161] After obtaining the maximum number N of pedestrian targets and interactive objects in different frames in the image frame sequence based on the maximum number and the target image sequence, the visual features of different targets obtained in the visual feature extraction process are cascaded to obtain the visual features of the entire target image sequence, that is, the visual feature sequence, whose size and shape are: target image sequence length × N × 512 (feature dimension) × 7 × 7 (feature size).

[0162] Step S303, determining a dynamic feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target frame sequence;

[0163] When obtaining the dynamic features of pedestrians and interactive objects in each frame of the target bounding box sequence, their respective bounding box coordinate information is recorded respectively. When the number of interactive objects is less than the maximum number, a matrix of all zeros of the same dimension size is used instead. Therefore, it is possible to support any type of interactive objects (such as crosswalks, vehicles, traffic lights, traffic signs, etc.) as input.

[0164] Based on the maximum number and the target border sequence, the dynamic features of the entire target border sequence, ie, the dynamic feature sequence, are obtained, whose size and shape are: target border sequence length×N×4, where 4 represents the upper left corner coordinates and the lower right corner coordinates of the target bounding box.

[0165] Step S304: determining the multi-source information feature sequence based on the visual feature sequence and the dynamic feature sequence.

[0166] In this step, the visual feature sequence and the dynamic feature sequence may be directly determined as the multi-source information feature sequence, or other feature sequences may be added in addition to the visual feature sequence and the dynamic feature sequence and determined together with the visual feature sequence and the dynamic feature sequence as the multi-source information feature sequence.

[0167] The embodiments of the present application can determine the visual feature sequence and the dynamic feature sequence based on the maximum number of interactive objects in different frames, ensure that the visual features in the visual feature sequence can be aligned, and ensure that the dynamic features in the dynamic feature sequence can be aligned.

[0168] In another embodiment of the present application, Figure 5 As shown, step S302 determines the visual feature sequence of the pedestrian target and the interactive object associated therewith based on the maximum number and the target image sequence, including:

[0169] Step S401, for each target image in the target image sequence, extracting visual features of pedestrian targets in the target image to obtain pedestrian target features;

[0170] In the process of visual feature extraction, VGG16 is used as the backbone network to perform downsampling feature extraction on each target image in the target image sequence. As a commonly used backbone network for extracting image features in deep learning models, the VGG16 network has fast speed and can extract features with strong expression capabilities at a higher level. VGG16 has a total of 16 layers, including 13 convolutional layers and 3 fully connected layers. The input target image first undergoes two convolutions with a convolution kernel size of 64, and then uses pooling. Then, it undergoes two convolutions with a convolution kernel size of 128, and then uses pooling again. Then, it repeats two convolutions with three convolution kernel sizes of 512, and then uses pooling again. Finally, after three full connections, the output is a 7×7×512 size pedestrian target feature and interactive object feature that contains rich semantic information of the target.

[0171] Step S402, extracting visual features of the interactive object in the target image to obtain interactive object features;

[0172] Step S403, splicing the pedestrian target feature and the interactive object feature based on the maximum number to obtain a spliced ​​feature;

[0173] For each frame, the pedestrian target features and the interactive object features are spliced ​​based on the maximum number to obtain a spliced ​​feature, whose size and shape are: N×512 (feature dimension)×7×7 (feature size).

[0174] Step S404: cascading the splicing features corresponding to each target image in the target image sequence to obtain the visual feature sequence.

[0175] After cascading each splicing feature, a visual feature sequence is obtained, and its size length is: target image sequence length × N × 512 (feature dimension) × 7 × 7 (feature size).

[0176] The embodiment of the present application can extract visual features from the target image sequence, obtain pedestrian target features and interactive object features, fill the interactive object features with the maximum number, and then perform feature splicing and cascading to obtain a visual feature sequence, thereby ensuring that each visual feature in the visual feature sequence can be aligned.

[0177] In another embodiment of the present application, Figure 6 As shown, step S303 determines the dynamic feature sequence of the pedestrian target and the interactive object associated therewith based on the maximum number and the target frame sequence, including:

[0178] Step S501, determining, for each target frame in the target frame sequence, first position information of a pedestrian target detection frame and second position information of an interactive object detection frame;

[0179] The first position information of the pedestrian target detection frame includes: the upper left corner coordinates and the lower right corner coordinates of the pedestrian target detection frame, and the second position information of the interactive object detection frame includes: the upper left corner coordinates and the lower right corner coordinates of the interactive object detection frame.

[0180] Step S502, splicing the first position information of the pedestrian target and the second position information of the interactive object based on the maximum number to obtain spliced ​​position information;

[0181] In the process of dynamic feature extraction, the position information of pedestrians and their different interactive objects in each frame is obtained according to the first position information of the pedestrian target detection frame and the second position information of the interactive object detection frame. After the first position information of the pedestrian target and the second position information of the interactive object are spliced ​​according to the maximum number, a multilayer perceptron composed of two fully connected layers is used to encode them to obtain the dynamic feature sequence of pedestrians and different interactive objects.

[0182] Step S503 , cascading the splicing position information corresponding to each target frame in the target frame sequence to obtain the dynamic feature sequence.

[0183] The embodiment of the present application can extract dynamic features from the target border sequence to obtain first position information and second position information, fill the second position information with the maximum number, and then cascade the splicing position information to obtain a dynamic feature sequence, thereby ensuring that each dynamic feature in the dynamic feature sequence can be aligned.

[0184] In another embodiment of the present application, Figure 7 As shown, step S304 determines the multi-source information feature sequence based on the visual feature sequence and the dynamic feature sequence, including:

[0185] Step S601, obtaining vehicle speed data;

[0186] In the embodiment of the present application, the vehicle speed data refers to the speed data of the vehicle within a target time period, and the target time period is determined according to the start time and the end time of the image frame sequence.

[0187] Step S602, determining vehicle speed features corresponding to different moments based on the vehicle speed data as a vehicle feature sequence;

[0188] Step S603: determining the vehicle speed feature sequence, the visual feature sequence and the dynamic feature sequence as the multi-source information feature sequence.

[0189] In this step, the vehicle speed features, visual features, and dynamic features may be aligned according to the acquisition time to obtain multi-source information features corresponding to different times as a multi-source information feature sequence.

[0190] In order to better model traffic scenarios, such as Figure 2 As shown, the present application also encodes the vehicle speed provided in the data set using a multi-layer perceptron composed of fully connected layers, and then forms a multi-source information feature sequence together with the visual feature sequence or the dynamic feature sequence to improve the robustness of the model.

[0191] In another embodiment of the present application, Figure 8 As shown, step S103 inputs the multi-source information feature sequence into the spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, including:

[0192] Step S701, determining the semantic relationship feature and the first temporal relationship feature between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset first spatiotemporal attention graph convolution module;

[0193] In this step, part of the information in the multi-source information feature sequence can be input into the first spatiotemporal attention graph convolution module to obtain the semantic relationship features and the first temporal relationship features between the pedestrian target and the interactive objects associated with it.

[0194] In one embodiment of the present application, the visual feature sequence in the multi-source information feature sequence may be input into a first spatiotemporal attention graph convolution module to obtain a semantic relationship feature between a pedestrian target and an interactive object associated therewith;

[0195] In order to improve the robustness of the model, in another embodiment of the present application, the visual feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence can be input into the first spatiotemporal attention graph convolution module to obtain the semantic relationship features between the pedestrian target and the interactive objects associated with it.

[0196] Step S702, determining the spatial relationship feature and the second temporal relationship feature between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset second spatiotemporal attention graph convolution module;

[0197] In this step, part of the information in the multi-source information feature sequence can be input into the second spatiotemporal attention graph convolution module to obtain the spatial relationship features and the second temporal relationship features between the pedestrian target and the interactive objects associated with it.

[0198] In one embodiment of the present application, the dynamic feature sequence in the multi-source information feature sequence can be input into the second spatiotemporal attention graph convolution module to obtain the spatial relationship feature between the pedestrian target and the interactive object associated therewith;

[0199] In order to improve the robustness of the model, in another embodiment of the present application, the dynamic feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence can be input into the second spatiotemporal attention graph convolution module to obtain the spatial relationship features between the pedestrian target and the interactive objects associated therewith.

[0200] Step S703: Fusing the semantic relationship feature, the spatial relationship feature, the first temporal relationship feature, and the second temporal relationship feature to obtain the spatiotemporal association feature.

[0201] The embodiment of the present application extracts semantic relationship features and first temporal relationship features through a first spatiotemporal attention graph convolution module, extracts spatial relationship features and second temporal relationship features through a second spatiotemporal attention graph convolution module, and finally performs feature fusion to obtain spatiotemporal correlation features, thereby realizing multi-source information fusion of semantic relationship features, spatial relationship features and temporal relationship features between pedestrian targets and interactive objects, and deeply mining the relationship between pedestrian targets and interactive objects for subsequent intention recognition.

[0202] In another embodiment of the present application, Figure 2 As shown, the first spatiotemporal attention graph convolution module includes: a first graph convolution unit, a first attention unit and a first time convolution unit;

[0203] The first graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0204] In order to more reasonably obtain the spatiotemporal correlation between pedestrians and their interactive objects in the image frame sequence instead of simply fusing multi-source information, thereby promoting the accuracy of pedestrian crossing intention recognition, it is necessary to first use the maximum number of pedestrian interactive objects N in different frames to construct the adjacency matrix in the first graph convolution unit, and assign 1 or 0 to the elements at the corresponding positions of the adjacency matrix according to the types of valid interactive objects contained in each frame. For example, if the maximum number of pedestrian interactive objects in different frames is 3, that is, it includes pedestrians, vehicles and crosswalks, and the current frame only contains pedestrians and vehicles, then the adjacency matrix A of the frame is:

[0205]

[0206] Among them, the 1 in the first row and first column of the adjacency matrix represents a pedestrian, the 1 in the second row and second column represents a vehicle, and the 1 in the first row and second column and the 1 in the second row and first column represent the association relationship between pedestrians and vehicles.

[0207] Since the goal of this application is to predict the crossing intention behavior of each pedestrian, the following Figure 2 The star graph centered on the pedestrian is shown to model each pedestrian, and the edges of the star graph are defined according to the spatial relationship between the pedestrian and the object it interacts with.

[0208] The first attention unit is used to determine a node visual feature sequence of each node in the star graph based on the visual feature sequence, and determine a semantic relationship feature between the pedestrian target and the interactive object according to each of the node visual feature sequences using an attention mechanism;

[0209] In view of the limited spatial capture capability of graph structure and over-reliance on graph structure of graph convolutional networks, this application adopts the node feature update of star graph based on the first attention unit to obtain the semantic relationship features of the pedestrian target to be predicted.

[0210] like Fig. 9 As shown in the figure, the first attention unit uses one-dimensional convolution to learn adaptive weights for each node feature in the star graph, and uses the product of the adaptive weight and the node feature as the new node feature to propagate and aggregate the features, that is, to pass the features of a node to its neighboring nodes; and feature aggregation refers to aggregating information from neighboring nodes to the current node to update its feature representation. Through this series of operations, the feature representation of the node can be gradually enriched to contain more information about the graph structure. Adaptive weights can effectively solve the problem of node feature redundancy, automatically suppress the transmission of invalid node features, and increase the weight of important node feature transmission. The use of one-dimensional convolution for weight learning can effectively reduce the amount of parameter calculation, which is conducive to subsequent real vehicle deployment.

[0211] The first temporal convolution unit is used to determine a first temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target image in the target image sequence.

[0212] The modeling of time series problems is generally modeled using recurrent neural networks, because the cyclic autoregressive structure of recurrent neural networks is able to express the time series characteristics of time series well. The temporal and spatial characteristics of interactive objects are important information for understanding their behavioral intentions. In this application, spatial graph convolution based on the attention mechanism has been used to learn complex topological structures to capture the spatial dependencies between different interactive objects, while the temporal dependencies between different sequence frames are obtained by using the dynamic changes of the time convolution network learning graph network in the first time convolution unit.

[0213] Temporal convolution is a neural network improved from convolutional networks for processing sequence problems. It allows parallel computing, has a flexible receptive field, and shares convolution kernels, occupies low memory resources. It is superior to recurrent neural networks such as long short-term memory networks and gated recurrent units in learning dynamic sequence data. Fig.10 The figure shows the temporal convolutional network used in the present invention. The input is the spatial features of nodes at continuous moments. The node features are propagated, accumulated and updated along the time dimension through the hidden layer in the middle. The step size is the sampling interval of the temporal convolution. The receptive field of the temporal convolution is controlled by adopting different step sizes. At the same time, the stacking time length can also be controlled. According to the topological relationship between node time and space, the intention of pedestrians at different time nodes is inferred.

[0214] Based on a similar principle, in another embodiment of the present application, the second spatiotemporal attention graph convolution module includes: a second graph convolution unit, a second attention unit and a second temporal convolution unit;

[0215] The second graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0216] The second attention unit is used to determine a node spatial feature sequence of each node in the star graph based on the dynamic feature sequence, and determine a spatial relationship feature between the pedestrian target and the interactive object according to each node spatial feature sequence using an attention mechanism;

[0217] In view of the limited spatial capture capability of graph structure and over-reliance on graph structure in graph convolutional networks, this application adopts the node feature update of star graph based on the first attention unit to obtain the spatial relationship features of pedestrian targets to be predicted.

[0218] The second temporal convolution unit is used to determine a second temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target frame in the target frame sequence.

[0219] In another embodiment of the present application, Fig.11 As shown, step S104 determines the pedestrian target's crossing intention based on the spatiotemporal correlation features, including:

[0220] Step S801, inputting the multi-source information features into a long short-term memory network to obtain decoding features corresponding to the pedestrian target;

[0221] Step S802: input the decoded features into a fully connected network to obtain the pedestrian target's crossing intention.

[0222] The embodiment of the present application can use a long short-term memory network to decode the multi-source information features obtained using the spatiotemporal attention graph convolution fusion, and then use a fully connected layer network to map the high-dimensional pedestrian intention features to a one-dimensional output to determine the category of the pedestrian's crossing intention.

[0223] In summary, the method designed in the present application first extracts the local context from the input time-series frame image and the provided vehicle speed encoding, and then combines them into the "visual" spatiotemporal attention graph network, and inputs the target motion information encoded from the target box into the "dynamic" spatiotemporal attention graph network, respectively extracts the spatiotemporal interaction features contained in the interaction process between pedestrians and interactive objects and performs feature fusion, and then uses the long short-term memory network to decode, and finally uses the fully connected layer network to output the pedestrian's crossing intention. After determining the maximum number of pedestrian interaction objects in different frames, it can support the input of any traffic element that interacts with pedestrians. At the same time, the present application can reasonably encode the temporal, spatial and semantic relationships between pedestrians and their interactive objects by using the spatiotemporal graph convolutional network based on the attention mechanism, thereby providing more effective modeling for the model to estimate pedestrian intentions, and also has the advantages of good effect and high generalization.

[0224] In another embodiment of the present application, a pedestrian intention recognition device is also provided, such as Fig.12 As shown, including:

[0225] An acquisition module 11 is used to acquire a sequence of image frames collected from a vehicle driving environment;

[0226] A first determination module 12, configured to determine a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence;

[0227] An input module 13, used for inputting the multi-source information feature sequence into the spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, wherein the spatiotemporal correlation features are used to characterize the spatiotemporal correlation relationship between the pedestrian target and the interactive object associated with it;

[0228] The second determination module 14 is used to determine the pedestrian's crossing intention based on the spatiotemporal correlation features.

[0229] Optionally, the first determining module includes:

[0230] A first determination module, configured to determine a target image sequence and a target frame sequence based on the image frame sequence;

[0231] The second determination module is used to determine the multi-source information feature sequence of the pedestrian target and the interactive objects associated therewith based on the target image sequence and the target frame sequence.

[0232] Optionally, the second determining module includes:

[0233] A first determination submodule, configured to determine the maximum number of pedestrian targets and interactive objects in each image frame of the image frame sequence;

[0234] A second determination submodule, configured to determine a visual feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target image sequence;

[0235] A third determination submodule, configured to determine a dynamic feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target frame sequence;

[0236] The fourth determination submodule is used to determine the multi-source information feature sequence based on the visual feature sequence and the dynamic feature sequence.

[0237] Optionally, the second determining submodule includes:

[0238] A first extraction unit is used to extract visual features of pedestrian targets in each target image in the target image sequence to obtain pedestrian target features;

[0239] A second extraction unit is used to extract visual features of the interactive object in the target image to obtain interactive object features;

[0240] A first splicing unit, configured to splice the pedestrian target feature and the interactive object feature based on the maximum number to obtain a spliced ​​feature;

[0241] The first cascading unit is used to cascade the splicing features corresponding to each target image in the target image sequence to obtain the visual feature sequence.

[0242] Optionally, the third determining submodule includes:

[0243] A first determining unit is used to determine, for each target frame in the target frame sequence, first position information of a pedestrian target detection frame and second position information of the interactive object detection frame;

[0244] A second splicing unit is used to splice the first position information of the pedestrian target and the second position information of the interactive object based on the maximum number to obtain spliced ​​position information;

[0245] The second cascading unit is used to cascade the splicing position information corresponding to each target frame in the target frame sequence to obtain the dynamic feature sequence.

[0246] Optionally, the fourth determining submodule includes:

[0247] An acquisition unit, used for acquiring vehicle speed data;

[0248] A second determining unit, configured to determine vehicle speed features corresponding to different moments based on the vehicle speed data as a vehicle feature sequence;

[0249] The third determining unit is used to determine the vehicle speed feature sequence, the visual feature sequence and the dynamic feature sequence as the multi-source information feature sequence.

[0250] Optionally, the input module includes:

[0251] A fifth determination submodule, configured to determine a semantic relationship feature and a first temporal relationship feature between a pedestrian target and an interactive object associated therewith based on the multi-source information feature sequence and a preset first spatiotemporal attention graph convolution module;

[0252] A sixth determination submodule, configured to determine a spatial relationship feature and a second temporal relationship feature between a pedestrian target and an interactive object associated therewith based on the multi-source information feature sequence and a preset second spatiotemporal attention graph convolution module;

[0253] A fusion submodule is used to fuse the semantic relationship feature, the spatial relationship feature, the first temporal relationship feature and the second temporal relationship feature to obtain the spatiotemporal correlation feature.

[0254] Optionally, the fifth determining submodule is further used for:

[0255] Inputting the visual feature sequence in the multi-source information feature sequence into a first spatiotemporal attention graph convolution module to obtain semantic relationship features between pedestrian targets and interactive objects associated therewith;

[0256] or,

[0257] The visual feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the first spatiotemporal attention graph convolution module to obtain the semantic relationship features between the pedestrian target and the interactive object associated with it.

[0258] Optionally, the sixth determining submodule is further used for:

[0259] Inputting the dynamic feature sequence in the multi-source information feature sequence into the second spatiotemporal attention graph convolution module to obtain the spatial relationship feature between the pedestrian target and the interactive object associated therewith;

[0260] or,

[0261] The dynamic feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the second spatiotemporal attention graph convolution module to obtain the spatial relationship features between the pedestrian target and the interactive objects associated with it.

[0262] Optionally, the first spatiotemporal attention graph convolution module includes: a first graph convolution unit, a first attention unit and a first temporal convolution unit;

[0263] The first graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0264] The first attention unit is used to determine a node visual feature sequence of each node in the star graph based on the visual feature sequence, and determine a semantic relationship feature between the pedestrian target and the interactive object according to each of the node visual feature sequences using an attention mechanism;

[0265] The first temporal convolution unit is used to determine a first temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target image in the target image sequence.

[0266] Optionally, the second spatiotemporal attention graph convolution module includes: a second graph convolution unit, a second attention unit and a second temporal convolution unit;

[0267] The second graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object;

[0268] The second attention unit is used to determine a node spatial feature sequence of each node in the star graph based on the dynamic feature sequence, and determine a spatial relationship feature between the pedestrian target and the interactive object according to each node spatial feature sequence using an attention mechanism;

[0269] The second temporal convolution unit is used to determine a second temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target frame in the target frame sequence.

[0270] Optionally, the second determining module includes:

[0271] A first input submodule is used to input the multi-source information features into a long short-term memory network to obtain a decoding feature corresponding to the pedestrian target;

[0272] The second input submodule is used to input the decoded features into a fully connected network to obtain the pedestrian's crossing intention.

[0273] In another embodiment of the present application, an electronic device is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0274] Memory, used to store computer programs;

[0275] The processor is used to implement the pedestrian intention recognition method described in any of the above method embodiments when executing the program stored in the memory.

[0276] In the electronic device provided by the embodiment of the present invention, the processor executes the program stored in the memory to extract the multi-source information feature sequence of pedestrian targets and the interactive objects associated with them in the image frame sequence, uses the multi-source information feature sequence as the input of the spatiotemporal attention graph convolution model, uses the spatiotemporal attention graph convolution model to extract the spatiotemporal correlation features between pedestrians and the interactive objects associated with them, and finally uses the spatiotemporal correlation features to determine the pedestrian target's crossing intention. The present application fully explores the interactive relationship between pedestrians and the traffic environment through the spatiotemporal attention graph convolution model, and while analyzing the pedestrian's crossing intention based on the pedestrian himself, fully considers the impact of the traffic environment on the pedestrian's crossing intention, and comprehensively identifies the pedestrian's crossing intention from more dimensions, so that the identification of the pedestrian's crossing intention is more accurate and more generalized.

[0277] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0278] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0279] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0280] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0281] In another embodiment of the present application, a computer-readable storage medium is also provided, on which a program of a pedestrian intention recognition method is stored. When the program of the pedestrian intention recognition method is executed by a processor, the steps of the pedestrian intention recognition method described in any of the aforementioned method embodiments are implemented.

[0282] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0283] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A pedestrian intention recognition method, characterized in that: include: Acquire a sequence of image frames collected from the vehicle driving environment; Determine a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence; Inputting the multi-source information feature sequence into a spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, wherein the spatiotemporal correlation features are used to characterize the spatiotemporal correlation relationship between the pedestrian target and the interactive object associated with it; The pedestrian's crossing intention is determined based on the spatiotemporal correlation features.

2. The pedestrian intention recognition method according to claim 1, characterized in that: Determining a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence includes: Determine a target image sequence and a target frame sequence based on the image frame sequence; A multi-source information feature sequence of a pedestrian target and an interactive object associated therewith is determined based on the target image sequence and the target frame sequence.

3. The pedestrian intention recognition method according to claim 2, characterized in that: Determining a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the target image sequence and the target frame sequence includes: Determining a maximum number of pedestrian targets and interactive objects in each image frame of the image frame sequence; Determine a visual feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target image sequence; Determine a dynamic feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target bounding box sequence; The multi-source information feature sequence is determined based on the visual feature sequence and the dynamic feature sequence.

4. The pedestrian intention recognition method according to claim 3, characterized in that: Determining a visual feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target image sequence includes: For each target image in the target image sequence, extracting visual features of pedestrian targets in the target image to obtain pedestrian target features; Extracting visual features of the interactive objects in the target image to obtain interactive object features; Based on the maximum number, the pedestrian target feature and the interactive object feature are spliced ​​to obtain a spliced ​​feature; The splicing features corresponding to each target image in the target image sequence are cascaded to obtain the visual feature sequence.

5. The pedestrian intention recognition method according to claim 3, characterized in that: Determining a dynamic feature sequence of a pedestrian target and an interactive object associated therewith based on the maximum number and the target frame sequence includes: For each target frame in the target frame sequence, determining first position information of a pedestrian target detection frame and second position information of an interactive object detection frame; splicing the first position information of the pedestrian target and the second position information of the interactive object based on the maximum number to obtain spliced ​​position information; The splicing position information corresponding to each target frame in the target frame sequence is cascaded to obtain the dynamic feature sequence.

6. The pedestrian intention recognition method according to claim 3, characterized in that: Determining the multi-source information feature sequence based on the visual feature sequence and the dynamic feature sequence includes: Get vehicle speed data; Determining vehicle speed features corresponding to different moments based on the vehicle speed data as a vehicle feature sequence; The vehicle speed feature sequence, the visual feature sequence and the dynamic feature sequence are determined as the multi-source information feature sequence.

7. The pedestrian intention recognition method according to claim 1, characterized in that: The multi-source information feature sequence is input into the spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, including: Determine the semantic relationship feature and the first temporal relationship feature between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset first spatiotemporal attention graph convolution module; Determine the spatial relationship feature and the second temporal relationship feature between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset second spatiotemporal attention graph convolution module; The semantic relationship feature, the spatial relationship feature, the first temporal relationship feature and the second temporal relationship feature are fused to obtain the spatiotemporal correlation feature.

8. The method for pedestrian intention recognition according to claim 7, characterized in that: Determining the semantic relationship features between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset first spatiotemporal attention graph convolution module includes: Inputting the visual feature sequence in the multi-source information feature sequence into a first spatiotemporal attention graph convolution module to obtain semantic relationship features between pedestrian targets and interactive objects associated therewith; or, The visual feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the first spatiotemporal attention graph convolution module to obtain the semantic relationship features between the pedestrian target and the interactive object associated with it.

9. The pedestrian intention recognition method according to claim 7, characterized in that: Determining the spatial relationship features between the pedestrian target and the interactive object associated therewith based on the multi-source information feature sequence and the preset second spatiotemporal attention graph convolution module includes: Inputting the dynamic feature sequence in the multi-source information feature sequence into the second spatiotemporal attention graph convolution module to obtain the spatial relationship feature between the pedestrian target and the interactive object associated therewith; or, The dynamic feature sequence and the vehicle speed feature sequence in the multi-source information feature sequence are input into the second spatiotemporal attention graph convolution module to obtain the spatial relationship features between the pedestrian target and the interactive objects associated with it.

10. The pedestrian intention recognition method according to claim 7, characterized in that: The first spatiotemporal attention graph convolution module includes: a first graph convolution unit, a first attention unit and a first temporal convolution unit; The first graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object; The first attention unit is used to determine a node visual feature sequence of each node in the star graph based on a visual feature sequence, and to determine a semantic relationship feature between the pedestrian target and the interactive object according to each of the node visual feature sequences using an attention mechanism; The first temporal convolution unit is used to determine a first temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target image in the target image sequence.

11. The pedestrian intention recognition method according to claim 7, characterized in that: The second spatiotemporal attention graph convolution module includes: a second graph convolution unit, a second attention unit and a second temporal convolution unit; The second graph convolution unit is used to generate a star graph based on an adjacency matrix, wherein the star graph has a pedestrian target as a central node and the interactive object associated with the pedestrian target as a non-central node, and the adjacency matrix contains the pedestrian target, the interactive object associated with the pedestrian target, and the association relationship between the pedestrian target and the interactive object; The second attention unit is used to determine a node spatial feature sequence of each node in the star graph based on the dynamic feature sequence, and determine a spatial relationship feature between the pedestrian target and the interactive object according to each node spatial feature sequence using an attention mechanism; The second temporal convolution unit is used to determine a second temporal relationship feature between the pedestrian target and the interactive object based on temporal information of each target frame in the target frame sequence.

12. The pedestrian intention recognition method according to claim 1, characterized in that: Determining the pedestrian target's crossing intention based on the spatiotemporal correlation features includes: Inputting the multi-source information features into a long short-term memory network to obtain decoding features corresponding to the pedestrian target; The decoded features are input into a fully connected network to obtain the pedestrian target's crossing intention.

13. A pedestrian intention recognition device, characterized in that: include: An acquisition module, used to acquire a sequence of image frames collected from a vehicle driving environment; A first determination module, configured to determine a multi-source information feature sequence of a pedestrian target and an interactive object associated therewith based on the image frame sequence; An input module, used for inputting the multi-source information feature sequence into a spatiotemporal attention graph convolution model to obtain spatiotemporal correlation features, wherein the spatiotemporal correlation features are used to characterize the spatiotemporal correlation relationship between the pedestrian target and the interactive object associated with it; The second determination module is used to determine the pedestrian target's crossing intention based on the spatiotemporal correlation characteristics.

14. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the pedestrian intention recognition method described in any one of claims 1 to 12 when executing the program stored in the memory.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program of a pedestrian intention recognition method, and when the program of the pedestrian intention recognition method is executed by a processor, the steps of the pedestrian intention recognition method described in any one of claims 1 to 12 are implemented.

Images