Pedestrian crossing intention identification method and device

Abstract

The invention relates to the technical field of automatic driving, in particular to a pedestrian crossing intention recognition method and device. Obtaining pedestrian data and street crossing data corresponding to the target pedestrian, wherein the pedestrian data comprises gender data corresponding to the target pedestrian; the crossing data comprises a pedestrian speed corresponding to the target pedestrian, a vehicle speed of a target vehicle corresponding to the target pedestrian, a first distance from the target pedestrian to a zebra crossing, a second distance from the target vehicle to the zebra crossing, and collision time of the target pedestrian relative to the target vehicle; performing data conversion from a time sequence to a space picture on the pedestrian data and the street crossing data to generate a target image; and inputting the target image into a preset pedestrian crossing intention recognition model, and outputting a target crossing intention corresponding to the target pedestrian. According to the method, the street crossing intention of the target pedestrian can be accurately identified, so that the automatic driving vehicle can make a pre-judgment in advance in the driving process, and a safer and more reliable decision can be taken.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and device for identifying pedestrians' intention to cross the street. Background Art

[0002] In the traffic system, pedestrians are a relatively vulnerable group. Since pedestrians lack external protection, even the slightest conflict with vehicles is likely to cause serious consequences. This not only causes great harm to pedestrians, but also has a serious impact on traffic order.

[0003] For autonomous vehicles, driving safely and reliably in complex urban road environments is a vital mission. Among the many influencing factors, accurately identifying pedestrians' intention to cross the street is the key to avoiding pedestrian-vehicle conflicts. The behavior of pedestrians on urban roads is uncertain and diverse, and they may show different tendencies to cross the street in different scenarios. Only by accurately identifying pedestrians' intention to cross the street can autonomous vehicles make predictions in advance during driving and take safer and more reliable decisions.

[0004] Therefore, how to accurately judge the pedestrians' intention to cross the street has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the present invention provides a method and device for identifying a pedestrian's intention to cross the street, so as to solve the problem of how to accurately judge a pedestrian's intention to cross the street.

[0006] In a first aspect, the present invention provides a method for identifying pedestrians' intention to cross the street, the method comprising:

[0007] Obtain pedestrian data and street crossing data corresponding to the target pedestrian, the pedestrian data including gender data corresponding to the target pedestrian; the street crossing data including pedestrian speed corresponding to the target pedestrian, vehicle speed of the target vehicle corresponding to the target pedestrian, a first distance from the target pedestrian to the zebra crossing, a second distance from the target vehicle to the zebra crossing, and a collision time of the target pedestrian relative to the target vehicle;

[0008] Convert pedestrian data and street crossing data from time series to spatial images to generate target images;

[0009] The target image is input into the preset pedestrian crossing intention recognition model, and the target crossing intention corresponding to the target pedestrian is output; the crossing intention includes walk-walk, walk-stop, and stop-walk.

[0010] The pedestrian crossing intention recognition method provided in the embodiment of the present application obtains pedestrian data and crossing data corresponding to the target pedestrian. The pedestrian data and crossing data are converted from time series to spatial images to generate a target image, thereby ensuring the accuracy of the generated target image. The target image is input into a preset pedestrian crossing intention recognition model, and the target crossing intention corresponding to the target pedestrian is output, thereby ensuring the accuracy of the outputted crossing intention corresponding to the target pedestrian. The above method can accurately identify the target pedestrian's crossing intention, so that the autonomous driving vehicle can make predictions in advance during driving and take safer and more reliable decisions.

[0011] In an optional implementation, the pedestrian data and the street crossing data are converted from a time series to a spatial image to generate a target image, including:

[0012] Convert gender data, pedestrian speed, vehicle speed, first distance, second distance and collision time into grayscale images respectively;

[0013] generating a first color image based on a first grayscale image corresponding to the gender data, a second grayscale image corresponding to the pedestrian speed, and a third grayscale image corresponding to the vehicle speed;

[0014] generating a second color image based on a fourth grayscale image corresponding to the first distance, a fifth grayscale image corresponding to the second distance, and a sixth grayscale image corresponding to the collision time;

[0015] The first color image and the second color image are spliced ​​to generate a target image.

[0016] The pedestrian crossing intention recognition method provided in the embodiment of the present application converts gender data, pedestrian speed, vehicle speed, first distance, second distance and collision time into grayscale images respectively, thereby ensuring the accuracy of the converted grayscale images. A first color image is generated based on the first grayscale image corresponding to the gender data, the second grayscale image corresponding to the pedestrian speed and the third grayscale image corresponding to the vehicle speed, thereby ensuring the accuracy of the generated first color image. A second color image is generated based on the fourth grayscale image corresponding to the first distance, the fifth grayscale image corresponding to the second distance and the sixth grayscale image corresponding to the collision time, thereby ensuring the accuracy of the generated second color image. The first color image and the second color image are spliced ​​to generate a target image, thereby ensuring the accuracy of the generated target image.

[0017] In an optional implementation, converting the pedestrian speed into a grayscale image includes:

[0018] The time series data corresponding to the pedestrian speed is divided into multiple bins; the amount of data in each bin is the same;

[0019] For each bin, calculate the frequency of the current bin transferring to the next bin;

[0020] Construct a transfer matrix based on the frequency of the current bin transferring to the next bin;

[0021] Based on the transfer matrix, a transfer field is constructed; each element in the transfer field represents the frequency of transfer from each quantile bin to another quantile bin;

[0022] According to the frequency value in the transfer field, the grayscale value of each pixel is mapped to the corresponding bin to generate a grayscale image corresponding to the pedestrian speed.

[0023] The pedestrian crossing intention recognition method provided in the embodiment of the present application divides the time series data corresponding to the pedestrian speed into multiple quantile boxes; for each quantile box, the frequency of the current quantile box transferring to the next quantile box is calculated, thereby ensuring the accuracy of the calculated frequency of the current quantile box transferring to the next quantile box. Based on the frequency of the current quantile box transferring to the next quantile box, a transfer matrix is ​​constructed, thereby ensuring the accuracy of the constructed transfer matrix. Based on the transfer matrix, a transfer field is constructed, thereby ensuring the accuracy of the constructed transfer field. According to the frequency value in the transfer field, the grayscale value of each pixel point is mapped to the corresponding quantile box, and a grayscale image corresponding to the pedestrian speed is generated, thereby ensuring the accuracy of the generated grayscale image corresponding to the pedestrian speed.

[0024] In an optional implementation, the target image is input into a preset pedestrian crossing intention recognition model, and the target crossing intention corresponding to the target pedestrian is output, including:

[0025] Dividing the target image into a plurality of sub-image blocks according to a first preset size; wherein the sub-image blocks do not overlap;

[0026] Input each sub-image block into the feature extraction module in the preset pedestrian crossing intention recognition model, perform local feature extraction and global feature extraction on each sub-image block, and output the target feature vector;

[0027] Based on the target feature vector, the target crossing intention corresponding to the target pedestrian is output.

[0028] The pedestrian crossing intention recognition method provided in the embodiment of the present application divides the target image into multiple sub-image blocks according to a first preset size; each sub-image block is input into a feature extraction module in a preset pedestrian crossing intention recognition model, local feature extraction and global feature extraction are performed on each sub-image block, and a target feature vector is output, thereby ensuring that the generated target feature vector includes the local features and global features of the target image, thereby ensuring the accuracy of the generated target feature vector. Based on the target feature vector, the target crossing intention corresponding to the target pedestrian is output, thereby ensuring the accuracy of the outputted crossing intention corresponding to the target pedestrian.

[0029] In an optional implementation, the feature extraction module includes four stage modules, inputting each sub-image block into the feature extraction module in the preset pedestrian crossing intention recognition model, performing local feature extraction and global feature extraction on each sub-image block, and outputting a target feature vector; including:

[0030] Input each sub-image block into the first stage module in the feature extraction module, map the dimension of each sub-image block, and generate a first feature vector corresponding to the target image;

[0031] Performing a first local feature extraction and a first global feature extraction on the first feature vector to generate a second feature vector; the second feature vector has the same size as the first feature vector;

[0032] The second feature vector is input into the second stage module in the feature extraction module, the second feature vector is downsampled, and the number of channels of the second feature vector is increased to generate a third feature vector;

[0033] Performing a second local feature extraction and a second global feature extraction on the third feature vector to generate a fourth feature vector; the fourth feature vector has the same size as the third feature vector;

[0034] Inputting the fourth feature vector into the third stage module in the feature extraction module, downsampling the fourth feature vector, and increasing the number of channels of the fourth feature vector to generate a fifth feature vector;

[0035] Performing a third local feature extraction and a third global feature extraction on the fifth feature vector to generate a sixth feature vector; the sixth feature vector has the same size as the fifth feature vector;

[0036] Inputting the sixth eigenvector into the fourth stage module in the feature extraction module, downsampling the sixth eigenvector, and increasing the number of channels of the sixth eigenvector to generate a seventh eigenvector;

[0037] A fourth local feature extraction and a fourth global feature extraction are performed on the seventh feature vector to generate a target feature vector; the target feature vector has the same size as the seventh feature vector.

[0038] The pedestrian crossing intention recognition method provided in the embodiment of the present application sequentially performs local feature extraction and global feature extraction through four-stage modules, thereby ensuring the accuracy of the generated target feature vector.

[0039] In an optional embodiment, the first stage module includes a local linear embedding module and a first level mixed attention module, the second stage module includes a first image block merging module and a second level mixed attention module, the third stage module includes a second image block merging module and a third level mixed attention module, and the fourth stage module includes a third image block merging module and a fourth level mixed attention module.

[0040] The pedestrian crossing intention recognition method provided in the embodiment of the present application comprises a first-stage module including a local linear embedding module and a first-level mixed attention module, a second-stage module including a first image block merging module and a second-level mixed attention module, a third-stage module including a second image block merging module and a third-level mixed attention module, and a fourth-stage module including a third image block merging module and a fourth-level mixed attention module, thereby ensuring the accuracy of the target feature vector finally generated.

[0041] In an optional implementation, the first-level hybrid attention module includes a local feature extraction network, a global feature extraction network, a compression and excitation network, and a deep separable network, performs a first local feature extraction and a first global feature extraction on the first feature vector, and generates a second feature vector, including:

[0042] The local feature extraction network performs local feature extraction on the first feature vector to generate a first sub-feature;

[0043] The global feature extraction network performs global feature extraction on the first feature vector to generate a second sub-feature;

[0044] The first sub-feature and the second sub-feature are fused to generate a fused feature;

[0045] The compression and excitation network identifies the fusion features and assigns weights to the features of different channels in the fusion features according to the identification results;

[0046] The deep separable network compresses the fused features according to the weights corresponding to the features of different channels in the fused features to reduce the channel dimension of the fused features and generate a second feature vector.

[0047] In the pedestrian crossing intention recognition method provided in the embodiment of the present application, the local feature extraction network performs local feature extraction on the first feature vector to generate the first sub-feature, thereby ensuring the accuracy of the generated first sub-feature. The global feature extraction network performs global feature extraction on the first feature vector to generate the second sub-feature, thereby ensuring the accuracy of the generated second sub-feature. The first sub-feature and the second sub-feature are fused to generate a fused feature, thereby ensuring that the generated fused feature contains both local features and global features. The compression and excitation network identifies the fused feature and assigns weights to the features of different channels in the fused feature according to the identification results; the deep separable network compresses the fused feature according to the weights corresponding to the features of different channels in the fused feature to reduce the channel dimension of the fused feature and generate a second feature vector, thereby ensuring that the generated second feature vector contains both local features and global features and has accurate dimensions.

[0048] In an optional implementation, the local feature extraction network performs local feature extraction on the first feature vector to generate a first sub-feature, including:

[0049] The first linear normalization layer in the local feature extraction network normalizes the first feature vector to generate a third sub-feature;

[0050] The depth-wise convolution layer in the local feature extraction network performs depth-wise convolution on the third sub-feature, extracts features of different channels of the third sub-feature, and obtains the fourth sub-feature;

[0051] The second linear normalization layer in the local feature extraction network normalizes the fourth sub-feature to generate a fifth sub-feature;

[0052] The linear transformation layer in the local feature extraction network performs a linear transformation on the fifth sub-feature to obtain a first query matrix, and extracts a first key value matrix and a first value matrix from the fifth sub-feature;

[0053] Multiplying the first query matrix by the first key value matrix to obtain an initial score matrix;

[0054] Scaling the initial score matrix to obtain the target score matrix;

[0055] Perform a convolution operation on the target score matrix to calculate the attention weights;

[0056] The sixth sub-feature is calculated based on the attention weight and the first value matrix;

[0057] The sixth sub-feature is normalized by application instance to obtain the first sub-feature.

[0058] In the pedestrian crossing intention recognition method provided by the embodiment of the present application, the first linear normalization layer in the local feature extraction network normalizes the first feature vector to generate the third sub-feature, thereby reducing the scale difference between different features. The depth-by-depth convolution layer in the local feature extraction network performs depth-by-depth convolution on the third sub-feature, extracts the features of different channels of the third sub-feature, and obtains the fourth sub-feature, thereby ensuring that the obtained fourth sub-feature contains the local features corresponding to each channel. The second linear normalization layer in the local feature extraction network normalizes the fourth sub-feature to generate the fifth sub-feature, thereby ensuring that the fifth sub-feature maintains a stable scale. The linear transformation layer in the local feature extraction network performs a linear transformation on the fifth sub-feature to obtain a first query matrix, and extracts a first key value matrix and a first value matrix from the fifth sub-feature; the first query matrix is ​​multiplied by the first key value matrix to obtain an initial score matrix, thereby ensuring the accuracy of the obtained initial score matrix. The initial score matrix is ​​scaled to obtain a target score matrix, thereby ensuring the accuracy of the obtained target score matrix. The target score matrix is ​​convolved to calculate the attention weight, thereby ensuring the accuracy of the calculated attention weight. The sixth sub-feature is calculated based on the attention weight and the first value matrix, which ensures the accuracy of the generated sixth sub-feature. The sixth sub-feature is normalized by the application instance to obtain the first sub-feature, which ensures the accuracy of the obtained first sub-feature.

[0059] In an optional implementation, the global feature extraction network performs global feature extraction on the first feature vector to generate a second sub-feature, including:

[0060] Performing a linear transformation on the first eigenvector to map and obtain a second query matrix, a second key value matrix, and a second value matrix;

[0061] For each attention mechanism head in the global feature extraction network, calculate the attention weight corresponding to each attention mechanism head;

[0062] The attention weights corresponding to each attention mechanism head are concatenated to generate the second sub-feature.

[0063] The pedestrian crossing intention recognition method provided in the embodiment of the present application performs a linear transformation on the first feature vector, maps the second query matrix, the second key value matrix and the second value matrix, and ensures the accuracy of the obtained second query matrix, the second key value matrix and the second value matrix. For each attention mechanism head in the global feature extraction network, the attention weight corresponding to each attention mechanism head is calculated to ensure the accuracy of the calculated attention weight corresponding to each attention mechanism head. The attention weights corresponding to each attention mechanism head are concatenated to generate a second sub-feature, ensuring that the generated second sub-feature contains the global feature of the first feature vector.

[0064] In a second aspect, the present invention provides a pedestrian crossing intention recognition device, the device comprising:

[0065] an acquisition module, for acquiring pedestrian data and street crossing data corresponding to the target pedestrian, the pedestrian data including age data and gender data corresponding to the target pedestrian; the street crossing data including pedestrian speed corresponding to the target pedestrian, vehicle speed of the target vehicle corresponding to the target pedestrian, a first distance from the target pedestrian to the zebra crossing, a second distance from the target vehicle to the zebra crossing, and a collision time of the target pedestrian relative to the target vehicle;

[0066] The data conversion module is used to convert the pedestrian data and street crossing data from time series to spatial images to generate the target image;

[0067] The input module is used to input the target image into a preset pedestrian crossing intention recognition model and output the target crossing intention corresponding to the target pedestrian; the crossing intention includes walk-walk, walk-stop, and stop-walk.

[0068] The pedestrian crossing intention recognition device provided in the embodiment of the present application obtains pedestrian data and crossing data corresponding to the target pedestrian. The pedestrian data and crossing data are converted from time series to spatial images to generate a target image, thereby ensuring the accuracy of the generated target image. The target image is input into a preset pedestrian crossing intention recognition model, and the target crossing intention corresponding to the target pedestrian is output, thereby ensuring the accuracy of the output crossing intention corresponding to the target pedestrian. The above method can accurately identify the target pedestrian's crossing intention, so that the autonomous driving vehicle can make predictions in advance during driving and take safer and more reliable decisions.

[0069] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for identifying pedestrian crossing intention according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0070] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for identifying pedestrian crossing intentions according to the first aspect or any corresponding embodiment thereof.

[0071] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for identifying pedestrian crossing intention according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 is a schematic flow chart of a method for identifying pedestrian crossing intention according to an embodiment of the present invention;

[0074] Figure 2 is a schematic diagram of data collection according to an embodiment of the present invention;

[0075] Figure 3 is a schematic diagram of a pedestrian's intention to cross the street according to an embodiment of the present invention;

[0076] Figure 4 is a flow chart of another method for identifying pedestrian crossing intention according to an embodiment of the present invention;

[0077] Figure 5 is a schematic diagram of a process of generating a target image according to an embodiment of the present invention;

[0078] Figure 6 is a flow chart of another method for identifying pedestrians' intention to cross the street according to an embodiment of the present invention;

[0079] Figure 7 is a schematic diagram of a process for determining a target pedestrian's intention based on a pedestrian intention recognition model according to an embodiment of the present invention;

[0080] Figure 8 It is a schematic diagram of a process in which a first-level hybrid attention module according to an embodiment of the present invention can perform a first local feature extraction and a first global feature extraction on a first feature vector to generate a second feature vector;

[0081] Fig. 9 is a schematic diagram of a process of performing local feature extraction on a first feature vector by a local feature extraction network according to an embodiment of the present invention to generate a first sub-feature;

[0082] Fig.10 is a schematic diagram of a process of performing global feature extraction on a first feature vector by a global feature extraction network according to an embodiment of the present invention to generate a second sub-feature;

[0083] Fig.11 is a structural block diagram of a device for identifying pedestrian crossing intention according to an embodiment of the present invention;

[0084] Fig.12 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0086] In the traffic system, pedestrians are a relatively vulnerable group. Since pedestrians lack external protection, even the slightest conflict with vehicles is likely to cause serious consequences. This not only causes great harm to pedestrians, but also has a serious impact on traffic order.

[0087] For autonomous vehicles, driving safely and reliably in complex urban road environments is a vital mission. Among the many influencing factors, accurately identifying pedestrians' intention to cross the street is the key to avoiding pedestrian-vehicle conflicts. The behavior of pedestrians on urban roads is uncertain and diverse, and they may show different tendencies to cross the street in different scenarios. Only by accurately identifying pedestrians' intention to cross the street can autonomous vehicles make predictions in advance during driving and take safer and more reliable decisions.

[0088] Therefore, how to accurately judge the pedestrians' intention to cross the street has become an urgent problem to be solved.

[0089] Based on this, an embodiment of the present application provides a method for identifying pedestrian crossing intentions, and obtains pedestrian data and crossing data corresponding to the target pedestrian. The pedestrian data and crossing data are converted from time series to spatial images to generate a target image, thereby ensuring the accuracy of the generated target image. The target image is input into a preset pedestrian crossing intention recognition model, and the target crossing intention corresponding to the target pedestrian is output, thereby ensuring the accuracy of the outputted crossing intention corresponding to the target pedestrian. The above method can accurately identify the target pedestrian's crossing intention, so that the autonomous driving vehicle can make predictions in advance during driving and take safer and more reliable decisions.

[0090] It should be noted that the method for pedestrian crossing intention recognition provided in the embodiment of the present application can be executed by a device for pedestrian crossing intention recognition, and the device for pedestrian crossing intention recognition can be implemented as part or all of an electronic device through software, hardware, or a combination of software and hardware, wherein the electronic device can be a server or a control device in an autonomous driving vehicle. The server in the embodiment of the present application can be a single server or a server cluster composed of multiple servers. In the following method embodiments, the execution subject is an electronic device as an example for explanation.

[0091] According to an embodiment of the present invention, an embodiment of a method for identifying pedestrian crossing intention is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0092] In this embodiment, a method for identifying pedestrian crossing intention is provided, which can be used in the above-mentioned electronic device. Figure 1 is a flow chart of a method for identifying pedestrian crossing intention according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0093] Step S101, obtaining pedestrian data and street crossing data corresponding to a target pedestrian.

[0094] Among them, the pedestrian data includes the gender data corresponding to the target pedestrian; the street crossing data includes the pedestrian speed corresponding to the target pedestrian, the vehicle speed of the target vehicle corresponding to the target pedestrian, the first distance from the target pedestrian to the zebra crossing, the second distance from the target vehicle to the zebra crossing, and the collision time of the target pedestrian relative to the target vehicle.

[0095] Specifically, the electronic device may take a photo of the target pedestrian through a camera device to obtain image data of the target pedestrian, and then identify the image data of the target pedestrian to determine the gender data corresponding to the target pedestrian.

[0096] The electronic device can also obtain the pedestrian speed corresponding to the target pedestrian and the vehicle speed of the target vehicle corresponding to the target pedestrian through sensor devices such as laser radar and millimeter wave radar.

[0097] The electronic device can also use a camera to capture a target pedestrian scene image, identify the target pedestrian scene image, and determine the first distance from the target pedestrian to the zebra crossing. Use a camera to capture a target vehicle scene image, identify the target vehicle scene image, and determine the first distance from the target vehicle to the zebra crossing.

[0098] Then, the electronic device can also obtain the relative distance between the target pedestrian and the target vehicle, and the relative speed between the target pedestrian and the target vehicle based on sensor devices such as laser radar and millimeter wave radar, and then divide the relative distance between the target pedestrian and the target vehicle by the relative speed between the target pedestrian and the target vehicle to generate the collision time of the target pedestrian and the target vehicle. Figure 2 The figure shows a schematic diagram of data collection.

[0099] Step S102, performing data conversion from time series to spatial images on pedestrian data and street crossing data to generate a target image.

[0100] Specifically, the electronic device can perform data preprocessing on pedestrian data and street crossing data, such as denoising, formatting, extracting useful features, etc.

[0101] Then, the electronic device can convert the pedestrian data and street crossing data from time series to spatial images based on a preset data conversion method to generate a target image.

[0102] Specifically, the electronic device can split the street crossing data to generate multiple non-intersecting street crossing data subsequences. According to the relationship between the street crossing data subsequences, the correlation integral between the street crossing data subsequences is calculated. According to the correlation integral between the street crossing data subsequences, the autocorrelation characteristics between the street crossing data subsequences are calculated. According to the autocorrelation characteristics between the street crossing data subsequences, the delay time and embedding dimension corresponding to the street crossing data sequence are determined. Based on the delay time and the embedding dimension, a street crossing phase space corresponding to the street crossing data sequence is generated. According to the distance between the street crossing vectors in the street crossing phase space, a grayscale image corresponding to the street crossing data is generated.

[0103] This step will be described in detail below.

[0104] Step S103: input the target image into a preset pedestrian crossing intention recognition model, and output the target crossing intention corresponding to the target pedestrian.

[0105] The intention to cross the street includes walk-walk, walk-stop, and stop-walk. Walk-walk means that the pedestrian crosses the street directly without stopping at the zebra crossing; walk-stop means that the pedestrian chooses to stay at the zebra crossing after observing the surrounding traffic environment; stop-walk means that the pedestrian chooses to cross directly after observing the traffic environment during the stay at the zebra crossing. Figure 3 Shown is a schematic diagram of pedestrian crossing.

[0106] Specifically, the electronic device can input the target image into a preset pedestrian crossing intention recognition model, the preset pedestrian crossing intention recognition model performs feature recognition on the target image, and outputs the target crossing intention corresponding to the target pedestrian based on the recognized features.

[0107] This step will be described in detail below.

[0108] The pedestrian crossing intention recognition method provided in the embodiment of the present application obtains pedestrian data and crossing data corresponding to the target pedestrian. The pedestrian data and crossing data are converted from time series to spatial images to generate a target image, thereby ensuring the accuracy of the generated target image. The target image is input into a preset pedestrian crossing intention recognition model, and the target crossing intention corresponding to the target pedestrian is output, thereby ensuring the accuracy of the outputted crossing intention corresponding to the target pedestrian. The above method can accurately identify the target pedestrian's crossing intention, so that the autonomous driving vehicle can make predictions in advance during driving and take safer and more reliable decisions.

[0109] In this embodiment, a method for identifying pedestrians' intention to cross the street is provided, which can be used in the above-mentioned electronic device. Figure 4 is a flow chart of a method for identifying pedestrian crossing intention according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0110] Step S201, obtaining pedestrian data and street crossing data corresponding to a target pedestrian.

[0111] Among them, the pedestrian data includes the gender data corresponding to the target pedestrian; the street crossing data includes the pedestrian speed corresponding to the target pedestrian, the vehicle speed of the target vehicle corresponding to the target pedestrian, the first distance from the target pedestrian to the zebra crossing, the second distance from the target vehicle to the zebra crossing, and the collision time of the target pedestrian relative to the target vehicle.

[0112] For details about this step, please refer to the above description of step S101, which will not be elaborated here.

[0113] Step S202: convert the pedestrian data and the street crossing data from time series to spatial images to generate a target image.

[0114] Specifically, the above step S202 may include the following steps:

[0115] Step S2021, converting the gender data, pedestrian speed, vehicle speed, first distance, second distance and collision time into grayscale images respectively.

[0116] Specifically, the “converting pedestrian speed into a grayscale image” in the above step S2021 may include the following steps:

[0117] Step a1, divide the time series data corresponding to the pedestrian speed into multiple bins.

[0118] The amount of data in each quantile box is the same.

[0119] Specifically, the electronic device can use quantiles to divide the time series data corresponding to the pedestrian speed into multiple quantile boxes. Among them, common quantiles are quartiles (Q1, Q2, Q3), where Q2 is the median.

[0120] For example, the pedestrian speed may be divided into five bins: Q1-Q2, Q2-Q3, Q3-Q4, Q4-maximum. Step a2: for each bin, calculate the frequency of the current bin transferring to the next bin.

[0121] Specifically,

[0122] Step a3, constructing a transfer matrix based on the frequency of transfer from the current bin to the next bin.

[0123] Specifically, the electronic device may construct a transfer matrix based on the frequency of the current bin transferring to the next bin.

[0124] For example, the transfer matrix can be as follows:

[0125] st∑ j w ij =1

[0126] Among them, w ij It represents the frequency of transfer from bin i to bin j.

[0127] Step a4: construct a transfer field based on the transfer matrix.

[0128] Each element in the transfer field represents the frequency of transfer from each bin to another bin.

[0129] Specifically, the electronic device can construct a transfer field according to the transfer matrix.

[0130] Exemplarily, the transition field may be as follows:

[0131]

[0132] Step a5, mapping the grayscale value of each pixel to the corresponding bin according to the frequency value in the transfer field, and generating a grayscale image corresponding to the pedestrian speed.

[0133] Specifically, the electronic device maps the grayscale value of each pixel to a corresponding bin according to the frequency value in the transfer field, and generates a grayscale image corresponding to the pedestrian speed.

[0134] Similarly, the electronic device can convert the gender data, vehicle speed, first distance, second distance and collision time into a grayscale image according to the above method.

[0135] Step S2022: Generate a first color image based on the first grayscale image corresponding to the gender data, the second grayscale image corresponding to the pedestrian speed, and the third grayscale image corresponding to the vehicle speed.

[0136] Specifically, the first grayscale image is generated based on gender, and the color of each pixel will reflect its corresponding gender data. The second grayscale image is generated based on pedestrian speed, and the color of each pixel will reflect the size of the pedestrian speed. The third grayscale image is generated based on vehicle speed, and the color of each pixel will reflect the size of the vehicle speed.

[0137] The electronic device can select different color maps for gender data, pedestrian speed, and vehicle speed. These color maps will be used to convert grayscale values ​​into color values. The electronic device then overlays the three grayscale images onto a color image, one for each color channel.

[0138] Exemplarily, the red channel corresponds to gender data (first grayscale image), the green channel corresponds to pedestrian speed data (second grayscale image), and the blue channel corresponds to vehicle speed data (third grayscale image). In order to make the generated color image easier to observe and understand, the electronic device can adjust the brightness and contrast to generate the first color image.

[0139] Step S2023: Generate a second color image based on the fourth grayscale image corresponding to the first distance, the fifth grayscale image corresponding to the second distance, and the sixth grayscale image corresponding to the collision time.

[0140] Specifically, the fourth grayscale image is generated based on the first distance, and the color of each pixel will reflect the distance of the first distance. The fifth grayscale image is generated based on the second distance, and the color of each pixel will reflect the distance of the second distance. The third grayscale image is generated based on the collision time, and the color of each pixel will reflect the size of the collision time.

[0141] The electronic device may select different color mappings for the first distance, the second distance, and the collision time. These color mappings will be used to convert the grayscale values ​​into color values. The electronic device then superimposes the three grayscale images onto a color image, one for each color channel.

[0142] Exemplarily, the red channel corresponds to the first distance (first grayscale image), the green channel corresponds to the second distance of the pedestrian (second grayscale image), and the blue channel corresponds to the collision time (third grayscale image). In order to make the generated color image easier to observe and understand, the electronic device can adjust the brightness and contrast to generate the second color image.

[0143] Step S2024, splicing the first color image and the second color image to generate a target image.

[0144] Specifically, the electronic device may splice the first color image and the second color image to generate a target image.

[0145] For example, Figure 5 Shown is the process of generating the target image.

[0146] Step S203: input the target image into a preset pedestrian crossing intention recognition model, and output the target crossing intention corresponding to the target pedestrian.

[0147] Crossing intentions include walk-walk, walk-stop, and stop-walk.

[0148] For details about this step, please refer to the above description of step S103, which will not be elaborated here.

[0149] The pedestrian crossing intention recognition method provided in this embodiment divides the time series data corresponding to the pedestrian speed into multiple bins; for each bin, the frequency of the current bin transferring to the next bin is calculated, ensuring the accuracy of the calculated frequency of the current bin transferring to the next bin. Based on the frequency of the current bin transferring to the next bin, a transfer matrix is ​​constructed to ensure the accuracy of the constructed transfer matrix. Based on the transfer matrix, a transfer field is constructed to ensure the accuracy of the constructed transfer field. According to the frequency value in the transfer field, the grayscale value of each pixel point is mapped to the corresponding bin, and a grayscale image corresponding to the pedestrian speed is generated to ensure the accuracy of the generated grayscale image corresponding to the pedestrian speed. Similarly, the gender data, vehicle speed, first distance, second distance and collision time are converted into grayscale images to ensure the accuracy of the converted grayscale images. Then, based on the first grayscale image corresponding to the gender data, the second grayscale image corresponding to the pedestrian speed and the third grayscale image corresponding to the vehicle speed, a first color image is generated to ensure the accuracy of the generated first color image. Based on the fourth grayscale image corresponding to the first distance, the fifth grayscale image corresponding to the second distance, and the sixth grayscale image corresponding to the collision time, a second color image is generated, thereby ensuring the accuracy of the generated second color image. The first color image and the second color image are spliced ​​to generate a target image, thereby ensuring the accuracy of the generated target image.

[0150] In this embodiment, a method for identifying pedestrians' intention to cross the street is provided, which can be used in the above-mentioned electronic device. Figure 6 is a flow chart of a method for identifying pedestrian crossing intention according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:

[0151] Step S301, obtaining pedestrian data and street crossing data corresponding to the target pedestrian.

[0152] Among them, the pedestrian data includes the gender data corresponding to the target pedestrian; the street crossing data includes the pedestrian speed corresponding to the target pedestrian, the vehicle speed of the target vehicle corresponding to the target pedestrian, the first distance from the target pedestrian to the zebra crossing, the second distance from the target vehicle to the zebra crossing, and the collision time of the target pedestrian relative to the target vehicle.

[0153] For details about this step, please refer to the above description of step S201, which will not be elaborated here.

[0154] Step S302: convert the pedestrian data and the street crossing data from time series to spatial images to generate a target image.

[0155] For details about this step, please refer to the above description of step S202, which will not be elaborated here.

[0156] Step S303: input the target image into a preset pedestrian crossing intention recognition model, and output the target crossing intention corresponding to the target pedestrian.

[0157] Among them, the intention to cross the street includes walk-walk, walk-stop, and stop-walk.

[0158] Specifically, the above step S303 may include the following steps:

[0159] Step S3031: divide the target image into a plurality of sub-image blocks according to a first preset size.

[0160] The sub-image blocks do not overlap with each other.

[0161] Specifically, the electronic device may divide the target image into a plurality of sub-image blocks according to a first preset size.

[0162] Among them, the first preset size can be 4*4, 5*5, or other sizes. The first preset size is set in the implementation of this application.

[0163] Exemplarily, the electronic device can divide the target image into multiple sub-image blocks according to the first preset size of 4*4. Wherein, each sub-image block is 4*4 in size, then the target image will be divided into H / 4*W / 4 sub-image blocks, and each sub-image block can be regarded as a local image area. This segmentation method enables the model to process the image in a more fine-grained manner, capture the local detail information of the image, and lay the foundation for subsequent feature extraction and processing.

[0164] Step S3032: input each sub-image block into a feature extraction module in a preset pedestrian crossing intention recognition model, perform local feature extraction and global feature extraction on each sub-image block, and output a target feature vector.

[0165] Specifically, Figure 7 As shown. The feature extraction module includes four stage modules. In an optional embodiment of the present application, the first stage module includes a local linear embedding module and a first-level mixed attention module, the second stage module includes a first image block merging module and a second-level mixed attention module, the third stage module includes a second image block merging module and a third-level mixed attention module, and the fourth stage module includes a third image block merging module and a fourth-level mixed attention module. The above step S3032 may include the following steps:

[0166] Step b1, input each sub-image block into the first stage module in the feature extraction module, map the dimension of each sub-image block, and generate a first feature vector corresponding to the target image.

[0167] Specifically, the electronic device inputs each sub-image block to the first stage module in the feature extraction module, and the local linear embedding module in the first stage module maps the dimension of each sub-image block to generate a first feature vector corresponding to the target image.

[0168] Exemplarily, the dimension of each sub-image block of the electronic device is 4*4*3, where 4*4 is the size of each sub-image block and 3 is the number of channels. The electronic device maps the dimension of each sub-image block to obtain 512 dimensions. At this time, the feature map is a sequence with a length of H / 4*W / 4, and each element is a 512-dimensional vector. These vectors are re-encoding and abstraction of the features of each sub-image block, extracting more semantic features, and preparing for the subsequent first-level hybrid attention module processing.

[0169] Step b2: performing a first local feature extraction and a first global feature extraction on the first feature vector to generate a second feature vector.

[0170] The second eigenvector has the same size as the first eigenvector.

[0171] Specifically, the first-level hybrid attention module in the first-stage module can perform a first local feature extraction and a first global feature extraction on the first feature vector to generate a second feature vector.

[0172] Specifically, the "performing a first local feature extraction and a first global feature extraction on the first feature vector to generate a second feature vector" in the above step b2 may include the following steps:

[0173] Step b21: The local feature extraction network performs local feature extraction on the first feature vector to generate a first sub-feature.

[0174] Specifically, the above step b21 may include the following steps:

[0175] Step b211: The first linear normalization layer in the local feature extraction network normalizes the first feature vector to generate a third sub-feature.

[0176] Specifically, the first linear normalization layer in the local feature extraction network normalizes the first feature vector and scales the features of the first feature vector to a standard range, such as [-1, 1] or [0, 1], to reduce the scale difference between different features, thereby generating a third sub-feature.

[0177] Step b212, the depth-by-depth convolution layer in the local feature extraction network performs depth-by-depth convolution on the third sub-feature, extracts features of different channels of the third sub-feature, and obtains the fourth sub-feature.

[0178] Specifically, the depth-wise convolution layer in the local feature extraction network applies convolution to each channel of the third sub-feature separately. This means that if the third sub-feature has C channels, there will be C independent convolution filters. Through depth-wise convolution, the depth-wise convolution layer can extract useful features from each channel separately. This operation allows the model to capture the unique information of each channel without fusing them. Then, after the depth-wise convolution process, each channel will produce a new set of features, and these feature maps together constitute the new fourth sub-features. These fourth sub-features can contain richer and more fine-grained information, providing more powerful feature representations for subsequent processing layers.

[0179] Step b213: the second linear normalization layer in the local feature extraction network normalizes the fourth sub-feature to generate a fifth sub-feature.

[0180] Specifically, the second linear normalization layer in the local feature extraction network normalizes the fourth sub-feature to generate the fifth sub-feature.

[0181] Step b214: the linear transformation layer in the local feature extraction network performs a linear transformation on the fifth sub-feature to obtain a first query matrix, and extracts a first key-value matrix and a first value matrix from the fifth sub-feature.

[0182] Specifically, the linear transformation layer in the local feature extraction network performs a linear transformation on the fifth sub-feature, wherein the linear transformation layer may include a weight matrix W and a bias vector b.

[0183] The formula of linear transformation can be expressed as: Q = W*F + b, where Q is the transformed first query matrix, F is the fifth sub-feature, W is the weight matrix, and b is the bias vector. Through this transformation, F is converted into Q, which can be used as the first query matrix for subsequent similarity calculations. At the same time, the electronic device can extract the first key value matrix and the first value matrix from the fifth sub-feature F.

[0184] Step b215, multiplying the first query matrix by the first key value matrix to obtain an initial score matrix.

[0185] Specifically, the local feature extraction network multiplies the first query matrix by the first key value matrix to obtain an initial score matrix.

[0186] Step b216, scaling the initial score matrix to obtain a target score matrix.

[0187] Specifically, the local feature extraction network performs a scaling operation on the initial score matrix to obtain the target score matrix, which can usually be achieved by adding a learnable scalar.

[0188] Step b217, perform a convolution operation on the target score matrix to calculate the attention weight.

[0189] Specifically, the electronic device performs a convolution operation on the target score matrix to calculate the attention weight.

[0190] Exemplarily, the attention weight can be calculated according to the following formula:

[0191]

[0192] Among them, Q1 is the first query matrix, K1 is the first key value matrix, d k It is the fifth sub-feature.

[0193] Step b218, calculating the sixth sub-feature based on the attention weight and the first value matrix.

[0194] Specifically, the local feature extraction network calculates the sixth sub-feature based on the attention weight and the first value matrix.

[0195] Specifically, the electronic device may calculate the sixth sub-feature according to the following formula.

[0196]

[0197] Among them, V1 is the first value matrix.

[0198] Step b219, normalize the sixth sub-feature based on the application instance to obtain the first sub-feature.

[0199] Specifically, the sixth sub-feature of the local feature extraction network is normalized by the application instance to obtain the first sub-feature.

[0200] Specifically, the first sub-feature is calculated according to the following formula:

[0201]

[0202] Step b22: The global feature extraction network performs global feature extraction on the first feature vector to generate a second sub-feature.

[0203] Specifically, the above step b22 may include the following steps:

[0204] Step b221, performing a linear transformation on the first eigenvector to map and obtain a second query matrix, a second key-value matrix, and a second value matrix.

[0205] Specifically, the global feature extraction network can perform a linear transformation on the first feature vector based on a preset linear transformation matrix, and map the first feature vector to obtain a second query matrix, a second key-value matrix, and a second value matrix.

[0206] The preset linear transformation matrix may be determined after the global feature extraction network has been trained multiple times.

[0207] Step b222, for each attention mechanism head in the global feature extraction network, calculate the attention weight corresponding to each attention mechanism head.

[0208] Specifically, for each attention mechanism head in the global feature extraction network, the attention weight corresponding to each attention mechanism head is calculated based on the second query matrix, the second key-value matrix and the second value matrix.

[0209] For example, the attention weight corresponding to each attention mechanism head can be calculated using the following formula:

[0210]

[0211] Among them, Q2 is the second query matrix, K2 is the second key value matrix, V2 is the second value matrix, d k is the first eigenvector.

[0212] Step b223, concatenate the attention weights corresponding to each attention mechanism head to generate the second sub-feature.

[0213] Specifically, the global feature extraction network can concatenate the attention weights corresponding to each attention mechanism head to generate a second sub-feature.

[0214] Specifically, the second sub-feature can be calculated using the following formula:

[0215] MultiHead(Q2,K2,V2)=Concat(head1···head a )*W O

[0216] Among them, W O is the weight.

[0217] Step b23, fusing the first sub-feature and the second sub-feature to generate a fused feature.

[0218] Specifically, the electronic device may perform splicing processing on the first sub-feature and the second sub-feature to generate a fusion feature.

[0219] Step b24, the compression and excitation network identifies the fused features and assigns weights to the features of different channels in the fused features according to the identification results.

[0220] Specifically, the compression and excitation network can perform global average pooling on the fused features to obtain the average response of each channel. Then, the result after global average pooling is processed by a nonlinear activation function (such as ReLU). The result after nonlinear mapping is channel normalized, and the weight of each channel is usually calculated using the normalization factor.

[0221] Then, the normalized features of each channel are weighted and summed with the original features. The result of the weighted summation is converted to values ​​between 0 and 1 through a nonlinear activation function (such as Sigmoid), which are used as the feature weights of each channel.

[0222] Step b25: the deep separable network compresses the fused features according to the weights corresponding to the features of different channels in the fused features to reduce the channel dimension of the fused features and generate a second feature vector.

[0223] Specifically, the deep separable network can compress channels whose weights are less than a preset weight threshold according to the weights corresponding to the features of different channels in the fused features, thereby reducing the channel dimension of the fused features and generating a second feature vector.

[0224] For example, Figure 8As shown, it is a schematic diagram of the process of the first-level hybrid attention module performing the first local feature extraction and the first global feature extraction on the first feature vector to generate the second feature vector.

[0225] For example, Fig. 9 As shown, the local feature extraction network performs local feature extraction on the first feature vector to generate a schematic flow chart of the first sub-feature.

[0226] For example, Fig.10 As shown, the global feature extraction network performs global feature extraction on the first feature vector to generate a flow chart of the second sub-feature.

[0227] Step b3: input the second feature vector to the second stage module in the feature extraction module, downsample the second feature vector, and increase the number of channels of the second feature vector to generate a third feature vector.

[0228] Specifically, the second eigenvector is input to the first image block merging module in the second stage module, and the first image block merging module downsamples the second eigenvector and increases the number of channels of the second eigenvector to generate a third eigenvector.

[0229] Exemplarily, after the first stage, the dimension of the second feature vector is H / 4*W / 4*C, where H / 4*W / 4 is the size and C is the number of channels. In the first image block merging module in the second stage module, the first image block merging module downsamples the second feature vector and increases the number of channels of the second feature vector to generate a third feature vector. The dimension of the third feature vector is H / 8*W / 8*2C.

[0230] Step b4: performing a second local feature extraction and a second global feature extraction on the third feature vector to generate a fourth feature vector.

[0231] The fourth eigenvector has the same size as the third eigenvector.

[0232] Specifically, the second-level hybrid attention module performs a second local feature extraction and a second global feature extraction on the third eigenvector to generate a fourth eigenvector.

[0233] Among them, the second-level hybrid attention module performs a second local feature extraction and a second global feature extraction on the third feature vector to generate a fourth feature vector. The process can refer to the first-level hybrid attention module performing a first local feature extraction and a first global feature extraction on the first feature vector to generate a second feature vector, which will not be repeated here.

[0234] Step b5, inputting the fourth eigenvector into the third stage module in the feature extraction module, downsampling the fourth eigenvector, and increasing the number of channels of the fourth eigenvector to generate a fifth eigenvector.

[0235] Specifically, the fourth feature vector is input to the third stage module in the feature extraction module. The second image block merging module in the third stage module downsamples the fourth feature vector and increases the number of channels of the fourth feature vector to generate a fifth feature vector.

[0236] Exemplarily, the dimension of the fourth eigenvector is H / 8*W / 8*2C, and the dimension of the fifth eigenvector is H / 16*W / 16*4C.

[0237] Step b6: performing a third local feature extraction and a third global feature extraction on the fifth feature vector to generate a sixth feature vector.

[0238] The sixth eigenvector has the same size as the fifth eigenvector.

[0239] Specifically, the third-level hybrid attention module in the third stage performs a third local feature extraction and a third global feature extraction on the fifth eigenvector to generate a sixth eigenvector.

[0240] Among them, the third-level hybrid attention module in the third stage performs a third local feature extraction and a third global feature extraction on the fifth eigenvector to generate the sixth eigenvector. The process can be referred to the first-level hybrid attention module performing the first local feature extraction and the first global feature extraction on the first eigenvector to generate the second eigenvector, which will not be repeated here.

[0241] Step b7, inputting the sixth eigenvector into the fourth stage module in the feature extraction module, downsampling the sixth eigenvector, and increasing the number of channels of the sixth eigenvector to generate a seventh eigenvector.

[0242] Specifically, the sixth eigenvector is input to the fourth stage module in the feature extraction module. The third image block merging module in the fourth stage module downsamples the sixth eigenvector and increases the number of channels of the sixth eigenvector to generate the seventh eigenvector.

[0243] Exemplarily, the dimension of the sixth eigenvector is H / 16*W / 16*4C, and the dimension of the seventh eigenvector is H / 32*W / 32*8C.

[0244] Step b8, performing a fourth local feature extraction and a fourth global feature extraction on the seventh feature vector to generate a target feature vector.

[0245] Among them, the target eigenvector has the same size as the seventh eigenvector.

[0246] Specifically, the fourth-level hybrid attention module in the fourth stage performs the fourth local feature extraction and the fourth global feature extraction on the seventh feature vector to generate a target feature vector.

[0247] Among them, the fourth-level hybrid attention module in the fourth stage performs the fourth local feature extraction and the fourth global feature extraction on the seventh eigenvector. The process of generating the target feature vector can refer to the first-level hybrid attention module performing the first local feature extraction and the first global feature extraction on the first feature vector to generate the second feature vector, which will not be repeated here.

[0248] Step S3033: output the target crossing intention corresponding to the target pedestrian based on the target feature vector.

[0249] Specifically, the preset pedestrian crossing intention recognition model can determine the probabilities of the target pedestrian's crossing intentions being walk-walk, walk-stop, and stop-walk based on the target feature vector. Then, according to the probabilities of the target pedestrian's crossing intentions being walk-walk, walk-stop, and stop-walk, the crossing intention with the highest probability is determined as the target crossing intention corresponding to the target pedestrian, and then the target crossing intention corresponding to the target pedestrian is output.

[0250] The pedestrian crossing intention recognition method provided in this embodiment divides the target image into multiple sub-image blocks according to a first preset size. Each sub-image block is input into a feature extraction module in a preset pedestrian crossing intention recognition model. The feature extraction module includes four stage modules, the first stage module includes a local linear embedding module and a first-level mixed attention module, the second stage module includes a first image block merging module and a second-level mixed attention module, the third stage module includes a second image block merging module and a third-level mixed attention module, and the fourth stage module includes a third image block merging module and a fourth-level mixed attention module.

[0251] Specifically, each sub-image block is input into the first stage module in the feature extraction module, and the dimensions of each sub-image block are mapped to generate the first feature vector corresponding to the target image. The first linear normalization layer in the local feature extraction network normalizes the first feature vector to generate the third sub-feature, thereby reducing the scale difference between different features. The depth-by-depth convolution layer in the local feature extraction network performs depth-by-depth convolution on the third sub-feature, extracts the features of different channels of the third sub-feature, and obtains the fourth sub-feature, thereby ensuring that the fourth sub-feature obtained contains the local features corresponding to each channel. The second linear normalization layer in the local feature extraction network normalizes the fourth sub-feature to generate the fifth sub-feature, thereby ensuring that the fifth sub-feature maintains a stable scale. The linear transformation layer in the local feature extraction network performs a linear transformation on the fifth sub-feature to obtain the first query matrix, and extracts the first key value matrix and the first value matrix from the fifth sub-feature; the first query matrix is ​​multiplied by the first key value matrix to obtain the initial score matrix, thereby ensuring the accuracy of the obtained initial score matrix. The initial score matrix is ​​scaled to obtain the target score matrix, thereby ensuring the accuracy of the obtained target score matrix. The target score matrix is ​​convolved to calculate the attention weight, which ensures the accuracy of the calculated attention weight. The sixth sub-feature is calculated based on the attention weight and the first value matrix, which ensures the accuracy of the generated sixth sub-feature. The sixth sub-feature is normalized by the application instance to obtain the first sub-feature, which ensures the accuracy of the obtained first sub-feature.

[0252] The global feature extraction network performs a linear transformation on the first feature vector, and maps the second query matrix, the second key value matrix, and the second value matrix, thereby ensuring the accuracy of the obtained second query matrix, the second key value matrix, and the second value matrix. For each attention mechanism head in the global feature extraction network, the attention weight corresponding to each attention mechanism head is calculated, thereby ensuring the accuracy of the calculated attention weight corresponding to each attention mechanism head. The attention weights corresponding to each attention mechanism head are concatenated to generate a second sub-feature, thereby ensuring that the generated second sub-feature contains the global features of the first feature vector.

[0253] The first sub-feature and the second sub-feature are fused to generate a fused feature, which ensures that the generated fused feature contains both local features and global features. The compression and excitation network identifies the fused feature and assigns weights to the features of different channels in the fused feature according to the identification results; the deep separable network compresses the fused feature according to the weights corresponding to the features of different channels in the fused feature to reduce the channel dimension of the fused feature and generate a second feature vector, which ensures that the generated second feature vector contains both local features and global features and has accurate dimensions.

[0254] The second feature vector is input into the second stage module in the feature extraction module, the second feature vector is downsampled, and the number of channels of the second feature vector is increased to generate a third feature vector. The third feature vector is subjected to a second local feature extraction and a second global feature extraction to generate a fourth feature vector, thereby ensuring the accuracy of the generated fourth feature vector. The fourth feature vector is input into the third stage module in the feature extraction module, the fourth feature vector is downsampled, and the number of channels of the fourth feature vector is increased to generate a fifth feature vector, thereby ensuring the accuracy of the generated fifth feature vector. The fifth feature vector is subjected to a third local feature extraction and a third global feature extraction to generate a sixth feature vector, thereby ensuring the accuracy of the generated sixth feature vector. The sixth feature vector is input into the fourth stage module in the feature extraction module, the sixth feature vector is downsampled, and the number of channels of the sixth feature vector is increased to generate a seventh feature vector; the seventh feature vector is subjected to a fourth local feature extraction and a fourth global feature extraction to generate a target feature vector, thereby ensuring the accuracy of the output target feature vector. Based on the target feature vector, the target crossing intention corresponding to the target pedestrian is output, thereby ensuring the accuracy of the output crossing intention corresponding to the target pedestrian.

[0255] In this embodiment, a pedestrian crossing intention recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0256] This embodiment provides a pedestrian crossing intention recognition device, such as Fig.11 As shown, including:

[0257] The acquisition module 401 is used to acquire pedestrian data and street crossing data corresponding to the target pedestrian, wherein the pedestrian data includes age data and gender data corresponding to the target pedestrian; the street crossing data includes pedestrian speed corresponding to the target pedestrian, vehicle speed of the target vehicle corresponding to the target pedestrian, a first distance from the target pedestrian to the zebra crossing, a second distance from the target vehicle to the zebra crossing, and a collision time of the target pedestrian relative to the target vehicle;

[0258] The data conversion module 402 is used to convert the pedestrian data and the street crossing data from a time series to a spatial image to generate a target image;

[0259] The input module 403 is used to input the target image into a preset pedestrian crossing intention recognition model, and output the target crossing intention corresponding to the target pedestrian; the crossing intention includes walk-walk, walk-stop, and stop-walk.

[0260] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0261] The pedestrian crossing intention recognition device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0262] The embodiment of the present invention also provides an electronic device having the above Fig.11 The pedestrian crossing intention recognition device shown.

[0263] See also Fig.12 , Fig.12 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Fig.12 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.12 A processor 10 is taken as an example.

[0264] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0265] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0266] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0267] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0268] The electronic device further comprises a communication interface 30 for the electronic device to communicate with other devices or a communication network.

[0269] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0270] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0271] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for identifying pedestrians' intention to cross the street, characterized in that: The method comprises: Obtain pedestrian data and street crossing data corresponding to the target pedestrian, the pedestrian data including gender data corresponding to the target pedestrian; the street crossing data including pedestrian speed corresponding to the target pedestrian, vehicle speed of the target vehicle corresponding to the target pedestrian, a first distance from the target pedestrian to the zebra crossing, a second distance from the target vehicle to the zebra crossing, and a collision time of the target pedestrian relative to the target vehicle; Performing data conversion from time series to spatial images on the pedestrian data and the street crossing data to generate a target image; The target image is input into a preset pedestrian crossing intention recognition model, and the target crossing intention corresponding to the target pedestrian is output; the crossing intention includes walk-walk, walk-stop, and stop-walk.

2. The method according to claim 1, characterized in that The step of converting the pedestrian data and the street crossing data from a time series to a spatial image to generate a target image includes: converting the gender data, the pedestrian speed, the vehicle speed, the first distance, the second distance, and the collision time into grayscale images respectively; generating a first color image based on a first grayscale image corresponding to the gender data, a second grayscale image corresponding to the pedestrian speed, and a third grayscale image corresponding to the vehicle speed; generating a second color image based on a fourth grayscale image corresponding to the first distance, a fifth grayscale image corresponding to the second distance, and a sixth grayscale image corresponding to the collision time; The first color image and the second color image are spliced ​​to generate the target image.

3. The method according to claim 2, characterized in that Converting the pedestrian speed into a grayscale image includes: Dividing the time series data corresponding to the pedestrian speed into a plurality of bins; the amount of data in each bin is the same; For each of the bins, calculating the frequency of the current bin transferring to the next bin; Constructing a transfer matrix based on the frequency of the current bin transferring to the next bin; Based on the transfer matrix, a transfer field is constructed; each element in the transfer field represents the frequency of transfer from each of the quantile bins to another quantile bin; According to the frequency value in the transfer field, the grayscale value of each pixel is mapped to the corresponding bin to generate a grayscale image corresponding to the pedestrian speed.

4. The method according to claim 1, characterized in that: The step of inputting the target image into a preset pedestrian crossing intention recognition model and outputting the target crossing intention corresponding to the target pedestrian includes: Dividing the target image into a plurality of sub-image blocks according to a first preset size; wherein the sub-image blocks do not overlap; Input each of the sub-image blocks into a feature extraction module in a preset pedestrian crossing intention recognition model, perform local feature extraction and global feature extraction on each of the sub-image blocks, and output a target feature vector; Based on the target feature vector, the target crossing intention corresponding to the target pedestrian is output.

5. The method according to claim 4, characterized in that The feature extraction module includes four stage modules, wherein each sub-image block is input into the feature extraction module in the preset pedestrian crossing intention recognition model, local feature extraction and global feature extraction are performed on each sub-image block, and a target feature vector is output; including: Inputting each of the sub-image blocks into the first stage module in the feature extraction module, mapping the dimensions of each of the sub-image blocks, and generating a first feature vector corresponding to the target image; Performing a first local feature extraction and a first global feature extraction on the first feature vector to generate a second feature vector; the second feature vector has the same size as the first feature vector; Inputting the second feature vector into the second stage module in the feature extraction module, downsampling the second feature vector, and increasing the number of channels of the second feature vector to generate a third feature vector; Performing a second local feature extraction and a second global feature extraction on the third feature vector to generate a fourth feature vector; the fourth feature vector has the same size as the third feature vector; Inputting the fourth feature vector into the third stage module in the feature extraction module, downsampling the fourth feature vector, and increasing the number of channels of the fourth feature vector to generate a fifth feature vector; Performing a third local feature extraction and a third global feature extraction on the fifth feature vector to generate a sixth feature vector; the sixth feature vector has the same size as the fifth feature vector; Inputting the sixth feature vector into the fourth stage module in the feature extraction module, downsampling the sixth feature vector, and increasing the number of channels of the sixth feature vector to generate a seventh feature vector; A fourth local feature extraction and a fourth global feature extraction are performed on the seventh feature vector to generate the target feature vector; the target feature vector has the same size as the seventh feature vector.

6. The method according to claim 5, characterized in that The first-stage module includes a local linear embedding module and a first-level mixed attention module, the second-stage module includes a first image block merging module and a second-level mixed attention module, the third-stage module includes a second image block merging module and a third-level mixed attention module, and the fourth-stage module includes a third image block merging module and a fourth-level mixed attention module.

7. The method according to claim 6, characterized in that The first-level hybrid attention module includes a local feature extraction network, a global feature extraction network, a compression and excitation network, and a deep separable network. The first feature vector is subjected to a first local feature extraction and a first global feature extraction to generate a second feature vector, which includes: The local feature extraction network performs local feature extraction on the first feature vector to generate a first sub-feature; The global feature extraction network performs global feature extraction on the first feature vector to generate a second sub-feature; Fusing the first sub-feature and the second sub-feature to generate a fused feature; The compression and excitation network identifies the fused features, and assigns weights to features of different channels in the fused features according to the identification results; The deep separable network compresses the fused feature according to the weights corresponding to the features of different channels in the fused feature to reduce the channel dimension of the fused feature and generate the second feature vector.

8. The method according to claim 7, characterized in that The local feature extraction network performs local feature extraction on the first feature vector to generate a first sub-feature, including: The first linear normalization layer in the local feature extraction network performs normalization processing on the first feature vector to generate a third sub-feature; The depth-by-depth convolution layer in the local feature extraction network performs depth-by-depth convolution on the third sub-feature to extract features of different channels of the third sub-feature to obtain a fourth sub-feature; The second linear normalization layer in the local feature extraction network performs normalization processing on the fourth sub-feature to generate a fifth sub-feature; The linear transformation layer in the local feature extraction network performs a linear transformation on the fifth sub-feature to obtain a first query matrix, and extracts a first key value matrix and a first value matrix from the fifth sub-feature; Multiplying the first query matrix by the first key value matrix to obtain an initial score matrix; Performing a scaling operation on the initial score matrix to obtain a target score matrix; Performing a convolution operation on the target score matrix to calculate the attention weight; Calculate a sixth sub-feature based on the attention weight and the first value matrix; The sixth sub-feature is normalized according to the application instance to obtain the first sub-feature.

9. The method according to claim 7, characterized in that: The global feature extraction network performs global feature extraction on the first feature vector to generate a second sub-feature, including: Performing a linear transformation on the first eigenvector to map and obtain a second query matrix, a second key value matrix, and a second value matrix; For each attention mechanism head in the global feature extraction network, calculate the attention weight corresponding to each attention mechanism head; The attention weights corresponding to the attention mechanism heads are concatenated to generate the second sub-feature.

10. A pedestrian crossing intention recognition device, characterized in that: The device comprises: an acquisition module, configured to acquire pedestrian data and street crossing data corresponding to a target pedestrian, wherein the pedestrian data includes age data and gender data corresponding to the target pedestrian; the street crossing data includes a pedestrian speed corresponding to the target pedestrian, a vehicle speed of a target vehicle corresponding to the target pedestrian, a first distance from the target pedestrian to a zebra crossing, a second distance from the target vehicle to the zebra crossing, and a collision time of the target pedestrian relative to the target vehicle; A data conversion module, used for performing data conversion from time series to spatial images on the pedestrian data and the street crossing data to generate a target image; An input module is used to input the target image into a preset pedestrian crossing intention recognition model, and output a target crossing intention corresponding to the target pedestrian; the crossing intention includes walk-walk, walk-stop, and stop-walk.

Images