Pedestrian intention prediction method and device and electronic equipment
By using a fusion neural network model to process and fusion visual and non-visual information in stages in autonomous driving systems, the challenge of pedestrian intent prediction is solved, the prediction accuracy is improved, and the system safety is enhanced.
Patent Information
- Application Number
- CN202310854868.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-12
- Publication Date
- 2025-01-14
AI Technical Summary
In autonomous driving systems, there are challenges in the prediction of pedestrians' intentions, especially due to the large differences in pedestrian dynamics and traffic rules, inaccurate predictions may lead to vehicle collisions, increasing hidden dangers.
The fusion neural network model is adopted to input visual information and non-visual information into different processing paths respectively, and data sharing and multiple information fusion are performed through the fusion module to improve the accuracy of pedestrian intention prediction. The specific steps include: performing staged feature extraction and vectorization of multiple information based on the convolutional neural network and the recurrent neural network, using the gated recurrent unit to splice and fusion, and finally generating the pedestrian intention prediction result.
Through multi-stage processing and information fusion, the accuracy of pedestrian intent prediction is improved, the safety of autonomous driving systems is enhanced, and the risk of conflict between vehicles and pedestrians is reduced.
Smart Images
Figure CN119314139A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of artificial intelligence and intelligent vehicles, and in particular to a method, device and electronic equipment for predicting pedestrian intentions. Background Art
[0002] In the vehicle's autonomous driving system, pedestrian intention prediction is essential. It plays a vital role in improving the safety of the system. Pedestrian crossing intention prediction is to estimate the future position of pedestrians and whether they will cross the road in the future, so as to help the planning and control of autonomous driving make decisions in order to avoid traffic accidents.
[0003] However, due to the high dynamics of pedestrians and the large differences in the constraints of traffic rules, it is quite challenging to predict pedestrians' crossing intentions. If the prediction is inaccurate, vehicles are likely to hit people, which will bring great hidden dangers to vehicles. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a method, device and electronic device for predicting pedestrian intentions. The method can effectively improve the accuracy of pedestrian intention prediction based on a variety of different types of scene information.
[0005] In order to achieve the above objectives, the present application provides a method for predicting pedestrian intentions, including:
[0006] Based on data attributes of the acquired scene information, dividing the scene information into at least one visual information and at least one non-visual information, the scene information being associated with pedestrians and / or vehicles in the application scene;
[0007] Inputting the visual information and the non-visual information into different processing pathways of the fusion neural network model respectively;
[0008] Predicting the pedestrian's intention based on the fused neural network model includes:
[0009] The visual information is processed in stages through a first processing path in the fused neural network model, and the non-visual information is processed in stages through a second processing path in the fused neural network model, wherein at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage, so that the fused neural network model predicts the intention of the pedestrian based on the fusion result.
[0010] Optionally, dividing the scene information into at least one visual information and at least one non-visual information based on the data attribute of the acquired scene information includes:
[0011] determining an image attribute of each type of the scene information, wherein the image attribute is used to characterize the image identity of the information;
[0012] Based on the image attributes, each type of the scene information is divided into information categories to determine the visual information and the non-visual information.
[0013] Optionally, the scene information includes at least one of the following: vehicle information, driving scene information, and pedestrian information;
[0014] The vehicle information includes: vehicle speed;
[0015] The driving scene information includes: global context information for semantically segmenting the external environment of the vehicle;
[0016] The pedestrian information includes at least one of the following: local context information, local box information, bounding box information and pedestrian posture key point information, wherein the local context information represents external information adjacent to the pedestrian, the local box information represents the shape of the pedestrian, the bounding box information represents the position change of the pedestrian during movement, and the pedestrian posture key point information represents the posture of the pedestrian and the changing trend of the posture.
[0017] Optionally, the step of processing the visual information in stages through the first processing path in the fused neural network model includes:
[0018] Based on a convolutional neural network and a recurrent neural network, feature extraction is performed on the plurality of visual information in stages, and the extracted features are sequentially spliced, wherein the recurrent neural network includes a gated recurrent unit;
[0019] Accordingly, the non-visual information is processed in stages through the second processing path in the fusion neural network model, including:
[0020] Based on the recurrent neural network, feature extraction is performed on the plurality of non-visual information in stages, and the extracted features are sequentially spliced.
[0021] Optionally, the visual information includes at least one of the following: pedestrian posture key point information, vehicle speed and bounding box information;
[0022] The method of extracting features of the plurality of visual information in stages based on the convolutional neural network and the recurrent neural network, and sequentially splicing the extracted features, includes:
[0023] Extract features of the pedestrian's posture key point information, the vehicle's speed and the bounding box information respectively;
[0024] All the extracted features are vectorized;
[0025] Based on the pedestrian posture key point information, the vehicle speed and the bounding box information, the respective information details and / or weights, the vectorized features are spliced in sequence.
[0026] Optionally, the non-visual information includes at least one of the following: global context information, local context information, and local box information;
[0027] The step of extracting features of the plurality of non-visual information in stages based on the recurrent neural network and sequentially splicing the extracted features comprises:
[0028] Performing feature extraction on the global context information, the local context information, and the local frame information respectively;
[0029] All the extracted features are vectorized;
[0030] Based on the respective information detail levels and / or weights of the global context information, the local context information, and the local frame information, the vectorized features are concatenated in sequence.
[0031] Optionally, the predicting the intention of the pedestrian based on the fused neural network model further includes:
[0032] In the multiple stages of feature extraction for the visual information and the non-visual information, the visual information and the corresponding non-visual information are fused multiple times through the multiple fusion modules to share data between the first processing path and the second processing path.
[0033] Optionally, the predicting the intention of the pedestrian based on the fused neural network model further includes:
[0034] splicing a first processing result of the first processing path with a second processing result of the second processing path to generate a first prediction result;
[0035] The first prediction result is subjected to a key point extraction operation to determine the pedestrian's intention, wherein the pedestrian's intention includes predicting whether the pedestrian will perform a crossing action.
[0036] The embodiment of the present application also provides a pedestrian intention prediction device, including:
[0037] a division module configured to divide the scene information into at least one visual information and at least one non-visual information based on data attributes of the acquired scene information, wherein the scene information is associated with pedestrians and / or vehicles in the application scene;
[0038] A preprocessing module configured to input the visual information and the non-visual information into different processing paths of the fusion neural network model respectively;
[0039] A prediction module is configured to predict the intention of the pedestrian based on the fused neural network model, including:
[0040] The visual information is processed in stages through a first processing path in the fused neural network model, and the non-visual information is processed in stages through a second processing path in the fused neural network model, wherein at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage, so that the fused neural network model predicts the intention of the pedestrian based on the fusion result.
[0041] An embodiment of the present application further provides an electronic device, including a processor and a memory, wherein the memory stores an executable program, and the memory executes the executable program to perform the steps of the method described above.
[0042] The method for predicting pedestrian intentions in the embodiment of the present application can use a fusion neural network model to classify and process scene information, and can also fuse the processing results of multiple different processing paths in stages, effectively improving the accuracy of the fusion neural network model in predicting pedestrian intentions. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flowchart of a method for predicting pedestrian intention according to an embodiment of the present application;
[0044] Figure 2 A flowchart of training and using a fusion neural network model according to an embodiment of the present application;
[0045] Figure 3 For the embodiments of this application Figure 1 A flowchart of an embodiment of step S100;
[0046] Figure 4 For the embodiments of this application Figure 1 A flowchart of an embodiment of step S300;
[0047] Figure 5 For the embodiments of this application Figure 1 A flowchart of another embodiment of step S300;
[0048] Figure 6 For the embodiments of this application Figure 1 A flowchart of another embodiment of step S300;
[0049] Figure 7 A schematic diagram of the structure of a fusion neural network model according to an embodiment of the present application;
[0050] Figure 8 A structural block diagram of a device for predicting pedestrian intention according to an embodiment of the present application;
[0051] Fig. 9 A structural block diagram of an electronic device according to an embodiment of the present application;
[0052] Fig.10 A schematic diagram of the structure of another electronic device according to an embodiment of the present application.
[0053] Description of Reference Numerals
[0054] 100-a device for predicting pedestrian intention; 101-a division module; 102-a preprocessing module; 103-a prediction module; 200-a first electronic device; 201-a processor; 202-a memory; 300-a second electronic device; 301-an input unit; 302-an output unit; 303-a storage unit; 304-a communication unit; 305-a computing unit; 306-a ROM; 307-a RAM; 308-an I / O bus. DETAILED DESCRIPTION
[0055] Various aspects and features of the present application are described herein with reference to the accompanying drawings.
[0056] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.
[0057] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0058] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.
[0059] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.
[0060] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.
[0061] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.
[0062] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.
[0063] A method for predicting pedestrian intentions in an embodiment of the present application can be applied to a vehicle, which can be a "car", "vehicle" or "entire vehicle" or other similar terms including a general motor vehicle. The prediction method can be applied to an automatic driving system of a vehicle to predict the behavior of pedestrians associated with the driving process, thereby improving the safety of the automatic driving system.
[0064] Combination Figure 1 and Figure 2 The method includes constructing a fusion neural network model, which can be used to predict the intention of pedestrians in a driving scene based on input scene information. The constructed fusion neural network model is trained, and the specific training process can use a training sample set for initial training, and the training sample set includes vehicle information, driving scene information, and pedestrian information. A cross-pathway fusion neural network model can be trained. When used, the trained fusion neural network model is used to calculate the prediction value of the new input. Specifically, the above-mentioned different types of scene information are classified to form visual information and non-visual information, and the fusion neural network model is used to process the visual information and non-visual information respectively. The processing results of different stages can be fused during the processing process, and then the intention of the pedestrian is accurately determined based on the fusion result, including predicting whether the pedestrian crosses the road.
[0065] The method for predicting pedestrian intention is described in detail below with reference to the accompanying drawings. Figure 1 Flow chart of the method for predicting pedestrian intention according to an embodiment of the present application, as shown in FIG. Figure 1 Shown and combined Figure 7 , the method comprises the following steps:
[0066] Step S100, based on data attributes of the acquired scene information, the scene information is divided into at least one visual information and at least one non-visual information, wherein the scene information is associated with pedestrians and / or vehicles in the application scene.
[0067] Exemplarily, scene information is information related to the scene of vehicle driving, specifically information associated with pedestrians and / or vehicles in the application scene. It can reflect the state, action, etc. of things related to the vehicle driving scene. For example, scene information includes at least one of the following: vehicle information, driving scene information, and pedestrian information. The above different types of information can represent relevant information of the driving scene from multiple aspects.
[0068] The scene information has data attributes, and the data attributes represent the information type, such as image type, text type, character type, sound type, etc. Each type of information has a corresponding data attribute. In this embodiment, the scene information can be divided into at least one visual information and at least one non-visual information based on the data attributes of the scene information. Visual information has visual attributes, such as graphic information, photo information, etc. Non-visual information does not have visual attributes, such as text information, character information, speed information, information describing the border of an object, etc.
[0069] In this embodiment, a large amount of scene information can be divided based on data attributes to form multiple visual information and multiple non-visual information, so as to process the visual information and the non-visual information respectively.
[0070] Step S200, inputting the visual information and the non-visual information into different processing pathways of the fusion neural network model respectively.
[0071] Exemplarily, the fused neural network model is a model constructed based on at least one type of neural network, and the fused neural network model is used to predict the intention of pedestrians in a driving scene based on input scene information.
[0072] The fused neural network model can be trained before use, and then used to predict the intentions of pedestrians after the training is completed.
[0073] In one embodiment, the fusion neural network model includes multiple processing paths, and different processing paths can process the corresponding information. For example, the first processing path can process the first type of information in the scene information, the second processing path can process the second type of information in the scene information, and the third processing path can process the third type of information in the scene information. Of course, the number of processing paths and the processing objects can be adjusted according to actual needs.
[0074] In this embodiment, the fusion neural network model has a first processing path and a second processing path. The structure of the first processing path makes it more accurate and efficient to process visual information, and visual information can be input into the first processing path to use the first processing path to process visual information; the structure of the second processing path makes it more accurate and efficient to process non-visual information, and non-visual information can be input into the second processing path to use the second processing path to process non-visual information. Since each processing path is adapted to the input information, each processing path can effectively improve the processing efficiency.
[0075] Step S300, predicting the pedestrian's intention based on the fused neural network model, includes:
[0076] The visual information is processed in stages through a first processing path in the fused neural network model, and the non-visual information is processed in stages through a second processing path in the fused neural network model, wherein at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage, so that the fused neural network model predicts the intention of the pedestrian based on the fusion result.
[0077] Exemplarily, in the process of using the fusion neural network model to process the input information, on the one hand, the first processing path processes the visual information in stages. Specifically, each stage can process one type of visual information, including performing feature extraction, feature vectorization and other operations on each type of visual information. In addition, the processing results of different stages can be spliced so that all visual information can be processed at multiple different stages and the overall features reflecting the important content of the visual information can be obtained. For example, feature extraction is performed on the global context information first, and then feature extraction is performed on the local context information, and then the features of the two aspects are spliced.
[0078] On the other hand, the second processing pathway processes non-visual information in stages in a similar manner to the above-mentioned processing method, and can process multiple different types of non-visual information in stages, including feature extraction, feature vectorization and other operations for each type of non-visual information. The processing results of different stages can also be spliced together, so that all non-visual information can be processed in multiple different stages and the overall characteristics reflecting the important content of the visual information can be obtained.
[0079] In this embodiment, at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage. Of course, the features obtained in the first processing path and the features obtained in the second processing path can be fused multiple times at multiple stages. For example, the global context is feature extracted in the first processing path, and the key point information of the pedestrian posture is feature extracted in the second processing path, and the two different types of features are fused through the fusion module. Thereby, the first processing path can take into account the content of the non-visual information in the process of processing the visual information, so that the feature extraction and judgment of the visual information are more accurate. Similarly, the second processing path can take into account the content of the visual information in the process of processing the non-visual information, so that the feature extraction and judgment of the non-visual information are more accurate. The fusion neural network model determines the final judgment result based on multiple fusion results, thereby realizing the prediction of the pedestrian's intention.
[0080] The method for predicting pedestrian intentions in the embodiment of the present application can use a fusion neural network model to classify and process scene information, and can also fuse the processing results of multiple different processing paths in stages, effectively improving the accuracy of the fusion neural network model in predicting pedestrian intentions.
[0081] In one embodiment of the present application, based on the data attributes of the acquired scene information, the scene information is divided into at least one visual information and at least one non-visual information, such as Figure 3 As shown, including:
[0082] Step S110 : determining image attributes of each type of the scene information, wherein the image attributes are used to characterize the image identity of the information.
[0083] Exemplarily, image attributes are used to characterize the image identity of information. Scene information with image attributes is associated with an image. For example, the scene information itself may be image information, or may contain other information of the image information. For example, scene information with image attributes may be global context information, local context information, and local frame information. These types of scene information all have image attributes.
[0084] Step S120: Based on the image attributes, each type of the scene information is divided into information types to determine the visual information and the non-visual information.
[0085] Exemplarily, based on the image attribute, it can be determined whether the identity of the scene information is associated with the image, and then whether the scene information is visual information. Based on the image attribute, the scene information can be divided into visual information and non-visual information.
[0086] Visual information has image attributes, and it is an image itself or is associated with an image. For example, visual information includes at least one of the following: global context information, local context information, and local box information. Non-visual information does not have image attributes, and it is not an image itself and is not associated with an image. For example, non-visual information includes at least one of the following: vehicle speed, bounding box information, and pedestrian posture key point information.
[0087] For visual information, the first processing path can be used to extract features based on its graphic attributes, thereby improving the efficiency of feature extraction. For non-visual information, the second processing path can be used to extract features based on its lack of graphic attributes, thereby also improving the efficiency of feature extraction.
[0088] In one embodiment of the present application, the scene information includes at least one of the following: vehicle information, driving scene information, and pedestrian information;
[0089] The vehicle information includes: vehicle speed;
[0090] The driving scene information includes: global context information for semantically segmenting the external environment of the vehicle;
[0091] The pedestrian information includes at least one of the following: local context information, local box information, bounding box information and pedestrian posture key point information, wherein the local context information represents external information adjacent to the pedestrian, the local box information represents the shape of the pedestrian, the bounding box information represents the position change of the pedestrian during movement, and the pedestrian posture key point information represents the posture of the pedestrian and the changing trend of the posture.
[0092] Specifically, the scene information includes at least one of the following: vehicle information, driving scene information, and pedestrian information. On the one hand, the vehicle information is the information of the vehicle itself in the usage scene, such as vehicle speed information, vehicle driving status information, fuel level information, power level information, driver information, etc.
[0093] On the other hand, the driving scene information includes: global context information for semantic segmentation of the vehicle's external environment. For example, the global context information includes: information such as the road surface, trees, lane lines, and signal lights outside the vehicle.
[0094] On the other hand, the pedestrian information includes at least one of the following: local context information, local box information, bounding box information and pedestrian posture key point information.
[0095] The local context information represents the external information of the neighboring pedestrians, which can be the local information after the image is enlarged and the pedestrians are removed. The local context information does not include the pedestrians themselves, but refers to the local information of the neighboring pedestrians at a certain distance, such as the step information of the neighboring pedestrians, the road surface information of the pedestrians walking, etc.
[0096] The local box information is the information of the local box of the pedestrian, which includes the box information of the entire pedestrian's appearance. The box information represents the shape and appearance of the pedestrian.
[0097] The bounding box information represents the position change of the pedestrian during movement, such as multiple vertex coordinates of the pedestrian's local box, such as the four vertex coordinates of the pedestrian's local box, which represents the coordinate position and change position of the pedestrian, thereby being able to represent the pedestrian's speed, movement route and other information.
[0098] The pedestrian posture key point information represents the pedestrian's posture and the trend of the posture change, and is an abstract representation of the pedestrian's posture points. For example, the pedestrian posture key point information includes multiple key points set on the pedestrian, such as six key points set on multiple active joints of the pedestrian, and the key points can represent the pedestrian's posture, such as walking posture, standing posture, running posture, etc.
[0099] In addition, it should be noted that the above-mentioned vehicle speed, global context information, local context information, local box information, bounding box information and pedestrian posture key point information can be divided into visual information and non-visual information. In this embodiment, the global context information, local context information and local box information are divided into visual information; the vehicle speed, bounding box information and pedestrian posture key point information are divided into non-visual information. After the division, the above-mentioned scene information is used.
[0100] In one embodiment of the present application, Figure 7 , the visual information is processed in stages through the first processing path in the fusion neural network model, comprising the following steps:
[0101] Based on a convolutional neural network and a recurrent neural network, feature extraction is performed on the plurality of visual information in stages, and the extracted features are sequentially spliced, wherein the recurrent neural network includes a gated recurrent unit;
[0102] Accordingly, the step of processing the non-visual information in stages through the second processing path in the fused neural network model comprises the following steps:
[0103] Based on the recurrent neural network, feature extraction is performed on the plurality of non-visual information in stages, and the extracted features are sequentially spliced.
[0104] Exemplarily, a convolutional neural network is a type of feedforward neural network that includes convolution calculations and has a deep structure, and is one of the deep learning algorithms. Convolutional neural networks have representational learning capabilities and can classify input information in a translation-invariant manner according to their hierarchical structure. A recurrent neural network is a type of recurrent neural network that takes sequence data as input, recurses in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain. A gated recurrent unit (GRU) is a gating mechanism in a recurrent neural network, similar to other gating mechanisms, that aims to solve the gradient vanishing / exploding problem in a standard recurrent neural network while retaining the long-term information of the sequence.
[0105] The first processing path is constructed based on a convolutional neural network and a recurrent neural network, so that the first processing path can effectively and accurately process visual information. The second processing path is constructed based on a recurrent neural network, so that the second processing path can effectively and accurately process non-visual information.
[0106] In this embodiment, feature extraction of multiple visual information is performed in stages based on convolutional neural networks and recurrent neural networks, and the extracted features are spliced in sequence. Specifically, a gated recurrent unit (GRU) is a gating mechanism in a recurrent neural network, and convolutional neural networks and gated recurrent units can be used to extract features of visual information in stages. For example, a convolutional neural network and a gated recurrent unit are used to extract features of global context information to generate a first feature, and then a convolutional neural network is used to extract features of local context information to generate a second feature, and the obtained first feature and second feature are spliced to form a first intermediate data. When a convolutional neural network is used to extract features of local frame information to generate a third feature, the first intermediate data and the third feature are spliced to form a second intermediate data. In one embodiment, after the second intermediate data is processed by the gated recurrent unit, the second intermediate data can also be fused with the data in the second processing path.
[0107] Similarly, based on a recurrent neural network, feature extraction is performed on the plurality of non-visual information in stages, and the extracted features are sequentially spliced. Specifically, a gated recurrent unit can be used to perform feature extraction on the non-visual information in stages. For example, a gated recurrent unit is used to extract features from the key point information of the pedestrian posture to generate a third feature, and then a gated recurrent unit is used to extract features from the bounding box information to generate a fourth feature, and the obtained third feature and fourth feature are spliced to form third intermediate data. The third intermediate data is then spliced with the feature of the vehicle's driving speed to generate fourth intermediate data. The fourth intermediate data is then processed using a gated recurrent unit. In this process, the third intermediate data and / or the fourth intermediate data source can be fused with the data in the first processing path.
[0108] Specifically, preferably, the convolutional neural network and the recurrent neural network are used to extract features of the plurality of visual information in stages, and the extracted features are sequentially spliced, such as Figure 4 As shown, the following steps are included:
[0109] Step S310, performing feature extraction on the pedestrian posture key point information, the vehicle speed and the bounding box information respectively;
[0110] Step S320, performing vectorization operation on the extracted features;
[0111] Step S330, based on the pedestrian posture key point information, the vehicle speed and the respective information detail levels and / or weights of the bounding box information, the vectorized features are sequentially spliced.
[0112] Exemplary, combined Figure 7 In the second processing path, after the pedestrian posture key point information, vehicle speed and bounding box information are respectively extracted in stages through the gated loop unit, the extracted features are gradually spliced. As described in the above embodiment, it will not be described in detail here. In this embodiment, the extracted features can be vectorized after feature extraction. Thereby facilitating the splicing of the vectorized features. In combination with the above embodiment, when splicing the third feature and the fourth feature, the third feature and the fourth feature can be vectorized respectively first, and then the splicing operation is performed. Thereby facilitating improving the efficiency of the splicing operation, and also ensuring that data will not be lost during the splicing process.
[0113] In one embodiment, in the process of processing data using the second processing path, the vectorized features are spliced in sequence based on the information detail and / or weight of the pedestrian posture key point information, the vehicle speed and the bounding box information. Specifically, the information detail and / or weight of the pedestrian posture key point information is higher than the information detail and / or weight of the vehicle speed, and the information detail and / or weight of the vehicle speed is higher than the information detail and / or weight of the bounding box information. This allows important information to be used preferentially in the staged processing process of the second processing path, and can participate in the data processing as much as possible, thereby ensuring the integrity of the data as much as possible.
[0114] Specifically, preferably, the recurrent neural network is used to extract features of the plurality of non-visual information in stages, and the extracted features are sequentially spliced, such as Figure 5 As shown, the following steps are included:
[0115] Step S340, performing feature extraction on the global context information, the local context information and the local frame information respectively;
[0116] Step S350, performing vectorization operation on the extracted features;
[0117] Step S360 , based on the respective information detail levels and / or weights of the global context information, the local context information, and the local frame information, the vectorized features are sequentially concatenated.
[0118] Exemplary, combined Figure 7 In the first processing path, after the global context information, local context information and local frame information are respectively subjected to staged feature extraction by a recurrent neural network and a gated recurrent unit, the extracted features are gradually spliced. As described in the above embodiment, no further detailed description is given here. In this embodiment, the extracted features can be vectorized after feature extraction. This facilitates the splicing of the vectorized features. In combination with the above embodiment, when splicing the first feature and the second feature, the first feature and the second feature can be first vectorized respectively, and then the splicing operation is performed. This facilitates improving the efficiency of the splicing operation, while also ensuring that data will not be lost during the splicing process.
[0119] Similar to the above-mentioned embodiment, in the process of processing data using the first processing path, the vectorized features are spliced in sequence based on the information detail level and / or weight of the global context information, the local context information, and the local frame information. Specifically, the information detail level and / or weight of the global context information is higher than the information detail level and / or weight of the local context information, and the information detail level and / or weight of the local context information is higher than the information detail level and / or weight of the local frame information. This allows important information to be used preferentially in the process of being processed in stages in the first processing path, and to participate in the processing of data as much as possible, thereby ensuring the integrity of the data as much as possible.
[0120] In one embodiment of the present application, the predicting the intention of the pedestrian based on the fused neural network model further includes:
[0121] In the multiple stages of feature extraction for the visual information and the non-visual information, the visual information and the corresponding non-visual information are fused multiple times through the multiple fusion modules to share data between the first processing path and the second processing path.
[0122] Exemplarily, fusion operations can be performed through a fusion module at multiple stages of feature extraction. For example, pedestrian posture key point information is fused with global context information, so that relevant features of the pedestrian posture key point information can also be used in the first processing path. At the same time, relevant features of the global context information can also be used in the second processing path.
[0123] In addition, in the fusion neural network model, a plurality of fusion modules may be arranged between the first processing path and the second processing path, and each fusion module is arranged at a plurality of processing nodes.
[0124] For example, a fusion module is provided between the relevant processing nodes of the pedestrian posture key point information and the relevant processing nodes of the global context information, for fusing the corresponding features. A fusion module is provided between the relevant processing nodes of the bounding box information and the relevant processing nodes of the local context information, for fusing the corresponding features. A fusion module is also provided between the relevant processing nodes of the vehicle speed and the relevant processing nodes of the local box information, for fusing the corresponding features. Through multiple staged data fusions, different types of information can be effectively utilized by the first processing path and the second processing path at the same time.
[0125] In one embodiment of the present application, the pedestrian's intention is predicted based on the fused neural network model, such as Figure 6 As shown, it also includes:
[0126] Step S370: concatenate the first processing result of the first processing path and the second processing result of the second processing path to generate a first prediction result.
[0127] Exemplarily, the first processing result is a processing result obtained at the last stage of the first processing path, for example, the first processing result is obtained after feature extraction and splicing operations are performed on the global context information, the local context information, and the local frame information.
[0128] The second processing result is the processing result obtained at the last stage of the second processing path, for example, the second processing result is obtained after feature extraction and splicing of pedestrian posture key point information, bounding box information and vehicle speed. The first processing result and the second processing result can be spliced.
[0129] Step S380: performing a key point extraction operation on the first prediction result to determine the pedestrian's intention, wherein the pedestrian's intention includes predicting whether the pedestrian will perform a crossing action.
[0130] Exemplarily, the key point extraction operation is performed on the first prediction result, which is to extract the important information and key information in the first prediction result. The key point extraction operation can be called an "attention" operation. It ensures that the important information in the first prediction result is used emphatically, and the pedestrian's intention can be predicted more accurately based on the important key point information. In addition, the prediction result can be "fully connected", that is, the prediction result is represented by mapping information, including predicting whether the pedestrian has performed the road crossing action.
[0131] Based on the same concept, the present application also provides a pedestrian intention prediction device 100, such as Figure 8 As shown, the prediction device comprises:
[0132] The division module 101 is configured to divide the scene information into at least one visual information and at least one non-visual information based on data attributes of the acquired scene information, wherein the scene information is associated with pedestrians and / or vehicles in the application scene.
[0133] Exemplarily, scene information is information related to the scene of vehicle driving, specifically information associated with pedestrians and / or vehicles in the application scene. It can reflect the state, action, etc. of things related to the vehicle driving scene. For example, scene information includes at least one of the following: vehicle information, driving scene information, and pedestrian information. The above different types of information can represent relevant information of the driving scene from multiple aspects.
[0134] The scene information has data attributes, and the data attributes represent the information type, such as image type, text type, character type, sound type, etc. Each type of information has a corresponding data attribute. In this embodiment, the division module 101 can divide the scene information into at least one visual information and at least one non-visual information based on the data attributes of the scene information. The visual information has visual attributes, such as graphic information, photo information, etc. The non-visual information does not have visual attributes, such as text information, character information, speed information, information describing the border of an object, etc.
[0135] In this embodiment, the division module 101 can divide a large amount of scene information into multiple visual information and multiple non-visual information based on data attributes, so as to process the visual information and the non-visual information respectively.
[0136] The pre-processing module 102 is configured to input the visual information and the non-visual information into different processing paths of the fusion neural network model respectively.
[0137] Exemplarily, the fused neural network model is a model constructed based on at least one type of neural network, and the fused neural network model is used to predict the intention of pedestrians in a driving scene based on input scene information.
[0138] The fused neural network model can be trained before use, and then used to predict the intentions of pedestrians after the training is completed.
[0139] In one embodiment, the fusion neural network model includes multiple processing paths, and different processing paths can process the corresponding information. For example, the first processing path can process the first type of information in the scene information, the second processing path can process the second type of information in the scene information, and the third processing path can process the third type of information in the scene information. Of course, the number of processing paths and the processing objects can be adjusted according to actual needs.
[0140] In this embodiment, the fusion neural network model has a first processing path and a second processing path. The preprocessing module 102 inputs visual information into the first processing path and inputs non-visual information into the second processing path. The structure of the first processing path makes it more accurate and efficient to process visual information. Visual information can be input into the first processing path so that the first processing path can be used to process visual information; the structure of the second processing path makes it more accurate and efficient to process non-visual information. Non-visual information can be input into the second processing path so that the second processing path can be used to process non-visual information. Since each processing path is adapted to the input information, each processing path can effectively improve the processing efficiency.
[0141] The prediction module 103 is configured to predict the intention of the pedestrian based on the fused neural network model, including:
[0142] The visual information is processed in stages through a first processing path in the fused neural network model, and the non-visual information is processed in stages through a second processing path in the fused neural network model, wherein at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage, so that the fused neural network model predicts the intention of the pedestrian based on the fusion result.
[0143] Exemplarily, the prediction module 103 uses a fused neural network model to predict the intention of the pedestrian. In the process of using the fused neural network model to process the input information, on the one hand, the first processing path processes the visual information in stages. Specifically, each stage can process one type of visual information, including performing feature extraction, feature vectorization and other operations on each type of visual information. In addition, the processing results of different stages can be spliced so that all visual information can be processed at multiple different stages and the overall features reflecting the important content of the visual information can be obtained. For example, feature extraction is performed on the global context information first, and then feature extraction is performed on the local context information, and then the features of the two aspects are spliced.
[0144] On the other hand, the second processing pathway processes non-visual information in stages in a similar manner to the above-mentioned processing method, and can process multiple different types of non-visual information in stages, including feature extraction, feature vectorization and other operations for each type of non-visual information. The processing results of different stages can also be spliced together, so that all non-visual information can be processed in multiple different stages and the overall characteristics reflecting the important content of the visual information can be obtained.
[0145] In this embodiment, at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage. Of course, the features obtained in the first processing path and the features obtained in the second processing path can be fused multiple times at multiple stages. For example, the global context is feature extracted in the first processing path, and the key point information of the pedestrian posture is feature extracted in the second processing path, and the two different types of features are fused through the fusion module. Thereby, the first processing path can take into account the content of the non-visual information in the process of processing the visual information, so that the feature extraction and judgment of the visual information are more accurate. Similarly, the second processing path can take into account the content of the visual information in the process of processing the non-visual information, so that the feature extraction and judgment of the non-visual information are more accurate. The fusion neural network model determines the final judgment result based on multiple fusion results, thereby realizing the prediction of the pedestrian's intention.
[0146] In one embodiment of the present application, the division module 101 is further configured as follows:
[0147] determining an image attribute of each type of the scene information, wherein the image attribute is used to characterize the image identity of the information;
[0148] Based on the image attributes, each type of the scene information is divided into information categories to determine the visual information and the non-visual information.
[0149] In one embodiment of the present application, the scene information includes at least one of the following: vehicle information, driving scene information, and pedestrian information;
[0150] The vehicle information includes: vehicle speed;
[0151] The driving scene information includes: global context information for semantically segmenting the external environment of the vehicle;
[0152] The pedestrian information includes at least one of the following: local context information, local box information, bounding box information and pedestrian posture key point information, wherein the local context information represents external information adjacent to the pedestrian, the local box information represents the shape of the pedestrian, the bounding box information represents the position change of the pedestrian during movement, and the pedestrian posture key point information represents the posture of the pedestrian and the changing trend of the posture.
[0153] In one embodiment of the present application, the prediction module 103 is further configured to use a fusion neural network model to:
[0154] Based on a convolutional neural network and a recurrent neural network, feature extraction is performed on the plurality of visual information in stages, and the extracted features are sequentially spliced, wherein the recurrent neural network includes a gated recurrent unit;
[0155] Accordingly, the non-visual information is processed in stages through the second processing path in the fusion neural network model, including:
[0156] Based on the recurrent neural network, feature extraction is performed on the plurality of non-visual information in stages, and the extracted features are sequentially spliced.
[0157] In one embodiment of the present application, the visual information includes at least one of the following: pedestrian posture key point information, vehicle speed and bounding box information;
[0158] The prediction module 103 is further configured to utilize the fusion neural network model to:
[0159] Extract features of the pedestrian's posture key point information, the vehicle's speed and the bounding box information respectively;
[0160] All the extracted features are vectorized;
[0161] Based on the pedestrian posture key point information, the vehicle speed and the bounding box information, the respective information details and / or weights, the vectorized features are spliced in sequence.
[0162] In one embodiment of the present application, the non-visual information includes at least one of the following: global context information, local context information, and local box information;
[0163] The prediction module 103 is further configured to utilize the fusion neural network model to:
[0164] Performing feature extraction on the global context information, the local context information, and the local frame information respectively;
[0165] All the extracted features are vectorized;
[0166] Based on the respective information detail levels and / or weights of the global context information, the local context information, and the local frame information, the vectorized features are concatenated in sequence.
[0167] In one embodiment of the present application, the prediction module 103 is further configured to use a fusion neural network model to:
[0168] In the multiple stages of feature extraction for the visual information and the non-visual information, the visual information and the corresponding non-visual information are fused multiple times through the multiple fusion modules to share data between the first processing path and the second processing path.
[0169] In one embodiment of the present application, the prediction module 103 is further configured to use a fusion neural network model to:
[0170] splicing a first processing result of the first processing path with a second processing result of the second processing path to generate a first prediction result;
[0171] The first prediction result is subjected to a key point extraction operation to determine the pedestrian's intention, wherein the pedestrian's intention includes predicting whether the pedestrian will perform a crossing action.
[0172] The embodiment of the present application further provides an electronic device, which is identified as a first electronic device 200 for easy distinction. Fig. 9 As shown, it includes a processor 201 and a memory 202, wherein the memory 202 stores an executable program, and the processor 201 executes the executable program to perform the steps of the method as described above.
[0173] The embodiment of the present application also provides another electronic device, which is identified as a second electronic device 300 for easy identification. The electronic device can implement the steps of the above method based on the cooperation of its internal components to predict the intention of the pedestrian. The electronic device includes an input unit 301, an output unit 302, a storage unit 303 and a communication unit 304. Fig.10As shown. They communicate with the computing unit 305, ROM (read-only memory) 306 and RAM (random access memory) 307 through the I / O bus 308. The input unit 301 may include devices such as vehicle-mounted cameras, vehicle-mounted radars, vehicle-mounted navigation, and global positioning system (GPS) sensors. The storage unit implements persistent storage of data, including solid-state drives (SSDs), disks, etc. The computing unit 305 may be a central processing unit (CPU) and a graphics processing unit (GPU). The communication unit 304 implements communication functions between in-vehicle devices and with the outside, including wireless transmission equipment and wired transmission equipment.
[0174] The embodiment of the present application also provides a vehicle, wherein the vehicle is equipped with the pedestrian intention prediction device 100 as described above, and the pedestrian intention prediction device 100 is used to identify the pedestrian intention, so that the vehicle has an artificial intelligence function capable of identifying the pedestrian intention during driving.
[0175] An embodiment of the present application further provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by a processor, the steps of the method described above are implemented.
[0176] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, electronic equipment, computer-readable storage media or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. When implemented by software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0177] The processor may be a general purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general purpose processor may be a microprocessor or any conventional processor.
[0178] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0179] The above-mentioned readable storage medium can be a disk, an optical disk, a DVD, a USB, a read-only memory (ROM) or a random access memory (RAM), etc. This application does not limit the specific form of the storage medium.
[0180] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.
Claims
1. A method for predicting pedestrian intention, characterized in that: include: Based on data attributes of the acquired scene information, dividing the scene information into at least one visual information and at least one non-visual information, the scene information being associated with pedestrians and / or vehicles in the application scene; Inputting the visual information and non-visual information into different processing pathways of the fusion neural network model respectively; Predicting the pedestrian's intention based on the fused neural network model includes: The visual information is processed in stages through a first processing path in the fused neural network model, and the non-visual information is processed in stages through a second processing path in the fused neural network model, wherein at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage, so that the fused neural network model predicts the intention of the pedestrian based on the fusion result.
2. The method according to claim 1, characterized in that: The step of dividing the scene information into at least one visual information and at least one non-visual information based on the data attribute of the acquired scene information comprises: determining an image attribute of each type of the scene information, wherein the image attribute is used to characterize the image identity of the information; Based on the image attributes, each type of the scene information is divided into information categories to determine the visual information and the non-visual information.
3. The method according to claim 1, characterized in that The scene information includes at least one of the following: vehicle information, driving scene information, and pedestrian information; The vehicle information includes: vehicle speed; The driving scene information includes: global context information for semantically segmenting the external environment of the vehicle; The pedestrian information includes at least one of the following: local context information, local box information, bounding box information and pedestrian posture key point information, wherein the local context information represents external information adjacent to the pedestrian, the local box information represents the shape of the pedestrian, the bounding box information represents the position change of the pedestrian during movement, and the pedestrian posture key point information represents the posture of the pedestrian and the changing trend of the posture.
4. The method according to claim 1, characterized in that: The step of processing the visual information in stages through the first processing path in the fusion neural network model includes: Based on a convolutional neural network and a recurrent neural network, feature extraction is performed on the plurality of visual information in stages, and the extracted features are sequentially spliced, wherein the recurrent neural network includes a gated recurrent unit; Accordingly, the non-visual information is processed in stages through the second processing path in the fusion neural network model, including: Based on the recurrent neural network, feature extraction is performed on the plurality of non-visual information in stages, and the extracted features are sequentially spliced.
5. The method according to claim 4, characterized in that The visual information includes at least one of the following: pedestrian posture key point information, vehicle speed and bounding box information; The method of extracting features of the plurality of visual information in stages based on the convolutional neural network and the recurrent neural network, and sequentially splicing the extracted features, includes: Extracting features of the pedestrian's posture key point information, the vehicle's speed and the bounding box information respectively; All the extracted features are vectorized; Based on the pedestrian posture key point information, the vehicle travel speed and the respective information detail and / or weight of the bounding box information, the vectorized features are spliced in sequence.
6. The method according to claim 4, characterized in that The non-visual information includes at least one of the following: global context information, local context information, and local box information; The step of extracting features of the plurality of non-visual information in stages based on the recurrent neural network and sequentially splicing the extracted features comprises: Performing feature extraction on the global context information, the local context information, and the local frame information respectively; All the extracted features are vectorized; Based on the respective information detail levels and / or weights of the global context information, the local context information, and the local frame information, the vectorized features are concatenated in sequence.
7. The method according to claim 1, characterized in that The predicting of the pedestrian's intention based on the fused neural network model also includes: In the multiple stages of feature extraction for the visual information and the non-visual information, the visual information and the corresponding non-visual information are fused multiple times through the multiple fusion modules to share data between the first processing path and the second processing path.
8. The method according to claim 1, characterized in that: The predicting of the pedestrian's intention based on the fused neural network model also includes: splicing a first processing result of the first processing path with a second processing result of the second processing path to generate a first prediction result; The first prediction result is subjected to a key point extraction operation to determine the pedestrian's intention, wherein the pedestrian's intention includes predicting whether the pedestrian will perform a crossing action.
9. A pedestrian intention prediction device, characterized in that: include: a division module configured to divide the scene information into at least one visual information and at least one non-visual information based on data attributes of the acquired scene information, wherein the scene information is associated with pedestrians and / or vehicles in the application scene; A preprocessing module configured to input the visual information and the non-visual information into different processing paths of the fusion neural network model respectively; A prediction module is configured to predict the intention of the pedestrian based on the fused neural network model, including: The visual information is processed in stages through a first processing path in the fused neural network model, and the non-visual information is processed in stages through a second processing path in the fused neural network model, wherein at least one fusion module is provided between the first processing path and the second processing path, and the fusion module is used to fuse the visual information and the non-visual information in at least one processing stage, so that the fused neural network model predicts the intention of the pedestrian based on the fusion result.
10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein an executable program is stored in the memory, and the memory executes the executable program to perform the steps of the method according to any one of claims 1 to 8.