Pedestrian intention prediction method and device, vehicle and storage medium

Abstract

The application relates to a pedestrian intention prediction method and device, a vehicle and a storage medium. The method comprises the following steps: acquiring a single frame image from sensing content collected by a vehicle sensor; inputting the single frame image into a prediction model, performing feature recognition on the single frame image by the prediction model, performing posture prediction and intention prediction according to the results of feature recognition respectively, combining the results of posture prediction and intention prediction to generate a pedestrian intention prediction result; and acquiring the pedestrian intention prediction result output by the prediction model. The scheme provided in the application can more quickly and accurately predict the behavior intention of pedestrians, and improve the safety of automatic driving.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to methods, devices, vehicles, and storage media for predicting pedestrian intentions. Background Technology

[0002] Pedestrian intent prediction is an important research area in the field of autonomous driving. It helps predict in advance whether pedestrians will cross the road, allowing vehicles to take measures such as slowing down and braking as early as possible to avoid traffic accidents. Intent prediction methods in related technologies are typically studied using trajectory prediction models or binary classification methods.

[0003] However, both trajectory prediction models and binary classification methods employ a top-down structure, requiring model detectors, such as pedestrian detectors, to detect pedestrians and their locations within the scene. This pedestrian detection information is then passed to the latter part of the model to complete the pedestrian intent recognition task; in other words, the intent recognition task depends on the results of pedestrian detection. If the model detector fails or has problems, the model will be unable to perform the intent prediction task or will fail to produce satisfactory prediction results. Furthermore, the model's runtime increases with the number of detected pedestrians. When there are many pedestrians in the scene, the increased number of pedestrians requires the model detector to detect each one, prolonging the detection time and thus extending the overall time for intent prediction. In autonomous driving scenarios, cars travel long distances in a short time due to high speeds, and model runtime is a crucial metric for evaluating autonomous driving models; models with long runtimes are unsuitable for autonomous driving scenarios.

[0004] Therefore, the aforementioned models and processing methods of related technologies cannot well meet the requirements of pedestrian behavior intention prediction tasks in autonomous driving scenarios, cannot effectively complete the prediction tasks, and affect the safety of autonomous driving. Summary of the Invention

[0005] To address or partially address the problems existing in related technologies, this application provides a pedestrian intention prediction method, device, vehicle, and storage medium, which can predict pedestrian behavioral intentions more quickly and accurately, thereby improving the safety of autonomous driving.

[0006] The first aspect of this application provides a method for predicting pedestrian intentions, including:

[0007] Obtain a single frame image from the perceived content collected by the vehicle's sensors;

[0008] The single-frame image is input into the prediction model, which performs feature recognition on the single-frame image. Based on the feature recognition results, pose prediction and intention prediction are performed respectively. The pedestrian intention prediction result is generated by combining the pose prediction result and the intention prediction result.

[0009] Obtain the pedestrian intent prediction results output by the prediction model.

[0010] In one embodiment, the step of performing feature recognition on the single-frame image by the prediction model, performing pose prediction and intent prediction based on the feature recognition results, and combining the pose prediction results and intent prediction results to generate a pedestrian intent prediction result includes:

[0011] The prediction model performs feature recognition on the single-frame image to generate a feature map;

[0012] The feature map is input into the pose head network for pose prediction, and the feature map is input into the intention head network for intention prediction. The pedestrian intention prediction result is generated by combining the pose prediction result and the intention prediction result.

[0013] In one embodiment, the step of inputting the feature map into the attitude head network for attitude prediction includes: inputting the feature map into the attitude head network, and having the attitude head network refine the feature map according to the feature map;

[0014] The step of inputting the feature map into the intention head network for intention prediction includes: inputting the feature map into the intention head network, and having the intention head network perform aggregation processing based on the refinement results of the feature map and the pose head network.

[0015] In one embodiment, the step of inputting the feature map into the attitude head network, and having the attitude head network refine the feature map, includes:

[0016] The feature map is input into the attitude head network, and the attitude head network performs initial processing on the feature map using a first preset function to obtain an initial activity map.

[0017] The feature map and the initial activity map are refined using a second preset function to obtain a refined activity map.

[0018] In one embodiment, the step of inputting the feature map into the intention head network, and having the intention head network perform aggregation processing based on the refinement results of the feature map and the pose head network, includes:

[0019] The feature map is input into the intention head network, which then aggregates the refined activity map based on the feature map and the attitude head network to obtain the group activity prediction results of pedestrians.

[0020] In one embodiment, the prediction model is pre-trained and optimized based on a loss function.

[0021] A second aspect of this application provides a pedestrian intention prediction device, comprising:

[0022] The image acquisition module is used to acquire a single-frame image from the perceived content collected by the vehicle's sensors;

[0023] The prediction processing module is used to input the single-frame image acquired by the image acquisition module into the prediction model, and the prediction model performs feature recognition on the single-frame image. Based on the feature recognition results, pose prediction and intention prediction are performed respectively. The pedestrian intention prediction result is generated by combining the pose prediction result and the intention prediction result.

[0024] The result acquisition module acquires the pedestrian intent prediction results output by the prediction model of the prediction processing module.

[0025] In one embodiment, the prediction processing module includes:

[0026] The information input submodule is used to input the single-frame image into the prediction model;

[0027] The prediction model submodule is used to perform feature recognition on the single frame image through the prediction model to generate a feature map; input the feature map into the pose head network for pose prediction; input the feature map into the intention head network for intention prediction; and combine the pose prediction result and the intention prediction result to generate a pedestrian intention prediction result.

[0028] In one embodiment, the prediction model submodule includes:

[0029] The feature map submodule is used to perform feature recognition on the single-frame image and generate a feature map.

[0030] The pose prediction submodule is used to input the feature map into the pose head network, and the pose head network refines the feature map according to the feature map.

[0031] The intent prediction submodule is used to input the feature map into the intent head network, and the intent head network performs aggregation processing based on the refinement results of the feature map and the pose head network.

[0032] In one embodiment, the apparatus further includes:

[0033] The model training module is used to pre-train a prediction model, which is then optimized based on a loss function.

[0034] A third aspect of this application provides a vehicle, comprising:

[0035] Processor; and

[0036] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0037] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0038] The technical solution provided in this application may include the following beneficial effects:

[0039] This application's solution involves inputting a single-frame image into a prediction model, which then performs feature recognition on the single-frame image. Based on the feature recognition results, it performs pose prediction and intent prediction respectively. Combining the pose prediction results and intent prediction results, it generates a pedestrian intent prediction result. The pedestrian intent prediction result output by the prediction model is then obtained. Since multi-frame images or videos introduce time bias, and if the time series contains multiple video frames, the biased model may have a greater impact, this application's solution eliminates this bias by inputting a single-frame image into the prediction model for processing, rather than multiple frames. Furthermore, the prediction model in this application performs pose prediction and intent prediction separately based on the feature recognition results of the single-frame image, ultimately generating a pedestrian intent prediction result. This eliminates the need for pedestrian detectors to predict whether pedestrians are crossing the road. Moreover, without pedestrian detectors, the computation time of this application's prediction model does not increase with the number of detected pedestrians, allowing it to run within a relatively stable timeframe, avoiding increased computation time and workload, and thus improving prediction efficiency and accuracy. Therefore, this application's solution can predict pedestrian behavior and intent more quickly and accurately, improving the safety of autonomous driving.

[0040] Furthermore, the solution proposed in this application can simultaneously achieve functions such as pose detection, intent prediction, and pedestrian detection through a prediction model without a pedestrian detector, and the detection accuracy is also high, while the efficiency is also improved.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0042] The above and other objects, features and advantages of this application will become more apparent from the following description of exemplary embodiments of this application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of this application.

[0043] Figure 1 This is a flowchart illustrating the pedestrian intention prediction method according to an embodiment of this application;

[0044] Figure 2 This is another schematic flowchart illustrating the pedestrian intention prediction method in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram illustrating the framework of a pedestrian intention prediction model as shown in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the pedestrian intention prediction device shown in the embodiments of this application;

[0047] Figure 5 This is another structural schematic diagram of the pedestrian intention prediction device shown in the embodiments of this application;

[0048] Figure 6 This is a schematic diagram of the vehicle structure shown in the embodiments of this application. Detailed Implementation

[0049] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0050] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0051] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0052] Existing pedestrian intent prediction models in related technologies cannot adequately meet the requirements of pedestrian intent prediction tasks in autonomous driving scenarios, thus hindering autonomous driving safety. To address these issues, this application provides a pedestrian intent prediction method that can predict pedestrian intent more quickly and accurately, thereby improving autonomous driving safety.

[0053] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0054] Figure 1 This is a flowchart illustrating the pedestrian intent prediction method in an embodiment of this application.

[0055] See Figure 1 The method includes:

[0056] S101, acquire a single frame image from the perceived content collected by the vehicle sensors.

[0057] A single frame image can be obtained from the perception data collected by the vehicle's cameras, LiDAR, and other sensors. In autonomous driving, vehicles are typically equipped with multiple sensors to acquire information about their surroundings, including images, LiDAR, and millimeter-wave radar. These sensors can collect both static and dynamic information about the vehicle's surroundings, such as static information like roads and buildings, and dynamic information like pedestrians and other vehicles.

[0058] S102, input a single frame image into the prediction model, the prediction model performs feature recognition on the single frame image, and performs pose prediction and intent prediction based on the feature recognition results. Combine the pose prediction results and intent prediction results to generate pedestrian intent prediction results.

[0059] In S102, the prediction model can perform feature recognition on a single frame image to generate a feature map; the feature map is input into the pose head network for pose prediction, and the feature map is input into the intention head network for intention prediction. The pedestrian intention prediction result is generated by combining the pose prediction result and the intention prediction result.

[0060] In this process, feature recognition of a single frame image by a prediction model can be achieved by extracting features from the acquired single frame image to obtain a feature map.

[0061] Predictive models can use Convolutional Neural Networks (CNNs) to extract features from single-frame images. CNNs are a type of feedforward neural network with a deep structure that includes convolutional computations, and are one of the representative algorithms of deep learning. CNNs possess representation learning capabilities, enabling them to perform shift-invariant classification of input information according to their hierarchical structure; therefore, they are also known as Shift-Invariant Artificial Neural Networks (SIANNs). A CNN consists of convolutional layers and pooling layers.

[0062] In this embodiment, the prediction model can use a 2D convolutional neural network or a 3D convolutional neural network to process the input single-frame image into a feature map. The single-frame image can be a photograph of the surrounding environment captured by the vehicle, a video frame from a video of the surrounding environment captured by the vehicle, or data collected from other scenes. A single-frame image can contain at least one row of people, and a group can contain multiple pedestrians. In the feature map, each pixel or spatial location corresponds to a set of feature vectors, which reflect information such as color, texture, shape, and motion in the original single-frame image.

[0063] The process of inputting the feature map into the attitude head network for attitude prediction can include: inputting the feature map into the attitude head network, where the attitude head network refines the feature map. For example, the feature map is input into the attitude head network, where the attitude head network performs initial processing on the feature map using a first preset function to obtain an initial activity map; then, a second preset function is used to refine the feature map and the initial activity map to obtain a refined activity map.

[0064] Based on the activities of individuals and groups, an intermediate spatial representation called an activity graph is learned and generated. The activities of individuals and groups can be obtained from acquired single-frame images. The activity graph contains spatial relationship information between individuals within the group activity. In other words, the activity graph is an intermediate spatial representation that helps to accurately capture the differences between different activities. The activity graph can also be viewed as another higher-level representation of the feature map, used for group activity recognition tasks.

[0065] In this embodiment, the feature map can be refined through a preset number of stages, T, with each stage generating a corresponding refined activity map. Using a multi-stage refinement component to refine the feature map in multiple stages can reduce prediction errors in the refined activity map. It should be noted that the number of stages, T, is typically chosen based on the specific application scenario and algorithm requirements. Generally, the value of T affects the accuracy and computational efficiency of the algorithm. If T is too small, it may fail to capture sufficiently high-level semantic information in the feature map, leading to decreased model accuracy; if T is too large, it may increase computational complexity and runtime, and may cause model overfitting. Therefore, when choosing the value of T, a balance between accuracy and efficiency needs to be considered, and adjustments should be made based on actual data. This embodiment does not limit the number of stages, T. By refining the feature map in multiple stages, prediction errors can be reduced, the accuracy of activity recognition can be improved, and the modeling of complex spatial relationships between individuals and groups can be enhanced.

[0066] The process of inputting feature maps into the intention head network for intent prediction can include: inputting feature maps into the intention head network, and then aggregating the data based on the refinement results of the feature maps and the pose head network. For example, inputting feature maps into the intention head network, and then aggregating the data based on the refined activity maps from the feature maps and the pose head network, yields the predicted group activity of pedestrians.

[0067] Intention Head Networks (INNs) are similar to Convolutional Relationship Machines (CRRMs), and are end-to-end convolutional neural networks that utilize spatial relationship information between individuals in images or videos to identify group activities. In end-to-end convolutional neural networks, single-frame images are used to extract spatial relationships between human activities, which can help identify different activities.

[0068] The activity map includes individual activity maps and group activity maps. Group activity maps are obtained through group activity identification, which involves classifying the group behavior of a group of individuals to predict the specific types of behavior they will perform. Unlike classifying the behavior of a single individual, group activity identification needs to consider the collaboration and mutual influence between individuals, thus requiring greater attention to spatial relationships and temporal information.

[0069] The pedestrian intent prediction result can be generated by combining the results of posture prediction and intent prediction. In the prediction model, the activity categories of each pedestrian in the entire scene are obtained by classifying the activities of each pedestrian. Group activities are determined based on the relationships between the activities of all pedestrians in the scene. Specifically, the relationships between the activities of all pedestrians in the scene can be calculated based on the feature map and the refined activity map corresponding to the last refinement stage, thereby generating predicted labels for pedestrian intent. This calculation process can be accomplished using different methods, such as building a graph model or analyzing the interaction relationships between pedestrians based on social network analysis methods. Different methods can be used to describe the group activity part for different task requirements. For example, the category of group activity can be determined by calculating the similarity and correlation between different categories in the activity map. Alternatively, the interaction relationships between pedestrians can be analyzed based on social network analysis methods to determine the type of group activity. In summary, in the prediction model, the classification and prediction of group activities can be calculated based on the relationships between the activities of all pedestrians in the scene.

[0070] S103, Obtain the pedestrian intent prediction results output by the prediction model.

[0071] After the prediction model generates the pedestrian intent prediction result, obtain the pedestrian intent prediction result output by the prediction model.

[0072] As can be seen from this embodiment, the present application's solution inputs a single-frame image into the prediction model, which then performs feature recognition on the single-frame image. Based on the feature recognition results, it performs pose prediction and intent prediction respectively. Combining the pose prediction results and intent prediction results, it generates a pedestrian intent prediction result; and obtains the pedestrian intent prediction result output by the prediction model. Since multiple frames of images or videos introduce time bias, if the time series contains multiple video frames, the bias model may have a greater impact. Therefore, the present application's solution eliminates this bias by selecting a single-frame image as the input for the prediction model, rather than inputting multiple frames. Furthermore, the prediction model in this application performs pose prediction and intent prediction based on the feature recognition results of the single-frame image, ultimately generating a pedestrian intent prediction result. This eliminates the need for pedestrian detectors to predict whether pedestrians are crossing the road. Moreover, without pedestrian detectors, the computation time of the prediction model in this application does not increase with the number of pedestrians detected, allowing it to run within a relatively stable timeframe, avoiding increased computation time and workload, and thus improving prediction efficiency and accuracy. Therefore, the present application's solution can predict pedestrian behavior and intent more quickly and accurately, improving the safety of autonomous driving.

[0073] Figure 2 This is another schematic diagram of the pedestrian intention prediction method shown in the embodiments of this application.

[0074] In this embodiment, this application provides a novel prediction model for predicting pedestrian intentions. This model can be a neural network model that combines environmental information to generate predictions of pedestrian intentions, such as predictions of pedestrian crossing behavior. Environmental information can include objects around the pedestrian, street signs, traffic lights, etc. This information helps the prediction model better understand the scene and environment in which the pedestrian is located, thereby more accurately predicting the pedestrian's behavioral intentions. For example, if there are stopped cars or roadblocks around the pedestrian, the pedestrian may choose to go around these obstacles to cross the road rather than walk directly through them. Therefore, environmental information can provide crucial context and background, helping the prediction model to more comprehensively understand pedestrian behavior. Simultaneously, environmental information can also provide decision support for autonomous driving systems, helping the system react accordingly based on the pedestrian's behavioral intentions, such as slowing down or braking, thereby ensuring driving safety.

[0075] The prediction model provided in this application can take a single frame image, such as a single frame RGB image, as input, and perform pose prediction and intent prediction based on the feature recognition results of the single frame image. It can generate two maps, namely a pose map and an intent map, which play different roles in predicting pedestrian intent, such as pedestrian intent to cross the street.

[0076] A pose map is a continuous map that describes the pose information of pedestrians in an image. For example, it labels each pixel in the image with a different pose of the pedestrian, including standing, walking, running, etc. Pose maps help predictive models better understand the state and behavior of pedestrians, thus more accurately predicting their behavioral intentions. Simultaneously, pose maps also provide decision support for autonomous driving systems, helping the system more accurately determine the direction and speed of pedestrian movement, thereby better planning vehicle routes.

[0077] The intent map is a binary map that describes the possible behavioral intentions of a pedestrian. For example, it labels each pixel in the image as "want to cross the road" or "don't want to cross the road," indicating that the pedestrian at that location is likely to cross the road or not. Therefore, the intent map is mainly used to predict whether a pedestrian will choose to cross the road. It can help autonomous driving systems react accordingly based on the pedestrian's behavioral intentions, such as slowing down or braking, thereby ensuring driving safety.

[0078] In this application, the prediction model can generate an intent map by predicting the probability that each pixel in the image belongs to a pedestrian, and obtain the pedestrian intent prediction result. In this way, the function of predicting whether a pedestrian is crossing the road can be realized without using a pedestrian detector. Moreover, without a pedestrian detector, the computation time of the prediction model does not increase with the increase of the number of pedestrians detected, so it can run in a relatively stable time, avoiding the increase of computation time and computation load, thereby improving prediction efficiency and prediction accuracy.

[0079] See Figure 2 The method includes:

[0080] S201, acquire a single frame image from the perceived content collected by the vehicle's sensors.

[0081] Single-frame images can be obtained from the perception data collected by sensors such as cameras and LiDAR in a vehicle. In autonomous driving, vehicles are typically equipped with multiple sensors to acquire information about their surroundings, including image sensors, LiDAR, and millimeter-wave radar. These sensors can collect both static and dynamic information about the vehicle's environment, such as roads and buildings, and dynamic information such as pedestrians and other vehicles. Among these sensors, image sensors are the most common because they provide visual information about the scene and can be processed using computer vision algorithms to perform many tasks, such as object detection and trajectory prediction. Therefore, in autonomous driving, single-frame images can be obtained from the perception data collected by sensors such as cameras and LiDAR in a vehicle.

[0082] For autonomous driving, the acquisition of a single frame image is scene-dependent. For example, on urban roads, autonomous vehicles typically capture single images at traffic lights, intersections, and pedestrian crossings. These images contain a wealth of environmental information, such as pedestrians, vehicles, traffic lights, and street signs, which can be used to achieve various functions of autonomous driving, such as traffic flow monitoring, obstacle detection, and pedestrian recognition and tracking. Furthermore, autonomous vehicles can acquire images from multiple angles using multiple cameras and fuse these images to improve the robustness and accuracy of scene perception.

[0083] The autonomous driving scenarios covered in this application can include, but are not limited to, urban roads, highways, and rural roads. In these scenarios, pedestrians may suddenly appear in front of or to the side of the vehicle, thus requiring pedestrian detection and behavioral intent prediction. For example: 1) On urban roads, pedestrians may walk from the sidewalk into the lane or step out from the doorway of a building on the side of the vehicle, necessitating pedestrian detection and behavioral intent prediction to avoid collisions. 2) On highways, pedestrians are rare, but if they do appear, such as when a pedestrian is stranded in the lane due to a car accident, pedestrian detection and behavioral intent prediction are crucial. 3) On rural roads, pedestrians may be working in fields beside the road or walking or cycling on rural roads, again requiring pedestrian detection and behavioral intent prediction. In short, pedestrians may appear in autonomous driving scenarios, whether on urban roads, highways, or rural roads, making pedestrian detection and behavioral intent prediction essential for ensuring driving safety.

[0084] S202, input a single frame image into the prediction model, and the prediction model performs feature recognition on the single frame image to generate a feature map.

[0085] This application can extract features from acquired single-frame images to obtain feature maps; wherein a single-frame image can contain at least one row of people. The single-frame image can be a photograph of the surrounding environment captured by a vehicle, a video frame from a video of the surrounding environment captured by a vehicle, or data collected from other scenes. A single-frame image can contain at least one row of people, and a group can contain multiple pedestrians.

[0086] A schematic diagram of the pedestrian intention prediction model in this application can be found in [reference needed]. Figure 3 See also Figure 3 In this embodiment, a 2D or 3D CNN (Convolutional Neural Network) can be used to process the input single-frame image into a feature map F. This feature map F can be used for subsequent activity map prediction and group activity recognition steps. In the feature map F, each pixel or spatial location corresponds to a set of feature vectors, which reflect information such as color, texture, shape, and motion in the original single-frame image. At this point, the feature map does not yet describe higher-level semantic information related to the task.

[0087] like Figure 3As shown, H and W represent the height and width of the input image or video frame, and K and C are the number of frames and channels, respectively. H', W', and D represent the height, width, and depth (i.e., the number of channels) of the feature map obtained after processing by the convolutional neural network. When K=1, the input is compatible with both 3D CNN on a 4D input frame sequence and regular CNN on a 3D RGB single frame after compressing the first dimension. The feature map is extracted by the CNN and spatially adjusted to H′×W′. This application uses a 3D feature map as an example but is not limited to it. When using a 3D CNN, the average value of the feature maps from different time dimensions is calculated to obtain the feature map F∈R representing the entire sequence. H′×W′×D In other words, when using 3D CNNs to process frame sequence data, after feature map extraction, the feature maps at each time dimension can be averaged to obtain a feature map representing the entire sequence. The purpose of this is to integrate features from different time periods within the sequence, thereby better representing the information of the entire sequence. This step is independent of the number of filters in the selected layer, as it occurs after feature map extraction, and the number of filters is a parameter pre-set in the design of the convolutional layer.

[0088] The prediction model in this application can be pre-trained and optimized based on a loss function. For example, for a ground-based real-world activity map A of individual and group activities, the loss function of the prediction model can be defined as:

[0089]

[0090] in, It is the total multi-task loss of the model. It was a loss from the group activity. It is the total loss of the activity graph at each stage, w A and w G These are the weights for controlling the total activity graph loss and the group activity loss, respectively.

[0091] in, The calculation formula is as follows:

[0092]

[0093] in, The calculation formula is as follows:

[0094]

[0095] in, The calculation formula is as follows:

[0096]

[0097] in, It is the activity graph loss of phase 1. The activity graph loss is for stage t, where 1 ≤ t ≤ T. It represents the overall loss across all stages of the activity map. It is a loss from group activities. This represents the model's total multi-task loss. The group activity label p is a one-hot encoded group activity label, which can be obtained by preprocessing the ground truth map. It is a predicted label.

[0098] To simplify the training of multi-task loss models, this application employs a two-step training strategy. The first step involves... G Set to 0. Therefore, the model focuses on learning to generate realistic activity graphs. After the first step, w... G and w A All weights should be assigned non-zero values ​​to optimize all weights together. Since the primary goal is to predict group activity, w should be... A Set to less than w G In order to place greater emphasis on accurate prediction of group activity categories.

[0099] During the training process, this application can optimize and adjust the pedestrian intention prediction model based on the results of the loss function, and retrain the model using several training samples until the training loss meets the preset training conditions, thereby obtaining a trained pedestrian intention prediction model. This application can set a loss threshold; when the training loss is less than the loss threshold, it can be determined that the current training loss meets the preset training conditions, and the pedestrian intention prediction model training is complete.

[0100] S203, input the feature map into the pose head network for pose prediction.

[0101] In this step, the feature map can be input into the attitude head network, which then performs refinement processing based on the feature map. Specifically, the feature map can be input into the attitude head network, which uses a first preset function to perform initial processing on the feature map to obtain an initial activity map; then, a second preset function is used to refine both the feature map and the initial activity map to obtain a refined activity map.

[0102] Activity maps are further learned from feature maps and are designed to capture the spatial relationships between human behaviors. They can be obtained by processing feature maps using convolutional neural networks, where each pixel or spatial location corresponds to an activity vector, reflecting information about human behavior at that location and its spatial relationship with surrounding human behaviors. Therefore, activity maps can be seen as a higher-level representation of feature maps, used for group activity recognition tasks.

[0103] See Figure 3In the model framework, a preliminary feature map is passed to the activity map prediction step to generate an initial activity map. Then, the activity map is refined in multiple stages to reduce prediction errors and enhance the modeling of complex spatial relationships between individuals and groups. Finally, an aggregation component combines information from different stages and the refined activity map to predict the final group activity label.

[0104] Therefore, the initial feature map is the basis for the generation of the activity map, which is a higher-level representation obtained by further learning and processing on the initial feature map. It can be used to more accurately capture the spatial relationships between human behaviors and help achieve the task of group activity recognition.

[0105] This application defines an activity graph as a set of two-dimensional fields representing individual and group activities, including person bounding boxes. A person bounding box is a rectangular border used to locate a person in an image or video; it is typically obtained using computer vision techniques such as object detection or pedestrian detection algorithms. Each individual and group activity in the activity graph has a mapping. Each map has a higher value within the bounding box of a person labeled with a specific individual behavior or group activity. Activity graphs enable predictive models to extract information from spatial relationships between people. By using person bounding boxes to describe individual and group activities, models can extract information from spatial relationships between people, further improving the accuracy and robustness of classification results.

[0106] This application processes the feature map F to obtain an individual activity map. and group activity diagram Individual activity diagram and group activity diagram Combine the data to generate a ground-based real-world activity map A∈R. H′×W′×N , where N = N I +N G .

[0107] This application can utilize a two-dimensional Gaussian probability density function and construct activity graphs based on human bounding boxes and their individual and group activities.

[0108] For example, during model training, for M people in the input training samples, the bounding boxes B∈R of the real people on the ground are obtained. M×4 Their individual action categories I∈{1,…,N I} M and group activity categories G∈{1,…,N G To create an activity graph, we need to define each person m and their bounding box. Its individual action i and group activity g, will be a person-specific activity diagram Am Generates equations (1) to (3):

[0109]

[0110]

[0111]

[0112] Where f m () represents the graph A for each point z in field i and the specific activity of the pedestrian. m N l+g The calculation is performed. μ m It is the center of the bounding box of person m. ∑ m It is a covariance matrix calculated using the height and width of the bounding box. The bounding box follows vertical and horizontal lines, making the covariance matrix diagonal. In the individual activity and group activity fields of the individual activity map, a bivariate Gaussian map is created at the location of the person's bounding box. Each field is normalized separately by dividing the values ​​by the maximum value in the field. Finally, all person-specific activity maps A are aligned and combined by taking the maximum value of the alignment points. m This is used to form the final activity graph A. Therefore, for a single input, people will have 2D Gaussian maps in different individual activity fields within the activity graph, depending on each person's individual activity label. However, in the group activity graph, each person will have a 2D Gaussian map in only one field, and in all other fields, it will be 0. It should be noted that when a people segmentation mask is available, a 2D Gaussian map is not necessary; a people segmentation mask will produce better results.

[0113] It should be noted that both ground truth person bounding boxes and person bounding boxes are rectangular boxes used to represent the position and size of a person in an image or video frame. For example, ground truth person bounding boxes typically refer to the labeled information of the position and size of a person given in the training dataset. This labeled information can be obtained manually or through automatic labeling algorithms and is used as a reference standard for model training. Person bounding boxes typically refer to the rectangular boxes representing the position and size of a person in an image, obtained during testing or inference using techniques such as object detection algorithms or pedestrian detection algorithms.

[0114] In this application, activity maps can be constructed using ground-based real-person bounding boxes, individual action categories, and group activity categories. Simultaneously, person-specific activity maps can be generated using personnel bounding boxes and individual / group activity categories. Both ground-based real-person bounding boxes and personnel bounding boxes are used to describe the position and size of a person within an image or video frame; however, the difference lies in that the former is based on annotation information, while the latter is based on the results obtained from a detection algorithm.

[0115] This application can generate an initial prediction in the first stage of the refinement process, which can be viewed as an initial activity graph. In subsequent stages of the refinement process, the prediction model further optimizes and refines this initial prediction to obtain more accurate representations of individual and group activities. These activity representations are then passed to the aggregation component for the final generation of group activity labels. The operation of the refinement process is similar to a convolutional pose machine, where the prediction model generates an initial prediction in the first stage and then refines it in subsequent stages. After refinement, the aggregation component is responsible for aggregating information from the entire scene and the refined activity graph to generate group activity labels.

[0116] This application uses the feature map F as input to the prediction model and performs T stages of refinement operations on the feature map F, which can generate a refinement activity map corresponding to each stage t of the refinement operation. Where N = N I +N G N I and N G These represent the number of individual and group activity categories, respectively, and T is the number of stages. After T stages of refinement, the final detailed activity diagram will be generated.

[0117] In the prediction model framework, the feature map F is passed to the activity map prediction step, where the feature map F is processed in the first stage to generate the initial activity map. Initial Activity Diagram This includes individual activity diagrams and group activity diagrams. Then, the function ψ can be used. t The activity diagram is refined at multiple stages.

[0118] For example, given the input feature map F, the first function, such as function... This is applied to the feature map ψ, thereby generating the initial activity map in the first stage. Then in the second stage, for example, a second function, such as another function ψ1, acts on F and The combination of these elements generates a detailed activity diagram. Starting from the second phase of operation, a refined activity map can be generated based on the initial activity map and feature map generated in the previous phase.

[0119] In other words, the prediction model can continue to analyze the initial activity graph generated in the first phase of the operation. Further refinement is then performed. Starting from the second stage, a refined activity map is generated based on the initial activity map and feature map produced in the previous stage. Refining the activity map across multiple stages can reduce prediction errors and enhance the modeling of complex spatial relationships between individuals and groups.

[0120] The initial activity graph generated in the first phase of operations can be represented by the following formula:

[0121]

[0122] The detailed activity diagram generated by the t-th stage operation can be represented by the following formula:

[0123]

[0124] Where T is the number of stages, ⊕ represents cascading, and φ and ψ t It consists of convolutional layers.

[0125] conv(x) represents a convolutional layer with a stride of 1 and a kernel size of x. φ consists of three conv(3) layers and two conv(1) layers. All ψ t They all have the same layers but are different from the layers in φ. Three conv(7) and two conv(1) are used to construct ψ. t .

[0126] S204. Input the feature map into the intention head network for intention prediction. Combine the results of pose prediction and intention prediction to generate pedestrian intention prediction results.

[0127] In this process, the feature map is input into the intention head network, which then aggregates the refined activity map based on the feature map and the pose head network to obtain the prediction results of pedestrian group activities.

[0128] The intention head network of this application is similar to a convolutional relational machine, which is an end-to-end convolutional neural network that uses spatial relationship information between individuals in images or videos to identify group activities.

[0129] In this application, a source of group activity refers to a group behavior or activity among multiple individuals, such as human collective movement, vehicular traffic, and pedestrian congestion. These sources of group activity can be identified by analyzing the spatial relationships between individuals in images or videos. The purpose of identifying different types of sources of group activity is to better understand and predict group activities. By classifying different sources of group activity, we can better understand the similarities and differences between different group activities and provide more accurate information for decision-making in application scenarios.

[0130] In this embodiment, given a feature map F as input to the prediction model, an activity map is generated in each stage of the refinement operation. After T stages of refinement operations, the final refined activity map is produced, where T is the number of stages. In this step, the corrected activity map... When the feature map F is input into the aggregation component of the intention head network, predicted labels representing the final group activity can be generated. In other words, the refined activity map and feature map can be input into the preset aggregation component of the intention head network, and the aggregation component generates predicted labels.

[0131] This application determines the corrected activity diagram. This is used as a representation of the spatial relationship between human activities and the activity graph F. The aggregation processing step in this application is mainly to gather information from all participants to generate the final group activity label, that is, by jointly processing the input feature graph F and the final activity graph. To predict group activities.

[0132] Predicted Labels This refers to the final group activity category predicted by the model for a given video or image sequence. In the prediction model, multiple stages of refinement components and aggregation components can be used to improve the ability to identify group behavior. Furthermore, model parameters can be optimized using Euclidean loss between the model's predicted outputs and the cross-entropy loss of group activities in the training data to obtain more accurate group activity label prediction results.

[0133] Among them, the predicted label It can be calculated using the following formula:

[0134]

[0135] in ζ is the probability vector for the predicted group activity category. ζ is defined in the feature map F and the activity map. Convolutional and pooling layers are used in a cascade to generate the final group activity prediction. If a pooling layer with kernel size x and stride 2 is denoted as pool(x), then ζ consists of a sequence of conv(7), pool(2), conv(7), pool(2), conv(7), pool(2), conv(1) and a global average pooling layer. In this way, the prediction model will be able to extract cues from the global context and predict group activities. For the function ζ, the common inference feature map F and the activity map are... Very important. If the activity diagram... If ζ is the only input, it will be unable to consider the matching between the visual representation and the corresponding activity graph, and thus will be unable to identify group activities.

[0136] Finally, the aggregation component ζ combines information from the feature map F with the refined activity map. Combined, the above formula is used to predict the final group activity label, i.e., the predicted label.

[0137] In other words, in the prediction model, the refinement activity graph corresponding to the last refinement stage is used as the corrected activity graph, and the corrected activity graph is used as the... The feature map F is used as input to the aggregation module to calculate the relationships between the activities of all pedestrians in the scene. This process can be accomplished using different methods, such as building a graphical model or analyzing pedestrian interactions based on social network analysis methods. Different methods can be used to describe the group activity component for different task requirements. For example, the category of group activity can be determined by calculating the similarity and correlation between different categories in the activity map. Alternatively, social network analysis methods can be used to analyze pedestrian interactions to determine the type of group activity. In summary, in the predictive model, the aggregation module can be used to classify and predict group activities based on the relationships between the activities of all pedestrians in the scene. Different aggregation modules can use different methods to describe the group activity component to adapt to different application scenarios and task requirements.

[0138] After identifying different types of group activity sources, the aggregation component can aggregate these sources according to their respective categories. Different types of group activity sources possess different characteristics and behavioral patterns, requiring different processing and analysis methods. For example, when processing pedestrian traffic, features such as pedestrian stride length and speed may need to be considered, while when processing vehicular traffic, features such as vehicle speed and direction need to be considered. Therefore, different aggregation methods are required for different types of group activity sources to accurately represent their behavioral characteristics.

[0139] S205, Obtain the pedestrian intent prediction results output by the prediction model.

[0140] After the prediction model generates the pedestrian intent prediction result, obtain the pedestrian intent prediction result output by the prediction model.

[0141] As can be seen from this embodiment, this application eliminates the bias caused by multiple frames by selecting a single-frame image as the input for the prediction model, rather than inputting multiple frames. Furthermore, the prediction model in this application performs pose prediction and intent prediction based on the feature recognition results of the single-frame image, ultimately generating a pedestrian intent prediction result. This eliminates the need for a pedestrian detector to predict whether a pedestrian is crossing the road. Moreover, without a pedestrian detector, the computation time of the prediction model does not increase with the number of detected pedestrians, allowing it to run within a relatively stable timeframe, avoiding increased computation time and workload, and thus improving prediction efficiency and accuracy. Additionally, this application can employ different aggregation methods to accurately represent the behavioral characteristics of different types of group activity sources. Different aggregation components can use different methods to describe the group activity portion, adapting to different application scenarios and task requirements, providing more accurate information for decision-making in application scenarios.

[0142] It should be noted that, in an optional embodiment of this application, the pedestrian intention prediction model can also output pedestrian pose maps in parallel, locate the human joints and joint connection points of each person, and output the human skeleton to display the detailed pose of each pedestrian. The pedestrian intention prediction model is easily extensible to perform various other tasks and has low overhead. In addition to implementing the main function of pedestrian intention prediction, the pedestrian intention prediction model can also perform pedestrian detection tasks and can serve as a general-purpose pedestrian detector. Finally, since the pedestrian intention prediction model can use only an RGB camera, it can be easily integrated into advanced driver assistance systems (ADAS), autonomous vehicles, or any other traffic light management systems, exhibiting good scalability and wide applicability.

[0143] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a pedestrian intention prediction device, a vehicle, and corresponding embodiments.

[0144] Figure 4 This is a schematic diagram of the pedestrian intention prediction device shown in the embodiments of this application.

[0145] See Figure 4 The pedestrian intention prediction device 40 includes: an image acquisition module 41, a prediction processing module 42, and a result acquisition module 43.

[0146] The image acquisition module 41 is used to acquire a single-frame image from the perceived content collected by the vehicle's sensors. This single-frame image can be acquired from the perceived content collected by the vehicle's cameras, LiDAR, and other sensors. In autonomous driving, vehicles are typically equipped with multiple sensors to acquire environmental information about their surroundings, including images, LiDAR, and millimeter-wave radar. These sensors can collect both static and dynamic information about the vehicle's surroundings, such as static information like roads and buildings, and dynamic information like pedestrians and vehicles.

[0147] The prediction processing module 42 is used to input the single-frame image acquired by the image acquisition module 41 into the prediction model. The prediction model performs feature recognition on the single-frame image, and performs pose prediction and intent prediction based on the feature recognition results. Combining the pose prediction results and intent prediction results, a pedestrian intent prediction result is generated. The prediction processing module 42 can perform feature recognition on the single-frame image by the prediction model to generate a feature map; input the feature map into the pose head network for pose prediction; input the feature map into the intent head network for intent prediction; and combine the pose prediction results and intent prediction results to generate a pedestrian intent prediction result.

[0148] The result acquisition module 43 acquires the pedestrian intent prediction results output by the prediction model of the prediction processing module 42.

[0149] Figure 5 This is another schematic diagram of the pedestrian intention prediction device shown in the embodiments of this application.

[0150] See Figure 5 The pedestrian intention prediction device 40 includes: an image acquisition module 41, a prediction processing module 42, a result acquisition module 43, and a model training module 44.

[0151] The prediction processing module 42 may include: an information input submodule 421 and a prediction model submodule 422.

[0152] The information input submodule 421 is used to input a single frame image into the prediction model.

[0153] The prediction model submodule 422 is used to perform feature recognition on a single frame image through the prediction model and generate a feature map; input the feature map into the pose head network for pose prediction, input the feature map into the intention head network for intention prediction, and combine the pose prediction result and the intention prediction result to generate the pedestrian intention prediction result.

[0154] The prediction model submodule 422 may include: feature map submodule 4221, pose prediction submodule 4222, and intention prediction submodule 4223.

[0155] The feature map submodule 4221 is used to perform feature recognition on a single frame image and generate a feature map.

[0156] The pose prediction submodule 4222 is used to input the feature map into the pose head network, which then refines the feature map. For example, the feature map is input into the pose head network, which uses a first preset function to perform initial processing on the feature map to obtain an initial activity map; then, a second preset function is used to refine the feature map and the initial activity map to obtain a refined activity map.

[0157] The intent prediction submodule 4223 is used to input feature maps into the intent head network, which then performs aggregation processing based on the refinement results of the feature maps and the pose head network. For example, the intent head network inputs feature maps into the intent head network, which then performs aggregation processing based on the refinement results of the feature maps and the pose head network. Alternatively, the intent head network inputs feature maps into the intent head network, which then performs aggregation processing based on the refined activity maps obtained from the feature maps and the pose head network, to obtain the predicted group activity of pedestrians.

[0158] The model training module 44 is used to pre-train the prediction model, which is then optimized based on a loss function. During training, this application can optimize and adjust the pedestrian intention prediction model based on the results of the loss function, and retrain the model using several training samples until the training loss meets preset training conditions, thus obtaining a fully trained pedestrian intention prediction model. This application can set a loss threshold; when the training loss is less than the threshold, it can be determined that the current training loss meets the preset training conditions, and the pedestrian intention prediction model training is complete.

[0159] The solution provided in this application addresses the temporal bias introduced by multiple frames of images or videos. If the time series contains multiple video frames, this bias can have a greater impact on the prediction model. Therefore, this application eliminates this bias by inputting a single frame image into the prediction model for processing, rather than multiple frames. Furthermore, the prediction model in this application performs pose prediction and intent prediction based on the feature recognition results of the single frame image, ultimately generating a pedestrian intent prediction result. This eliminates the need for pedestrian detectors to predict whether a pedestrian is crossing the road. Moreover, without pedestrian detectors, the computation time of the prediction model does not increase with the number of detected pedestrians, allowing it to run within a relatively stable timeframe, avoiding increased computation time and workload, and thus improving prediction efficiency and accuracy. Therefore, this application's solution can predict pedestrian behavior and intent more quickly and accurately, improving the safety of autonomous driving.

[0160] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0161] Figure 6 This is a schematic diagram of the vehicle structure shown in the embodiments of this application.

[0162] See Figure 6 The vehicle 600 includes a memory 610 and a processor 620.

[0163] The processor 620 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0164] Memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 620 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 610 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0165] The memory 610 stores executable code, which, when processed by the processor 620, can cause the processor 620 to execute part or all of the methods described above.

[0166] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0167] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0168] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for predicting pedestrian intentions, characterized in that, include: Obtain a single frame image from the perceived content collected by the vehicle's sensors; The single-frame image is input into the prediction model, which performs feature recognition on the single-frame image. Based on the feature recognition results, pose prediction and intention prediction are performed respectively. The pedestrian intention prediction result is generated by combining the pose prediction result and the intention prediction result. Obtain the pedestrian intent prediction results output by the prediction model; The step of performing feature recognition on the single-frame image by the prediction model, performing pose prediction and intent prediction based on the feature recognition results, and combining the pose prediction results and intent prediction results to generate a pedestrian intent prediction result includes: The prediction model performs feature recognition on the single-frame image to generate a feature map; The feature map is input into the pose head network for pose prediction, and the feature map is input into the intention head network for intention prediction. The pedestrian intention prediction result is generated by combining the pose prediction result and the intention prediction result. The intention prediction result is obtained by processing the feature map and the pose prediction result. The step of inputting the feature map into the attitude head network for attitude prediction includes: inputting the feature map into the attitude head network, and having the attitude head network refine the feature map according to the feature map; The step of inputting the feature map into the intention head network for intention prediction includes: inputting the feature map into the intention head network, and having the intention head network perform aggregation processing based on the refinement results of the feature map and the pose head network.

2. The method according to claim 1, characterized in that, The step of inputting the feature map into the pose head network, and having the pose head network refine the feature map, includes: The feature map is input into the attitude head network, and the attitude head network performs initial processing on the feature map using a first preset function to obtain an initial activity map. The feature map and the initial activity map are refined using a second preset function to obtain a refined activity map.

3. The method according to claim 2, characterized in that, The step of inputting the feature map into the intention head network, and then having the intention head network perform aggregation processing based on the refinement results of the feature map and the pose head network, includes: The feature map is input into the intention head network, which then aggregates the refined activity map based on the feature map and the attitude head network to obtain the group activity prediction results of pedestrians.

4. The method according to any one of claims 1 to 3, characterized in that, The prediction model is pre-trained and optimized based on a loss function.

5. A pedestrian intention prediction device, characterized in that, include: The image acquisition module is used to acquire a single-frame image from the perceived content collected by the vehicle's sensors; The prediction processing module is used to input the single-frame image acquired by the image acquisition module into the prediction model, and the prediction model performs feature recognition on the single-frame image. Based on the feature recognition results, pose prediction and intention prediction are performed respectively. The pedestrian intention prediction result is generated by combining the pose prediction result and the intention prediction result. The result acquisition module acquires the pedestrian intent prediction results output by the prediction model of the prediction processing module; The prediction processing module includes: The information input submodule is used to input the single-frame image into the prediction model; The prediction model submodule is used to perform feature recognition on the single frame image through the prediction model to generate a feature map; input the feature map into the pose head network for pose prediction; input the feature map into the intention head network for intention prediction; and combine the pose prediction result and the intention prediction result to generate a pedestrian intention prediction result, wherein the intention prediction result is obtained by processing the feature map and the pose prediction result. The prediction model submodule includes: The feature map submodule is used to perform feature recognition on the single-frame image and generate a feature map. The pose prediction submodule is used to input the feature map into the pose head network, and the pose head network refines the feature map according to the feature map. The intent prediction submodule is used to input the feature map into the intent head network, and the intent head network performs aggregation processing based on the refinement results of the feature map and the pose head network.

6. The apparatus according to claim 5, characterized in that, The device further includes: The model training module is used to pre-train a prediction model, which is then optimized based on a loss function.

7. A vehicle, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-4.

8. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-4.

Images