System and method for predicting movement trajectory of pedestrian

Abstract

Some embodiments of the present specification provide systems and methods for predicting a movement trajectory of a pedestrian. The method system comprises a communication interface used for receiving a map of an area where a pedestrian is walking and obtaining sensor data associated with the pedestrian; at least one processor for extracting pedestrian features from the sensor data; identifying one or more objects surrounding the pedestrian based on the location of the pedestrian; extracting object features of one or more objects from the sensor data; determining a score of each candidate trajectory by using a learning model based on the extracted pedestrian features and object features; in response to the condition that the score of the candidate trajectory is greater than a threshold value, predicting a moving trajectory and a moving speed of the pedestrian based on one or more candidate trajectories greater than the threshold value; updating an angle of the lidar and / or adjusting a focus of the camera in response to the score of absence of the candidate trajectory being greater than a threshold to obtain additional sensor data; pedestrian features and object features are determined based on the additional sensor data.

Description

Description of the case

[0001] This application is a divisional application of Chinese patent application CN 201980100948.5, entitled “System and method for predicting pedestrian movement trajectory”, filed on September 30, 2019. Technical Field

[0002] The present application relates to a system and method for predicting the movement trajectory and movement speed of a pedestrian, and more specifically, to a system and method for predicting the movement trajectory and movement speed of a pedestrian based on features extracted from a map and sensor data. Background Art

[0003] Vehicles share the road with other vehicles, pedestrians, and objects such as traffic signs, roadblocks, fences, etc. Therefore, the driver needs to constantly adjust driving maneuvers to avoid collisions with such obstacles. Some obstacles that are easy to avoid are usually static, but some obstacles can move. For moving obstacles, the driver must not only observe their current location, but also predict their movement trajectory to determine their future location. For example, a pedestrian near the vehicle can cross the road in front of the vehicle, walk in a direction parallel to the vehicle's direction of travel, or stop. The driver usually makes predictions based on observations, for example, based on the pedestrian's walking speed, the direction the pedestrian is facing, and any hand signals provided by the pedestrian.

[0004] Autonomous vehicles need to make similar decisions to avoid collisions with obstacles. Therefore, autonomous driving technology relies heavily on automatically predicting the movement trajectory of other moving obstacles. However, existing prediction systems and methods are limited by the vehicle's ability to "observe" (such as collecting relevant data), its ability to process data, and its ability to make accurate predictions based on the data. Therefore, improvements to existing prediction systems and methods can benefit the development of autonomous vehicles.

[0005] The system and method of the embodiments of the present application that predict the movement trajectory and movement speed of pedestrians based on features extracted from maps and sensor data improve the existing prediction systems and methods in autonomous driving. Summary of the invention

[0006] An embodiment of the present application provides a system for predicting the movement trajectory of a pedestrian. The system includes a communication interface, which is used to receive a map of an area where a pedestrian is walking and obtain sensor data associated with the pedestrian. The system includes at least one processor, which is used to locate the pedestrian on the map and extract pedestrian features from the sensor data. The at least one processor is also used to identify one or more objects surrounding the pedestrian based on the positioning of the pedestrian, and extract object features of one or more objects from the sensor data. The at least one processor is also used to predict the movement trajectory and movement speed of the pedestrian using a learning model based on the extracted pedestrian features and object features.

[0007] An embodiment of the present application also provides a method for predicting the movement trajectory of a pedestrian. The method includes receiving a map of an area where a pedestrian is walking and sensor data associated with the pedestrian through a communication interface. The method also includes locating the pedestrian on the map and extracting pedestrian features from the sensor data through at least one processor. The method also includes identifying one or more objects surrounding the pedestrian based on the positioning of the pedestrian; and extracting object features of the one or more objects from the sensor data through the at least one processor. The method also includes predicting the movement trajectory and movement speed of the pedestrian using a learning model based on the extracted pedestrian features and object features through at least one processor.

[0008] Embodiments of the present application also provide a non-temporary computer-readable medium for storing instructions, which, when executed by at least one processor, cause the at least one processor to perform the above-mentioned operations. The operations include receiving a map of an area where a pedestrian is walking and obtaining sensor data related to the pedestrian. The operations also include locating the pedestrian on the map and extracting pedestrian features from the sensor data. The operations also include identifying one or more objects surrounding the pedestrian based on the positioning of the pedestrian; and extracting object features of the one or more objects from the sensor data. The operations also include predicting the movement trajectory and movement speed of the pedestrian using a machine learning model based on the extracted pedestrian features and the object features.

[0009] As stated herein, the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of an exemplary road section including a sidewalk and a crosswalk next to a lane according to an embodiment of the present application.

[0011] Figure 2is a schematic diagram of an exemplary system for predicting the movement trajectory of pedestrians according to an embodiment of the present application.

[0012] Figure 3 is a schematic diagram of an exemplary vehicle with sensors on wheels according to an embodiment of the present application.

[0013] Figure 4 This is an exemplary server block diagram for predicting the movement trajectory of pedestrians according to an embodiment of the present application.

[0014] Figure 5 This is a flowchart of an exemplary method for predicting the movement trajectory of pedestrians according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0016] Figure 1 1 is a schematic diagram of an exemplary road segment 100 according to an embodiment of the present application, wherein the road segment 100 includes a sidewalk 106 and a crosswalk 110 near lanes 102 and 104. Figure 1 As shown, road segment 100 extends eastward, toward a traffic light 140 located at an intersection. It is contemplated that road segment 100 may extend in any other direction, and not necessarily adjacent to a traffic light.

[0017] The road segment 100 may be a one-way road or a portion of a two-way road. Figure 1 Only two lanes in a single direction are shown. However, it is contemplated that the road segment 100 may include more or fewer lanes, and that the lanes may be opposite each other in both directions and separated by a median strip. Figure 1 As shown, the road segment 100 includes lanes 102 and 104, and a sidewalk 106 on the right side of the lane. In some embodiments, the sidewalk 106 may be separated from the lane 104 by a dividing strip 108, which may be, for example, a guardrail, a fence, a tree, a bush, or a no-entry zone. In some embodiments, the sidewalk 106 may not be separated from the lane 104, or may be separated only by a marking line.

[0018] Various vehicles may travel in lanes 102 and 104. For example, vehicle 101 may travel eastbound in lane 104. In some embodiments, vehicle 101 may be an electric vehicle, a fuel cell vehicle, a hybrid vehicle, or a conventional internal combustion engine vehicle. In some embodiments, vehicle 101 may be an autonomous or semi-autonomous vehicle.

[0019] Pedestrians can walk in one or both directions on the sidewalk 106. For example, pedestrian 130 can walk east or west on the sidewalk 106. In some embodiments, the sidewalk 106 can be indicated as a sidewalk with road markings. For example, the character "行" can be marked on the sidewalk 106, as Figure 1 shown. As another example, a pedestrian icon or additional text description can be marked on the sidewalk 106.

[0020] The traffic of vehicles and pedestrians on the road section 100 can be scheduled by traffic lights 140 and pedestrian traffic lights 142 (e.g., including pedestrian traffic lights 142-A and 142-B). For example, traffic lights 140 can schedule the traffic of vehicles, and pedestrian traffic lights 142 can schedule the traffic of pedestrians. In some embodiments, traffic lights 140 can include lights of three colors: red, yellow, and green, to indicate whether the intersection is passable. In some embodiments, traffic lights 140 can also include turning protection lights to schedule vehicles making left turns, right turns, and / or U-turns. For example, a left-turn protection light can allow vehicles in certain lanes (usually the leftmost lane) to make a left turn without having to yield to oncoming vehicles going straight in the opposite direction.

[0021] Pedestrian traffic lights 142-A and 142-B can be located at different corners of the intersection, facing pedestrians walking in various directions. For example, pedestrian traffic light 142-A may face eastbound pedestrians, and pedestrian traffic light 142-B may face northbound pedestrians. The pedestrian traffic lights can switch between two modes: a "通行 (Pass)" mode and a "禁止通行 (No Pass)" mode. By design, the pedestrian traffic lights can display different words or icons to indicate different modes. For example, when pedestrians and cross-traffic are allowed, the pedestrian traffic lights can display a pedestrian icon; and a hand-shaped icon can be displayed to block the passage of pedestrians. In some embodiments, pedestrian traffic signal lights 142 can additionally use different colors, sounds (e.g., beeps), and / or flashes to indicate different modes. For example, the "通行 (Pass)" mode can be displayed in green, and the "禁止通行 (No Pass)" mode can be displayed in red.

[0022] In some embodiments, pedestrian 130 can cross the road from the crosswalk 110. In some embodiments, the crosswalk 110 can be marked with white stripes on the road surface (referred to as zebra crossings). The direction of passage of the crosswalk is perpendicular to the stripes. For example, the crosswalk 110 contains stripes extending east-west, and pedestrian 130 can walk north or south on the crosswalk 110 to cross the road. Pedestrians walking on the crosswalk have the right of way, and other vehicles will stop and yield to the pedestrians until the pedestrians have crossed the crosswalk. Although Figure 1Only one crosswalk 110 is shown, but it is contemplated that there may be more crosswalks extending in different directions. The crosswalk 110 need not be located at an intersection with a traffic light. In some embodiments, the crosswalk may exist in the middle of a road segment.

[0023] Pedestrian 130 may cross in places that are not supervised by traffic lights and / or do not have crosswalks, etc. For example, pedestrian 130 may cross the road to enter a side road on the other side of the road. In such cases, the pedestrian may sometimes signal the vehicle with a hand gesture before entering the lane. For example, the pedestrian may raise his palm to signal the vehicle to stop or point in the direction he intends to walk.

[0024] In some embodiments, the vehicle 101 may be equipped with a pedestrian trajectory prediction system (e.g., Figure 2 200) shown in FIG. 1 , or in communication therewith, to predict the movement trajectory of a pedestrian (e.g., pedestrian 130) so as to make a decision to avoid the pedestrian from appearing in the path of the vehicle. Figure 1 In the setting of , the pedestrian 130 heading north may follow four candidate trajectories: candidate trajectory 151 of crossing the road to the north, candidate trajectory 152 of turning left and walking west, candidate trajectory 153 of turning right and walking east, and candidate trajectory 154 of stopping.

[0025] Consistent with embodiments of the present invention, the pedestrian trajectory prediction system may "observe" (e.g., through various sensors) pedestrians 130 and objects surrounding the pedestrians (e.g., traffic lights 140, pedestrian traffic lights 142, crosswalks 110, and any traffic signs along road segment 100). The pedestrian trajectory prediction system then predicts candidate walking trajectories that pedestrian 130 may follow based on these observations. In some embodiments, a learning model such as a neural network model may be used to perform the prediction. In some embodiments, scores (e.g., probabilities and rankings) may be determined for each candidate trajectory 151-154 or 161-164.

[0026] Figure 2 2 is a schematic diagram of an exemplary system 200 for predicting the movement trajectory of a pedestrian according to an embodiment of the present application. In some embodiments, the system 200 may include a pedestrian trajectory prediction server 210 (also referred to as server 210 for simplicity). The server 210 may be a general-purpose server configured or programmed with functions for predicting the movement trajectory of pedestrians, or a proprietary device specifically designed to predict the movement trajectory of pedestrians. It is conceivable that the server 210 may be an independent server or an integrated component of an independent server. In some embodiments, the server 210 may be integrated into a system on a vehicle, for example, a system of the vehicle 101.

[0027] like Figure 2As shown, the server 210 can receive and analyze data collected from various sources. For example, data can be continuously, periodically, or intermittently acquired by sensors 220 (e.g., including sensors 220-A and 220-B) installed along the road and / or one or more sensors 230 installed on the vehicle 101 traveling in the lane 104. The sensors 220 and 230 may include radars, lidars, cameras (e.g., surveillance cameras, monocular / binocular cameras, video cameras), speedometers, or any other suitable sensors to acquire data representing pedestrians 130 and objects surrounding pedestrians 130, such as traffic lights 140, pedestrian traffic lights 142, and crosswalks 110. By way of example only, the sensors 220 may include one or more surveillance cameras to acquire images of pedestrians 130, traffic lights 140, pedestrian traffic lights 142, and crosswalks 110.

[0028] In some embodiments, sensor 230 may include a laser radar that measures the distance between vehicle 101 and pedestrian 130, and the laser radar may also determine the position of pedestrian 130 in a three-dimensional map. In some embodiments, sensor 230 may also include a GPS / IMU (inertial measurement unit) sensor for acquiring position / attitude data of vehicle 101. In some embodiments, sensor 230 may also include a camera for acquiring images of pedestrian 130 and objects surrounding pedestrian 130. Since the images acquired by sensor 220 and sensor 230 are from different angles, they can complement each other to provide more detailed information of pedestrian 130 and objects surrounding pedestrians. In some embodiments, sensors 220 and 230 can acquire data that tracks the trajectory of moving objects, such as vehicles, bicycles, pedestrians, etc.

[0029] In some embodiments, the sensor 230 may be mounted on the vehicle 101 so as to travel with the vehicle 101. For example, according to an embodiment of the present invention, Figure 3 An exemplary vehicle 101 equipped with sensors 340-360 is shown. Vehicle 101 may have a body 310, which may be any body type, such as a sports car, a coupe, a sedan, a pickup truck, a station wagon, a sport utility vehicle (SUV), a minivan, or a modified van. In some embodiments, as Figure 3 As shown, the vehicle 101 may include a pair of front wheels and a pair of rear wheels 320. The vehicle 101 may have fewer wheels or an equivalent structure that enables the vehicle 101 to travel. The vehicle 101 may be configured as all-wheel drive (AWD), front-wheel drive (FWR), or rear-wheel drive (RWD). In some embodiments, the vehicle 101 may be an autonomous or semi-autonomous vehicle.

[0030] like Figure 3 As shown, Figure 2The sensors 230 may include various sensors 340, 350, and 360 according to embodiments of the present application. The sensor 340 may be mounted to the body 310 via the mounting structure 330. The mounting structure 330 may be an electromechanical device mounted or otherwise connected to the body 310 of the vehicle 101. In some embodiments, the mounting structure 330 may include screws, adhesives, or other mounting structures. The vehicle 101 may be additionally equipped with sensors 350 and 360 or an external body 310 using any suitable mounting structure. It will be appreciated that the manner in which the sensors 340-360 are mounted on the vehicle 101 is not limited to the embodiment of the present invention. Figure 3 The examples shown are limitations and may be modified depending on the type of sensor 340 - 360 and / or vehicle 101 to achieve desired sensing performance.

[0031] In some embodiments, sensor 340 may be a laser radar that measures the distance to a target by illuminating the target with a pulsed laser and detecting the reflected pulses. The difference in laser return time and wavelength may be used to create a digital three-dimensional depiction of the target. For example, sensor 340 may measure the distance between vehicle 101 and pedestrian 130 or between vehicle 101 and other objects. The light used for laser radar scanning may be ultraviolet, visible, or near infrared. Because narrow laser beams may map physical features at very high resolution, laser radar scanners are particularly suitable for locating objects in three-dimensional maps. For example, a laser radar scanner may capture point cloud data that may be used to locate vehicles 101, pedestrians 130, and / or other objects.

[0032] In some embodiments, sensor 350 may include one or more cameras mounted on body 310 of vehicle 101. Figure 3 The sensor 350 is shown mounted on the front of the vehicle 101, but the sensor 350 can be mounted in other locations of the vehicle 101, such as on the side, behind the rearview mirror, on the windshield, on the rack, or at the rear. The sensor 350 can be used to obtain images of objects around the vehicle 101, and the objects around the vehicle 101 can be, for example, pedestrians 130 on the road, traffic lights (e.g., 140 and 142), crosswalks 110, and / or traffic signs. In some embodiments, the camera can be a monocular or binocular camera. The binocular camera can obtain data indicating the depth of the object (i.e., the distance of the object from the camera). In some embodiments, the camera can be a video camera that captures image frames over time, thereby recording the movement of the object.

[0033] like Figure 3As shown, the vehicle 101 may be additionally equipped with sensors 360, which may include sensors used in a navigation unit, such as a GPS receiver and one or more IMU sensors. GPS is a global navigation satellite system that provides geographic location and time information to a GPS receiver. An IMU is an electronic device that is capable of measuring and providing vehicle-specific forces, angular rates, and sometimes magnetic field data around the vehicle using various inertial sensors (such as accelerometers and gyroscopes, and sometimes magnetometers). By combining a GPS receiver and an IMU sensor, the sensor 360 can provide real-time posture information of the vehicle 101 while the vehicle 101 is traveling, including the position and orientation (e.g., Euler angles) of the vehicle 101 at each point in time.

[0034] Sensors 340-360 can communicate with server 210 via a network to continuously, frequently, or intermittently send sensor data. In some embodiments, any suitable network can be used for communication, such as a wireless local area network (WLAN), a wide area network (WAN), a wireless communication network using radio waves, a cellular network, a satellite communication network, and / or a local or short-range wireless network (e.g., Bluetooth TM ).

[0035] like Figure 2 As shown, the system 200 may also include a three-dimensional map database 240. The three-dimensional map database 240 may store three-dimensional maps. The three-dimensional map may include maps covering different areas. For example, a three-dimensional map (or a portion of a map) may cover an area including a crossroad section 100. In some embodiments, the server 210 may communicate with the three-dimensional map database 240 to retrieve a related three-dimensional map (or a portion of a map) based on the location of the vehicle 101. For example, map data containing the GPS location of the vehicle 101 and its surrounding area may be retrieved. In some embodiments, the three-dimensional map database 240 may be an internal component of the server 210. For example, the three-dimensional map may be stored in a memory of the server 210. In some embodiments, the three-dimensional map database 240 may be located outside the server 210, and the communication between the three-dimensional map database 240 and the server 210 may be carried out via a network (e.g., the various networks described above).

[0036] Server 210 may be configured to analyze sensor data received from sensors 230 (eg, sensors 340 - 360 ) and map data received from 3D map database 240 to predict a movement trajectory of a pedestrian (eg, pedestrian 130 ). Figure 44 is a block diagram of an exemplary server 210 for predicting the movement trajectory of pedestrians according to an embodiment of the present application. The server 210 may include a communication interface 402, a processor 404, a memory 406, and a storage 408. In some embodiments, the server 210 may have different modules in a single device, such as an integrated circuit (IC) chip (implemented as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)) or a separate device with a dedicated function. The components of the server 210 may be integrated devices, or devices distributed in different locations but communicating with each other through a network (not shown).

[0037] The communication interface 402 may be connected via a direct communication link, a wireless local area network (WLAN), a wide area network (WAN), a wireless communication network using radio waves, a cellular network, and / or a local wireless network (such as Bluetooth TM The communication interface 402 may be used to send and receive data to components such as sensors 220 and 230 using a wireless or WiFi (such as a wireless or WiFi network), or may be used to communicate using other communication methods. In some embodiments, the communication interface 402 may be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection. For another example, the communication interface 402 may be a local area network (LAN) card to provide a data communication connection with a compatible LAN. A wireless link may also be implemented by the communication interface 402. In such an implementation, the communication interface 402 may send and receive electrical, electromagnetic, or optical signals over a network, which carry digital data streams representing various types of information.

[0038] In some embodiments, the communication interface 402 may receive sensor data 401 acquired by the sensor 220 and / or 230 and map data 403 provided by the three-dimensional map database 240, and provide the received data to the memory 406 and / or the storage 408 for storage, or to the processor 404 for processing. The sensor data 401 may include information about pedestrians (e.g., pedestrian 130) and other objects around the pedestrians. The sensor data 401 may include data representing the movement of objects acquired over time. In some embodiments, the map data 403 may include point cloud data.

[0039] The communication interface 402 may also receive a learning model 405. In some embodiments, the learning model 405 may be applied by the processor 404 and predict the movement trajectory of the pedestrian based on the features extracted from the sensor data 401 and the map data 403. In some embodiments, the learning model 405 may be a prediction model, such as a decision tree learning model, a logistic regression model, or a convolutional neural network (CNN) model. Other suitable machine learning models may also be used as the learning model 405.

[0040] A decision tree uses the observed values ​​of an item (represented by branches) to predict the target value of an item (represented by leaves). For example, a decision tree model may predict the probabilities of several hypothetical outcomes, such as the probability of candidate trajectories for pedestrian 130. In some embodiments, gradient boosting may be combined with a decision tree learning model to form a prediction model as a set of decision trees. For example, learning model 405 may include a gradient boosting decision tree model formed by staged decision trees.

[0041] In some embodiments, the learning model 405 can be a logistic regression model that predicts the value of a discrete variable. For example, a logistic regression model can be used to rank several hypothetical results, for example, to rank candidate trajectories of the pedestrian 130. In some embodiments, the learning model 405 can be a convolutional neural network (CNN) model, which includes multiple layers, such as one or more convolutional layers or full convolutional layers, nonlinear operator layers, pooling or subsampling layers, fully connected layers, and / or final loss layers. Each layer of the convolutional neural network model produces one or more feature maps. Convolutional neural network models are generally suitable for tasks such as image recognition, video analysis, and image classification, for example, identifying objects from image or video data.

[0042] In some embodiments, the learning model 405 can be trained using known pedestrian movement trajectories and their respective sample features, such as semantic features, which include pedestrian speed, pedestrian direction (i.e., the direction the pedestrian is facing), pedestrian gestures, pedestrian crossing markings, pedestrian traffic light status, and the type of dividing strip between the sidewalk and the lane. The sample features may also include non-semantic features extracted from data describing pedestrian movement. In some embodiments, the learning model 405 may be trained in advance by the server 210 or another computer / server.

[0043] "Training" a learning model refers to determining one or more parameters of at least one layer in the learning model. For example, a convolutional layer of a convolutional neural network model may include at least one filter or kernel. One or more parameters of at least one filter (e.g., kernel weight, size, shape, and structure) may be determined by, for example, a back-propagation-based training process. The learning model 405 is trained so that when it requires sample features as input, it will provide a predicted structure of a pedestrian movement trajectory that is substantially close to a known trajectory.

[0044] Processor 404 may include any suitable type of general or special purpose microprocessor, digital signal processor, or microcontroller. Processor 404 may include a separate processor module dedicated to predicting the movement trajectory of pedestrians. Alternatively, processor 404 may be configured as a shared processor module for performing other functions related or unrelated to pedestrian trajectory prediction. For example, the shared processor may further make an autonomous driving decision based on the predicted pedestrian movement trajectory.

[0045] like Figure 4 As shown, the processor 404 may include multiple modules, such as a positioning unit 440, an object recognition unit 442, a feature extraction unit 444, a trajectory prediction unit 446, etc. These modules (and any corresponding sub-modules or sub-units) may be hardware units (e.g., parts of an integrated circuit) of the processor 404 for use with other components, or part of an execution program. The program may be stored on a computer-readable medium (e.g., memory 406 and / or storage 408), and when executed by the processor 404, the processor 404 may perform one or more functions. Although Figure 4 Units 440 - 446 are shown as all located within one processor 404 , however, it will be appreciated that these units may be distributed among multiple processors that are located adjacent to each other or remotely located.

[0046] The positioning unit 440 may be configured to locate the trajectory of the pedestrian being predicted (e.g., pedestrian 130) in the map data 403. In some embodiments, the sensor data 401 may include various data of the pedestrian acquired to assist in positioning. For example, the laser radar data acquired by the sensor 340 mounted on the vehicle 101 may reveal the location of the pedestrian 130 in the point cloud data. In some embodiments, the point cloud data acquired for the pedestrian 130 may be matched with the map data 401 to determine the location of the pedestrian. In some embodiments, a positioning method such as simultaneous localization and mapping (SLAM) may be used to locate the pedestrian.

[0047] In some embodiments, the positions of pedestrians (e.g., pedestrian 130) may be marked on map data 401. For example, a subset of point cloud data P1 is marked as corresponding to pedestrian 130 at time T1, a subset of point cloud data P2 is marked as corresponding to pedestrian 130 at time T2, a subset of point cloud data P3 is marked as corresponding to pedestrian 130 at time T3, and so on. The marked subsets represent the existing movement trajectory and movement speed of the pedestrian.

[0048] The object recognition unit 442 can recognize pedestrians 130 and objects surrounding pedestrians. Objects surrounding pedestrians may include, for example, traffic lights 140, pedestrian traffic lights 142, crosswalks 110, traffic signs, lane markings, dividers 108, and other vehicles. In some embodiments, various image processing methods such as image segmentation, image classification, and image recognition may be applied to recognize pedestrians and objects surrounding pedestrians. In some embodiments, machine learning techniques such as convolutional neural network models may also be applied to recognize pedestrians and objects surrounding pedestrians.

[0049] The feature extraction unit 444 may be configured to extract features indicating the future trajectory of the pedestrian from the sensor data 401 and the map data 403. The extracted features may include pedestrian features and object features. The pedestrian features may be associated with the pedestrian 130, such as the pedestrian's speed, the direction the pedestrian is facing, the pedestrian's movement and action, etc. The object features may be related to surrounding objects, such as the direction of the crosswalk, the lane markings of the sidewalk, the state of the pedestrian traffic light, the pedestrian hand signal, and the type of dividing line between the sidewalk and the lane, etc.

[0050] Various feature extraction tools may be used to extract features, such as facial recognition, gesture detection, movement detection, gait recognition, and the like. For example, the feature extraction unit 444 may perform facial recognition to identify the face of a pedestrian. The face of a pedestrian provides important information about where the pedestrian is going. For another example, the feature extraction unit 444 may also perform a gesture detection method to detect the movement of the pedestrian's arms and legs. The pedestrian's gestures may characterize where the pedestrian intends to go. For another example, the feature extraction unit 444 may perform gait recognition to extract features indicating how the pedestrian walks, such as features of body displacement, body mechanics, and muscle activity. Such gait features provide information about the pedestrian's movement, such as whether the pedestrian walks, runs, jogs, jumps, limps, or moves with the aid of an assistive device. The pedestrian's movement may represent his mobility. For example, a person who walks with the aid of a cane has poor mobility. In some embodiments, facial features may also help determine the gender and age of the pedestrian, thereby further helping to determine the pedestrian's mobility.

[0051] In addition, lane markings and crosswalk markings can be detected from sensor data based on color and / or contrast information, because the markings are usually white and the road surface is usually black or gray. When color information is available, markings can be identified based on different colors (e.g., white) in the color information. When grayscale information is available, markings can be identified based on shades (e.g., light gray) that are different from the background (e.g., the conventional road surface is dark gray). The direction of the crosswalk can be determined based on the direction in which the markings of the crosswalk extend. As another example, traffic light signals can be detected by detecting intensity changes in image pixels (e.g., changes caused by flickering, flashing, or color changes). In some embodiments, machine learning techniques can also be applied to extract features.

[0052] The features of these surrounding objects can also provide additional information useful for pedestrian trajectory prediction. For example, if the pedestrian traffic light controlling pedestrian traffic indicates no-go, the pedestrian may not move immediately. For another example, if a pedestrian is standing in a crosswalk, this indicates that the pedestrian plans to cross the road.

[0053] The trajectory prediction unit 446 can use the extracted pedestrian features and object features to predict the movement trajectory of the pedestrian. In some embodiments, the trajectory prediction unit 446 can determine at least two candidate trajectories. In some embodiments, the candidate trajectories can be determined based on the direction the pedestrian is facing. For example, if the pedestrian is detected facing north, the trajectory prediction unit 446 can determine candidate trajectories 151-154 (such as Figure 1 That is, pedestrian 130 may cross the road at crosswalk 110 (candidate trajectory 151), turn left onto sidewalk 106 (candidate trajectory 152), turn right onto sidewalk 106 (candidate trajectory 153), or stop (candidate trajectory 154). For another example, if a pedestrian is detected facing east, trajectory prediction unit 446 may determine candidate trajectories 161-164 of pedestrian 130 (e.g., Figure 1 That is, pedestrian 130 may go straight along sidewalk 106 (candidate trajectory 161), turn left and cross the road at crosswalk 110 (candidate trajectory 162), turn around on sidewalk 106 and walk westward (candidate trajectory 163), or stop (candidate trajectory 164).

[0054] In some embodiments, the trajectory prediction unit 446 can apply the learning model 405 for prediction. For example, the learning model 405 can determine the score of each candidate trajectory based on the extracted features. In some embodiments, the score can indicate the probability that the pedestrian follows the candidate trajectory. In some other embodiments, the score can be a ranking number assigned to each trajectory. In some embodiments, the candidate trajectory with the highest score (e.g., the highest probability or the highest ranking) can be used as the predicted movement trajectory of the pedestrian.

[0055] In some embodiments, before applying the learning model 405, the trajectory prediction unit 446 can first remove one or more candidate trajectories that conflict with any features. For example, candidate trajectory 163 can be eliminated because the probability that a pedestrian facing east will turn around and walk west is very low. For another example, if the pedestrian traffic light 142-B is in "no passing" mode, candidate trajectory 151 can be eliminated. By removing certain candidate trajectories, the trajectory prediction unit 446 can simplify the prediction task and save processing power of the processor 404.

[0056] In some embodiments, the trajectory prediction unit 446 can compare the determination scores (e.g., probabilities) of the various candidate trajectories to a threshold. If none of the candidate trajectories has a score that exceeds the threshold, the trajectory prediction unit 446 can determine that the prediction is not reliable enough and that additional "observations" are needed to improve the prediction. In some embodiments, the trajectory prediction unit 446 can determine which additional sensor data to acquire and generate control signals to be sent to the sensors 220 and / or 230 for acquiring the additional data. For example, it can be determined that the lidar should be tilted at a different angle, or that the camera should adjust its focus. The control signals can be provided to the sensors 220 and / or 230 via the communication interface 402.

[0057] In addition to predicting the movement trajectory, the trajectory prediction unit 446 can also predict the movement speed of the pedestrian. In some embodiments, the current speed of the pedestrian and the movement and mobility information can be used to estimate the future movement speed of the pedestrian. For example, a running pedestrian may cross the road at a very fast speed, while a person walking with crutches may walk very slowly.

[0058] Memory 406 and storage 408 may include any appropriate type of mass storage for storing any type of information that the processor 404 may need to operate. Memory 406 and storage 408 may be volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable or other types of storage devices or tangible (i.e., non-temporary) computer-readable media, including but not limited to ROM, flash memory, dynamic RAM, and static RAM. Memory 406 and / or storage 408 may be configured to store one or more computer programs that can be executed by the processor 404 to implement the pedestrian motion trajectory prediction function disclosed in this specification. For example, memory 406 and / or storage 408 may be configured to store programs that can be executed by the processor 404 to predict the movement trajectory of pedestrians based on features extracted from sensor data 401 and map data 403 obtained from various sensors 220 and / or 230.

[0059] The memory 406 and / or the storage 408 may also be configured to store information and data used by the processor 404. For example, the memory 406 and / or the storage 408 may be configured to store sensor data 401 acquired by the sensors 220 and / or 230, map data 403 received from the three-dimensional map database 240, and the learning model 405. The memory 406 and / or the storage 408 may also be configured to store intermediate data generated by the processor 404 during feature extraction and movement trajectory prediction, such as pedestrian features, object features, candidate trajectories, and scores of candidate trajectories. After processing each data frame, various types of data may be permanently stored, periodically deleted, or immediately ignored.

[0060] Figure 5 is a flowchart of an exemplary method 500 for predicting the movement trajectory of a pedestrian according to an embodiment of the present application. For example, the method 500 may be implemented by the system 200, which includes the server 210 and the sensors 220 and 230. However, the method 500 is not limited to this exemplary embodiment. The method 500 may include steps S502-S522 as described below. It should be understood that some steps may be optional to perform the disclosure of the present application provided in this specification. In addition, some steps may be performed simultaneously or in conjunction with Figure 5 The method 500 will be described as predicting the movement trajectory of the pedestrian 130 to assist the autonomous driving decision of the vehicle 101 (eg Figure 1 However, the method 500 may be used in other applications that may benefit from accurate prediction of the movement trajectory of pedestrians.

[0061] In step S502, the server 210 receives a map of the area where the pedestrian 130 is walking. In some embodiments, the server 210 can determine the location of the vehicle 101 based on, for example, GPS data collected by the sensor 360, and identify a map area around the location. The server 210 can receive relevant three-dimensional mapping data, such as the map data 403, from the three-dimensional map database 240.

[0062] In step S504, the server 210 receives information about the pedestrian 130 and objects surrounding the pedestrian acquired by the sensor data. In some embodiments, the sensor data may be acquired by various sensors, such as sensors 220 installed along the road and / or sensors 230 equipped on the vehicle 101 (including, for example, sensors 340-360). The sensor data may include pedestrian speed acquired by a speedometer, images (including video images) acquired by a camera, point cloud data acquired by a laser radar, etc. In some embodiments, the sensor data may be acquired over time to track the movement of the pedestrian 130 and objects surrounding the pedestrian. The sensor may communicate with the server 210 via a network to continuously, periodically, or intermittently transmit sensor data, such as sensor data 401.

[0063] Method 500 proceeds to step S506, where server 210 locates pedestrian 130 in the map. In some embodiments, point cloud data of pedestrian 130 (e.g., acquired by sensor 340) may be matched with map data 403 to determine the location of the pedestrian in the map. In some embodiments, a positioning method such as SLAM (Simultaneous Localization and Mapping) may be used to locate pedestrian 130. In some embodiments, the location of pedestrian 130 at different points in time may be marked on map data 403 to track the pedestrian's previous trajectory and movement speed. The marking of point cloud data may be performed automatically by server 210 or with the assistance of a human.

[0064] In step S508, the server 210 identifies the pedestrian 130 and other objects surrounding the pedestrian 130. For example, these objects may include the traffic light 140, the pedestrian traffic light 142, the crosswalk 110, the sidewalk 106, the dividing strip 108, traffic signs and lane markings, etc. The features of the pedestrian and the features of the objects surrounding the pedestrian can provide information for predicting the movement trajectory of the pedestrian 130. In some embodiments, various image processing methods and machine learning methods (e.g., convolutional neural networks) can be used to identify pedestrians and objects surrounding pedestrians.

[0065] In step S510, the server 210 extracts pedestrian features of the pedestrian 130 and object features of objects surrounding the pedestrian from the sensor data 401 and the map data 403. In some embodiments, the extracted features may include semantic or non-semantic features that indicate the future walking trajectory of the pedestrian. For example, pedestrian features may include, for example, pedestrian speed, the direction the pedestrian is facing, the movement and mobility of the pedestrian, any gesture signals of the pedestrian, etc. Object features of objects surrounding the pedestrian may include, for example, lane markings of the sidewalk, the direction of strip markings and crosswalks, the state of pedestrian traffic lights, the type of dividing strip between the sidewalk and the lane, and information on traffic signs. In some embodiments, various feature extraction methods including image processing methods and machine learning methods can be used.

[0066] In step S512, server 210 determines the direction that pedestrian 130 is facing. For example, facial recognition may be performed to identify the face of the pedestrian and the direction it is facing. In step S514, server 210 determines a plurality of candidate trajectories of pedestrian 130 based on the direction that pedestrian 130 is facing. Candidate trajectories are trajectories that pedestrian 130 may move along. For example, pedestrian 130 facing north may follow four candidate trajectories 151-154 (e.g., Figure 1 Similarly, pedestrian 130 heading east can walk along one of four candidate trajectories 161-164, i.e., continue walking eastward on sidewalk 106, turn left on crosswalk 110 to cross the road, turn around on sidewalk 106 and walk westward, or stop.

[0067] In some embodiments, server 210 may remove one or more candidate tracks that conflict with any of the features. For example, for pedestrian 130 heading east, candidate track 163 may be removed because the probability that the pedestrian turns around and walks west is very low. This optional filtering step may help simplify the prediction task and reduce the amount of data processing by server 210.

[0068] The method 500 proceeds to step S516 to determine a score for each candidate trajectory. In some embodiments, the score may be a probability that the pedestrian will follow the corresponding candidate trajectory or a ranking assigned to the candidate trajectory. In some embodiments, the server 210 may apply the learning model 405 to the prediction. In some embodiments, the learning model 405 may be a prediction model, such as a decision tree learning model, a logistic regression model, or a convolutional neural network model. For example, the learning model 405 may be a gradient boosted decision tree model. In some embodiments, the learning model 405 may be trained using known pedestrian movement trajectories and their respective sample features.

[0069] For example, in step S516, the learning model 405 may be applied to determine the probability of each candidate trajectory based on the extracted pedestrian features and object features. For example, it may be determined that the probability of pedestrian 130 walking along candidate trajectory 151 is 60%, the probability of walking along candidate trajectory 152 is 20%, the probability of walking along candidate trajectory 153 is 5%, and the probability of walking along candidate trajectory 154 is 15%.

[0070] In step S518, server 210 may compare the score (e.g., probability) to a predetermined threshold. In some embodiments, the predetermined threshold may be a percentage above 50%, such as 60%, 70%, 80%, or 90%. If no probability is above the threshold (S516: No), the prediction may be considered unreliable. In some embodiments, method 500 may return to step S504 to receive additional sensor data to improve the prediction. In some embodiments, server 210 may determine which additional sensor data may be acquired and generate a control signal to instruct sensor 220 and / or 230 to acquire the additional sensor data in step S504.

[0071] If the highest score is higher than the threshold (S518: Yes), the server 210 may predict the movement trajectory of the pedestrian by selecting the corresponding candidate trajectory from the candidate trajectories in step S520. In some embodiments, the candidate trajectory with the highest probability may be identified as the predicted movement trajectory of the pedestrian. For example, when the candidate trajectory 152 has the highest probability, it may be selected as the predicted movement trajectory of the pedestrian 130. In some other embodiments, in step S518, when the server 210 ranks the candidate trajectories instead of calculating the probabilities, the method 500 may skip step S518 and select the candidate trajectory with the highest ranking in step S520.

[0072] In step S522, the server 210 further predicts the pedestrian's moving speed. In some embodiments, the pedestrian's current speed and motion and mobility information may be used to estimate the pedestrian's future moving speed. For example, a running pedestrian may cross the road at a faster speed, while a pedestrian with crutches may walk very slowly.

[0073] The prediction results provided by method 500 can be provided to vehicle 101 and used to help the driver control the vehicle or make driving decisions. For example, the autonomous vehicle can make automatic control decisions based on the predicted movement trajectory of pedestrians to avoid hitting pedestrians. The prediction can also be used to help remind the driver to adjust his expected driving path and / or speed to avoid accidents. For example, an audio alarm such as a beep can be provided to warn the driver and / or pedestrians.

[0074] It should be noted that the above-mentioned video and / or image data containing the faces of pedestrians are only used for local vehicle analysis. These data will not be transmitted outside the vehicle and will be destroyed immediately after use.

[0075] Another aspect of the present invention relates to a non-transitory computer-readable medium for storing instructions, which is used to perform the above method by one or more processors. The computer-readable medium may include a volatile, non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable or other type of computer-readable medium or computer-readable storage device. For example, the computer-readable medium can be a storage device or memory module having computer instructions stored thereon. In some embodiments, the computer-readable medium can be a disk or a flash drive having computer instructions stored thereon.

[0076] Various modifications and variations of the disclosed systems and related methods are possible to those skilled in the art. Other embodiments will be apparent to those skilled in the art from considerations of the specification and practice of the disclosed systems and related methods.

[0077] It is intended that the specification and examples be considered as exemplary only, with a detailed scope being indicated by the following claims and their equivalents.

Claims

1. A system for predicting the movement trajectory of pedestrians, characterized in that: include: a communication interface for receiving a map of an area in which a pedestrian is walking and obtaining sensor data associated with the pedestrian; as well as at least one processor configured to: locating the pedestrian on the map; extracting pedestrian features from the sensor data; identifying one or more objects surrounding the pedestrian based on the location of the pedestrian; extracting object features of the one or more objects from the sensor data; determining at least two candidate trajectories; Determine a score for each candidate trajectory using a learning model based on the extracted pedestrian features and the object features; In response to the existence of a candidate trajectory having a score greater than a threshold, predicting a moving trajectory and a moving speed of the pedestrian based on one or more candidate trajectories greater than the threshold; In response to the absence of a candidate trajectory having a score greater than the threshold, Updating the angle of the lidar and / or adjusting the focus of the camera to obtain additional sensor data; The pedestrian characteristics and the object characteristics are determined based on the additional sensor data.

2. The system according to claim 1, characterized in that Before determining the score of each candidate trajectory, the at least one processor is further configured to: One or more candidate tracks that conflict with the pedestrian feature and the object feature are deleted.

3. The system according to claim 1, characterized in that The at least one processor is further configured to: determining the direction the pedestrian is facing based on the sensor data; and determining the at least two candidate trajectories based on the directions.

4. The system according to claim 1, characterized in that The score is the probability that the pedestrian follows the corresponding candidate trajectory.

5. The system according to claim 1, characterized in that The learning model is a decision tree model, a logistic regression model or a convolutional neural network model.

6. The system according to claim 1, wherein: In order to extract the pedestrian features, the at least one processor is further configured to detect the mobility of the pedestrian.

7. The system according to claim 1, wherein: In order to extract pedestrian features and object features of one or more objects surrounding the pedestrian from the sensor data, the at least one processor is also used to extract the pedestrian features and the object features using a feature extraction tool; the feature extraction tool includes at least one of facial recognition, gesture detection, movement detection, gait recognition, and detection of image pixels.

8. The system according to claim 1, characterized in that The one or more objects include a pedestrian traffic light that the pedestrian faces, wherein, in order to extract the object features of the one or more objects, the at least one processor is further configured to detect a state of the pedestrian traffic light.

9. The system according to claim 1, characterized in that The one or more objects include a crosswalk on which the pedestrians walk, wherein in order to extract the object features of the one or more objects, the at least one processor is further configured to detect a passing direction of the crosswalk.

10. The system according to claim 1, characterized in that The sensor data is acquired by a sensor installed on a vehicle traveling in an area where the pedestrian is walking, wherein the communication interface is also used to provide the vehicle with the predicted movement trajectory and movement speed of the pedestrian.

11. A method for predicting the movement trajectory of a pedestrian, characterized in that: include: receiving, via a communication interface, a map of an area in which the pedestrian is walking and sensor data associated with the pedestrian; locating the pedestrian on the map by at least one processor; extracting pedestrian features from the sensor data by the at least one processor; identifying, by the at least one processor, one or more objects surrounding the pedestrian based on the location of the pedestrian; extracting, by the at least one processor, object features of the one or more objects from the sensor data; determining, by the at least one processor, at least two candidate trajectories; Determine, by the at least one processor, a score for each of the candidate trajectories using a learning model based on the extracted pedestrian features and the object features; In response to the existence of a candidate trajectory having a score greater than a threshold, predicting a moving trajectory and a moving speed of the pedestrian based on one or more candidate trajectories greater than the threshold; In response to the absence of a candidate trajectory having a score greater than the threshold, Updating the angle of the lidar and / or adjusting the focus of the camera to obtain additional sensor data; The pedestrian characteristics and the object characteristics are determined based on the additional sensor data.

12. The method according to claim 11, characterized in that Before determining the score of each candidate trajectory, the method further includes: One or more candidate tracks that conflict with the pedestrian feature and the object feature are deleted.

13. The method according to claim 11, characterized in that The extracting pedestrian features and object features of one or more objects surrounding the pedestrian from the sensor data comprises: The pedestrian features and the object features are extracted using a feature extraction tool; the feature extraction tool includes at least one of facial recognition, gesture detection, movement detection, gait recognition, and detection of image pixels.

14. The method according to claim 11, characterized in that Extracting the object features further includes: detecting a state of a pedestrian traffic light facing the pedestrian; and The direction of travel of the pedestrian on the crosswalk is detected.

15. The method according to claim 11, characterized in that The sensor data is acquired by a sensor installed on a vehicle traveling in an area where the pedestrian is walking, wherein the method further includes providing the vehicle with the predicted movement trajectory and movement speed of the pedestrian.

16. A non-transitory computer-readable medium, characterized in that Instructions are stored thereon, and when the instructions are executed by at least one processor, the instructions cause the at least one processor to perform the following operations: receiving a map of an area in which a pedestrian is walking and obtaining sensor data associated with the pedestrian; locating the pedestrian on the map; extracting pedestrian features from the sensor data; identifying one or more objects surrounding the pedestrian based on the positioning of the pedestrian; extracting object features of the one or more objects from the sensor data; determining at least two candidate trajectories; Determine a score for each candidate trajectory using a learning model based on the extracted pedestrian features and the object features; In response to the existence of a candidate trajectory having a score greater than a threshold, predicting a moving trajectory and a moving speed of the pedestrian based on one or more candidate trajectories greater than the threshold; In response to there not being a candidate trajectory with a score greater than the threshold, Updating the angle of the lidar and / or adjusting the focus of the camera to obtain additional sensor data; The pedestrian characteristics and the object characteristics are determined based on the additional sensor data.

Images