OBJECT CLASSIFICATION METHOD AND SYSTEM FOR CONTROLLING AN AUTONOMOUS VEHICLE

Abstract

Object classification procedures, including: the receipt of sensor data (402) associated with an object observed by a sensor system (28) of an autonomous vehicle (10); determining a limiting curve (810, 820, 901, 902) associated with the sensor data (402) using a processor; determining a variety of boundary curve features based on a set of convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922) that are assigned to the boundary curve (810, 820, 901, 902); and classifying the object by applying the multitude of boundary curve features to a machine learning model and obtaining a classification output that classifies the object to assist in controlling the autonomous vehicle (10); where the boundary curve features include slopes (971, 972, 973, 974) between adjacent convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922).

Description

TECHNICAL AREA

[0001] The present disclosure relates generally to autonomous vehicles and in particular to systems and methods for classifying objects observed by various sensors of an autonomous vehicle. BACKGROUND

[0002] An autonomous vehicle (AV) is a vehicle capable of perceiving its surroundings and navigating with little or no user input. This is achieved through the use of sensor devices such as radar, lidar, image sensors, and the like. Autonomous vehicles also utilize information from global positioning systems (GPS), navigation systems, vehicle-to-vehicle communication, vehicle infrastructure technologies, and / or wired systems to aid in navigation.

[0003] While significant progress has been made in AVs in recent years, such systems could still be improved in a number of aspects. For example, it could be advantageous for an AV to be able to classify an object detected in its environment more precisely—for instance, regardless of whether the detected object is a person, a vehicle, or something else.

[0004] Accordingly, it is desirable to provide systems and methods capable of classifying objects detected in the environment more precisely. Furthermore, other desirable functions and features of the present invention will become apparent from the following detailed description and the accompanying claims, in conjunction with the accompanying drawings, as well as from the preceding technical field and background.

[0005] ABBAS, Ammar, et al. Stereo vision based pedestrian detection using B-spline modeling; In: 2008 IEEE International Conference on Vehicular Electronics and Safety. IEEE, 2008, pp. 63-68, describes an approach for pedestrian detection in the vehicle environment using a stereo view. Here, segmented pedestrians, represented as a series of segments with equal disparity, are refined using B-spline arithmetic. The segments are derived from a non-dense disparity map estimated based on a contour-following algorithm. The approximated segments are matched with an existing pedestrian model, taking into account attributes of the approximated segments such as concavity, convexity, and local curvature for the matching process. Further state of the art is described in STEIN, Simon Christoph, et al.Convexity based object partitioning for robot applications; In: 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014, pp. 3213-3220. SUMMARY

[0006] The object of the invention is to provide an improved object classification method and an improved system for controlling an autonomous vehicle.

[0007] To solve the problem, an object classification method with the features of claim 1 and a system with the features of claim 8 are provided. Advantageous embodiments of the invention can be found in the dependent claims, the description, and the drawings.

[0008] Systems and methods are provided for controlling an autonomous vehicle. In one embodiment, an object classification method involves obtaining sensor data associated with an object detected by a sensor system of an autonomous vehicle and determining a boundary curve associated with the sensor data using a processor. The method further involves determining a plurality of boundary curve features based on a set of convexities and concavities associated with the boundary curve and classifying the object by applying the plurality of boundary curve features to the machine learning model and obtaining a classification output that classifies the object to assist in controlling the autonomous vehicle. The boundary curve features include slopes between adjacent convexities and concavities.

[0009] In one embodiment, a vehicle control system includes an object classification module, including a processor. The object classification module is configured to receive sensor data associated with an object detected by a sensor system of an autonomous vehicle; to determine a boundary curve associated with the sensor data; to determine a plurality of boundary curve features based on a set of convexities and concavities associated with the boundary curve; and to classify the object by applying the plurality of boundary curve features to a machine learning model, which classifies the object to assist in controlling the autonomous vehicle. The boundary curve features include slopes between adjacent convexities and concavities. DESCRIPTION OF THE DRAWINGS

[0010] The exemplary embodiments are described below in conjunction with the following drawings, wherein the same reference numerals denote the same elements, and wherein: Fig. Figure 1 is a functional block diagram illustrating an autonomous vehicle with a control system according to various embodiments; Fig. Figure 2 is a functional block diagram representing a transport system with one or more autonomous vehicles. Fig. 1 illustrated according to different embodiments; Fig. Figure 3 is a functional block diagram illustrating an autonomous driving system (ADS) in conjunction with an autonomous vehicle according to various embodiments; Fig. 4 is a data flow diagram that represents an object classification system according to various embodiments; Fig. Figure 5 is a conceptual block diagram of an artificial neural network (ANN) according to various embodiments; Fig. 6 is a flowchart illustrating a control procedure for object classification according to different embodiments; Fig. Figure 7 illustrates various boundary curves that correspond to exemplary point clouds according to different embodiments; Fig. Figure 8 illustrates the boundary curves from Fig. 7 with various corresponding convexities and concavities; Fig. Figure 9 illustrates the extraction of features from a section of the exemplary boundary curves according to different embodiments; and Fig. Figure 10 illustrates the extraction of features from a section of exemplary boundary curves according to different embodiments. DETAILED DESCRIPTION

[0011] The following detailed description serves only as an example and is not intended to restrict application and use in any way. Furthermore, there is no intention to be bound by any theory explicitly or implicitly presented in the preceding technical section, background, summary, or the following detailed description.The term "module" as used herein refers to all hardware, software, firmware products, electronic control components, processing logic and / or processor devices, individually or in any combination, including, but not limited to, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an electronic circuit, a processor (shared, dedicated or group processor) and memory executing one or more software or firmware programs, a combinational logic circuit and / or other suitable components providing the described functionality.

[0012] Embodiments of this disclosure may be described herein as functional and / or logical block components and various processing steps. It should be noted that such block components may be composed of any number of hardware, software, and / or firmware components configured to perform the required functions. For example, an embodiment of this disclosure of a system or component may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, value tables, or the like, which can perform multiple functions under the control of one or more microprocessors or other control devices.Furthermore, experts in the field will recognize that the exemplary embodiments of the present disclosure can be used in conjunction with any number of systems, and that the system described herein is merely an exemplary embodiment of the present disclosure.

[0013] For the sake of brevity, conventional techniques relating to signal processing, data transmission, signaling, control, autonomous vehicles, machine learning, image analysis, neural networks, lidar, analytical geometry, and other functional aspects of the systems (and the individual operating components of the systems) will not be described in detail herein. Furthermore, the connecting lines shown in the various figures are intended to represent exemplary functional relationships and / or physical connections between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in an embodiment of this disclosure.

[0014] With reference to Fig. 1 is an object classification system (or simply “system”) 100 assigned to an autonomous vehicle (AV) 10 according to various embodiments. In general, the classification system 100 includes a machine learning (ML) model (e.g., a neural network) that is capable of classifying objects located near the vehicle 10 based on the boundary curves of these objects—e.g., the attributes of the various “bulges” and “indentations” (also referred to as convexities and concavities, respectively) captured in the boundary curve (or “contour”). According to the invention, these attributes or features include the slope of the boundary curve between adjacent convexities and concavities.These attributes, which can also include the distances between adjacent convexities and concavities, the elevation (based on an appropriate mass reference) of these convexities and concavities, and any other geometric features, could be used to train a neural network or other classification-type machine learning model. The resulting machine learning model can be distributed across any number of vehicles and can be automatically updated at regular or configurable intervals.

[0015] Referring to Fig. 1 comprises an autonomous vehicle (“AV” or simply “vehicle”) 10, generally comprising a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is mounted on the chassis 12 and essentially encloses the other components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The wheels 16-18 are each rotatably connected to the chassis 12 near a respective corner of the body 14.

[0016] In various embodiments, the vehicle 10 is an autonomous vehicle, and the object classification system 100 is integrated into the autonomous vehicle 10. The autonomous vehicle 10 is, for example, a vehicle that is automatically controlled to transport passengers from one place to another. In the illustrated embodiment, the vehicle 10 is depicted as a passenger car; however, it should be noted that any other vehicle, including motorcycles, trucks, sports vehicles (SUVs), recreational vehicles (RVs), ships, aircraft, etc., can also be used.

[0017] In an exemplary embodiment, the autonomous vehicle 10 corresponds to a Level 4 or Level 5 automation system according to the Society of Automotive Engineers (SAE) Standard Taxonomy of Automated Driving Levels “J3016”. Using this terminology, a Level 4 system denotes a “high degree of automation” with reference to a driving mode in which the automated driving system performs all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request for intervention. A Level 5 system, on the other hand, exhibits “full automation” and denotes a driving mode in which the automated driving system performs all aspects of the dynamic driving task under all road and environmental conditions that a human driver can handle.It is estimated that the embodiments according to the present subject matter are not limited to a specific taxonomy or category of automation. Furthermore, systems and methods according to the present embodiment can be used in conjunction with any autonomous vehicle that uses a navigation system to provide route guidance.

[0018] As shown, the autonomous vehicle 10 generally includes a drive system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one control unit 34, and a communication system 36. The drive system 20 may, in various embodiments, include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell drive system. The transmission system 22 is configured to transmit power from the drive system 20 to the vehicle wheels 16 and 18 according to selectable gear ratios. Depending on the embodiment, the transmission system 22 may include a gear-ratio automatic transmission, a continuously variable transmission, or another suitable transmission.

[0019] The braking system 26 is configured to provide a braking torque to the vehicle wheels 16 and 18. The braking system 26 can include, in various embodiments, friction brakes, bake-by-wire, a regenerative braking system such as an electric motor, and / or other suitable braking systems.

[0020] The steering system 24 influences the position of the vehicle wheels 16 and / or 18. While in some embodiments within the scope of the present disclosure it is shown for illustration as a steering wheel 25, the steering system 24 may not include a steering wheel.

[0021] The sensor system 28 includes one or more sensor devices 40a-40n that detect observable states of the external environment and / or the internal environment of the autonomous vehicle 10. The scanning devices 40a-40n may include, but are not limited to, radar devices, lidar, global positioning systems, optical cameras, thermal imaging cameras, ultrasonic sensors and / or other sensors.

[0022] The actuator system 30 includes one or more actuator devices 42a-42n that control, but are not limited to, one or more vehicle features, such as the drive system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the autonomous vehicle 10 may also include vehicle interior and / or exterior features not included in Fig. 1 shown, such as various doors, trunk and cabin equipment, such as air, music, lighting, touchscreen display components (as used in conjunction with navigation systems) and the like.

[0023] The data storage device 32 stores data for use in the automatic control of the autonomous vehicle 10. In various embodiments, the data storage device 32 stores defined maps of the navigable environment. In various embodiments, the defined maps are predefined and provided by a remote system (for further details regarding Fig. 2 described). For example, the defined maps can be assembled by the remote system and communicated to the autonomous vehicle 10 (wirelessly and / or wired) and stored in the data storage device 32. Route information can also be stored in the data storage device 32—that is, in a series of road segments (geographically linked to one or more of the defined maps) that together define a route the user can travel from a starting point (e.g., the user's current location) to a destination. As can be seen, the data storage device 32 can be part of the controller 34, separate from the controller 34, or part of the controller 34 and part of a separate system.

[0024] The controller 34 includes at least one processor 44 and a computer-readable memory device or media 46. The processor 44 can be a custom-designed or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU) among multiple processors connected to the controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a combination thereof, or generally any device for executing instructions. The computer-readable memory device or media 46 can include volatile and non-volatile memory in the form of read-only memory (ROM), direct-access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operating variables while the processor 44 is powered off.The computer-readable storage device or media 46 can be implemented using any number of known storage devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or any other electrical, magnetic, optical, or combined storage devices capable of storing data, some of which may be executable instructions used by the controller 34 in controlling the autonomous vehicle 10.

[0025] The instructions can include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by the processor 44, the instructions receive and process signals from the sensor system 28, perform logic, calculations, procedures, and / or algorithms for the automatic control of the components of the autonomous vehicle 10, and generate control signals that are transmitted to the actuator system 30 to automatically control the components of the autonomous vehicle 10 based on the logic, calculations, procedures, and / or algorithms. Although in Fig. While only one controller 34 is shown in Figure 1, embodiments of the autonomous vehicle 10 can include any number of controllers 34 that communicate and interact via a suitable communication medium or combination of communication media to process sensor signals, perform logic, calculations, procedures, and / or algorithms, and generate control signals to automatically control the functions of the autonomous vehicle 10. In one embodiment, as discussed in detail below, the controller 34 is configured to classify objects in the environment using a machine learning model that has been previously trained based on the type of boundary curve associated with such objects.

[0026] The communication system 36 is configured to wirelessly transmit information to and from other units 48, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote transport systems and / or user devices (with respect to Fig. 2 (described in more detail). In an exemplary embodiment, the wireless communication system 36 is configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard, via Bluetooth, or by means of mobile data communication. However, within the scope of this disclosure, additional or alternative communication methods, such as a dedicated short-range communication (DSRC) channel, are also considered. DSRC channels refer to one-way or two-way short-range to medium-range radio communication channels that have been specifically developed for the automotive industry and a corresponding set of protocols and standards.

[0027] With further reference to Fig. 2 in various embodiments, the autonomous vehicle 10, which with reference to Fig. As described in section 1, it should be suitable for use within a taxi or shuttle company in a specific geographic area (e.g., a city, a school or business campus, a shopping mall, an amusement park, an event center, or the like). For example, the autonomous vehicle 10 can be assigned to an autonomous vehicle-based transport system. Fig. Figure 2 illustrates an exemplary embodiment of an operating environment, generally shown at Figure 50, which includes an autonomous vehicle-based transportation system (or simply “remote transportation system”) 52, which, as with reference to Fig. 1 described, is assigned to one or more autonomous vehicles 10a-10n. In various embodiments, the operating environment 50 includes (which wholly or partially comprises the in Fig. 1 units 48 shown) furthermore one or more user devices 54 that communicate with the autonomous vehicle 10 and / or the remote transport system 52 via a communication network 56.

[0028] The communication network 56 supports communication between devices, systems, and components supported by the operating environment 50 (e.g., via physical communication links and / or wireless communication links). For example, the communication network 56 may include a wireless carrier system 60, such as a mobile phone system, comprising multiple cell towers (not shown), one or more mobile switching centers (MSCs) (not shown), and any other network components necessary to connect the wireless carrier system 60 to the fixed network. Each cell tower includes transmit and receive antennas and a base station, with the base stations of different cell towers connected to the MSCs, either directly or via intermediate devices, such as a base station controller.The Wireless Carrier System 60 can implement any suitable communication technology, such as digital technologies like CDMA (e.g., CDMA2000), LTE (e.g., 4G LTE or 5G LTE), GSM / GPRS, or other current or emerging wireless technologies. Other cell tower / base station / MSC configurations are possible and could be used with the Mobile Carrier System 60. For example, the base station and cell tower could be located in the same location or at a distance from each other; each base station could be responsible for a single cell tower; a single base station could serve multiple cell towers; or multiple base stations could be coupled to a single MSC, to name just a few of the possible configurations.

[0029] Apart from using the wireless carrier system 60, a second wireless carrier system in the form of a satellite communication system 64 can be used to provide unidirectional or bidirectional communication with the autonomous vehicle 10a-10n. This can be done using one or more communication satellites (not shown) and an up-facing transmitting station (not shown). Unidirectional communication can include, for example, satellite radio services, in which programmed content data (news, music, etc.) is received from the transmitting station, packaged for uploading, and then sent to the satellite, which broadcasts the programming to the subscribers. Bidirectional communication can include, for example, satellite telephone services, which use the satellite to relay telephone communications between the vehicle 10 and the station.Satellite telephony can be used either in addition to or instead of the mobile network operator system 60.

[0030] A fixed-line communication system 62 can include a conventional fixed-line telecommunications network connected to one or more fixed-line telephones and linking the wireless carrier system 60 to the remote transport system 52. For example, the fixed-line communication system 62 can be a public switched telephone network (PSTN) such as that used to provide fixed-line telephony, packet-switched data communications, and internet infrastructure. One or more segments of the fixed-line communication system 62 could be implemented using a standard wired network, a fiber optic or other optical network, a cable network, power lines, other wireless networks such as wireless local area networks (WLANs) or networks providing wireless broadband access (BWA), or any combination thereof.Furthermore, the remote transport system 52 does not need to be connected via the fixed-line communication system 62, but could include radio telephone equipment so that it can communicate directly with a wireless network, such as the wireless carrier system 60.

[0031] Although in Fig. 2. While only one user device 54 is shown, embodiments of the operating environment 50 can support any number of user devices 54, including multiple user devices 54 that are owned, operated, or otherwise used by a person. Each user device 54 supported by the operating environment 50 can be implemented using a suitable hardware platform. In this respect, the user device 54 can be realized in a common form factor, including: a desktop computer; a mobile computer (e.g., a tablet computer, a laptop computer, or a netbook computer); a smartphone; a video game console; a digital media player; a component of a home entertainment device; a digital camera or video camera; a wearable computing device (e.g., a smartwatch, smart glasses, smart clothing); or the like.Each user device 54 supported by the operating environment 50 is implemented as a computer-implemented or computer-aided device with the hardware, software, firmware, and / or processing logic required to perform the various techniques and procedures described herein. For example, the user device 54 includes a microprocessor in the form of a programmable device containing one or more instructions stored in an internal memory structure, which are used to receive binary inputs and produce binary outputs. In some embodiments, the user device 54 includes a GPS module that can receive GPS satellite signals and generate GPS coordinates based on these signals.In further embodiments, the user device 54 includes cellular communication functionality, enabling the device to conduct voice and / or data communications over the communication network 56 using one or more cellular communication protocols, as described herein. In various embodiments, the user device 54 includes a visual display, such as a graphical touchscreen display or other display.

[0032] The remote transportation system 52 includes one or more backend server systems (not shown) that may be cloud-based, network-based, or resident at the specific campus or geographic location served by the transportation system 52. The remote transportation system 52 may be staffed by a live advisor, an automated advisor, an artificial intelligence system, or a combination thereof. The remote transportation system 52 may communicate with the user devices 54 and the autonomous vehicles 10a-10n to schedule trips, relocate autonomous vehicles 10a-10n, and the like. In various embodiments, the remote transportation system 52 stores account information, such as subscriber authentication data, vehicle registration numbers, profile records, biometric data, behavioral patterns, and other relevant subscriber information.In one embodiment, as described in more detail below, the remote transport system 52 includes a route database 53 that stores information relating to navigation system routes.

[0033] According to a typical use case workflow, a registered user of the remote transportation system 52 can create a ride request via the user device 54. The ride request typically specifies the desired passenger pick-up location (or current GPS location), the desired destination (which may identify a predefined vehicle stop and / or a user-defined passenger destination), and a pick-up time. The remote transportation system 52 receives the ride request, processes the request, and dispatches a selected autonomous vehicle 10a-10n (if and when available) to pick up the passenger at the designated pick-up location and time. The transportation system 52 can also generate and send a suitably configured confirmation message or notification to the user device 54 to inform the passenger that a vehicle is en route.

[0034] As can be seen, the subject matter disclosed herein offers certain improved features and functions for what may be considered a standard or baseline autonomous vehicle 10 and / or an autonomous vehicle-based transportation system 52. For this purpose, an autonomous vehicle-based transportation system may be modified, extended, or otherwise augmented to provide the additional functions described in more detail below.

[0035] According to various embodiments, the control unit 34 implements an autonomous driving system (ADS) 70, as shown in Fig. 3 shown. This means that suitable software and / or hardware components of the controller 34 (e.g. the processor 44 and the computer-readable storage medium 46) are used to provide an autonomous drive system 70 that is used in conjunction with the vehicle 10.

[0036] In various embodiments, the instructions of the autonomous drive system 70 can be structured according to function or system. For example, the autonomous drive system 70 can be structured as shown in Fig. Figure 3 shows a sensor fusion system 74, a positioning system 76, a steering system 78, and a vehicle control system 80. As can be seen, the instructions can be divided into any number of systems (e.g., combined, further subdivided, etc.) in various embodiments, since the disclosure is not limited to the examples shown.

[0037] In various embodiments, the sensor fusion system 74 synthesizes and processes sensor data and predicts the presence, location, classification and / or movement of objects and features in the vehicle's environment 10. In various embodiments, the sensor fusion system 74 can incorporate information from multiple sensors, including but not limited to cameras, lidars, radars and / or any number of other types of sensors.

[0038] The positioning system 76 processes sensor data together with other data to determine a position (e.g., a local position relative to a map, a precise position relative to a road lane, vehicle direction, speed, etc.) of the vehicle 10 in relation to its environment. The guidance system 78 processes sensor data together with other data to determine a route that the vehicle 10 is to follow. The vehicle control system 80 generates control signals to steer the vehicle 10 according to the determined route.

[0039] In various embodiments, the controller 34 implements machine learning techniques to support the functionality of the controller 34, such as feature recognition / classification, obstacle mitigation, route crossing, mapping, sensor integration, ground truth determination, and the like.

[0040] As briefly mentioned above, the system is 100% Fig. 1 is able to classify objects located near the vehicle 10 based on the boundary curves of these objects. In various embodiments, classifications are performed based on the attributes of the various "bulges" and "indentations" (also referred to as convexities and concavities, respectively) detected in the object's boundary curve. According to the invention, these attributes or features include the slope of the boundary curve between adjacent convexities and concavities, and may include the distance between adjacent convexities and concavities, the slope of these convexities and concavities, and any other geometric features that can be used to train a neural network or other classification-type machine learning model.

[0041] Fig. Figure 4 is a data flow diagram illustrating various embodiments of the system 100 that can be embedded in the controller 34. With reference to Fig. Figure 5 includes an exemplary system, generally comprising an object classification module 420, which receives sensor data 402 relating to the vehicle's environment (e.g., camera images, lidar data, or any other sensor data obtained from the sensor system 28) and, as its output 403, provides a determination regarding the class (or category) of the various objects detected in the vehicle's environment. In various embodiments, the module 420 implements a machine learning model that has been previously trained using a corpus of exemplary images (e.g., exemplary images of known object types).

[0042] It is understood that different embodiments of the system 100 according to the present disclosure can include any number of submodules embedded in the controller 34. As can be seen, all modules / submodules that are in Fig. 4 shown, combined and / or further subdivided to perform the various procedures described herein in a similar manner. Inputs to the system 100 can be received from the sensor system 28, which are received by other control modules (not shown) assigned to the autonomous vehicle 10, which are received by the communication system 36, and / or by other submodules (not shown) within the control system 34 of Fig. 1. Module 420 can, for example, be determined / modeled in one of the various in Fig. The 3 illustrated modules will be implemented.

[0043] The object classification module 420 can implement a variety of machine learning methods, such as an artificial neural network that uses a set of images that have been previously captured and stored (e.g., in Server 53 from Fig. 2) is trained. In this respect, Fig. 5 A conceptual overview of a configuration of an artificial neural network 500 that could be used in conjunction with various embodiments. In general, ANNs, such as the one described in Fig. Figure 5 shows mathematical models (implemented using any suitable combination of hardware and / or software) that mimic, to some degree, the neural structure of the cerebral cortex of mammals.

[0044] The ANN 500 includes a set of input nodes 501 (e.g., 501a-501n), a set of output nodes 504 (e.g., 504a-504n), and one or more interconnected node layers configured as "hidden layers" (in this case, two hidden layers: nodes 502a-502n and nodes 503a-503n). Each node layer 502, 503, and 504 receives inputs from preceding layers via a network of weighted connections (as arrows in the diagram). Fig. (5 illustrated). Each node has a corresponding "activation function," which generally varies depending on the specific application. The input (in the form of optical patterns, numerical features, etc.) is presented to ANN 500 via the input layer (node ​​501), which is connected to hidden layers (nodes 502 and 503), with the actual "learning" occurring through training. The hidden layers (nodes 502 and 503) are connected to an output layer (node ​​504).

[0045] The ANN 500 is "trained" via the learning rule, which modifies the connection weights according to the input patterns provided to input layer 501. This allows the ANN 500 to learn from examples. Such learning can be supervised (with known examples as input), unsupervised (with uncategorized examples as input), or involve reinforcement learning (where a reward is provided during training). Once the neural network is "trained" to a satisfactory level, it can be used as an analytical tool to make predictions.This means that new inputs are presented to the input node 501, into which they are filtered, and processed by the middle layers 502-503 as if the training were taking place; however, at this point, the output of one execution of the propagation forward is the predicted model for the input data, which can then be used for further analysis and interpretation.

[0046] As will be described in more detail below, the ANN 500 can be made from Fig. 5 can be used to select the classification module 420 from Fig. 4. This is to be implemented by accepting inputs 501 corresponding to the attributes of the various convexities and concavities of an observed boundary curve of the object, and creating a classification of the one or more observed objects as an output (504). For example, output 504a may correspond to a "human" classification (along with some confidence values ​​associated with that classification), output 504b may correspond to a "motorcycle" classification, and output 504c may correspond to a "vehicle" classification, and so on. However, these classifications are not intended to be restrictive and may include any number of object types typically observed by the AV 10 during operation, such as cars, trucks, strollers, bicycles, people, dogs, light poles, and the like.

[0047] It is understood that the present embodiments are not limited to the ANN model 500 described above. A variety of machine learning techniques can be used, including, for example, other artificial neural networks such as recurrent neural networks (RNNs), as well as random forest classifiers, Bayesian classifiers (e.g., naive Bayes), principal component analysis (PCA), support vector machines, linear discriminant analysis, and the like.

[0048] Fig. Figure 6 is a flowchart that represents a control procedure for object classification according to various embodiments and is now presented in conjunction with Fig. 1-5 as well as Fig. 7-10 described. In general, and as further described below, illustrated. Fig. 7 different boundary curves, which correspond to exemplary point clouds according to different embodiments. Fig. Figure 8 illustrates the boundary curves from Fig. 7 with various corresponding convexities and concavities, and the Fig. Figures 9-10 illustrate the extraction of features from a section of the exemplary boundary curves according to different embodiments.

[0049] Referring to Fig. 6. A tax procedure 600 can be carried out by the system 100. Fig. 1 in conjunction with object classification module 420 from Fig. 4. As will be evident from the disclosure, the order of operation within Procedure 600 is not limited to sequential execution as described in Fig. 6 is illustrated, but can be carried out in one or more different orders, as is applicable and in accordance with the present disclosure.

[0050] The control procedure 600 begins at 601 with the training of a machine learning model based on a set (or "corpus") of training data. As in conjunction with the ANN 500 from Fig. As described in section 5, this corpus of training data can include a large set of images containing objects that are highly likely to be found by AV 10 (e.g., various vehicles, people, etc.), along with known classifications associated with these images. This training step 601 generally involves determining the boundary curve of the objects and extracting features, which will be described in more detail below. Once the ANN has been trained (which is generally done within a system located outside of AV 10), the model can be provided to one or more vehicles (e.g., AV 10) (e.g., via the communication network 56).

[0051] Next, sensor data is acquired by the AV 10 at 602. As mentioned above, the sensor data can include any available data acquired by the sensor system 28 during the operation of the AV 10. In various embodiments, optical camera data and lidar data are of particular interest with regard to the classification procedure described above. However, all available sensor data can be used.

[0052] A boundary curve, which is assigned to the acquired sensor data, is then determined at 603. As used herein, the term "boundary curve" refers to a curve that essentially corresponds to the contour of an object within the field of view of the sensors used to acquire the sensor data. Fig. Figure 7 illustrates, for example, a lidar cloud 710 corresponding to the return from the rear of a vehicle, alongside a lidar cloud 720 corresponding to the return of a pedestrian (as might be seen from the viewpoint of AV 10). An exemplary boundary curve 701, as substantially illustrated, corresponds to the contour of the vehicle's lidar point cloud 710, while a boundary curve 702 essentially corresponds to the contour of a human lidar cloud 720.

[0053] While the in the Fig. 7 and Fig. While the boundary curves 701 and 702 shown in Figure 8 are illustrated as closed curves, the subject matter described herein is not so restricted. In some embodiments, the boundary curve may only partially encompass an object and therefore be an open curve. Depending on the nature of the object, the type of sensor data, and other factors, the boundary curve may be continuous or non-continuous, smooth (i.e., continuously differentiable) or non-smooth, rectilinear (e.g., polygonal) or curved, etc. Furthermore, the boundary curve need not necessarily include all sensor data associated with an object and may be an approximation derived from a suitable algorithm known in the art (e.g., edge detection algorithms used in the field of optical image analysis).Accordingly, the type of a boundary curve can generally vary widely, as long as suitable features of the curve can be extracted to effectively train a machine learning model to classify the relevant objects.

[0054] Subsequently, the boundary curve determined at 603 is characterized at 604 based on convexities and concavities captured within the boundary curve (i.e., the features of the boundary curve are extracted). With further reference to Fig. 8. The limiting curves 701 and 702 are formed from... Fig. Figure 7 is shown as boundary curves 810 and 820. In these examples, boundary curves 810 and 820, as described above, are closed, non-self-intersecting continuous loops in a plane (e.g., "simple closed curves" or "Jordan" curves) which, as will become apparent in the figures, have corresponding interior and exterior regions. Each of boundary curves 810 and 820 can be characterized by a corresponding centroid 831 and 832.

[0055] In various embodiments, the boundary curves 810 and 820 are characterized by their corresponding sets of concavities and convexities (also referred to as "indentations" and "bulges," respectively). The concavities generally correspond intuitively to the curve segments that are "opening outwards" (i.e., away from the interior of the boundary curve), while the convexities generally correspond to those curve segments that are "opening inwards" (i.e., towards the interior of the boundary curve).

[0056] Referring to the limiting curve 810 from Fig. Figure 8 can, for example, represent a number of alternating convexities and concavities, including convexities 811, 813, 815, and 817, and concavities 812, 814, 816, and 818. For the sake of clarity (taking into account their asymmetry), only half of the boundary curve 810 is characterized in this figure. Similarly, the boundary curve 820 includes convexities 821, 823, 825, 827, and 829, and concavities 822, 824, 826, and 828.

[0057] In accordance with various embodiments, the properties of the convexities and concavities assigned to the boundary curves 810 and 820 are extracted in a manner that allows a properly trained ANN to distinguish between a vehicle (represented by boundary 810) and a human (represented by boundary curve 820). The extracted features assigned to a given boundary curve can be expressed as a vector of real numbers, integers, or other appropriate variable types, which serve as inputs to the ML model (e.g., inputs 501a-n from Fig. 5) be used, be represented.

[0058] In some embodiments (using the boundary curve 810 as a non-restrictive example), the extracted features for each concavity and convexity (811-818) include the elevation (i.e., the distance between along the z-axis) of the corresponding concavity / convexity. Thus, convexity 811 has a greater elevation than concavity 813, convexity 813 has a greater elevation than convexity 815, and so on. This feature intuitively corresponds to the observation that certain classes of objects will generally be "larger" than other objects (as seen by AV 10). This means that the concavities / convexities in a conventional, mid-sized sedan will generally have a greater elevation than those of a typical baby carriage.

[0059] According to the invention, the slope between adjacent convexities / concavities is used as an extracted feature. This feature intuitively corresponds to the idea that certain objects (such as humans) generally contain a series of closely spaced and “deep” concavities, while others (such as motor vehicles) generally contain less closely spaced concavities. In this respect, the term “widely spaced” denotes a greater distance along the boundary curve. Consider, for example, a point 841, as depicted on the boundary curve 810, moving around the boundary curve 810 (in this case counterclockwise). The parametric distance traveled to this point, while progressing around the boundary curve 810, can be referred to herein as a distance s (to distinguish it from other distances within the Fig. (to distinguish it from the Cartesian frame shown in section 8). The same applies to point 842 of the boundary curve 820, as illustrated.

[0060] The nature of such "gradients" is further described in Fig. Figure 9 illustrates two exemplary boundary curves: 901 (top) and 902 (bottom). For the sake of simplicity, the curves are shown in Fig. Figure 9 is shown as "unwrapped" and placed in a Cartesian coordinate system rather than a polar coordinate system, as this might be more natural for a boundary curve that broadly surrounds an object. This means that while the slopes below can be expressed in the context of a Cartesian coordinate system (e.g., dz / dx), it will be understood that the same principles of curvature and slope also apply in a polar coordinate system (e.g., dr / dθ).

[0061] The boundary curve 901, which can correspond to the contours of a vehicle, has an inner region 932 and an outer region 931, and includes convexity 911, concavity 912, and convexity 913. Similarly, the boundary curve 902, which can correspond to the contours of a human being, has an inner region 942, an outer region 941, convexity 921, concavity 922, and convexity 923.

[0062] Referring to the limiting curve 901 from Fig. Figure 9 shows that the transition from convexity 911 to concavity 912 can be characterized by a tangent vector 971 having an angle θ1, and the transition from concavity 912 to convexity 913 can be characterized by a tangent vector 972 having an angle θ2. Similarly, with reference to the boundary curve 902, it can be seen that the transition from convexity 921 to concavity 922 can be characterized by a tangent vector 973 having an angle θ3, and the transition from concavity 922 to convexity 923 can be characterized by a tangent vector 974 having an angle θ4.

[0063] Thus, it is evident that the slopes 971 and 972 vary less than those of slopes 973 and 974 (and are of a smaller order of magnitude). Furthermore, the distance between adjacent convexities (i.e., as separated by an intervening concavity) is generally also greater in boundary curve 901 than in boundary curve 902. Therefore, the magnitude of these slopes and distances (and others between successive convexities and concavities) can be used as extractable features to be fed into ANN 500.

[0064] Fig. Figure 10 further shows another way to define the boundary curves 901 and 902, which are in Fig. 9 can be represented to characterize. This means that the radius of the curve at points along each curve 901, 902 can be calculated using standard algorithms, which, for example, can fit a circle at each inflection point and then use the radius of each circle to characterize the curvature. In Fig. For example, the boundary curve 901 is characterized by circles 951, 952, and 953, while the boundary curve 902 is characterized by circles 954, 955, and 956. The curvatures of these circles can be used as extracted features for the ANN 500 from Fig. 5 can be used. As in Fig. In simplified terms, the radii of circles 951, 952, and 953 are on average larger than those of circles 954, 955, and 956.

[0065] With renewed reference to Fig. In step 6, the object is classified by the ML model using the features extracted in step 604. As above in conjunction with Fig. As mentioned in section 5, in one embodiment this classification generally involves feeding the extracted boundary curve features into the inputs 501 of the ANN 500. Fig. 5, and then using output 504 to determine the washing unity that different classes of objects are within the field of view of the AV 10 as it moves within its surroundings.

Claims

[1] Object classification procedures, including: the receipt of sensor data (402) associated with an object observed by a sensor system (28) of an autonomous vehicle (10); determining a limiting curve (810, 820, 901, 902) associated with the sensor data (402) using a processor; determining a variety of boundary curve features based on a set of convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922) that are assigned to the boundary curve (810, 820, 901, 902); and classifying the object by applying the multitude of boundary curve features to a machine learning model and obtaining a classification output that classifies the object to assist in controlling the autonomous vehicle (10); where the boundary curve features include slopes (971, 972, 973, 974) between adjacent convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922). [2] Method according to claim 1, wherein the machine learning model is an artificial neural network model. [3] Method according to claim 1, further comprising transmitting the machine learning model to the autonomous vehicle (10) via a communication network (56). [4] Method according to claim 1, wherein the boundary curve features include the elevation of each of the convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922). [5] Method according to claim 1, wherein the boundary curve features include the radius of the curve of each of the convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922). [6] Method according to claim 1, wherein the boundary curve features include the distance between adjacent convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923). [7] Method according to claim 1, wherein the sensor data are lidar data and the boundary curve (810, 820, 901, 902) substantially corresponds to the contour of the lidar data. [8] System for controlling an autonomous vehicle (10), comprising: an object classification module (420) including a processor configured to: Receiving sensor data (402) associated with an object observed by a sensor system (28) of an autonomous vehicle (10); Determining a boundary curve (810, 820, 901, 902) that is assigned to the sensor data (402); Determining a variety of boundary curve features based on a set of convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922) associated with the boundary curve (810, 820, 901, 902); and Classifying the object by applying the multitude of boundary curve features to a machine learning model; where the boundary curve features include slopes (971, 972, 973, 974) between adjacent convexities (811, 813, 815, 817, 821, 823, 825, 827, 829, 911, 913, 921, 923) and concavities (812, 814, 816, 818, 822, 824, 826, 828, 912, 922). [9] System according to claim 8, wherein the machine learning model is an artificial neural network model.

Images