CLASSIFICATION SYSTEM FOR A VEHICLE

Abstract

A computer-implemented procedure, when executed by data processing hardware, causes the data processing hardware to perform operations. These operations include acquiring image data by an imaging sensor, wherein the image data contains an object; acquiring radar points corresponding to the object by a radar sensor; and projecting the acquired radar points onto the acquired image data. The operations also include estimating key points of the object and / or one or more enveloping bodies of the object by a classification algorithm; classifying the object based on the estimated key points of the object or one or more enveloping bodies of the object; and executing a response function in response to the classified object and its position.

Description

INTRODUCTION

[0001] The information provided in this section serves the purpose of providing a general overview of the context of the disclosure. Neither the work of the inventors currently named, to the extent described in this section, nor those aspects of the description that could not otherwise qualify as prior art at the time of filing, are expressly or implicitly recognized as prior art against the present disclosure.

[0002] The present disclosure relates generally to a classification system and in particular to a classification system for a vehicle.

[0003] Vehicles often have interior monitoring systems, such as camera systems. These camera systems are frequently used to monitor driving behavior or detect the presence of other occupants. Occasionally, occupants sit in an unnatural position on a passenger seat, such that the occupant's position may be unsafe or otherwise inconsistent with the seat's design. While conventional camera systems can detect an improper occupant position, conventional vehicles typically lack the ability to assess an occupant's position. Therefore, there is a need for improved passenger monitoring within a vehicle interior. SUMMARY

[0004] In certain aspects, a computer-implemented procedure, when executed by data processing hardware, causes the data processing hardware to perform operations. These operations include acquiring image data by an imaging sensor, where the image data contains an object; acquiring radar points corresponding to the object by a radar sensor; and projecting the acquired radar points onto the acquired image data. The operations also include estimating key points of the object and / or one or more of the object's envelopes by a classification algorithm; classifying the object based on the estimated key points or envelopes; and executing a response function in response to the classified object and its position.

[0005] In some examples, the processes may include identifying radar points near an area of ​​the object's estimated key points and estimating a three-dimensional (3D) location of the object's estimated key points based on the identified radar points. The processes may also include determining the object's section lengths based on the 3D location and estimating an object classification based on these section lengths. In some cases, the processes may include using the classification algorithm to identify radar points that overlap with one or more estimated areas. The processes may also include creating 3D envelopes based on the identified radar points and estimating the dimensions of the 3D envelopes, including height, width, and depth.Optionally, the classification algorithm can classify the object based on the dimensions of its 3D envelope. In other examples, the response function can include adaptive restraints and / or airbag suppression and / or warnings and / or notifications. In some cases, the processes can include creating a digital image database.

[0006] According to further aspects, a vehicle classification system comprises data processing hardware and storage hardware that communicates with the data processing hardware. The storage hardware stores instructions which, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include acquiring image data by an imaging sensor, where the image data contains an object; acquiring radar points corresponding to the object by a radar sensor; and projecting the acquired radar points onto the acquired image data.The processes also include estimating key points of the object and / or one or more enclosing bodies of the object using a classification algorithm, classifying the object based on the estimated key points of the object or one or more enclosing bodies, and executing a response function in response to the classified object and a position of the object.

[0007] In some examples, the processes may include identifying radar points near an area of ​​the object's estimated key points and estimating a three-dimensional (3D) location of the object's estimated key points based on the identified radar points. The processes may also include determining the object's section lengths based on the 3D location and estimating an object classification based on these section lengths. Optionally, the processes may include using the classification algorithm to identify radar points that overlap with one or more estimated areas. In some cases, the processes may include creating 3D envelopes based on the identified radar points and estimating the dimensions of the 3D envelopes, including height, width, and depth.In some examples, classification by the classification algorithm involves classifying the object based on the dimensions of its 3D hulls. Optionally, the response function can include adaptive restraints and / or airbag suppression and / or warnings and / or notifications. The processes can also include the creation of a digital image database.

[0008] According to another aspect, a computer-implemented procedure, when executed by data processing hardware, causes the data processing hardware to perform operations. These operations include acquiring image data by an imaging sensor, where the image data contains an object; acquiring radar points corresponding to the object by a radar sensor; and projecting the acquired radar points onto the acquired image data.The processes also include estimating key points of the object and / or one or more of the object's enclosing bodies using a classification algorithm, classifying the object based on the object's key points or one or more of its enclosing bodies using a classification function of the classification algorithm, and executing a response function in response to the classified object and its position, where the response function includes adaptive constraints and / or airbag suppression. The processes further include issuing a warning to a vehicle's user interface in response to the executed response function and creating a digital image database using the classification algorithm.

[0009] In some examples, the processes may include identifying radar points near an area of ​​the object's estimated key points, estimating a three-dimensional (3D) location of the object's estimated key points based on the identified radar points, estimating object section lengths based on the 3D location, estimating an object classification based on the object section lengths, and identifying radar points that overlap with one or more estimated areas using the classification algorithm. The processes may also include creating 3D envelopes based on the identified radar points and estimating the dimensions of the 3D envelopes, including height, width, and depth.Optionally, the processes can include classifying the object using the classification algorithm based on the dimensions of the 3D envelope. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described here serve only to illustrate selected configurations and are not intended to limit the scope of this disclosure; they show: Fig. 1 a schematic representation of a vehicle equipped with a classification system according to the present disclosure; Fig. 2 an exemplary block diagram of a classification system according to the present disclosure; Fig. 3 a perspective partial view from above of a passenger compartment of a vehicle according to the present disclosure, wherein the passenger compartment contains occupants; Fig. 4 Another perspective partial view from above of the passenger compartment of Fig. 3, where one occupant is outside his position; Fig. 5 a schematic representation of a classification system according to the present disclosure, which executes a classification algorithm that includes radar points; Fig. 6 a further schematic representation of a classification system according to the present disclosure, which executes a classification algorithm that includes envelope bodies; Fig. 7 a perspective partial view of a passenger compartment of a vehicle according to the present disclosure, wherein the passenger compartment contains an object and an occupant; Fig. 8 a perspective partial view of the passenger compartment of Fig. 7, where the occupant leaves the vehicle and a classification system issues a notification; and Fig. 9 an exemplary flowchart of a classification system according to the present disclosure.

[0011] In all drawings, corresponding reference symbols denote corresponding parts. DETAILED DESCRIPTION

[0012] Exemplary configurations are now described in more detail with reference to the accompanying drawings. Exemplary configurations are provided to ensure that this disclosure is thorough and fully conveys its scope to those skilled in the art. Specific details, such as examples of specific components, devices, and processes, are presented to provide a precise understanding of the configurations of this disclosure. It is evident to those skilled in the art that specific details need not be used, that exemplary configurations can be embodied in many different forms, and that the specific details and exemplary configurations are not intended to limit the scope of the disclosure.

[0013] The terminology used here serves only to describe certain exemplary configurations and is not intended to be restrictive. As used here, the singular articles "a," "an," and "the" may be intended to include the plural forms unless the context clearly indicates otherwise. The terms "includes," "contain," and "exhibit" are inclusive and therefore establish the presence of features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more further features, steps, operations, elements, components, and / or groups thereof.The procedural steps, processes, and operations described herein are not intended to necessarily require their execution in the specific order discussed or illustrated, unless explicitly identified as such. Additional or alternative steps may be employed.

[0014] When an element or layer is described as "attached to," "intervening with," "connected to," "attached to," or "coupled to" another element or layer, it may be directly attached to, intervening with, connected to, attached to, or coupled to that other element or layer, or there may be intermediate elements or layers. Conversely, when an element is described as "directly attached to," "directly intervening with," "directly connected to," "directly attached to," or "directly coupled to" another element or layer, there need not be any intermediate elements or layers. Other words used to describe the relationship between elements should be interpreted similarly (e.g., "between" or "directly between," "adjacent" or "directly adjacent," etc.).As used herein, the expression “and / or” includes all combinations of one or more of the associated listed elements.

[0015] The terms "first," "second," "third," etc., can be used here to describe different elements, components, areas, layers, and / or sections. These elements, components, areas, layers, and / or sections are not intended to be limited by these terms. These terms can only be used to distinguish one element, component, area, layer, or section from another. Terms such as "first," "second," and other numerical terms do not imply any sequence or order unless clearly indicated by the context.Thus, a first element, a first component, a first area, a first layer or a first section discussed below can be referred to as a second element, a second component, a second area, a second layer or a second section without deviating from the instructions of the exemplary configurations.

[0016] In this application, including the definitions below, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor (shared, dedicated, or a group) that executes code; a working memory (shared, dedicated, or a group) that stores code executed by a processor; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0017] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes a processor that, in combination with additional processors, executes some or all of the code from multiple modules. The term "shared memory" includes a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium."The term "computer-readable medium" encompasses non-transient electrical and electromagnetic signals that propagate through a medium and can therefore be considered a physical, non-transient storage medium. Non-restrictive examples of non-transient storage include physical computer-readable media such as non-volatile memory, magnetic storage, and optical storage.

[0018] The devices and methods described in this application can be partially or completely implemented by one or more computer programs executed by one or more processors. The computer programs contain processor-executable instructions stored on at least one non-transient, computer-readable physical medium. The computer programs may also contain and / or access stored data.

[0019] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In certain examples, a software application may be called an "application," an "app," or a "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0020] Non-transient memory can be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transient memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (which is typically used, for example, for firmware such as boot programs).Examples of volatile memory include, but are not limited to, write / read memory (RAM), dynamic write / read memory (DRAM), static write / read memory (SRAM), phase change memory (PCM), and disks or tapes.

[0021] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in assembly / machine language. As used here, the terms "machine-readable medium" and "computer-readable medium" refer to a computer program product, a non-transient computer-readable medium, a device, and / or a setup (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, which includes a machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to a signal that is used to provide machine instructions and / or data to a programmable processor.

[0022] Various implementations of the systems and techniques described herein can be realized in digital electronics and / or an optical circuit arrangement, an integrated circuit arrangement, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include an implementation in one or more computer programs that are executable and / or interpretable in a programmable system comprising at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0023] The processes and logic operations described in this application can be performed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by working on input data and generating outputs. The processes and logic operations can also be performed by a special-purpose logic circuit arrangement, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing a computer program include, by way of example, general-purpose microprocessors, special-purpose microprocessors, and one or more processors of any type of digital computer. Generally, a processor receives instructions and data from read-only memory and / or read / write memory.The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes, or is functionally coupled to, a means of receiving data from and / or sending data to one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical media. However, a computer does not necessarily have to possess such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory); magnetic media (e.g., internal hard disks or removable media); magneto-optical media; and CD-ROM and DVD-ROM media.The processor and memory can be supplemented or incorporated by a logic circuit arrangement for a specific purpose.

[0024] To provide interaction with a user, one or more aspects of the disclosure may be implemented in a computer that has a display device, such as a CRT (cathode ray tube), an LCD (liquid crystal display), or a touchscreen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, with which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as...Visual, auditory, or haptic feedback; and input can be received from the user in any form, including auditory, verbal, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to an internet browser in a user's client device in response to requests received from the internet browser.

[0025] In the Fig. Figures 1-3 show a vehicle 100 equipped with a classification system 10. The classification system 10 is configured, in combination with a sensor system 200, to monitor a passenger compartment 102 of the vehicle 100. In some cases, the classification system 10 may be configured as part of a cloud-based server 300 and communicate with the sensor system 200 of the vehicle 100. For illustrative purposes, the classification system 10 is described as being executed in the vehicle 100. However, the classification system 10 can be executed in the vehicle 100, the cloud-based server 300, or a combination of the vehicle 100 and the cloud-based server 300 without deviating from the instructions described here.Regardless of the location of execution, the classification system 10 is configured to monitor and assess objects 400 within the passenger compartment 102 of the vehicle in combination with the sensor system 200. The objects 400 described herein may include, but are not limited to, occupants, crates, animals, or any other practical cargo that may be arranged in the passenger compartment 102. Furthermore, the objects 400 may be arranged arbitrarily within the passenger compartment 102 of the vehicle 100, such that the object 400 may be located arbitrarily within the seating arrangements, a cargo area, the floor, or any other practical location within the passenger compartment 102.

[0026] The classification system 10 includes a controller 12 configured to execute a classification algorithm 14. For example, the controller 12 includes data processing hardware 16 configured to execute the classification algorithm 14. The controller 12 also includes memory hardware 18 communicating with the data processing hardware 16. The memory hardware stores instructions which, when executed on the data processing hardware 16, cause the data processing hardware 16 to perform the operations described herein. The controller 12 is communicatively coupled to the sensor system 200. For example, the sensor system 200 includes an imaging sensor 202 configured to acquire image data 204 and a radar sensor 206 configured to acquire radar points 208. The sensor system 200 can be located at any practical location in the vehicle 100.For example, the sensor system 200 can be located at locations which, without being limited to, include the front, middle, rear, and side pillars of the vehicle 100. It is further provided that the sensor system 200 can use a single sensor 202, 206, or any practical number of sensors 202, 206, to acquire the image data 204 and the radar points 208. The image data 204 and the radar points 208 are communicated to the controller 12 and used by the classification algorithm 14, which is described in more detail below.

[0027] The image data 204 generally contains images of one or more objects 400 within the passenger compartment 102. The classification system 10 can use the image data 204 to identify different object positions 402, which are used by the classification algorithm 14 to determine whether a response function 20 should be executed. For example, shows Fig. 3 places an occupant 400 (e.g., an object 400) in a first position 402a and shows Fig. 4. The occupant 400 is in a second position 402b. The classification algorithm 14 is configured to identify the object 400 in the second position 402b as being outside its position relative to the first position 402b, as described in more detail below. The imaging sensor 202 acquires the image data 204 containing the object 400 and communicates the image data 204 to the controller 12. Simultaneously, the radar sensor 206 acquires the radar points 208 and communicates the radar points 208 to the controller 12. The classification algorithm 14 of the controller 12 receives both the image data 204 and the radar points 208 and uses both to determine the response function 20, which is assigned to a safety system 110 of the vehicle 100.

[0028] The safety system 110 can execute various safety functions 112 in response to the response function 20, which is executed by the classification system 10. In some cases, the response function 20 may include the safety functions 112, such that the response function 20 and the safety functions 112 may encompass the same or similar functions. For example, the response function 20 and the safety functions 112 may include, but are not limited to, adaptive restraint devices, airbag suppression, warnings, and / or notifications. The response function 20 is determined by the classification algorithm 14 and, when executed, causes the controller 12 to communicate the response function 20 to the safety system 110. In response to the received response function 20, the safety system 110 executes the corresponding safety function 112.For example, a warning 112a can be issued at a user interface 104 of the vehicle 100.

[0029] In the Fig. The classification algorithm 14 is configured in 2-6 to determine an object classification 22 based on the image data 204 and the radar points 208 received by the sensor system 200. The object classification 22 comprises classes 24 and 24a-n, which are used to categorize or classify the objects 400. Classes 24 and 24a-n can be stored in the memory hardware 18 for optional use by the classification algorithm 14. As mentioned above, the objects 400 can include children, adults, cargo, animals, etc. Each object 400 can undergo an object classification 22 to be classified into a corresponding class 24. Classes 24, 24a-n may, but are not limited to, include a Class 24a of an adult male, a Class 24b of an adult female, a Child Class 24c and a Cargo Class 24d.Each class 24, 24a-n can correspond to specific safety functions 112 assigned to each class. For example, the classification algorithm 14 can determine that object 400 is classified in a child class 24c and can execute the response function 20, which corresponds to the airbag suppression safety function 112. In other examples, the response function 20 can correspond to a notification safety function 112, reminding a driver to take the child with them when exiting the vehicle 100. The safety functions 112 can be cross-referenced for different classes 24, 24a-n, such that the safety functions 112 can overlap between classes 24, 24a-n and are not restricted to a specific class 24, 24a-n.

[0030] The classification algorithm 14 can be activated in response to the controller 12 receiving the image data 204 and the radar points 208. The classification algorithm 14 is configured to project the radar points 208 onto the image data 204. For example, the radar points 208 are three-dimensional (3D) points of the object 400, detected by the radar sensor 206, and contain spatial information about the detected radar points 208 in 3D space. The image data 204 is a two-dimensional (2D) representation of the object 400. The radar points 208 are projected from 3D space into the 2D image space of the imaging sensor 202. Once the radar points 208 are projected onto the image data 204, the classification algorithm 14 can execute a classification function 26.The classification function 26 is used to estimate, via an estimation function 28, the key points 30 of the object and / or one or more envelope bodies 32, which are described in more detail below.

[0031] With particular reference to the Fig. Figures 2-5 show the classification function 26 of the classification algorithm 14, where the projected radar points 208 are superimposed on the image data 204. The estimation function 28 analyzes the image data 204 to estimate the key points 30 of the object, which can correspond to different areas 404 of the object 400. For example, the classification function 26 can identify the object 400 as an occupant and identify key points 30 of the body (i.e., head point, shoulder points, elbow points, hip points, knee points, ankle points, etc.) that can correspond to the different areas 404 (i.e., head, shoulders, elbows, hips, knees, ankles, etc.) of the object 400. When the estimation function 28 is executed, the classification function 26 identifies the radar points 208 near the estimated key points 30 of the object.The identified radar points 208 can be a single point or multiple points that are closest to the key point 30 of the body in the image data 204. The classification function 26 uses the identified radar points 208 to estimate the 3D position of the key point 30 of the object 400's body.

[0032] For example, the classification function 26 can execute the estimation function 28 to estimate a 3D location of the object's estimated key points 30 based on the identified radar points 208 and the body's key point 30. The classification function 26 of the classification algorithm 14 can use the location of the object's key points 30 to determine the object's section lengths 38 using the estimated 3D positions of the identified key points 30. The object's section lengths 38 can generally correspond to the lengths of the object's various regions 404.In an example of an occupant 400, the classification algorithm 14 can, using the 3D positions of the identified key points 30 of the body, determine a section length 38 which, without being restricted thereto, includes a shoulder width, a torso height, a torso width, a hip width, a limb length and a torso-to-head height of the occupant 400.

[0033] The classification algorithm 14 can use the determined section lengths 38 to perform object classification 22 and classify object 400. For example, each class 24, 24a-n can store estimated ranges 40, 40a-n and estimated section ranges 42, 42a-n, which can be used by the classification function 26 when estimating object classification 22. The section lengths 38 determined by the classification algorithm 14 can be compared with the estimated section ranges 42, 42a-n of each class 24, 24a-n to classify object 400 (i.e., a child, an adult male, an adult female, etc.). Once object 400 is classified, the classification algorithm 14 can execute the response function 20 to trigger one or more safety functions 112.

[0034] With special reference to the Fig. Figures 2-4 and 6 show the classification function 26 of the classification algorithm 14 with the projected radar points 208 superimposed on the image data 204, where the areas 404 have been identified by the enclosing bodies 32 described below. As mentioned above, the classification function 26 can identify different areas 404 of the object 400 from the image data 204, and the radar points 208 are superimposed on the image data 204. The classification algorithm 14 evaluates the radar points 208 that are enclosed by an estimated area 404 relative to the image data 204. Using the spatial information of the radar points 208, the classification function 26 can construct 3D enclosing bodies 32 for each of the areas 404.

[0035] The estimation function 28 can also estimate the dimensions 50 of the 3D envelopes 32. The dimensions 50 can include the height, width, and depth of the 3D envelopes 32. For example, the radar points 208 can be used by the classification algorithm 14 to identify the dimensions 50 of the envelopes 32. The dimensions 50 of the envelopes 32 are used during the object classification 32 of the classification function 26, which is described below.

[0036] The dimensions 50 of the encapsulation bodies 32 can also be used by the classification algorithm 14 to determine the section length 38 of the object 400. As mentioned above, the determined section length 38 can be compared with the stored estimated section ranges 42, 42a-n for each of the estimated ranges 40, 40a-n. Based on the comparison of the dimensions 50 of the encapsulation bodies 32 with the estimated section lengths 42, 42a-n, the classification algorithm 14 can classify the object 400. Once the object 400 is classified into an object classification 22, the classification algorithm 14 can execute the response function 20, and the security system 110 can execute one or more security features 112 corresponding to the response function 20 and the corresponding class 24, 24a-n.

[0037] As mentioned above, classification system 10 can also be used to determine whether object 400 is outside a given position. Based on the spatial 3D positions of different areas 404 of object 400, object 400 can be detected in the first, normal position 402a and later in the second, anomalous position 402b.

[0038] For example, the anomalous position 402b can correspond to object 400 being outside of a position or outside of normal position 402a. Examples of anomalous position 402b include, but are not limited to, object 400 being at least partially outside vehicle 100, and the legs of an occupant 400 being on the dashboard of vehicle 100. In addition to the object classification 22 described above, the classification system 10 can use object positions 400 to evaluate object 400. For example, when determining object position 402, the classification system 10 can store normal position 402a and anomalous position 402b as references in memory hardware 18.

[0039] The classification system 10 can monitor the time period during which object 400 is located at the second position 402b to determine whether object 400 is temporarily repositioned or whether object 400 is outside of a position at the second position 402b. For example, a time threshold 56 can be stored on the memory hardware 18. The classification algorithm 14 can determine, based on the image data 204 and / or the radar points 208, that object 400 is located at the second position 402b. For example, the classification system 10 can include a spatial threshold 58, which is used by the classification algorithm 14 to identify a position classification 22a of object 400 that includes object position 402 (i.e., the normal position as opposed to the anomalous position 402a, 402b).If object 400 remains in second position 402b for a period of time exceeding the time threshold 56, the classification system 10 can issue a response function 20 to trigger a safety function 112.

[0040] As especially in the Fig. 2, Fig. 7 and Fig. As shown in Figure 8, the classification algorithm 14 can be used to detect, identify, and classify inanimate objects 400, such as cargo, in addition to the occupant objects 400 described above in general terms. For example, the Fig. 7 and Fig. 8. An object 400 is located on a passenger seat 106 of vehicle 100. The classification algorithm 14 can execute one of the processes described above (i.e., using the key points 30 of the object and / or the envelopes 32) to classify the object 400 and execute the response function 20 based on the class 24, 24d of the object 400. The example shown is Fig. Figure 8 shows a driver exiting vehicle 100 and receiving a warning 112a on user interface 104, corresponding to object 400. Warning 112a is triggered by classification algorithm 14, which classifies object 400 into load class 24d, with response function 20 triggering warning 112a from safety function 112. For example, warning 112a can remind the driver not to leave object 400 in vehicle 100.

[0041] As in the Fig. As shown in Figures 2-8, the classification algorithm 14 can also be configured to generate a digital inventory 60 of the objects 400 located inside the vehicle 100. The digital inventory 60 can be stored on the storage hardware 18 and contain a digital directory 62 of the identified objects 400 and their corresponding object classifications 22, as well as the location of the object 400. In some cases, when executing the classification algorithm 14, the controller 12 can refer to the digital inventory 60 to determine whether any of the objects 400 captured in the image data 204 are reflected in the digital inventory 60. If an object 400 is listed in the digital list 62, the classification algorithm 14 can retrieve the corresponding object classification 22. Therefore, the classification algorithm 14 can determine the class 24, 24a-n of object 400 by referring to the digital inventory 60.Furthermore, a user can access the digital inventory 60 to monitor the object 400, which can remain in the vehicle 100, as soon as the user leaves the vehicle 100.

[0042] Fig. Figure 9 shows an exemplary flowchart 900 of the classification system 10, which relates to the Fig. 1-9 refers to. In 902, the classification system 10 acquires image data 204 by an imaging sensor 202. The image data 204 contains an object 400. In 904, the classification system 10 acquires radar points 208 corresponding to the object 400 by a radar sensor 206. In 906, the classification system 10 projects the acquired radar points 208 onto the acquired image data 204. In 908, the classification algorithm 14 estimates the key points 30 of the object and / or one or more enveloping bodies 32 of the object. The classification system 10 classifies the object 400 in 910 using a classification function 26 of the classification algorithm 14 based on the key points 30 of the object or the one or more enveloping bodies 32 of the object. In 912, the classification system 10 executes a response function 20 in response to the classified object 400.Response function 20 includes adaptive restraint devices and / or airbag suppression. In 914, the classification system 10, in response to the executed response function 20, issues a warning 112a at a user interface 104 of a vehicle 100. The classification algorithm 14 creates a digital inventory 60 of the objects 400 in 916.

[0043] As in the Fig.As shown in Figures 1-9, the classification system 10 advantageously classifies the objects 400 into the object classification 22, which can be used to execute various safety functions 112 of a safety system 110. The safety functions 112 can be adapted based on the object classification 22 to correspond to the object 400. For example, if the object 400 is a child, the classification system 10 can execute a response function 20 corresponding to a child class 24, 24c, which can trigger safety functions 112 adapted to child classes 24, 24c (i.e., adaptive restraints, airbag suppression, etc.). Furthermore, the classification system 10 can present warnings and notifications (i.e., safety functions 112) to an occupant to remind the occupant to remove the object 400 (i.e., cargo, a child, etc.) from the vehicle 100 when exiting.The classification system 10 can advantageously employ two separate procedures for identifying and classifying the objects 400 (i.e., the key points 30 of the object or the encapsulation bodies 32 of the object). Ultimately, as outlined here, each of the procedures can be used to classify the objects 400 and to determine the position 402 of the object 400 within the vehicle 100.

[0044] Several implementations have been described. However, it should be understood that various modifications can be made without deviating from the concept and scope of the disclosure. Accordingly, further implementations fall within the scope of the following claims.

[0045] The preceding description is provided for illustrative and descriptive purposes only. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular configuration are generally not restricted to that particular configuration but are, where applicable, interchangeable and may be used in any chosen configuration, even if not specifically shown or described. They may also be varied in many ways. Such variations are not to be considered a deviation from the disclosure, and it is intended that all such modifications are included within the scope of the disclosure.

Claims

[1] A computer-implemented procedure which, when executed by data processing hardware, causes the data processing hardware to perform operations which include: Capturing image data by an imaging sensor, wherein the image data contains an object; Detection of radar points by a radar sensor, wherein the radar points correspond to the object; Projecting the detected radar points onto the captured image data; Estimating key points of the object and / or one or more of the object's envelopes using a classification algorithm; Classifying the object based on the estimated key points of the object or of one or more enclosing bodies of the object; and Executing a response function in response to the classified object and the object's position. [2] The method of claim 1, further comprising identifying the radar points near an area of ​​the estimated key points of the object and estimating a three-dimensional location (3D location) of the estimated key points of the object based on the identified radar points. [3] Method according to claim 2, further comprising determining section lengths of the object based on 3D localization and estimating an object classification based on the section lengths of the object. [4] The method of claim 1, further comprising the identification of radar points that overlap with one or more estimated areas by the classification algorithm. [5] The method of claim 4, further comprising creating 3D envelopes based on the identified radar points and estimating the dimensions of the 3D envelopes, wherein the dimensions comprise a height, a width and a depth of the 3D envelopes. [6] Method according to claim 5, wherein the classification by the classification algorithm comprises classifying the object based on the dimensions of the 3D envelopes. [7] Method according to claim 1, wherein the response function comprises adaptive restraint devices and / or airbag suppression and / or warnings and / or notifications. [8] Method according to claim 1, further comprising the creation of a digital image database. [9] Classification system configured to perform the method according to claim 1. [10] Vehicle containing the classification system according to claim 9.

Images