Vehicle cabin-based target detection and tracking

Abstract

A target tracking system for use in a vehicle interior of a motor vehicle includes a target sensor and an electronic control unit (ECU). A target sensor, such as a red, green, blue (RGB) camera, RGB infrared or RGB depth (RGBD) camera, is located in the interior of the vehicle and outputs a flow of sensor data of the seat surface. The ECU includes a processor and a non-transitory computer readable storage medium on which instructions are recorded, execution of the instructions by the processor causes the ECU to receive a flow of sensor data of the seat surface and segment the flow of sensor data into output files. The ECU also uses the output file, such as a child restraint system, to detect targets on the seat surface and classify the targets into target classes. The ECU then performs a control action in response to the target class.

Description

BACKGROUND

[0001] In the passenger compartment or cabin of a motor vehicle, vehicle seats are surrounded by or attached to one or more passenger restraint systems. For example, a given seat in a modern vehicle interior is equipped with at least one passenger restraint system in the form of a lap-and-shoulder seatbelt, an airbag, a seatbelt pretensioner, an adjustable headrest, a knee bolster, or an energy-absorbing device. Additional passenger restraint systems are specially configured for securing infants and toddlers within the vehicle interior and are thus referred to as child restraint systems.

[0002] Child restraint systems can take the form of rear-facing or forward-facing car seats to provide protection to infant, toddler, or slightly older child passengers based on their height, weight, and age. Rear-facing car seats are specially configured for carrying newborn and infant passengers. Once a child passenger has grown larger than the car seat, the child passenger can still use a booster seat to elevate the child relative to the seat surface. This in turn helps to ensure that the lap-and-shoulder belt is properly disposed across the child's torso. SUMMARY

[0003] The solutions described in detail below are collectively operable to perceive the position and orientation of a target within a vehicle cabin or interior and to track that position and orientation through a series of collected images or other sensor data. The target can include a child restraint system (CRS) in one or more example embodiments, with other targets in the form of objects or human or animal occupants being possible within the scope of the present disclosure. For example, implementations of the present teachings enable the identification and tracking of a cell phone, a purse, a package, a pet, or other predefined target that can be located on the surface of a vehicle seat and transported within the vehicle interior.

[0004] Detection as contemplated herein can occur within the vehicle interior using one or more target sensors, such as image sensors / cameras, radar sensors, Wi-Fi devices, ultrasonic sensors, structured light, ultra-wideband sensors, or other image-based or non-image-based sensors in different embodiments.

[0005] The target sensors are in communication with an electronic control unit (ECU) operable to perform the methods as set forth herein. As part of the methods and depending on the nature of the sensors used in a particular application, the ECU can perform data (e.g., image) processing and deep learning methods on the received stream of sensor data to correctly identify the target object and thereafter trigger one or more control responses as needed.

[0006] According to an example embodiment, a target tracking system for use in a vehicle interior of a motor vehicle includes a target sensor and an ECU. The target sensor is located in the vehicle interior and is configured to output a sensor data stream of a seat surface in the vehicle interior. The ECU is in communication with the target sensor and includes a processor and a non-transient computer-readable storage medium. Instructions are recorded on the computer-readable storage medium that, upon execution by the processor, cause the ECU to receive the sensor data stream of the seat surface, segment the sensor data stream into an output file, and use the output file to detect a target on the seat surface. The ECU also classifies the target into a target class and performs a control action in response to the target class.

[0007] The target on the seat surface can optionally include a child restraint system (CRS), the motor vehicle can include a plurality of airbags, and the function of the motor vehicle can include inflation of one or more of the airbags. In such embodiments, the target class includes an orientation of the CRS relative to the seat surface.

[0008] The target sensor includes, in one or more optional embodiments, a red, green, blue (RGB) camera, such as a RGB depth (RGBD) camera and / or a RGB infrared (RGB-IR) camera.

[0009] The ECU can include a fully convolutional neural network (FCN) that the ECU uses to segment the sensor data stream into the output file. In some implementations, execution of the instructions by the processor causes the ECU to segment the sensor data stream into the output file includes processing the sensor data stream via the FCN. In some implementations, execution of the instructions by the processor causes the ECU to: compute a segmentation mask (M) from the output file; and / or detect the target on the seat surface using the output file by extracting a largest connected region in the segmentation mask (M).

[0010] The ECU can be configured to: extract the largest connected region in the segmentation mask (M) using a depth-first search (DFS) to identify a cluster or region of pixels; and select the largest connected region from among the cluster or region of pixels.

[0011] Execution of the instructions by the processor can cause the ECU to perform the control action by selectively disabling the function of the motor vehicle based on the target class.

[0012] Aspects of the present disclosure include an external device in communication with the ECU. Execution of the instructions by the processor causes the ECU, in such embodiments, to: transmit a message to the external device.

[0013] Also disclosed herein is a method for tracking a target in a vehicle interior of a motor vehicle. Embodiments of the method include: receiving, via an ECU of the motor vehicle, a sensor data stream of a seat surface in the vehicle interior from a target sensor; and then segmenting, via a fully convolutional neural network (FCN) of the ECU, the sensor data stream into an output file. The method can include: detecting, using the output file from the FCN, a target on the seat surface; classifying, via the ECU, the target into a target class; and performing, via the ECU, a control action in response to the target class.

[0014] Also disclosed herein is a motor vehicle, embodiments of which include a vehicle body defining a vehicle interior and a target tracking system for use in the vehicle interior. The target tracking system can include: at least one target sensor connected to the vehicle body and located in the vehicle interior. The at least one target sensor is configured to output a sensor data stream of a seat surface in the vehicle interior, and wherein the target sensor includes an RGB camera. The target tracking system further includes an ECU in communication with the at least one target sensor. The ECU receives, in one or more implementations, the sensor data stream of the seat surface from the at least one target sensor, segments the sensor data stream into an output file using a fully convolutional neural network (FCN), and detects a target on the seat surface using the output file. The ECU also classifies the target into a target class, and thereafter performs a control action in response to the target class.

[0015] The above features and advantages of the present teachings, and other features and advantages, are readily apparent from the following detailed description, taken in connection with the accompanying drawings, from the claims, and from the appended Abstract, wherein: BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings incorporated in and forming a part of the specification, illustrate implementations of the present disclosure and serve to explain the principles of the present disclosure serve to explain the principles of the present disclosure.

[0017] Figure 1 A representative motor vehicle according to the present disclosure is schematically illustrated, having a vehicle interior and an electronic control unit (ECU) configured to identify and track a target in the vehicle interior.

[0018] Figure 2This illustration shows an exemplary use case in identifying and tracking targets. Figure 1 The flowchart of the ECU.

[0019] Figure 3A and 3B They depicted as in Figure 1 Representative targets detected inside vehicles and those with applied masks.

[0020] Figure 4 This is a flowchart illustrating an exemplary embodiment of a method for identifying and tracking targets inside a vehicle.

[0021] Figure 5 This is an illustration of a fully convolutional neural network for generating output data, according to a representative embodiment.

[0022] The accompanying drawings need not be drawn to scale, but may present simplified representations of various preferred features of the present disclosure as disclosed herein, including, for example, specific dimensions, orientations, positions, and shapes. Details associated with such features will be determined in part by the specific intended application and environment of use. Detailed Implementation

[0023] The components of the disclosed embodiments may be arranged in various configurations. Therefore, the following detailed description is not intended to limit the scope of this disclosure, but merely illustrates possible embodiments. Furthermore, although numerous specific details are set forth in the following description to provide a thorough understanding of various representative embodiments, some embodiments may be practiced without some of the details disclosed. Additionally, for the sake of clarity, certain technical materials understood in the related art have not been described in detail. Moreover, the disclosure illustrated and described herein may be practiced without the presence of elements not specifically disclosed herein.

[0024] Referring now to the accompanying drawings, which are found throughout several views, similar reference numerals indicate similar features. Figure 1 A mobile system is depicted in a non-limiting representative form, namely a motor vehicle 10. The motor vehicle 10 includes a target tracking system 11 for use within the vehicle interior 14 (i.e., the passenger compartment defined by the vehicle body 12). The motor vehicle 10 includes a set of road wheels (not shown) connected to the vehicle body. Although the motor vehicle 10 is depicted as being driven by an operator 13 and configured as a passenger vehicle, this teaching can be applied to other vehicles and mobile systems, including but not limited to fully autonomous vehicles such as self-driving taxis, rail vehicles, agricultural equipment, etc. For the sake of clarity only, this document will... Figure 1 Motor vehicle 10 is described in a non-restrictive representative scenario of passenger transport shown.

[0025] The vehicle interior 14 can be equipped with one or more passenger restraint systems, e.g., airbags 15 arranged about various vehicle seats 16. For the sake of simplicity, without limitation, two airbags 15 are illustrated in Figure 1 . That is, the airbags 15 can include various front (driver and passenger side), seat-mounted, overhead track-mounted, etc. In addition, a child restraint system (CRS) 18 can be secured to one or more of the vehicle seats 16 when an infant, toddler, or other child passenger 130 or 230 is present in the transport vehicle interior 14. Other passengers can be present in the vehicle interior 14 at various times, or the motor vehicle 10 can be used to transport packages 20 or other objects 25 (see Figure 2 ).

[0026] As set forth in detail below, the object tracking system 11 includes an electronic control unit (ECU) 50 configured to identify and track the CRS 18, packages 20, and / or other objects 25 (see Figure 2 ) with the aid of one or more object sensors 26. The object sensors 26 are in wired or wireless communication with the ECU 50 such that a sensor data stream CC 26 is provided to the ECU 50 for processing by resident control logic 100L. Thus, using the sensor data stream CC 26 and the control logic 100L, the ECU 50 is configured to identify and track detected objects 25 within the vehicle interior 14. That is, within the scope of the present disclosure, the ECU 50 is configured to execute a computer-readable / executable instruction set embodying a method 100, non-limiting example embodiments of which are described below with reference to Figure 4 . By executing the constituent steps or logic blocks of the method 100, the ECU 50 is able to generate an output signal CC O requesting or executing a control action based on the results of the method 100, if needed. Implementations of the method 100 are described below with particular reference to Figures 3A-5 a non-limiting use case in which the object 25 is a CRS 18.

[0027] Briefly referring to Figure 2 , Figure 1 , the ECU 50 of the object tracking system 11 also includes a set of object sensors (S) 26, which are also labeled C1,..., C n to represent Figure 1the possibility of more than one target sensor 26 within the vehicle interior 14. In one or more non-limiting exemplary implementations, the target sensor 26 can be embodied as an image sensor in the form of one or more red, green, blue (RGB) cameras, infrared (IR) cameras, RFG-IR cameras, RGB depth (RGBD) cameras adding depth information, e.g., via a depth map / image created by a time-of-flight, stereo, or another three-dimensional depth sensor, as appreciated in the art. Thus, the sensor data stream CC 26 may be human visible and / or belong to another category within the scope of the present disclosure. Other implementations as mentioned above can include, in different embodiments, radar sensors, lidar sensors, Wi-Fi devices, ultrasound sensors, ultra-wideband sensors, and / or other image-based or non-image-based sensors, and thus the image-based implementations described herein are intended to be representative and non-limiting.

[0028] The target 25 as observed and tracked by the ECU 50 within the scope of the present disclosure can vary with the intended application as mentioned above. For example, the target 25 can include Figure 1 a CRS 18 or a parcel 20. Other possible targets 25 include a cell phone 21 or other portable electronic device, a purse 22 or other hand-held object such as a handbag, a shopping bag, other human or animal passenger / occupant, etc., or other article that can be located Figure 1 on one of the vehicle seats 16. Ultimately, the ECU 50 identifies and tracks the target 25 and performs a suitable control action in response to certain criteria via an output signal (CC O ) to at least one external device (D) 28, where the configuration of the external device(s) 28 varies with the intended application.

[0029] For example, when the target 25 includes a CRS 18, the ECU 50 can prevent one or more airbags 15 Figure 1That is, rather than ECU 50 relying solely on weight-based detection of a possible occupant of a given vehicle seat 16, ECU 50 is informed via method 100 of the location and orientation of CRS 18 on vehicle seat 16 within vehicle interior 14. ECU 50 can also issue an alert at an appropriate time when motor vehicle 10 is turned off and one or more of the doors are opened, such as by displaying a message on a display screen while operator 13 is still seated, or causing a light to flash, a horn to sound or another audible alert, etc. Similar actions can be taken for other objects 25, for example, to alert a passenger that they can have left a package 20, cell phone 21, wallet 22 or other object 25 in vehicle interior 14 upon exiting vehicle interior 14. Such an exiting passenger can receive a generated "retrieve object" message or alert, for example.

[0030] As configured herein Figure 1 and 2 ECU 50 can be embodied as a microcontroller, electronic control unit, application specific integrated circuit(s) (ASIC), field programmable gate array (FPGA), electronic circuit(s), central processing unit(s) (e.g., microprocessor(s)), and associated transitory and non-transitory memory / storage components. ECU 50 is schematically depicted as having a processor 52 of one or more of such types, and a computer readable storage medium or memory 54. Memory 54 includes at least one tangible non-transitory computer storage medium (e.g., read only, programmable read only, solid state, random access, optical, magnetic, etc.). Computer readable instructions embodying method 100 Figure 4

[0031] Input / output circuitry and devices include analog to digital converters and associated devices that monitor inputs from object sensors 26 and other possible sensors, where such inputs are monitored at a preset sampling frequency or in response to a triggering event. Software, firmware, programs, instructions, control routines, code, algorithms and like terms mean controller executable instruction sets including calibration and lookup tables. Finally, ECU 50 outputs the above-mentioned control signals CC O .

[0032] Reference is now made to Figure 3A ​And consistent with the non-limiting example embodiments of CRS18, in one or more implementations, it is possible to capture Figure 1 and 2 Sensor data stream CC 26 As a real-time video feed, the collected sensor data stream CC in this embodiment... 26 In this context, child passenger 130 and / or 230 should be visible in most of the collected frames. However, child passenger 130 and / or 230 can move back and forth within CRS18, causing her position and orientation within CRS18 to change dynamically. Similarly, Figure 1 Other passengers or objects inside the vehicle 14 may temporarily obstruct [the view / control]. Figure 1 and 2 The field of view of one or more target sensors 26 makes child occupants 130 and / or 230, as well as CRS18, invisible in some representative image frames. In other words, child occupants 130 and / or 230 can be dynamic subjects as seen from the perspective of target sensors 26. In contrast, CRS18 is largely static.

[0033] Therefore, this scheme, as performed by ECU 50, identifies and tracks the static CRS18, instead of the potentially dynamic child passenger 130 and / or 230. To do this, ECU 50 uses resident image processing and machine learning capabilities to automatically perceive the position and orientation of the CRS18, while ECU 50 also monitors the data stream CC. 26 CRS18 is tracked frame by frame. Within the scope of method 100, such as... Figure 3B As shown in the masked area 30, CRS18 is effectively masked. Since the clear boundary of CRS18 captured by the masked area 30 forms a static background from the perspective of ECU 50, the movement and identity of child passengers 130 and / or 230 are not taken into account. This reduces the computational load on ECU 50.

[0034] Now turn to Figure 4 For clarity, method 100 is described as a series of process steps or logic blocks. Each of logic blocks B102-B120 can be constructed by... Figure 1 and 2 The processor 52 of the ECU 50 shown in the diagram executes during the driving cycle of the vehicle 10. For example, when a driving cycle is initiated via an on / off event of the vehicle 10, the ECU 50 may initialize, establish communication with the target sensor 26 on the controller area network or other suitable communication channel, and begin execution of method 100.

[0035] As explained above, it is provided for use inside the vehicle 14. Figure 1The object detection system includes one or more target sensors 26 located inside the vehicle interior 14. Each target sensor 26 is configured to output a sensor data stream CC of the seat surface 16S inside the vehicle interior 14 when searching for the presence or absence of CRS 18 or other targets 25. 26 At least part of it. As part of this effort, ECU 50 is configured to receive sensor data stream CC from the seat surface 16S. 26 , to transmit sensor data stream CC 26 Segmented into output file 46 ( Figure 5 ), and use output file 46 to detect on seat surface 16S. Figure 2 Target 25. Additionally, in this embodiment, ECU 50 is configured to classify target 25 into a target class (e.g., in...). Figure 1 In a non-limiting representative embodiment, CRS18 (given orientation) and in response to the target class, performs control actions such that output file 46 may include the target class.

[0036] A non-limiting exemplary embodiment of method 100, in which the target sensor 26 is configured as an image sensor or camera (or multiple sensors or an array thereof), begins at block B102, wherein the ECU 50 from... Figure 1 and 2 One or more target sensors 26, schematically shown in the diagram, receive sensor data stream CC. 26 (In this example, the image data stream). Box B102 may include receiving the sensor data stream CC in a specific category (e.g., RGB, IR, or depth) or in a combined category (e.g., RGB-IR, RGBD, etc.). 26 As mentioned above. When using other sensor types (such as radar, UWB, Wi-Fi, structured light, etc.), the specific scope will vary depending on the specific sensor type, as is understood in the art. Method 100 then proceeds to box B104.

[0037] Box B104 (image segmentation) includes: in this non-limiting embodiment where the target sensor 26 is an image sensor, the sensor data stream CC of box B102 is... 26 The data is fed into the image segmentation routine to generate output file 46. For example, for image (I) with height H and width W, where... Then O can be described as As appreciated in the art, the process of image segmentation is used in machine vision applications to partition a digital image into pixel sets when identifying and locating specific target objects within the image. Thus, block B104 can include pre-processing for noise reduction, contrast improvement, edge sharpening, color balancing, etc. To distinguish between constituent pixels or clusters of pixels in the image, ECU 50 can also extract identifying features of the CRS 18 (in this example, the child passenger 130 and / or 230) from other pixels. Each image in this embodiment is then segmented based on these identifying features, e.g., using edge detection methods or clustering, possibly followed by post-processing steps such as filtering for further noise reduction. The output file 46 is the result of block B104. Figure 3A

[0038] Briefly referring to Figure 5 in one or more embodiments, block B104 can be performed using a fully convolutional neural network (FCN) 40 as an exemplary machine learning (ML) technique. Here, the sensor data stream CC 26 is fed into an encoder layer 42 to a decoder layer 44, with the output file 46 ultimately being generated via the decoder layer 44. To this end, the FCN 40 is pre-trained with labeled images of the target 25. For example, when training the ECU 50 to detect the CRS 18, a training data set of different CRS 18 arranged in various positions, orientations, and possibly at some level of obstructions (from child passengers 130 and / or 230, coats, blankets, toys, etc.) can be used. The various respective encoder and decoder layers 42 and 44 can be constructed using suitable algorithms such as Fast R-CNN or Mask R-CN and trained on the labeled data set described above. Thus, block B104 pre-supposes the existence of a pre-trained model or FCN 40 for this purpose. In theory, multiple FCNs 40 can be used when detecting other targets 25. However, in practice, a single FCN 40 can be trained to segment pixels of different targets 25 in a scene, which can be more efficient and less computationally intensive than using multiple FCNs. Figure 2

[0039] As appreciated in the art, FCNs are a class of deep learning architectures often used to perform image segmentation tasks such as target detection as set forth herein. In contrast to convolutional neural networks that output a single class score for an entire input image, Figure 5 the contemplated FCN 40 assigns a class label to each pixel in a given input image. As Figure 2 ​​As shown in FIG. 2, FCN 40 includes one or more encoder layers 42, one or more decoder layers 44, and various skip connections 45A, 45B. The pre-trained encoder layer(s) 42 are operable to extract features from the input image, i.e., the encoder layer(s) 42 are pre-trained using a large-scale image classification task. The decoder layer(s) 44 then up-sample the feature maps from the encoder layer(s) 42 to produce an output file 46 as a segmentation map (M) as described below. Within FCN 40, the skip connections 45A, 45B combine feature maps from the encoder layer(s) 42 with corresponding feature maps from the decoder layer(s) 44 to facilitate recovery of spatial and contextual information that can be lost during encoding, as appreciated by one skilled in the art of FCNs.

[0040] With respect to the segmentation mask (M), the output file (O) 46 of FCN 40 is a three-dimensional (3D) tensor having a height (H), a width (W), and a predetermined number of possible classes. Each pixel location in the output file 46 contains a vector of class scores / probabilities for each of the classes to which a given pixel belongs. The argmax operation referred to below is used to select a particular target class having the highest probability or score for each pixel location, where the result of the argmax operation is a two-dimensional (2D) tensor, i.e., a width (W) and a height (H). Thus, the 2D tensor is the segmentation mask (M) herein, which in this exemplary embodiment assigns a class label to each image pixel. For example, as shown in FIG. 3, the segmentation mask (M) can be overlaid on the displayed image of the collected image to depict the detection results. For example, a readily distinguishable color such as yellow can be overlaid on the displayed image of the collected image such that the CRS 18 is clearly visible, or this can only occur in the logic of the ECU 50. Figure 3B With respect to the segmentation mask (M), the output file (O) 46 of FCN 40 is a three-dimensional (3D) tensor having a height (H), a width (W), and a predetermined number of possible classes. Each pixel location in the output file 46 contains a vector of class scores / probabilities for each of the classes to which a given pixel belongs. The argmax operation referred to below is used to select a particular target class having the highest probability or score for each pixel location, where the result of the argmax operation is a two-dimensional (2D) tensor, i.e., a width (W) and a height (H). Thus, the 2D tensor is the segmentation mask (M) herein, which in this exemplary embodiment assigns a class label to each image pixel. For example, as shown in FIG. 3, the segmentation mask (M) can be overlaid on the displayed image of the collected image to depict the detection results. For example, a readily distinguishable color such as yellow can be overlaid on the displayed image of the collected image such that the CRS 18 is clearly visible, or this can only occur in the logic of the ECU 50.

[0041] At block B106, the ECU 50 next computes a segmentation mask (M) from the output file 46. For example, the FCN 40 can generate a prediction in the form of a bounding box or segmentation mask (e.g., the masked region 30 of FIG. 2) via the output file 46. This occurs by finding the 2D tensor of size H x W with the largest score. Thus, the ECU 50 can compute the segmentation mask (M) as: Figure 5 Figure 3B M(x, y) = arg max O[x, y, :]; M e {0, 1} HxW The method 100 then continues to block B108.

[0042] Figure 4 ​​Box B108 needs to close or fill the holes in the segmented mask (M), for example, using morphological closure. To close the segmented mask M with respect to kernel K: Method 100 then proceeds to box B110. Box B110 includes: extracting the largest fully connected region to select the largest fully connected region in the segmented mask M from various pixel clusters or regions. As part of this effort, ECU 50 may apply depth-first search (DFS) to find pixels. The cluster / region. Once an unlabeled pixel is found, the ECU 50 can initiate a DFS from that pixel and recursively explore every neighboring pixel belonging to the same connected component (i.e., those with the same label or value in the segmented mask (M)). The ECU 50 can then calculate the maximum region as Method 100 then continues to box B111.

[0043] Still referencing Figure 4 In this embodiment of method 100, box B111 includes determining, via ECU 50, whether the area (A), height (H), and width (W) of the segmented mask (M) exceed a corresponding threshold (T). As an example, ECU 50 can calculate whether ∑ x，y [R * [x, y]≥T (in this case, R) * This corresponds to CRS18 (or, in other implementations, other target 25). When the area (A), height (H), and width (W) of the segmented mask (M) do not exceed their corresponding thresholds (T), method 100 then proceeds to box B112, and alternatively, when the area, height, and width of the segmented mask (M) exceed their corresponding thresholds (T), method 100 then proceeds to box B114.

[0044] At box B112, Figure 1 and 2 The ECU 50 determined that CRS18 was not present in the sensor data stream CC. 26 In response, given the absence of target 25, ECU 50 may refrain from performing a control action. For example, if ECU 50 is programmed to detect CRS18 and act accordingly when CRS18 is present, then given the absence of CRS18, ECU 50 can determine that such a control action is not necessary.

[0045] Using the airbag suppression example mentioned above, for instance, ECU 50 can transmit bitcode to another controller for... Figure 1The inflatable airbag 15 or other passenger restraints within the vehicle interior 14 fully realize the airbag deployment for the inflatable airbag 15 located near the monitored seat surface 16S. Similarly, the ECU 50 may not perform any control action or may perform a positive control action for other targets 25, such as indicating that the passenger has not placed target 25 on the seat surface 16S by illuminating a green light for disembarking passengers. Method 100 then completes and returns to box B102.

[0046] At box B114, Figure 1 and 2 The ECU 50 determined that CRS18 exists in the sensor data stream CC. 26 In the middle, box B114 may include: recording the bit code that confirms CRS18 resides in the segmented mask (M). In response, method 100 continues to box B116.

[0047] Box B116 includes: extracting encoder features (F) corresponding to the pixels of CRS18, i.e., from Figure 5 The features of FCN 40 described above. This feature (F) is available in the memory 54 of ECU 50 from the execution of the prior block B104. Method 100 then proceeds to block B118.

[0048] Figure 4 Box B118 includes classifying the orientation of the detected target 25 into a target class. For example, for a non-limiting example of CRS18, the target class could be forward or backward, or it could be an incorrect or undefined orientation. Other target classes could be used for other targets 25, such as the horizontal or vertical orientation relative to the seat surface 16S on which the target 25 is detected. However, for the use case of CRS18, the orientation of CRS18 can be particularly useful in informing control decisions because the orientation indicates the possible posture and facing position of child passengers 130 and / or 230 who may be located in CRS18. Infants are typically secured in a rear-facing seat, such as in the back seat, and therefore, the position and orientation of the detected CRS18 are important in determining how to respond. Once the ECU 50 determines the object class, method 100 then proceeds to box B120.

[0049] At box B120, ECU 50 performs a control action in response to the target class in box B118. Examples include disabling or suppressing. Figure 1 One or more of the inflatable airbags 15 are inflated, as mentioned above. When the ECU 50 detects the opening of the CRS 18 and one of the doors of the vehicle 10, the ECU 50 can... Figure 2The device 28 is used to activate audible and / or visual alarms to draw attention to the possible presence of child passenger 130 or another target 25.

[0050] Therefore, using the above reference Figures 1-5 The exposition of teachings aims to achieve the goal of Figure 1 A vision- and machine learning-based scheme for detecting, locating, and tracking a target 25 within the interior 14 of the vehicle shown is possible. The solution described below is highly invariant to lighting levels, background, passenger posture, passenger size, CRS orientation, and vehicle interior / model, and therefore can be used in a wide range of applications to provide the disclosed advantages. In addition to other accompanying advantages of this application, the owner or operator of the motor vehicle 10 can enjoy enhanced situational awareness of the current state of the target 25.

[0051] Regarding the operation of vehicle 10, ECU 50 can be based on Figure 4 The results of method 100 shown are used for seamless intervention. Figure 1 The control of the inflatable airbag 15 or other vehicle systems, such as by suppressing or preventing the inflation of the airbag 15, as mentioned above. Where appropriate, the ECU 50 can amplify the alarm to the owner / operator to draw attention to the seat surface 16S in response to the possible presence of the target 25. That is, instead of constantly alarming the operator to inspect the seat 16 for objects or passengers that may be excessive (e.g., for drivers who have never transported child passengers 130 and / or 230), this teaching can achieve a similar protective end in a less reckless manner. These and other potential advantages will readily be appreciated by those skilled in the art who now possess the advantages of this teaching.

[0052] Aspects of this disclosure have been described in detail with reference to the illustrated embodiments; however, those skilled in the art will recognize that many modifications can be made thereto without departing from the scope of this disclosure. This disclosure is not limited to the precise construction and composition disclosed herein; any and all modifications, alterations, and variations apparent from the foregoing description are within the scope of this disclosure as defined by the appended claims. Furthermore, this concept explicitly includes any and all combinations and sub-combinations of the foregoing elements and features.

Claims

1. A target tracking system for use in a vehicle interior of a motor vehicle, comprising: a target sensor configured to output a sensor data stream of a seating surface in the vehicle interior; and an electronic control unit (ECU) in communication with the target sensor, the ECU comprising: a processor; and a non-transient computer readable storage medium having instructions recorded thereon, wherein execution of the instructions by the processor causes the ECU to: receive the sensor data stream of the seating surface; segment the sensor data stream into an output file; use the output file to detect a target on the seating surface; classify the target into a target class; and perform a control action in response to the target class.

2. The target tracking system of claim 1, wherein the ECU comprises a fully convolutional neural network (FCN), and wherein execution of the instructions by the processor causes the ECU to segment the sensor data stream into the output file comprises processing the sensor data stream via the FCN.

3. The target tracking system of claim 1, wherein execution of the instructions by the processor causes the ECU to: compute a segmentation mask (M) from the output file.

4. The target tracking system of claim 1, wherein execution of the instructions by the processor causes the ECU to: detect a target on the seating surface using the output file by extracting a largest connected component in the segmentation mask (M).

5. The target tracking system of claim 4, wherein the ECU is configured to: extract the largest connected component in the segmentation mask (M) using a depth-first search (DFS) to identify a cluster or region of pixels; and select the largest connected component from among the cluster or region of pixels.

6. The target tracking system of claim 1, wherein the target sensor comprises a red, green, blue (RGB) camera.

7. The target tracking system of claim 6, wherein the RGB camera comprises an RGB depth (RGBD) camera.

8. The target tracking system of claim 6, wherein the RGB camera comprises an RGB infrared (RGB-IR) camera.

9. The target tracking system of claim 1, wherein execution of the instructions by the processor causes the ECU to: perform the control action in response to the target class by selectively disabling a function of the motor vehicle based on the target class.

10. The target tracking system of claim 9, wherein: the target on the seating surface comprises a child restraint system (CRS); the motor vehicle comprises a plurality of airbags; the function of the motor vehicle comprises inflation of one or more of the airbags; and the target class comprises an orientation of the CRS relative to the seating surface.

Images