Vehicle interior camera system

Abstract

An example vehicle interior camera system includes a memory configured to store at least one Gaussian process image segmentation model; a vehicle interior camera; and a vehicle control module configured to obtain an image of an interior of a vehicle, extract one or more image cuts surrounding a region of a target object in the image, detect and segment a passenger mask in the image, supply the image to the at least one Gaussian process image segmentation model, and determine a target object in the at least one Gaussian process image segmentation model. The method includes receiving a passenger mask from a target object to obtain a region mask associated with a region surrounding the target object, removing the passenger mask from the region mask to extract a target object mask, identifying a target object condition from the target object mask, and controlling at least one vehicle system or feature in response to the identification of the target object condition.

Description

[0001] introduction

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent it is described in this section, and aspects of this description that may not otherwise qualify as prior art at the time of filing, are not admitted, either expressly or impliedly, to be prior art against the present disclosure. Technical Field

[0003] The present disclosure generally relates to vehicle interior camera systems, including a process for identifying a target object mask for images captured by a vehicle interior camera. Background Art

[0004] Interior vehicle cameras capture images of objects within the vehicle's interior. Computer vision and image processing algorithms can use masks to identify objects, but human annotation services for labeling objects to train image processing models are expensive, time-consuming, and may require sharing sensitive passenger data with third-party annotation companies. Summary of the Invention

[0005] An example vehicle interior camera system includes: a memory configured to store at least one Gaussian process image segmentation model; a vehicle interior camera configured to capture an image of at least a portion of an interior of a vehicle; and a vehicle control module configured to: obtain an image of at least a portion of an interior of a vehicle from the vehicle interior camera, extract one or more image crops of a region surrounding a target object in the image, detect and segment an occupant mask in the image, supply the image to the at least one Gaussian process image segmentation model to obtain a region mask associated with the region surrounding the target object, remove the occupant mask from the region mask to extract the target object mask, identify a target object condition based on the target object mask, and control at least one vehicle system or feature in response to the identification of the target object condition.

[0006] In some examples, the vehicle control module is configured to output a target object mask to train a machine learning model to identify target objects in an image sequence.

[0007] In some examples, the target object includes at least one of a child restraint system (CRS) and a passenger seat belt.

[0008] In some examples, controlling a vehicle system or feature includes inhibiting deployment of an air bag in response to detecting a child in a child restraint system.

[0009] In some examples, the system includes a human-machine interface (HMI) of the vehicle, wherein controlling the vehicle system or feature includes generating an alert via the HMI, and the alert includes at least one of: a warning not to place a child in the front seat of the vehicle, a warning that a child is still in the vehicle after the key fob has left the vehicle, or a warning to passengers to wear seat belts while the vehicle is in motion.

[0010] In some examples, the vehicle control module is configured to obtain a time series of k images from a vehicle interior camera and generate a background motion model based on an image crop from each of the k images.

[0011] In some examples, the vehicle control module is configured to apply a background motion model to the temporal sequence of k images to generate a foreground mask.

[0012] In some examples, the vehicle control module is configured to remove the foreground mask from an image captured via a vehicle interior camera.

[0013] In some examples, removing the foreground mask from the image occurs before removing the passenger mask from the region mask to extract the target object mask.

[0014] In some examples, the vehicle control module is configured to determine whether a background motion model update condition is satisfied, and in response to the background motion model update condition being satisfied, update the background motion model based on a specified number of recent image frames.

[0015] In some examples, the background motion model update condition includes at least one of processing a specified number of image frames since a last background motion model update and detecting movement of a target object across a sequence of image frames exceeding a threshold movement distance.

[0016] In some examples, the vehicle control module is configured to estimate a homography matrix between adjacent frames in a temporal sequence of k images, and transform a target object mask from a first adjacent frame to a next adjacent frame using the homography matrix.

[0017] An example method for processing a vehicle interior camera image includes obtaining an image of at least a portion of an interior of a vehicle from a vehicle interior camera, extracting one or more image crops of a region surrounding a target object in the image, detecting and segmenting an occupant mask in the image, supplying the image to at least one Gaussian process image segmentation model to obtain a region mask associated with the region surrounding the target object, removing the occupant mask from the region mask to extract a target object mask, identifying a target object condition based on the target object mask, and controlling at least one vehicle system or feature in response to the identification of the target object condition.

[0018] In some examples, the target object mask is supplied to a machine learning model to train the machine learning model to identify the target object in an image sequence.

[0019] In some examples, the target object includes at least one of a child restraint system (CRS) and a passenger seat belt.

[0020] In some examples, controlling a vehicle system or feature includes inhibiting deployment of an air bag in response to detecting a child in a child restraint system.

[0021] In some examples, controlling a vehicle system or feature includes generating an alert via a human-machine interface (HMI), and the alert includes at least one of: a warning not to place a child in the front seat of the vehicle, a warning that a child is still in the vehicle after the key fob has left the vehicle, or a warning to passengers to wear seat belts while the vehicle is in motion.

[0022] In some examples, the method includes obtaining a time series of k images from a vehicle interior camera, and generating a background motion model based on an image crop from each of the k images.

[0023] In some examples, the method includes applying a background motion model to a temporal sequence of k images to generate a foreground mask.

[0024] In some examples, the method includes removing a foreground mask from an image captured via a vehicle interior camera before removing the passenger mask from the region mask to extract the target object mask.

[0025] The present disclosure may also include the following technical solutions:

[0026] 1. A vehicle interior camera system comprising:

[0027] a memory configured to store at least one Gaussian process image segmentation model;

[0028] a vehicle interior camera configured to capture an image of at least a portion of an interior of the vehicle; and

[0029] A vehicle control module is configured to:

[0030] obtaining an image of at least a portion of an interior of the vehicle from an interior vehicle camera;

[0031] extracting one or more image crops of a region surrounding a target object in the image;

[0032] Detect and segment passenger masks in images;

[0033] supplying the image to the at least one Gaussian process image segmentation model to obtain a region mask associated with a region surrounding the target object;

[0034] removing the passenger mask from the region mask to extract the target object mask;

[0035] identifying a target object condition based on the target object mask; and

[0036] At least one vehicle system or feature is controlled in response to the identification of the target object condition.

[0037] 2. A vehicle interior camera system according to technical solution 1, wherein the vehicle control module is configured to output a target object mask to train a machine learning model to identify target objects in an image sequence.

[0038] 3. A vehicle interior camera system according to technical solution 1, wherein the target object includes at least one of a child restraint system (CRS) and a passenger seat belt.

[0039] 4. A vehicle interior camera system according to technical solution 3, wherein controlling the vehicle system or feature includes inhibiting the deployment of an airbag in response to detecting a child in a child restraint system.

[0040] 5. The vehicle interior camera system according to claim 3, further comprising a vehicle human-machine interface (HMI), wherein:

[0041] controlling a vehicle system or feature including generating an alert via a human-machine interface; and

[0042] The alert includes at least one of a warning not to place a child in a front seat of the vehicle, a warning that a child is in the vehicle after the key fob has left the vehicle, or a warning to passengers to wear seat belts while the vehicle is in motion.

[0043] 6. The vehicle interior camera system according to technical solution 1, wherein the vehicle control module is configured to:

[0044] Obtain a time series of k images from a vehicle interior camera; and

[0045] A background motion model is generated from an image crop from each of the k images.

[0046] 7. A vehicle interior camera system according to technical solution 6, wherein the vehicle control module is configured to apply a background motion model to a time series of k images to generate a foreground mask.

[0047] 8. The vehicle interior camera system according to technical solution 7, wherein the vehicle control module is configured to remove the foreground mask from the image captured by the vehicle interior camera.

[0048] 9. A vehicle interior camera system according to technical solution 8, wherein removing the foreground mask from the image occurs before removing the passenger mask from the area mask to extract the target object mask.

[0049] 10. The vehicle interior camera system according to technical solution 6, wherein the vehicle control module is configured to:

[0050] determining whether a background motion model update condition is satisfied; and

[0051] In response to the background motion model update condition being satisfied, the background motion model is updated based on a specified number of latest image frames.

[0052] 11. A vehicle interior camera system according to technical solution 10, wherein the background motion model update condition includes at least one of the following: processing a specified number of image frames since the last background motion model update, and detecting movement of a target object across a sequence of image frames exceeding a threshold movement distance.

[0053] 12. The vehicle interior camera system according to technical solution 6, wherein the vehicle control module is configured to:

[0054] estimating a homography matrix between adjacent frames in a temporal sequence of k images; and

[0055] The target object mask is transformed from the first adjacent frame to the next adjacent frame using a homography matrix.

[0056] 13. A method for processing an image from a camera inside a vehicle, the method comprising:

[0057] obtaining an image of at least a portion of an interior of the vehicle from an interior vehicle camera;

[0058] extracting one or more image crops of a region surrounding a target object in the image;

[0059] Detect and segment passenger masks in images;

[0060] supplying the image to at least one Gaussian process image segmentation model to obtain a region mask associated with a region surrounding the target object;

[0061] removing the passenger mask from the region mask to extract the target object mask;

[0062] identifying a target object condition based on the target object mask; and

[0063] At least one vehicle system or feature is controlled in response to the identification of the target object condition.

[0064] 14. The method according to technical solution 13 further includes supplying the target object mask to the machine learning model to train the machine learning model to identify the target object in the image sequence.

[0065] 15. A method according to technical solution 13, wherein the target object includes at least one of a child restraint system (CRS) and a passenger seat belt.

[0066] 16. The method of claim 15, wherein controlling the vehicle system or feature includes inhibiting deployment of an airbag in response to detecting a child in a child restraint system.

[0067] 17. The method according to technical solution 15, wherein:

[0068] controlling a vehicle system or feature including generating an alert via a human machine interface (HMI); and

[0069] The alert includes at least one of a warning not to place a child in a front seat of the vehicle, a warning that a child is in the vehicle after the key fob has left the vehicle, or a warning to passengers to wear seat belts while the vehicle is in motion.

[0070] 18. The method according to technical solution 13 further comprises:

[0071] Obtain a time series of k images from a vehicle interior camera; and

[0072] A background motion model is generated from an image crop from each of the k images.

[0073] 19. The method according to technical solution 18 further includes applying a background motion model to a time series of k images to generate a foreground mask.

[0074] 20. The method according to technical solution 19 further includes removing a foreground mask from an image captured by a vehicle interior camera before removing the passenger mask from the region mask to extract the target object mask.

[0075] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims, and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The present disclosure will become more fully understood from the detailed description and accompanying drawings.

[0077] Figure 1is a diagram of an example vehicle including a vehicle interior camera.

[0078] Figure 2 is a flow chart depicting an example process for identifying a target object mask for an image captured by a vehicle interior camera.

[0079] Figure 3 is a flow chart depicting an example process for identifying a target object mask by removing the foreground from an image.

[0080] Figure 4 is a flow chart depicting an example process for identifying a target object mask using a homography matrix.

[0081] Figure 5A and 5B is a graphical representation of an example neural network for predicting target object masks.

[0082] Figure 6 is a flowchart illustrating an example process for training a machine learning model.

[0083] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0084] In some examples, image processing algorithms and deep learning models can automatically extract pixel masks of target objects in images and videos captured by vehicle interior cameras, such as target object masks for child restraint systems (CRS). The extracted target object masks can be used to train machine learning models to improve object detection within the vehicle interior and can be used to modify the operation or functionality of one or more systems of the vehicle (such as suppressing airbag deployment when a child is detected in a child restraint system).

[0085] In some examples, the target object mask detection algorithm can be used for daytime (e.g., color) images and video, nighttime (e.g., infrared (IR)) images and video, monochrome images (including depth maps), etc. The target object localization and mask segmentation algorithm can be invariant to vehicle seat position, vehicle seat height, and orientation of the child restraint system (or other target object).

[0086] Now refer to Figure 1 , the vehicle 10 includes front wheels 12 and rear wheels 13. Figure 1 , drive unit 14 selectively outputs torque to front wheels 12 and / or rear wheels 13 via drive lines 16 and 18, respectively. Vehicle 10 may include different types of drive units. For example, the vehicle may be an electric vehicle, such as a battery electric vehicle (BEV), a hybrid vehicle, or a fuel cell vehicle, a vehicle including an internal combustion engine (ICE), or other types of vehicles.

[0087] Some examples of drive unit 14 may include any suitable electric motor, a power inverter, and a motor controller configured to control a power switch within the power inverter to adjust motor speed and torque during propulsion and / or regeneration. The battery system provides power to or receives power from the electric motor of drive unit 14 via the power inverter during propulsion or regeneration.

[0088] Although the vehicle 10 is Figure 1 1 includes a drive unit 14, but the vehicle 10 can have other configurations. For example, two separate drive units can drive the front wheels 12 and rear wheels 13, one or more individual drive units can drive individual wheels, etc. As can be appreciated, other vehicle configurations and / or drive units can be used.

[0089] The vehicle control module 20 can be configured to control the operation of one or more vehicle components, such as the drive unit 14 (e.g., by controlling the torque setting of the electric motor of the drive unit 14). The vehicle control module 20 can receive inputs for controlling the vehicle components, such as signals received from a steering wheel, an accelerator pedal, a brake pedal, etc. The vehicle control module 20 can monitor the vehicle's telematics, such as vehicle speed, vehicle position, vehicle braking and acceleration, etc., for safety purposes.

[0090] The vehicle control module 20 can receive signals from any suitable components for monitoring one or more aspects of the vehicle, including one or more vehicle sensors (such as cameras, microphones, pressure sensors, steering wheel position sensors, brake sensors, position sensors such as global positioning system (GPS) antennas, wheel height and / or position sensors, accelerometers, etc.). Some sensors can be configured to monitor the current motion of the vehicle, the acceleration of the vehicle, the braking of the vehicle, the current steering direction of the vehicle, the current height and / or position of one or more wheels, etc. In some examples, the vehicle interior camera 22 is configured to capture images of the interior of the vehicle 10, which can include capturing images of a target object 24 (e.g., a child restraint system).

[0091] The vehicle control module 20 can communicate with another device via a wireless communication interface, which may include one or more wireless antennas for transmitting and / or receiving wireless communication signals. For example, the wireless communication interface can communicate via any suitable wireless communication protocol, including but not limited to vehicle-to-everything (V2X) communication, Wi-Fi communication, wireless area network (WAN) communication, cellular communication, personal area network (PAN) communication, short-range wireless communication (e.g., Bluetooth), etc. The wireless communication interface can communicate with a remote computing device via one or more wireless and / or wired networks. With respect to vehicle-to-vehicle (V2X) communication, the vehicle 10 may include one or more V2X transceivers (e.g., V2X signal transmission and / or reception antennas).

[0092] Figure 2 is a flow chart depicting an example process for identifying a target object mask for an image captured by a vehicle interior camera. The process may be performed by, for example Figure 1 The process begins at 204 by obtaining an image including a target object, such as obtaining an image including a target object 24 from a vehicle interior camera 22 .

[0093] At 208, the process includes extracting an image crop of one or more regions surrounding the target object. For example, if the target object is a car seat (e.g., a child restraint system (CRS)), the process may crop the region of interest around the car seat.

[0094] At 212, the process includes passing the image through a Gaussian process (GP) segmentation algorithm to obtain a region mask. The GP segmentation algorithm can be any suitable model that uses a Gaussian process to generate a probability distribution of two-dimensional (2D) pixel regions based on the image, which can produce consistent and consistent image segmentations with the presence of salient objects of interest in the image. In other examples, other suitable image processing and segmentation algorithms can be used.

[0095] At 216, the process includes detecting and segmenting the passenger mask. For example, image processing and segmentation algorithms can be applied to the image to detect passengers (e.g., children sitting in car seats), and the passengers can be identified via segmentation.

[0096] The process removes the passenger mask from the region mask to extract the target object mask at 220. For example, frame differencing can be used to remove portions of the passenger mask from an image where the target object mask remains (e.g., a child restraint system mask where the child mask portion is removed).

[0097] At 224, the process includes determining whether the identified target object mask will be used for machine learning model training. For example, automated detection of target object masks using the example processes described herein can facilitate the generation of training data for a computer vision deep learning model to avoid or reduce the need for manual identification of target objects (such as child restraint systems) by human reviewers.

[0098] If the target object mask is used for model training at 224, control proceeds to 228 to train an object detection machine learning model using the extracted target object mask. The machine learning model can be trained for any suitable purpose, such as improving the identification of the presence of a specific target object in real time based on a sequence of captured images from a vehicle interior camera. In some examples, the machine learning model can be trained to predict or localize the target pixel mask. The machine learning model can include, but is not limited to, a segmentation model, a convolutional neural network configured to perform segmentation as a downstream task, and the like.

[0099] At 232 , the process includes determining whether the object mask is used for vehicle control. For example, detection of the presence of a target object may be used to control the operation of one or more vehicle systems or features, to generate an alert, or the like.

[0100] If the target object mask is to be used for vehicle control at 232 , control proceeds to 236 to determine whether a target object condition is identified or satisfied based on the target object mask. If so, control proceeds to 240 to control a vehicle system or feature based on the identified object condition.

[0101] In some examples, the target objects may include a child restraint system, a passenger seat belt, the presence of a child in the vehicle, etc. Example vehicle systems or features that may be controlled based on the identified presence of the target object in an image from an interior vehicle camera may include, but are not limited to, changing air bag deployment settings (such as inhibiting air bag deployment when a child is detected seated in a child restraint system), issuing a warning via the vehicle's human machine interface (HMI) when a passenger is detected not wearing a seat belt, a child is seated in the front seat of the vehicle, a child remains in the vehicle after the key fob has left the vehicle within a specified threshold proximity distance, etc.

[0102] Figure 3 is a flow chart depicting an example process for identifying a target object mask by removing the foreground from an image. The process may be performed by, for example Figure 1 The process begins at 304 by obtaining a temporal sequence of image frames, such as a plurality of images including a target object 24 from a vehicle interior camera 22. The images may be captured at any suitable interval, such as every 100 milliseconds, every 500 milliseconds, every 1 second, every 5 seconds, every minute, etc.

[0103] At 308, control extracts image crops of the area surrounding the target object. For example, each frame in the image sequence can be individually cropped. Then, at 312, control uses the first K crops to create a background motion model. For example, control can use any suitable background motion modeling algorithm to identify moving objects and / or static objects in the background of the image over time. The K image crops can be any suitable number, such as 10 crops, 50 crops, 100 crops, etc.

[0104] At 316, the process includes selecting image frame K+i. At 320, control then uses the background model up to frame K+i-1 to obtain a foreground mask. For example, control may use the background motion model on a new frame (e.g., a recently captured image from a vehicle interior camera) to generate a foreground mask, such as by identifying portions of the new image that were not part of a previously identified background in a prior image sequence.

[0105] At 324, control determines whether background update conditions are met. If so, control updates the background motion model based on a specified number of recent image frames (such as the ten most recent image frames, the fifty most recent image frames, etc.) at 328. Example background update conditions may include, but are not limited to, processing a specified number of image frames since the last background motion model update, detecting movement of a target object across the image frame sequence exceeding a threshold movement distance, a specified time period having elapsed since the last background motion model update, etc. In some examples, the background motion model may be updated when, for example, a child restraint seat or booster seat is displaced due to movement of the child.

[0106] At 332, the process includes removing the foreground from the scene. For example, after obtaining the foreground mask at 320, the foreground mask can be used to remove the portion of the image corresponding to the foreground mask at 332. Then, at 336, control can optionally extract the target object mask (e.g., after removing the child from the foreground of the image, the car seat object can be identified in the background). After removing the foreground from the image at 332, the image can be re-imaged by performing, for example, Figure 2 The target object mask is extracted by the process of

[0107] At 340, control determines whether there are any additional frames to process, such as subsequent frames in the temporal sequence of images. For example, the process may first identify the target object and then continuously monitor for the presence of the object in the background. If control determines that there are additional frames to process, control selects the next image frame in the sequence at 344 and returns to 320 to obtain a foreground mask for the selected frame (e.g., frame i) using the background model up to frame K+i-1.

[0108] Figure 4 is a flow chart depicting an example process for identifying a target object mask using image registration via estimating a homography matrix. The process may be performed by, for example Figure 1 The process begins at 404 by obtaining a target object mask. For example, Figure 2 The process of can be used to extract the mask of the target object (such as a child restraint system) in the image.

[0109] At 408, control selects an image frame from a temporal sequence of image frames, such as a video captured by a vehicle interior camera. Control then estimates a homography matrix between frame i-1 and frame i at 412. For example, control may estimate a 3x3 homography matrix Hi* between these frames. An example equation for estimating the homography matrix is:

[0110]

[0111] In some examples, gradient descent or another iterative method can be used to solve the homography matrix. At 416, control transforms the target object mask to frame i using the homography matrix. For example, the original target object mask can be projected to the current frame i using the homography matrix, such as by transforming the target object mask to coordinates of frame i using the following terms:

[0112]

[0113] Control extracts the foreground of frame i at 420. For example, Figure 3 The process illustrated in can be applied to the image of frame i after transformation using the homography matrix to extract the foreground mask from the transformed frame i. The foreground pixels are then removed from the transformed mask at 424 to obtain mask i.

[0114] At 428 , control determines whether the object mask is used for vehicle control. For example, the mask output from 424 may be used for tracking in a live vehicle environment, such as detecting the presence or absence of a child restraint system during vehicle operation.

[0115] If the object mask is used for vehicle control at 428, control proceeds to 432 to determine whether the target object condition is met based on the object mask. If the target object condition is met, the process controls the vehicle components (e.g., systems or features) according to the identified object condition. For example, as described in the example above, if the target object condition is met, such as the presence of a child restraint system, the vehicle system or feature can be controlled based on the detected target object (such as inhibiting airbag deployment when a child restraint system is detected, or issuing an alert via the HMI when an unsafe child or passenger object condition is detected). Control then selects the next frame from the time series of images at 440 and returns to 412 to estimate the homography matrix between the next adjacent frames.

[0116] Figure 5A and 5B An example of a neural network used to generate a model (such as the one described above) using machine learning techniques is shown. Machine learning is a method for designing complex models and algorithms that facilitate predictions (e.g., patient and provider matching predictions). Models generated using machine learning (such as the one described above) can produce reliable, repeatable decisions and results and reveal hidden insights by learning from historical relationships and trends in the data.

[0117] The purpose of using a neural network-based model and using machine learning as described above to train the model can be to directly predict the dependent variable without projecting the relationship between the variables into a mathematical form. The neural network model includes a large number of virtual neurons that operate in parallel and are arranged in layers. The first layer is the input layer and receives the original input data. Each subsequent layer modifies the output from the previous layer and sends them to the next layer. Each subsequent layer optionally applies a nonlinear transformation function to the output from the previous layer before sending it to the next layer. The last layer is the output layer and generates the output of the system.

[0118] Figure 5A A fully connected neural network is shown, where each neuron in a given layer is connected to every neuron in the next layer. In the input layer, each input node is associated with a value, which can be any real number. In each layer, each connection leaving an input node has a weight associated with it, which can also be any real number (see Figure 5B ). In the input layer, the number of neurons is equal to the number of elements (columns) in the dataset. The output layer can have multiple continuous outputs.

[0119] The layers between the input and output layers are hidden layers. The number of hidden layers can be one or more (for most applications, one hidden layer is probably sufficient). A neural network with no hidden layers can represent linearly separable functions or decisions. A neural network with one hidden layer can perform continuous mappings from one finite space to another. A neural network with two hidden layers can approximate any smooth mapping with any degree of accuracy.

[0120] The number of neurons can be optimized. At the beginning of training, the network configuration is more likely to have redundant nodes. During training, which will not significantly affect the network performance, some of the nodes can be removed from the network. For example, nodes with weights approaching zero after training can be removed (this process is called pruning). The number of neurons may cause underfitting (unable to fully capture the signal in the data set) or overfitting (not enough information to train all neurons; the network performs well on the training data set, but poorly on the test data set).

[0121] Various methods and criteria can be used to measure the performance of a neural network model. For example, the root mean square error (RMSE) measures the average distance between the observations and the model predictions. The coefficient of determination (R2) measures the correlation (rather than the accuracy) between the observed and predicted outcomes. If the data has a large variance, this method may be unreliable. Other performance metrics include irreducible noise, model bias, and model variance. A high model bias of a model indicates that the model is not able to capture the true relationship between the predictor and the outcome. Model variance can indicate whether the model is stable (slight perturbations in the data will significantly change the model fit). A neural network can receive an input, such as a vector, which can be used to generate a model that can be used for image processing and target object mask identification based on images from a camera inside a vehicle.

[0122] Figure 6 An example process for generating a machine learning model is illustrated. At 707, control obtains data from a database 702 (e.g., a data warehouse). The data can include any suitable data for developing a machine learning model.

[0123] At 711, control separates the data obtained from database 702 into training data 715 and test data 719. The training data 715 is used to train the model at 723, and the test data 719 is used to test the model at 727. Typically, depending on the desired model development parameters, the set of training data 715 is selected to be larger than the set of test data 719. For example, the training data 715 may include approximately 70 percent of the data, approximately 80 percent of the data, approximately 90 percent of the data, etc. obtained from the database 702. The remaining 30 percent, 20 percent, or 10 percent is then used as the test data 719.

[0124] Separating a portion of the acquired data into test data 719 allows the trained model to be tested against actual output data, facilitating more accurate training and development of the model at 723 and 727. At 723, the model can be trained using any suitable machine learning model technique, including those described herein, such as random forests, generalized linear models, decision trees, and neural networks.

[0125] At 731, control evaluates the model test results. For example, the trained model can be tested using the test data 719 at 727, and the results from the output data of the tested model can be compared with the actual output of the test data 719 to determine the level of accuracy. The model results can be evaluated using any suitable machine learning model analysis, such as the example techniques described further below.

[0126] After evaluating the model test results at 731, if the model test results are satisfactory, the model can be deployed at 735. Deploying the model can include using the model to make predictions for a large-scale input data set with unknown outputs. If the evaluation of the model test results at 731 is unsatisfactory, the model can be further developed using different parameters, using different modeling techniques, using other model types, etc. Figure 6 The machine learning model method can receive an input, such as a vector, which can be used to generate a model that can be used, for example, for image processing and target object mask identification based on images from a vehicle interior camera.

[0127] The foregoing description is merely illustrative in nature and is in no way intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in many forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, as other modifications will become apparent after studying the drawings, the description and the following claims. It should be understood that one or more steps within the method can be performed in a different order (or simultaneously) without changing the principles of the present disclosure. In addition, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in the features of any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the permutation of one or more embodiments with each other is still within the scope of the present disclosure.

[0128] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "immediately adjacent," "on," "above," "below," and "disposed." Unless explicitly described as "direct," when a relationship between a first and a second element is described in the above disclosure, the relationship can be a direct relationship in which no other intervening elements exist between the first and second elements, but can also be an indirect relationship in which one or more intervening elements exist (either spatially or functionally) between the first and second elements. As used herein, the phrase "at least one of A, B, and C" should be interpreted to mean a logical (A or B or C) using a non-exclusive logical "OR" and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0129] In the various figures, the direction of the arrow, as indicated by the arrow head, generally indicates the flow of information (such as data or instructions) of interest with respect to the diagram. For example, when component A and component B exchange various information, but the information transmitted from component A to component B is relevant to the diagram, the arrow may point from component A to component B. This unidirectional arrow does not imply that no other information is being transmitted from component B to component A. In addition, for information sent from component A to component B, component B may send a request for the information or an acknowledgment of receipt to component A.

[0130] In this application, including the definitions below, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor circuit (shared, dedicated, or grouped) that executes code; a memory circuit (shared, dedicated, or grouped) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0131] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules connected via the interface circuit. For example, multiple modules may allow for load balancing. In a further example, a server (also referred to as a remote or cloud) module may implement some functions on behalf of a client module.

[0132] The term "code" as used above may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" encompasses a single processor circuit that executes some or all code from multiple modules. The term "group processor circuit" encompasses a processor circuit that executes some or all code from one or more modules in combination with additional processor circuits. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or combinations of the above. The term "shared memory circuit" encompasses a single memory circuit that stores some or all code from multiple modules. The term "group memory circuit" encompasses a memory circuit that stores some or all code from one or more modules in combination with additional memory.

[0133] The term "memory circuit" is a subset of the term "computer-readable medium." As used herein, the term "computer-readable medium" does not encompass transitory electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); thus, the term "computer-readable medium" may be considered tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0134] The apparatus and methods described in this application can be implemented in part or in whole by a special-purpose computer, which is created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flow chart components, and other elements described above serve as software specifications that can be translated into a computer program through routine work by a skilled technician or programmer.

[0135] The computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0136] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated by a compiler from source code, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code may be written using syntax from languages ​​including: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language Version 5), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and

Claims

1. A vehicle interior camera system comprising: a memory configured to store at least one Gaussian process image segmentation model; a vehicle interior camera configured to capture an image of at least a portion of an interior of the vehicle; as well as A vehicle control module is configured to: obtaining an image of at least a portion of an interior of the vehicle from an interior vehicle camera; extracting one or more image crops of a region surrounding a target object in the image; Detect and segment passenger masks in images; supplying the image to the at least one Gaussian process image segmentation model to obtain a region mask associated with a region surrounding the target object; removing the passenger mask from the region mask to extract the target object mask; identifying a target object condition according to the target object mask; as well as At least one vehicle system or feature is controlled in response to the identification of the target object condition.

2. The vehicle interior camera system of claim 1 , wherein the vehicle control module is configured to output a target object mask to train a machine learning model to identify target objects in an image sequence. 3 . The vehicle interior camera system of claim 1 , wherein the target object comprises at least one of a child restraint system (CRS) and a passenger seat belt. 4 . The vehicle interior camera system of claim 3 , wherein controlling the vehicle system or feature includes inhibiting deployment of an air bag in response to detecting a child in a child restraint system.

5. The vehicle interior camera system of claim 3, further comprising a human machine interface (HMI) of the vehicle, wherein: controlling a vehicle system or feature including generating an alert via a human-machine interface; and The alert includes at least one of a warning not to place a child in a front seat of the vehicle, a warning that a child is in the vehicle after the key fob has left the vehicle, or a warning to passengers to wear seat belts while the vehicle is in motion.

6. The vehicle interior camera system of claim 1 , wherein the vehicle control module is configured to: Obtain a time series of k images from a vehicle interior camera; and A background motion model is generated from an image crop from each of the k images.

7. The vehicle interior camera system of claim 6, wherein the vehicle control module is configured to apply a background motion model to the temporal sequence of k images to generate a foreground mask. 8 . The vehicle interior camera system of claim 7 , wherein the vehicle control module is configured to remove a foreground mask from an image captured via the vehicle interior camera.

9. The vehicle interior camera system of claim 8, wherein removing the foreground mask from the image occurs before removing the passenger mask from the region mask to extract the target object mask.

10. A method for processing an image from a camera inside a vehicle, the method comprising: obtaining an image of at least a portion of an interior of the vehicle from an interior vehicle camera; extracting one or more image crops of a region surrounding a target object in the image; Detect and segment passenger masks in images; supplying the image to at least one Gaussian process image segmentation model to obtain a region mask associated with a region surrounding the target object; removing the passenger mask from the region mask to extract the target object mask; identifying a target object condition according to the target object mask; as well as At least one vehicle system or feature is controlled in response to the identification of the target object condition.

Images