Method for providing protection to an occupant of a vehicle

Abstract

A method for providing protection to an occupant of a vehicle includes acquiring at least one real-time image of an interior of the vehicle. The occupant is detected within the at least one real-time image. The detected occupant is classified based on the at least one real-time image. The detection classification is communicated to an operator of the vehicle. At least one deployment characteristic of an airbag associated with the detected occupant is set based on the classification.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to vehicle assist systems, and in particular to vision systems for helping to protect vehicle occupants. BACKGROUND

[0002] Current driver assist systems (ADAS - Advanced Driver Assist Systems) provide a range of monitoring functions in vehicles. In particular, ADAS can monitor the environment within a vehicle and notify the driver of the vehicle of conditions in the environment. To do so, ADAS can capture images of the interior of the vehicle and digitally process the images to extract information. In response to the extracted information, the vehicle can perform one or more functions. SUMMARY

[0003] In one example, a method for providing protection to an occupant of a vehicle includes acquiring at least one real-time image of an interior of the vehicle. The occupant is detected within the at least one real-time image. The detected occupant is classified based on the at least one real-time image. The detected classification is notified to an operator of the vehicle. At least one deployment characteristic of an airbag associated with the detected occupant is set based on the classification.

[0004] In another example, a method for providing protection to an occupant of a vehicle includes acquiring at least one real-time image of an interior of the vehicle. The occupant is detected within the at least one real-time image. An age and a weight of the detected occupant are estimated. The detected occupant is classified based on the estimated age and weight. The detected classification is notified to an operator of the vehicle. Feedback from the operator is received in response to the notification. At least one deployment characteristic of an airbag associated with the detected occupant is set based on the classification and the feedback.

[0005] Other objects and advantages of the present invention will be grasped by a more complete understanding of the present invention and will be mastered upon reading the following detailed description in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1A is a top view of a vehicle including an example vision system in accordance with the present invention.

[0007] Figure 1B is a cross-sectional view taken along line 1B-1B of the vehicle. Figure 1A

[0008] Figure 2A is a schematic representation of an ideal alignment image of an interior of the vehicle.

[0009] Figure 2B is a schematic representation of another example ideal alignment image.

[0010] Figure 3 ​It is a schematic display of real-time images inside the vehicle.

[0011] Figure 4 It uses generated key points to compare an ideally aligned image with a real-time image.

[0012] Figure 5 It is a schematic representation of a calibrated real-time image with ideal alignment to the region of interest.

[0013] Figure 6 It is a schematic representation of a real-time image with a calibrated region of interest.

[0014] Figure 7 It is a schematic representation of continuous real-time images acquired by a vision system.

[0015] Figure 8 It is an illustrative representation used to assess the confidence level of real-time images.

[0016] Figure 9 yes Figure 8 A magnified view of a portion of the confidence level.

[0017] Figure 10 It is an illustrative display of children and adults in the front seats of the vehicle.

[0018] Figure 11 It is an illustrative display of elderly people and teenagers in the front seats of the vehicle.

[0019] Figure 12 This is a schematic diagram of a controller connected to a vehicle component.

[0020] Figure 13 It is a schematic representation of the interior of a vehicle, including occupant protection equipment. Detailed Implementation

[0021] This invention relates generally to vehicle assistance systems, and more particularly to vision systems for helping to protect vehicle occupants. Figure 1A and Figure 1B A vehicle 20 is shown with an example vehicle assistance system in the form of a vision system 10 for acquiring and processing images within the vehicle. The vehicle 20 extends along a centerline 22 from a first end or front end 24 to a second end or rear end 26. The vehicle 20 extends to the left side 28 and the right side 30 on opposite sides of the centerline 22. Front doors 36 and rear doors 38 are provided on both sides 28, 30. The vehicle 20 includes a roof 32 that cooperates with the front door 36 and rear door 38 on each side 28, 30 to define a passenger compartment or interior 40. The exterior of the vehicle 20 is indicated by 41.

[0022] The front end 24 of the vehicle 20 includes a dashboard 42 facing the interior 40. A steering wheel 44 extends from the dashboard 42. Alternatively, if the vehicle 20 is an autonomous vehicle, the steering wheel 44 (not shown) can be omitted. Either way, a windshield or windscreen 50 can be located between the dashboard 42 and the roof 32. Rearview mirrors 52 are connected to the interior of the windshield 50. A rear window 56 at the rear end 26 of the vehicle 20 helps enclose the interior 40.

[0023] Seats 60 are positioned in the interior 40 for receiving one or more occupants 70. In one example, the seats 60 can be arranged in a front row 62 and a rear row 64, respectively, oriented in a forward-facing manner. In an autonomous vehicle configuration (not shown), the front row 62 can face rearward. Safety belts 59 are associated with each seat 60 to help restrain the occupants 70 in the associated seats. A center console 66 is positioned between the seats 60 in the front row 62.

[0024] The vision system 10 includes at least one camera 90 positioned within the vehicle 20 for acquiring images of the interior 40. As shown, the camera 90 is connected to the rearview mirror 52, but other locations are contemplated, such as the roof 32, the rear window 56, etc. In any case, the camera 90 has a field of view 92 that extends rearward through the interior 40 over a large percentage of the interior (e.g., the space between the doors 36, 38 and from the windshield 50 to the rear window 56). The camera 90 produces signals indicative of the images taken and sends the signals to a controller 100. It is understood that the camera 90 can alternatively be mounted on the vehicle 20 so that the field of view 92 extends over or includes the exterior 41 of the vehicle. The controller 100 in turn processes the signals for future use.

[0025] As Figure 2A shown, when the vehicle 20 is manufactured, a template or ideal alignment image 108 of the interior 40 is created for helping to calibrate the camera 90 when it is installed or periodically thereafter. The ideal alignment image 108 reflects the ideal position of the camera 90 to be aligned with the interior 40 in a prescribed manner to produce the desired field of view 92. To this end, for each make and model of the vehicle 20, the camera 90 is positioned so that its real-time images (i.e., images taken during vehicle use) most closely match the desired orientation of the ideal alignment in the interior 40, including the desired position, depth, and boundaries. The ideal alignment image 108 captures the portion of the interior 40 in which objects, such as the seats 60, occupants 70, pets, or personal belongings, are expected to be monitored / detected during operation of the vehicle 20.

[0026] The ideal alignment image 108 is defined by a boundary 110. The boundary 110 has a top boundary 110T, a bottom boundary 110B, and a pair of side boundaries 110L, 110R. That is, the illustrated boundary 110 is rectangular, but other shapes of the boundary are contemplated, e.g., triangular, circular, etc. Since the camera 90 faces rearward in the vehicle 20, the side boundary 110L is on the left side of the image 108 but on the right side 30 of the vehicle 20. Similarly, the side boundary 110R is on the right side of the image 108 but on the left side 28 of the vehicle 20. The ideal alignment image 108 is overlaid with a global coordinate system 112 having x, y, and z axes.

[0027] The controller 100 can divide the ideal alignment image 108 into one or more regions of interest 114 (abbreviated as “ROI” in the figure) and / or one or more regions of non-interest 116 (indicated as “outside ROI” in the figure). In the illustrated example, a boundary line 115 demarcates a middle region of interest 114 from regions of non-interest 116 on either side thereof. The boundary line 115 extends between boundary points 111, which in this example intersect the boundary 110. The region of interest 114 is between the boundaries 110T, 110B, 115. The left region of non-interest 116 (as viewed in FIG. 2) is between the boundaries 110T, 110B, 110L, 115. The right region of non-interest 116 is between the boundaries 110T, 110B, 110R, 115.

[0028] In Figure 2A In the illustrated example, the region of interest 114 can be the region that includes both rows (62, 64) of seats 60. The region of interest 114 can coincide with a region of the interior 40 in which one or more particular objects would reasonably reside. For example, it is reasonable for an occupant 70 to be in a seat 60 in either row 62, 64, and thus the illustrated region of interest 114 extends generally to the lateral extent of each row. In other words, the illustrated region of interest 114 is sized and shaped specifically to the occupant 70— a region of interest that can be said to be occupant-specific.

[0029] It will be appreciated that different regions of interest (e.g., for pets, laptops, etc.) can have regions of interest of particular sizes and shapes that predefine reasonable locations for that particular object in the vehicle 20. These different regions of interest have predetermined and known locations within the ideal alignment image 108. Depending on the object of interest associated with each region of interest, these different regions of interest can overlap one another.

[0030] With this in mind, Figure 2BThe ideal alignment image 108 is shown displaying different regions of interest for different objects of interest, i.e., the region of interest 114a is for the pet in the back row 64, the region of interest 114b is for the occupant in the driver seat 60, and the region of interest 114c is for the laptop. Each region of interest 114a-c is bounded between the associated boundary points 111. In each case, the region of interest 114a-c is the inverse of the region(s) of disinterest 116, such that these regions collectively form the entire ideal alignment image 108. In other words, anywhere in the ideal alignment image 108 that is not bounded by a region of interest 114a-c is considered the region(s) of disinterest 116.

[0031] Returning to Figure 2A In the example shown, the region of disinterest 116 is the region outside the lateral sides of the rows 62, 64 and adjacent to the doors 36, 38. The region of disinterest 116 coincides with the region of the interior 40 in which it is unreasonable for an object (here, the occupant 70) to reside. For example, it is unreasonable for the occupant 70 to be located inside the roof 32.

[0032] During operation of the vehicle 20, the camera 90 acquires images of the interior 40 and sends signals indicative of these images to the controller 100. In response to the received signals, the controller 100 performs one or more operations on the images and then detects objects of interest in the interior 40. The images taken during operation of the vehicle 20 are referred to herein as "live images." An example live image 118 taken during operation of the vehicle 20 is shown in Figure 3

[0033] The live image 118 shown is bounded by a border 120. The border 120 includes a top border 120T, a bottom border 120B, and a pair of side borders 120L, 120R. Since the camera 90 faces rearward in the vehicle 20, the side border 120L is on the left side of the live image 118 but on the right side 30 of the vehicle 20. Similarly, the side border 120R is on the right side of the live image 118 but on the left side 28 of the vehicle 20.

[0034] ​From the perspective of camera 90, real-time image 118 has a local coordinate system 122 superimposed on it, or is associated with, having x, y, and z axes. That is, real-time image 118 can indicate the position / orientation of camera 90 has a deviation from the position / orientation of the camera that generated ideal alignment image 108 for several reasons. First, camera 90 can be improperly or otherwise installed in the orientation that captures field of view 92, which deviates from the field of view generated by the camera that took ideal alignment image 108. Second, after installation, the position of camera 90 can be affected by vibrations from, for example, road conditions and / or impacts to rearview mirror 52. In any case, coordinate systems 112, 122 can not be identical, and thus, it is desirable to calibrate camera 90 to account for any orientation differences between the position of the camera that captures real-time image 118 and the ideal position of the camera that captures ideal alignment image 108.

[0035] In one example, controller 100 uses one or more image matching techniques, such as Oriented FAST and rotated BRIEF (ORB) feature detection, to generate key points in each image 108, 118. Controller 100 then generates a homography matrix from the matched key points pairs, and uses the homography matrix along with known intrinsic properties of camera 90 to identify camera position / orientation deviations in eight degrees of freedom to help controller 100 calibrate the camera. This allows the vision system to ultimately better detect objects within real-time image 118, and make decisions in response thereto.

[0036] Figure 4 One example implementation of the process is illustrated. For illustrative purposes, ideal alignment image 108 and real-time image 118 are placed adjacent to one another. Controller 100 identifies key points within each image 108, 118 - the illustrated key points are indicated as ①, ②, ③, ④. Key points are different locations in images 108, 118 that attempt to match one another in the images and correspond to the same precise point / position / speck. Features can be, for example, corners, stitches, etc. Although only four key points are specifically identified, it should be understood that vision system 10 can rely on hundreds or thousands of key points.

[0037] In any case, the key points are identified and their positions mapped between the image 108, 118. The controller 100 computes a homography matrix based on the matching of the key points of the ideal alignment image 108 to the real-time image 118. With additional information of the intrinsic properties of the camera, the homography matrix is then decomposed to identify any translation (x, y, and z axes), rotation (yaw, pitch, and roll), and sheer and scale of the camera 90 capturing the real-time image 118 relative to the ideal camera capturing the ideal alignment image 108. Thus, the decomposition of the homography matrix quantifies the misalignment in eight degrees of freedom between the camera 90 capturing the real-time image 118 and the ideal camera capturing the ideal alignment image 108.

[0038] The misalignment threshold range can be associated with each degree of freedom. In one example, the threshold range can be used to identify which degrees of freedom deviations of the real-time image 118 are negligible and which degrees of freedom deviations are considered large enough to warrant a physical correction to the position and / or orientation of the camera 90. In other words, the deviation of one or more particular degrees of freedom between the image 108 and the image 118 can be small enough to warrant being ignored - no correction to such degree of freedom is made. The threshold range for each degree of freedom can be symmetrical or asymmetrical.

[0039] For example, if the threshold range for rotation about the x-axis is + / - 0.05°, then a calculated x-axis rotation deviation of the real-time image 118 from the ideal alignment image 108 within the threshold range would not be considered when making physical adjustments to the camera 90. On the other hand, a rotation deviation about the x-axis outside of the corresponding threshold range would cause a significant misalignment and necessitate recalibration or physical repositioning of the camera 90. Thus, the threshold range acts as a pass / fail filter for the deviation of each degree of freedom.

[0040] The homography matrix information can be stored in the controller 100 and used to calibrate any real-time image 118 taken by the camera 90 so that the vision system 10 can better react to the real-time image, e.g., better determine changes in the interior 40. To this end, the vision system 10 can use the homography matrix to transform the entire real-time image 118 and produce a calibrated or adjusted real-time image 119 as shown in Figure 5 When this occurs, the calibrated real-time image 119 can be rotated or skewed relative to the borders 120 of the real-time image 118. The region of interest 114 is then projected onto the calibrated real-time image 119 via the border points 111. In other words, the uncalibrated region of interest 114 is projected onto the calibrated real-time image 119. However, this transformation of the real-time image 118 can involve a significant amount of computation by the controller 100.

[0041] That is, the controller 100 can alternatively transform or calibrate only the region of interest 114 and project the calibrated region of interest 134 onto the uncalibrated live image 118 to form a calibrated image 128 as shown. Figure 6 In other words, the region of interest 114 can be transformed via translation, rotation, and / or steering / scale data stored in a homography matrix and projected or mapped onto the untransformed live image 118 to form the calibrated image 128.

[0042] More specifically, the generated homography matrix is used to calibrate the boundary points 111 of the region of interest 114 by transformation to produce corresponding boundary points 131 in the calibrated image 128. However, it will be appreciated that when the region of interest is projected onto the live image 118, one or more of the boundary points 131 can lie outside the boundary 120, in which case the intersection of the line connecting the boundary points with the boundary 120 helps to define the calibrated region of interest 134 (not shown). Either way, when the original region of interest 114 is aligned on the ideal alignment image 108, the new calibrated region of interest 134 is aligned on the live image 118 (in the calibrated image 128). This calibration effectively pins the region of interest 114 so that no image transformation needs to be applied to the entire live image 118, thereby reducing the processing time and processing power required.

[0043] To this end, using the homography matrix to calibrate a few boundary points 111 that define the region of interest 114 is far easier, quicker and more efficient than transforming or calibrating the entire live image 118 as performed in the prior art. Figure 5 The calibration of the region of interest 114 ensures that any misalignment of the camera 90 from the ideal position will have minimal, if any, adverse impact on the accuracy with which the vision system 10 detects objects in the interior 40. The vision system 10 can perform the calibration of the region of interest 114 at predetermined time intervals or on the occurrence of an event (e.g. the start-up of the vehicle 20 or at five second intervals) - each time generating a new homography matrix based on a new live image.

[0044] The calibrated region of interest 134 can be used to detect objects in the interior 40. The controller 100 analyses the calibrated image 128 or the calibrated region of interest 134 and determines which objects, if any, are located therein. In the example shown, the controller 100 detects the occupant 70 within the calibrated region of interest 134. However, it will be appreciated that the controller 100 can calibrate any alternative or additional regions of interest 114a to 114c to form associated calibrated regions of interest and detect specific objects of interest (not shown) therein.

[0045] In analyzing the calibrated image 128, the controller 100 can detect objects that intersect or cross outside the calibrated region of interest 134 and thus exist both inside and outside the calibrated region of interest. When this occurs, the controller 100 can rely on a threshold percentage that determines whether the detected object is ignored. More specifically, the controller 100 can confirm or “pass” a detected object that has at least, for example, 75% overlap with the calibrated region of interest 134. Thus, a detected object that has less than the threshold percentage of overlap with the calibrated region of interest 134 will be ignored or “failed.” Only detected objects that meet this criterion will be considered for further processing or action.

[0046] The vision system 10 can perform one or more operations in response to detecting and / or recognizing an object within the calibrated real-time image 128. This can include, but is not limited to, deploying one or more airbags based on the position of the occupant(s) in the interior 40.

[0047] Referring to Figure 7 to Figure 9 , the vision system 10 includes additional safety measures, including a confidence level in the form of a counter, to help ensure that objects are accurately detected within the real-time image 118. The confidence level can be used in conjunction with the aforementioned calibration or separately therefrom. During operation of the vehicle 20, the camera 90 rapidly and continuously takes a plurality of real-time images 118 (see Figure 7 ). For clarity, each successive real-time image 118 is given an index, e.g., first, second, third... up to nth image and corresponding suffix “a,” “b,” “c”... “n.” Thus, the first real-time image is indicated at 118a in Figure 7 . The second real-time image is indicated at 118b. The third real-time image is indicated at 118c. The fourth real-time image is indicated at 118d. Although only four real-time images 118a-118d are shown, it should be understood that the camera 90 can take more or fewer real-time images. Regardless, the controller 100 performs object detection in each real-time image 118.

[0048] With this in mind, the controller 100 evaluates the first real-time image 118a and uses image inference to determine which object(s) (in this example, the occupant 70 in the back row 64) are located within the first real-time image. The image inference software is configured such that an object will not be indicated as detected without at least a predetermined confidence level (e.g., at least 70% confidence that the object is in the image).

[0049] It is to be understood that this detection can occur after calibrating the first live image 118a (and subsequent live images) as described above, or without calibration. In other words, object detection can occur in every live image 118, or specifically in the calibrated region of interest 134 projected onto the live image 118. The following discussion focuses on detecting objects / passengers 70 in live images 118 without first calibrating the live images and without using a region of interest.

[0050] When the controller 100 detects one or more objects in the live image 118, a unique identification number and a confidence level 150 (see Figure 8 ) is associated with or assigned to each detected object. Although multiple objects can be detected, in the example shown only a single object (in this case a passenger 70) is detected, and thus only a single confidence level 150 associated therewith is shown and described for the sake of brevity. The confidence level 150 helps to rate the reliability of the object detection. Figure 7 to Figure 9

[0051] The confidence level 150 has a range between a first value 152 and a second value 154, for example, a range from -20 to 20. The first value 152 can serve as a minimum value for the counter 150. The second value 154 can serve as a maximum value for the counter 150. A value of 0 for the confidence level 150 indicates that no live image 118 has yet been evaluated, or it is not possible to determine whether a detected object is actually present or not present in the live image 118. A positive value for the confidence level 150 indicates that the detected object is more likely to be actually present in the live image 118. A negative value for the confidence level 150 indicates that the detected object is more likely to be actually not present in the live image 118.

[0052] Furthermore, as the confidence level 150 decreases from the value 0 towards the first value 152, the confidence that the detected object is not actually present in the live image 118 (an "incorrect" indication) increases. On the other hand, as the confidence level 150 increases from the value 0 towards the second value 154, the confidence that the detected object is actually present in the live image 118 (a "correct" indication) increases.

[0053] Prior to evaluating the first live image 118a, the value of the confidence level 150 is 0 (see also Figure 9 ). If the controller 100 detects a passenger 70 within the first live image 118a, the value of the confidence level 150 increases to 1. This increase is schematically shown by arrow A in Figure 9 . Alternatively, detecting an object in the first live image 118a can keep the confidence level 150 at the value 0, but trigger or initiate a multi-image evaluation process.

[0054] ​For each subsequent live image 118b to 118d, the controller 100 detects whether the occupant 70 is present. When the controller 100 detects the occupant 70 in each of the live images 118b to 118d, the value of the confidence level 150 will increase (move closer to the second value 154). Each time the controller 100 does not detect the occupant 70 in each of the live images 118b to 118d, the value of the confidence level 150 will decrease (move closer to the first value 152).

[0055] The amount by which the confidence level 150 increases or decreases can be the same for each successive live image. For example, if the occupant 70 is detected in five successive live images 118, the confidence level 150 can increase as follows: 0, 1, 2, 3, 4, 5. Alternatively, the confidence level 150 can increase in a non-linear fashion as the number of successive live images in which the occupant 70 is detected increases. In this example, after the occupant 70 is detected in each live image 118, the confidence level 150 can increase as follows: 0, 1, 3, 6, 10, 15. In other words, the reliability or confidence of the object detection assessment can increase rapidly when the object is detected in more successive images.

[0056] Similarly, if the occupant 70 is not detected in five successive live images 118, the confidence level 150 can decrease as follows: 0, -1, -2, -3, -4, -5. Alternatively, if the occupant 70 is not detected in five successive live images 118, the confidence level 150 can decrease in a non-linear fashion as follows: 0, -1, -3, -6, -10, -15. In other words, the reliability or confidence of the object detection assessment can decrease rapidly when the object is not detected in more successive images. In all cases, the confidence level 150 is adjusted, i.e. increased or decreased, when the object detection assessment is made for each successive live image 118. It will be appreciated that this process is repeated for each confidence level 150 associated with each detected object, and thus each detected object will undergo the same object detection assessment.

[0057] It will also be appreciated that once the counter 150 reaches the minimum value 152, any subsequent non-detection will not change the value of the counter from the minimum value. Similarly, once the counter 150 reaches the maximum value 154, any subsequent detection will not change the value of the counter from the maximum value.

[0058] With the example shown, after detecting the occupant 70 in the first live image 118a, the controller then detects the occupant in the second live image 118b, does not detect the occupant in the third live image 118c, and detects the occupant in the fourth live image 118d. The failure to detect in the third live image 118c can be attributed to a change in lighting, rapid movement of the occupant 70, etc. As shown, the third live image 118c is darkened due to lighting conditions in / around the vehicle 20, causing the controller 100 to fail to detect the occupant 70. That is, in response to detecting the occupant 70 in the second live image 118b, the value of the confidence level 150 is increased by 2 in the manner indicated by arrow B.

[0059] Then, in response to not detecting the occupant 70 in the third live image 118c, the value of the confidence level 150 is decreased by 1 in the manner indicated by arrow C. Then, in response to detecting the occupant 70 in the fourth live image 118d, the value of the confidence level 150 is increased by 1 in the manner indicated by arrow D. After evaluating all of the live images 118a-118d, the value of the final confidence level 150 is 3.

[0060] The final value of the confidence level 150 between the first value 152 and the second value 154 can indicate the situation in which the controller 100 determines that the detected occupant 70 is actually present and the degree of confidence in that determination. The final value of the confidence level 150 can also indicate the situation in which the controller 100 determines that the detected occupant 70 is actually not present and the degree of confidence in that determination. The controller 100 can be configured to finally determine whether the detected occupant 70 is actually present or not after evaluating a predetermined number of consecutive live images 118 (four live images in this case) or after a predetermined time frame (e.g., seconds or minutes) of live images being acquired.

[0061] After examining the four live images 118a-118d, the positive value of the confidence level 150 indicates that the occupant 70 is more likely to actually be present in the vehicle 20. The value of the final confidence level 150 indicates that this assessment has a lower degree of confidence than a final value closer to the second value 154, but a higher degree of confidence than a final value closer to 0. The controller 100 can be configured to associate a particular percentage or value with each final confidence level 150 value or a range of values between and including the values 152 and 154.

[0062] The controller 100 can be configured to enable, disable, actuate, and / or deactivate one or more vehicle functions in response to the value of the final confidence level 150. This can include, for example, controlling vehicle airbags, seatbelt pretensioners, door locks, emergency braking, HVAC, etc. It should be appreciated that different vehicle functions can be associated with different final confidence level 150 values. For example, vehicle functions associated with occupant safety can require a relatively higher final confidence level 150 value to initiate actuation than vehicle functions not associated with occupant safety. To this end, in some cases, object detection assessments with a final confidence level 150 value of 0 or below can be discarded or ignored altogether.

[0063] During operation of the vehicle 20, the real-time images 118 can be assessed multiple times or periodically. Assessments can be made in the vehicle interior 40 when the field of view 92 of the camera 90 is facing inward, or around the vehicle exterior 41 when the field of view is facing outward. In each case, the controller 100 individually examines a plurality of real-time images 118 for object detection and ultimately determines whether the detected object actually exists in the real-time images with an associated confidence value.

[0064] An advantage of the vision system shown and described herein is that it provides improved reliability in object detection within and around a vehicle. When multiple images of the same field of view are taken within a short timeframe, the quality of one or more of the images can be affected by, for example, lighting conditions, shadows, objects passing in front of and obstructing the camera, and / or motion blur. As a result, the current camera can produce false positive and / or false negative detections of objects in the field of view. This erroneous information can adversely affect downstream applications that rely on object detection.

[0065] By individually analyzing a series of consecutive real-time images to determine a cumulative confidence score, the vision system of the present invention helps mitigate the aforementioned deficiencies that can exist in a single frame. As a result, the vision system shown and described herein helps reduce false positives and negatives in object detection.

[0066] That is, the controller 100 can not only detect objects within the vehicle 20, but also classify the detected objects. In a first stage of classification, the controller 100 determines whether the detected object is a human / occupant or an animal / pet. In the former case, the detected occupant can be classified a second time based on age, height, weight, or any combination thereof.

[0067] In Figure 10 In the example shown, the controller 100 detects and identifies a child 190 and an adult 192 in the vehicle interior 40 (e.g., front row 62 seat 60). In Figure 11In the illustrated example, the controller 100 detects and identifies the elderly person 194 and the teenager 196 in the front row 62 of seats 60. It should be appreciated that any of the child 190, adult 192, elderly person 194, or teenager 196 can also be located in the rear row 64 (not shown).

[0068] In each instance, occupant detection can be performed with or without calibrating the real-time image 118 or the region of interest 114 associated with the ideal alignment image 108. Detection can also be performed with or without utilizing the confidence level / counter 150. In any case, the above-described processes occur after the controller 100 determines that an occupant is in the vehicle 20.

[0069] Either way, in response to receiving a signal from the camera 90, the controller 110 uses an artificial intelligence (AI) model, image inference software, and / or pattern recognition software to estimate the age of each detected occupant. The AI model can be prepared and trained under supervised learning for this application. Other characteristics of the detected occupant (e.g., seated height and weight) can also be estimated with the AI model, image inference software, and / or pattern recognition software.

[0070] With reference to Figure 12 and Figure 13 , the controller 100 is connected to or includes an integral airbag controller 200. One or more weight sensors 212 are located in the seat base 65 of each seat 60 in the vehicle 20 and are connected to the airbag controller 200. The weight sensors 212 detect the weight of any object on the seat base 65 and send a signal indicative of the detected weight to the controller 200. As a result, the vision system 10 can rely on both the camera 90 and the weight sensors 212 to help estimate the weight of each detected occupant.

[0071] The controller 100 can include a look-up table or the like that associates seated height and weight (or ranges thereof) with a particular age classification. That is, the controller 100 can use the estimated age in combination with the estimated seated height and weight to make an age-based classification determination for each detected occupant with high reliability.

[0072] The age-based classification can be based on an estimate that the detected occupant has an age within a prescribed range, e.g., under 12 years old is a child 190, 12 to 19 years old is a teenager 196, 20 to 60 years old is an adult 192, and over 60 years old is an elderly person 194. Other age ranges can be envisioned for each identification, however.

[0073] The controller 100 is also connected to a display 220 in the vehicle interior 40 and is visible to the occupants 70. In one example, the display 220 is located on the dashboard 42 (see Figure 13 ).

[0074] The airbag controller 200 is connected to one or more inflators that are fluidly connected to associated airbags. In the illustrated example, a first inflator 222 is fluidly connected to a passenger-side frontal airbag 232 that is mounted in the dashboard 42. Another inflator 224 is fluidly connected to a driver-side frontal airbag 234 that is mounted in the steering wheel 240.

[0075] The inflators 222, 224 can be single-stage or multi-stage inflators that are capable of delivering an inflation fluid to the associated airbags 232, 234 at one or more rates and / or pressures. The airbags 232, 234 can include passive or active adaptive features, such as tethers, vents, tear lines, ramps, etc. Thus, the deployment characteristics (e.g., size, shape, profile, stiffness, speed, pressure, and / or direction) of each airbag 232, 234 can be controlled by the inflators 222, 224 and / or by operating the adaptive features. The controller 100, which is connected to the inflators 222, 224 and the airbags 232, 234 (more specifically, the adaptive features) by the airbag controller 200, can affect the deployment characteristics of each airbag.

[0076] With this in mind, each occupant classification can have a particular set of airbag deployment characteristics associated with it that depend on the type of airbag and the location of that airbag in the vehicle 20. In other words, the airbag controller 200 can be equipped with a table or the like that correlates each type of occupant classification with particular airbag deployment characteristics. These correlations can also take into account the type of airbag (e.g., frontal airbag, side curtain, knee bolster, etc.) and the location of that airbag in the vehicle (e.g., front or rear). Each combination of deployment characteristics can have a corresponding set of inflator 222, 224 and / or airbag 232, 234 commands or controls associated with it. The airbag controller 200 can associate each unique set of commands / controls with a unique “mode.”

[0077] The airbag controller 200 can be connected to additional inflators associated with additional airbags (not shown) positioned throughout the vehicle 20 (e.g., side curtain airbags along the left side 28 or right side 30, floor-mounted airbags, roof-mounted airbags, and / or seat-mounted airbags). Thus, the airbag controller 200 and the controller 100 can affect or control the deployment characteristics of these additional airbags.

[0078] With this in mind, once the controller 100 identifies the occupant(s) and classifies them, it sends a signal to the display 220 to inform the operator of the vehicle 20 of the location in the vehicle where the occupant has been detected (e.g., the front row 62 or the rear row 64) and the classification of each detected occupant. This includes information related to the classification of the operator themselves.

[0079] For example, when the controller detects a child 190 in the front row 62 of the right / passenger side 30 Figure 10 ) and an adult 192 in the left / driver side 28, the controller 100 can send a notification to the display 220. The operator of the vehicle 20 (in this case, the adult 192) can provide feedback, e.g., touching the display 220 or a voice command, confirming whether the occupant classification is accurate. If the operator indicates that the classification of the child 190 is accurate, the controller 100 instructs the airbag controller 200 to set the deployment characteristics of the passenger airbag 232 to a “baby” or “child” mode, which corresponds to an airbag deployment that provides a relatively reduced impact force upon occurrence of a vehicle impact. These reduced impact forces can be forces comparable to child airbag safety standards.

[0080] On the other hand, if the operator indicates that the classification of the child 190 is inaccurate (e.g., the occupant classified as a child is actually an adult), the controller 100 instructs the airbag controller 200 to set the deployment characteristics of the passenger airbag 232 to an “adult” mode, which corresponds to an airbag deployment that provides a standard impact force upon occurrence of a vehicle impact. These impact forces can be forces comparable to adult airbag safety standards.

[0081] The remaining age-related classifications can have associated airbag deployment characteristics that are the same as or different from the “adult mode” or “child mode.” In particular, in response to classifying a detected occupant as Figure 11 the elderly person 194 or the teenager 196, the controller 100 can instruct the airbag controller 200 to set the deployment characteristics to an “intermediate mode.” This “intermediate mode” can correspond to airbag deployment characteristics that provide a reaction force value that is intermediate between the “child mode” reaction force and the “adult mode” reaction force.

[0082] It should be understood that, although the height, weight, and age of the detected occupant are used to collectively determine an age-based classification of the occupant, the controller can alternatively classify the occupant in a different manner (e.g., based on weight) and use the remaining collected data to adjust the deployment characteristics of the airbag. In other words, the controller can initially determine weight-based deployment characteristics and subsequently adjust these deployment characteristics based on the remaining height and age information.

[0083] In each scenario, the controller receives signals from the camera and weight sensor(s), classifies the detected occupants in the vehicle based on these signals, and notifies the vehicle operator of these classifications. In response, the operator provides feedback by confirming or correcting each classification. The controller then sets the deployment characteristics or "mode" of each airbag accordingly.

[0084] An advantage of the vision system of the present application is that improved reliability is provided in classifying occupants of a vehicle and thereafter adjusting occupant protection measures (e.g., deployment of airbags) in response to these classifications. Furthermore, by allowing the vehicle operator to provide feedback to the classifications before setting a particular protection measure, the operator can check the determinations made in classifying, thereby helping to ensure that appropriate protection measures are implemented.

[0085] What has been described above are examples of the present application. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the present application, but one of ordinary skill in the art will recognize that many other combinations and permutations of the present application are possible. Accordingly, the present application is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A method for providing protection to an occupant of a vehicle, characterized by, The method comprises: acquiring a plurality of live images of a vehicle interior; calibrating a region of interest in the plurality of live images using a homography matrix; projecting the calibrated region of interest onto the live images that are not calibrated to form a calibrated image; detecting occupants within the calibrated image individually; determining, using a confidence value associated with the occupants, whether the occupants are actually present in the calibrated image; classifying the detected occupants based on the calibrated image; informing an operator of the vehicle of the classification of the detected occupants; and setting at least one deployment characteristic of an airbag associated with the detected occupants based on the classification, The method further comprises: acquiring data indicative of a weight of the detected occupants, and classifying the detected occupants based on the weight, wherein the weight data is a weight of the detected occupants in the plurality of live images estimated using an artificial intelligence model.

2. The method of claim 1, wherein, The classification is based on an estimated age of the detected occupants in the plurality of live images using an artificial intelligence model.

3. The method of claim 1, wherein, The classification is based on an estimated seated height of the detected occupants.

4. The method of claim 1, wherein, The weight data is further based on signals acquired by a weight sensor disposed in a seat occupied by the detected occupants.

5. The method of claim 1, wherein, The occupants are classified as one of a juvenile and a senior.

6. The method of claim 1, further comprising: receiving feedback from the operator in response to the informing; and setting the at least one deployment characteristic of the airbag based on the feedback. The feedback includes confirming the classification.

7. The method of claim 6, wherein, The feedback includes changing the classification.

8. The method of claim 6, wherein, The step of setting at least one deployment characteristic includes setting an inflation rate of an airbag associated with the detected occupants.

9. The method of claim 1, wherein, The step of setting at least one deployment characteristic includes setting an inflation pressure of an airbag associated with the detected occupants.

10. The method of claim 1, wherein, 11. A method for providing protection to occupants of a vehicle, the method comprising: acquiring a plurality of live images of a vehicle interior; calibrating a region of interest in the plurality of live images using a homography matrix; projecting the calibrated region of interest onto the live images that are not calibrated to form a calibrated image; detecting occupants within the calibrated image individually; determining, using a confidence value associated with the occupants, whether the occupants are actually present in the calibrated image; estimating an age and a weight of the detected occupants; classifying the detected occupants based on the estimated age and weight; informing an operator of the vehicle of the classification of the detected occupants; receiving feedback from the operator in response to the informing; and setting at least one deployment characteristic of an airbag associated with the detected occupants based on the classification and the feedback, wherein the weight is estimated using an artificial intelligence model and the plurality of live images. The age is estimated using an artificial intelligence model and the plurality of live images. The classification is based on an estimated seated height of the occupants.

12. The method of claim 11, wherein, The feedback includes confirming the classification.

13. The method of claim 11, wherein, The feedback includes changing the classification.

14. The method of claim 11, wherein, ​ 15. The method of claim 11, wherein, ​ 16. The method of claim 11, wherein, The step of setting at least one deployment characteristic includes setting an inflation rate of the airbag associated with the detected occupant.

17. The method of claim 11, wherein, The step of setting at least one deployment characteristic includes setting an inflation pressure of the airbag associated with the detected occupant.

Images