System, device and method for post-crash vehicle support

Abstract

A system and method for providing useful information about the medical status and injuries of occupants of a vehicle to first responders after a vehicle accident is provided. The system includes an in-cabin sensor, the in-cabin sensor including at least one or more of an image sensor, a depth sensor, and a micro-vibration sensor, for capturing sensory data of the vehicle cabin, including pre-crash data, during-crash data, and post-crash data. The system also includes at least one processor, the at least one processor being configured to analyze the sensory data.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Application No. 62 / 785,724, filed on December 28, 2018, entitled “SYSTEM, DEVICE AND METHOD FOR VEHICLE POST-CRASH SUPPORT” (Attorney Docket No. GR003 / USP), which is incorporated herein by reference in its entirety. Technical Field

[0003] In some embodiments of the invention, the invention relates to a system and method for analyzing in-cabin data to provide medically relevant information after a vehicle accident.

[0004] Incorporation by reference

[0005] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. Background Art

[0006] When a vehicle incident such as a car accident occurs, the correct actions of the first responder team and the medical team have long-term implications. Understanding the nature and severity of the injuries to the car occupants can improve the response in terms of prioritization and immediate treatment. This also helps guide the correct first responder team, thereby reducing the time it takes for the right equipment to arrive on the scene.

[0007] While significant efforts have been made to reduce the number and severity of vehicle accidents, vehicle accidents do continue to occur and, regrettably, will continue to occur for the foreseeable future.

[0008] Currently, rescue teams have no advance knowledge of what is happening inside a vehicle and typically receive updates about the medical condition of occupants only upon arrival. Furthermore, even when rescue teams arrive on scene, the information available is only available through visual inspection and communication with the occupants involved, if such communication is possible. While such communication provides important information, it is limited by the occupants’ level of awareness and knowledge.

[0009] In light of the above, there remains an unmet need for in-cabin systems that can, for example, provide medically relevant information about an occupant's post-crash status in real time. Such a system would increase the likelihood of surviving a vehicle accident and reduce long-term medical effects. This would also have economic benefits, potentially impacting insurance costs. Summary of the invention

[0010] The present disclosure provides a system, device and method for evaluating the medical condition of one or more vehicle occupants, such as in a post-crash situation. According to one embodiment, a system for detecting the breathing or heartbeat of a vehicle occupant is provided. The system includes: at least one illumination source configured to project light in a predetermined light pattern on a scene; an imaging device configured to capture a plurality of images, the plurality of images including reflections of the light pattern from one or more occupants in the scene; and at least one processor configured to extract the breathing or heartbeat data of the one or more occupants by analyzing the reflections of the light pattern.

[0011] The present disclosure also provides a method for analyzing a heartbeat or breathing signal from the multiple images of the reflected light pattern. According to an embodiment, the analysis includes the following steps: detecting one or more changes in one or more speckle patterns of at least one of the reflections of the light pattern in at least some consecutive images of the multiple images; identifying micro-vibrations of the at least one object based on the speckle pattern analysis; and analyzing the micro-vibrations to extract a breathing or heartbeat signal. In some embodiments, the breathing or heartbeat signal is used to assess the medical condition of the occupant. In some embodiments, the system includes using an illumination source in the near infrared (NIR) spectral range.

[0012] In an embodiment, the system comprises a communication module configured to transmit the medically relevant data to a rescue team of first responders.

[0013] In an embodiment, the data is uploaded to a cloud service for extraction by a rescue team of first responders.

[0014] In an embodiment, the system comprises a memory.In a further embodiment, the system is configured to record post-crash images captured by the imaging device.

[0015] In an embodiment, the post-crash image is sent to a team of first responders via a communication module or uploaded to a cloud service.

[0016] In an embodiment, the system is configured to store the pre-crash image. In a further embodiment, the system is further configured to send the pre-crash image to a first responder team or upload the image to a cloud service for retrieval by a first responder team.

[0017] In an embodiment, the processor is configured to analyze one or more post-crash images and extract post-crash body posture to provide an assessment of the nature or severity of the injury.

[0018] In an embodiment, the processor is configured to analyze one or more stored pre-collision images to extract pre-collision body posture. In a further embodiment, the processor is configured to analyze the pre-collision body posture, optionally in conjunction with any available information about the physical parameters of the impact, to provide an assessment of the nature or severity of the injury.

[0019] In an embodiment, the processor is configured to analyze one or more pre-collision images to extract at least one physical attribute, including but not limited to weight, height, width, volume, age, and gender. In some embodiments, the processor is further configured to use the at least one physical attribute, optionally in conjunction with any available information about the physical parameters of the impact, to analyze and assess the nature or severity of the injury.

[0020] In an embodiment, the assessment is sent to a team of first responders via a communication module or uploaded to a cloud service.

[0021] In an embodiment, the system is triggered to initiate analysis of the nature or severity of the injury by using a vehicle impact sensor. In various embodiments, the vehicle impact sensor comprises a micro switch located inside the vehicle bumper, the micro switch being configured to detect a high impact. In an embodiment, the system uses the same sensor as the vehicle's airbag deployment system.

[0022] In an embodiment, the system is triggered to initiate the analysis by visually detecting rapid, frantic movements of occupants within the vehicle interior.An example of such a detection mechanism is based on an optical flow algorithm configured to analyze relative displacements of objects between two or more consecutive frames.

[0023] According to an embodiment, the system includes at least one high-speed imaging device. In some embodiments, the high-speed imaging device is configured to capture multiple images at a high rate during the accident. In other embodiments, the system processor is configured to analyze the high-rate images to assess the nature or severity of injuries. In additional embodiments, the assessment is based on visual tracking of impacts occurring within the cabin during the accident.

[0024] According to an embodiment, the system comprises a depth sensor configured to generate a depth map within the cabin.

[0025] In an embodiment, the system is configured to store depth data.

[0026] In an embodiment, the processor is configured to analyse post-crash body posture using the depth map to assess the nature and severity of the injury.

[0027] In an embodiment, the processor is configured to analyze the stored pre-impact depth map to provide information about the pre-impact body posture of the vehicle occupant. In several embodiments, the processor is further configured to use the body posture, optionally in conjunction with any available information about the physical parameters of the impact, to analyze and assess the nature or severity of the injury.

[0028] In an embodiment, the processor is configured to analyze one or more pre-impact depth maps to extract physical attributes, including but not limited to weight, height, width, volume, age, and gender. In further embodiments, the processor is further configured to use the at least one physical attribute, optionally in conjunction with any available information about the physical parameters of the impact, in order to analyze and assess the nature or severity of the injury.

[0029] In an embodiment, said analyzing comprises using a combination of at least one image and at least one depth map.

[0030] In an embodiment, the system uses deep learning methods such as, but not limited to, convolutional neural networks, in order to provide the analysis.

[0031] According to another aspect, a system for providing medical information of at least one occupant in a vehicle cabin is provided, the system comprising: a sensing module comprising at least one sensor configured to capture sensory data of the vehicle cabin; and a control module comprising at least one processor configured to: receive the sensory data from the sensor; and

[0032] The sensory data is analyzed using one or more analysis methods to provide the medical information for the at least one occupant.

[0033] In an embodiment, the analysis method is one or more of the following: a computer vision method, a machine learning method, a deep neural network method, a signal processing method.

[0034] In an embodiment, said medical information comprises a medical status of said at least one vehicle occupant following an accident with said vehicle.

[0035] In an embodiment, the medical information includes a medical condition assessment or an injury assessment of the at least one occupant.

[0036] In an embodiment, said information comprises said medical status of said at least one vehicle occupant following an accident with said vehicle.

[0037] In an embodiment, the system comprises a communication module configured to transmit the information to a first responder team.

[0038] In an embodiment, the at least one sensor is an image sensor.

[0039] In an embodiment, the sensing module further comprises at least one illuminator comprising a light source configured to project a light pattern onto the vehicle cabin.

[0040] In an embodiment, the light source comprises a laser or a light emitting diode (LED).

[0041] In an embodiment, the light source comprises one or more optical elements for splitting the single light beams generated by the light source, the one or more optical elements being selected from the group consisting of: a DOE, a splitting mirror and a diffuser.

[0042] In an embodiment, the sensing module comprises a depth sensor.

[0043] In an embodiment, the depth sensor is configured to capture depth data by projecting a light pattern onto the vehicle cabin, and wherein the at least one processor is configured to analyze the depth data for positions of known light pattern elements.

[0044] In an embodiment, the posture of the at least one occupant is estimated by analyzing the depth data using the computer vision method.

[0045] In an embodiment, the sensing module comprises a micro-vibration sensor.

[0046] In an embodiment, the micro-vibration sensor is configured to: project light onto a scene in the vehicle cabin; capture a plurality of images, wherein each of the plurality of images includes a reflected diffuse light element; and

[0047] The processor is configured to: receive captured images; and analyze one or more temporal changes in a speckle pattern of at least one of the plurality of reflected diffuse light elements in at least some consecutive images of the plurality of images to generate micro-vibration data.

[0048] In an embodiment, the sensing module comprises a combination of at least two sensors selected from the group consisting of: an image sensor, a depth sensor, and a micro-vibration sensor.

[0049] In an embodiment, the processor is further configured to classify or identify a property of the one or more objects based on the at least one micro-vibration data.

[0050] In an embodiment, the sensory data comprises a plurality of images, and wherein the at least one processor is further configured to classify the at least one vehicle occupant using the computer vision method by visually analyzing at least one image of the plurality of images captured by the at least one sensor.

[0051] In an embodiment, the sensing module is configured to detect an impending collision of the vehicle.

[0052] In an embodiment, said detection of an impending collision is performed by one or more of:

[0053] Vehicle impact sensors, hectic movement detection sensors; and external sensing systems.

[0054] In an embodiment, the at least one processor is configured to: analyze the sensory data, wherein the sensory data is captured before, during, and after the detection of the impending collision; and evaluate the medical status of the at least one vehicle occupant after the collision based on the analyzed sensory data.

[0055] In an embodiment, said sensory data captured prior to said detection of said impending collision is analyzed using said machine vision method to extract a pre-collision classification.

[0056] In an embodiment, the pre-crash classification includes one or more of body posture, weight, age, body size and gender of the at least one occupant.

[0057] In an embodiment, the system is configured to provide a measure of the likelihood and severity of injury resulting from an automobile accident.

[0058] In an embodiment, the system is configured to provide high rate data of the vehicle cabin subsequent to said detection of said impending collision.

[0059] In an embodiment, the high rate data is used to extract the trajectory of the at least one occupant or one or more objects in the vehicle.

[0060] In an embodiment, the extracted trajectory is used to assess the likelihood and severity of the injury to the at least one occupant.

[0061] In an embodiment, the system is configured to record post-crash information within the cabin; and

[0062] The medical condition of at least one vehicle occupant is assessed.

[0063] In an embodiment, the system is configured to provide at least one of a pre-crash, during-crash, and post-crash medical status assessment of at least one vehicle occupant.

[0064] In an embodiment, the control module is configured to wirelessly communicate with an external party and transmit the information to a first responder team.

[0065] In an embodiment, the information is uploaded to a cloud service.

[0066] In an embodiment, the sensing module is mounted on a roof or a cabin roof of the vehicle.

[0067] In an embodiment, the analyzing is performed using at least one deep learning algorithm.

[0068] According to another aspect, there is provided a system for estimating a medical status of one or more occupants in a vehicle cabin after an accident of the vehicle, the system comprising:

[0069] A sensing module, comprising: an illuminator, comprising one or more illumination sources, the one or more illumination sources being configured to project light in a structured light pattern onto the vehicle cabin; at least one image sensor, configured to capture sensory data before, during, and after a collision of the vehicle, the sensory data comprising a series of 2D (two-dimensional) images and 3D (three-dimensional) images of the vehicle cabin, wherein at least one of the 2D images comprises a reflection of the structured light pattern from the one or more occupants in the cabin; a control module, comprising: a memory module; at least one processor, the at least one processor being configured to: receive the captured sensory data from the sensing module; receive the A collision detection signal of an impending collision of a vehicle; storing received sensory data captured before the collision detection is received in the memory module; analyzing the received sensory data using a computer vision algorithm or a machine learning algorithm to generate pre-collision assessment data including an identification of the state of the one or more occupants before the collision; receiving sensory data captured during the collision from the sensing module; analyzing the received sensory data captured during the collision to generate during-collision assessment data including body trajectories of the one or more occupants; and providing medical information of the one or more occupants after the collision based on the during-collision assessment data and the pre-collision assessment data.

[0070] In an embodiment, the at least one processor is further configured to: receive sensory data captured after the collision; analyze the received sensory data captured after the collision data using a computer vision algorithm or a machine learning algorithm to generate one or more of the post-collision assessment data of the one or more occupants.

[0071] In an embodiment, the post-crash assessment data comprises one or more of the following: heartbeat, breathing rate, body posture, body movement, visible wounds.

[0072] In an embodiment, the processor is further configured to combine the post-crash assessment data with the during-crash assessment data and the pre-crash assessment data to generate the medical information of the one or more occupants after the collision.

[0073] In an embodiment, the heartbeat or breathing rate is identified by analyzing one or more changes in one or more speckle patterns of at least one of the reflections of the structured light pattern in at least some consecutive images of the plurality of images; and vibrations of the one or more occupants are identified based on the speckle pattern analysis.

[0074] In an embodiment, said one or more of said body posture, body movement and visible wound is detected by analysing said series of 2D images or 3D images of said vehicle cabin.

[0075] In an embodiment, the body pose or body movement is identified using one or more of a skeletal model, optical flow of a visual tracking method.

[0076] According to another aspect, a method for providing medical information of at least one occupant in a vehicle cabin is provided, the method comprising: receiving captured sensory data of the vehicle cabin from a sensing module; receiving an impact detection signal of an impending collision of the vehicle; storing the received sensory data captured before the impact detection is received in a memory module; analyzing the received sensory data using a computer vision algorithm or a machine learning algorithm to generate pre-collision assessment data including an identification of a state of the at least one occupant before the collision; receiving sensory data captured during the collision from the sensing module; analyzing the received sensory data captured during the collision to generate during-collision assessment data including body trajectories of the one or more occupants; and providing medical information of the one or more occupants after the collision based on the during-collision assessment data and the pre-collision assessment data.

[0077] According to another aspect, there is provided a method for providing information regarding the medical status or injury of a vehicle occupant of a vehicle to a first responder following an accident, the method comprising the steps of: i. monitoring a cabin of the vehicle using in-cabin sensors;

[0078] ii. combining the analysis of at least one of the pre-crash data, the during-crash data, and the post-crash data; and iii. providing the analysis to a first responder. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] A better understanding of the features and advantages of the present disclosure will be obtained by referring to the following detailed description and accompanying drawings that set forth illustrative embodiments, in which the principles of the present disclosure are employed.

[0080] Figures 1A-1C side views of a vehicle before, during, and after an accident of the vehicle, respectively, according to some embodiments of the present disclosure;

[0081] Figure 2A shows an example of system raw data according to some embodiments of the present disclosure;

[0082] Figure 2B is a flow chart illustrating steps for identifying objects, such as hidden objects in a vehicle cabin, and providing information about the identified objects according to some embodiments of the present disclosure;

[0083] Figure 2C shows a more detailed block diagram of a vehicle including a monitoring system according to some embodiments of the present disclosure;

[0084] Figure 3A shows a high-level block design of a system including a sensing module and a control module according to some embodiments of the present disclosure;

[0085] Figure 3B shows a high-level block design of a system including different types of sensors according to some embodiments of the present disclosure;

[0086] Figure 4 A flowchart for capturing and processing in-cabin data according to some embodiments of the present disclosure is shown;

[0087] Figure 5A A high-level block diagram illustrating a vehicle post-crash data, during-crash data, and pre-crash data collection process according to some embodiments of the present disclosure;

[0088] Figure 5B A flow chart showing a method for detecting the heartbeat and / or respiration of one or more occupants according to some embodiments of the present disclosure;

[0089] Figure 6 A schematic diagram showing possible ways of communicating between a processing system and a team of first responders according to some embodiments of the present disclosure;

[0090] Fig. 7A shows an exemplary configuration for capturing light pattern images and converting identified speckle patterns to measure the heartbeat and / or breathing rhythm of one or more occupants according to some embodiments of the present disclosure;

[0091] Figure 7Bshows an exemplary captured image according to some embodiments of the present disclosure;

[0092] Figure 7C-7E shows exemplary graph results suitable for incorporation according to an embodiment;

[0093] Fig. 8A showing an exemplary enlargement of a captured image of an occupant's chest suitable for incorporation according to an embodiment; and

[0094] Figures 8B-8D Exemplary graph results suitable for incorporation according to an embodiment are shown. DETAILED DESCRIPTION

[0095] In the following description, various aspects of the present invention will be described. For the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the present invention. It is obvious to those skilled in the art that other embodiments of the present invention differ in detail but do not affect the essence thereof. Therefore, the present invention is not limited to that shown in the drawings and described in the specification, but only as indicated in the appended claims, wherein the proper scope is determined only by the broadest interpretation of the claims.

[0096] It should be emphasized that the details shown herein are by way of example and only for the purpose of illustrative discussion of the preferred embodiments of the present invention and are presented in order to provide what is considered to be the most useful and easily understood description of the principles and conceptual aspects of the present invention.

[0097] The configurations disclosed herein can be combined in one or more of a variety of ways to provide a way to notify first responders or rescue teams of information such as medical information including the status of injured occupants, such as in a post-crash accident situation. One or more components and methods of the configurations disclosed herein can be combined with each other in a variety of ways. Systems and methods as described herein use an in-cabin sensing module to monitor the medical status of vehicle occupants and analyze possible injuries and their severity.

[0098] According to some embodiments, the system may be installed and / or mounted and / or integrated and / or embedded in a vehicle, in particular in a cabin of the vehicle.

[0099] According to some embodiments, the system includes one or more imaging devices, and optionally or additionally one or more illumination sources, the one or more illumination sources being configured and capable of projecting light onto a scene in the cabin in a predetermined pattern. The one or more imaging devices are configured to capture multiple images of the scene. The multiple images may contain reflections of the light pattern from one or more objects in the scene (e.g., vehicle occupants). The system may also include one or more processors, the one or more processors being configured and capable of analyzing the multiple captured images to perform post-crash medical status analysis, and / or monitor the medical condition of the vehicle occupants and / or analyze the possible injuries of the occupants and their severity, as described in detail below.

[0100] Alternatively or in combination, the one or more processors are configured to analyze the speckle content of a light pattern induced on the scene by the one or more illumination sources.The speckle pattern may be used to obtain a micro-vibration pattern of one or more objects in the scene.

[0101] According to one embodiment, when one or more occupants are humans present at the scene, the micro-vibration signal can be associated with skin movement caused by breathing movement or heartbeat, or a combination thereof. Thus, the disclosed system can monitor the vital signs of vehicle occupants, for example, in a post-crash situation.

[0102] Advantageously, the system provides both a vital sign signal of breathing or heartbeat and multiple image data of the post-crash cabin, all using a single device or system.

[0103] Additionally, the system provides post-crash and pre-crash data that can be used to further assess details and extent of injuries.

[0104] As used herein, like characters refer to like elements.

[0105] As used herein, the term "light" encompasses electromagnetic radiation having wavelengths in one or more of the ultraviolet, visible, or infrared (IR) portion of the electromagnetic spectrum including shortwave IR, near IR, and long IR.

[0106] As used herein, the term "light pattern" is defined as the process of projecting a known pattern of pixels onto a scene. The term "pattern" is used to refer to the form and shape produced by any non-uniform illumination, specifically patterned illumination using a plurality of pattern features, such as lines, stripes, dots, geometric shapes, etc., having uniform or varying characteristics such as shape, size, intensity, etc. As a non-limiting example, a patterned light illumination pattern may include a plurality of parallel lines as pattern features.

[0107] As used herein, the term "depth map" is defined as an image containing information about the distance from the viewpoint to the surface of scene objects. The depth map can be in the form of a grid connecting all points with z-axis data.

[0108] As used herein, the term "occupant" is defined as any individual present in a vehicle, including the driver and any passengers. The term also includes non-human occupants.

[0109] As used herein, the term "vehicle" refers to a personal automobile, commercial automobile, truck, bus, transporter, drivable mechanical equipment, or any vehicle used to transport people on roads or tracks.

[0110] As used herein, the term "normal vehicle operating mode" refers to any operation of the vehicle, such as while driving or while the vehicle is stopped prior to a vehicle accident.

[0111] As used herein, the term "pre-crash data" refers to any type of data, such as visual images, 3D images of objects within a vehicle, obtained prior to a vehicle collision.

[0112] As used herein, the term "post-crash data" refers to any type of data, such as visual images, 3D images of objects within a vehicle, obtained after a vehicle collision.

[0113] Now referring to the accompanying drawings, Figure 1A , Figure 1B and Figure 1C The vehicle before the accident according to the embodiment is shown respectively ( Figure 1A ), during the accident ( Figure 1B ) and after the accident ( Figure 1C ) and a side view of a vehicle 100 and a passenger cabin 120. The vehicle 100 includes a sensing system 110 that is configured and capable of capturing and obtaining data before an accident (e.g., pre-crash data), data during an accident (e.g., during crash data), and data after an accident (post-crash data). Each data may include visual data (e.g., video images) and stereo data (e.g., depth maps), such as 2D (two-dimensional) images and / or 3D (three-dimensional) images and vibration (e.g., micro-vibration) data of areas and objects within the vehicle 100. According to an embodiment, each data may be analyzed, for example, in real time or near real time to assess the medical status of one or more occupants after a vehicle accident.

[0114] Specifically, the sensing system 110 is configured to monitor areas and objects within the vehicle before, during, and after an accident to obtain various types of sensory data of the areas and objects (e.g., occupants), and analyze the obtained data using one or more processors to extract visual data and depth data and detect speckle pattern dynamics for identifying vibrations (e.g., micro-vibrations), and use the extracted data to estimate the medical status of the occupants. According to an embodiment, a non-limiting example of such an object may be one or more of the occupants of the vehicle, such as the driver 111 or the passenger 112.

[0115] More specifically, methods and systems are provided that are configured to analyze captured sensory data to detect the posture and position of an occupant prior to an accident, as well as the trajectory 118 of the occupant resulting from the vehicle collision and the final accident results (e.g., post-crash data) containing medical information such as information related to the occupant's wounds and heartbeat. In some cases, according to an embodiment, the medical information can be transmitted to external units and forces, such as rescue teams, for example, in real time.

[0116] According to some embodiments, the sensing system 110 may be connected to or may communicate with vehicle units such as vehicle sensors, such as impact detection sensors or 105 and / or airbag control unit (ACU) 108 and / or vehicle computing system (VCS) 109 and / or collision avoidance system (CAS) 105. For example, the sensing system may receive one or more signals containing additional sensory data from CAS 105. The sensing system 110 may communicate with other types of vehicle sensors such as accelerometers, and the additional data may be fused and used with the extracted data to estimate the medical status of the occupant.

[0117] Specifically, according to some embodiments, the sensing system 110 can be connected to or can communicate with the vehicle's CAS, such as wirelessly, which is configured to prevent severe collisions or reduce the severity of collisions. Examples of such CAS can include radar (all-weather) and / or lasers (LIDAR) and cameras (using image recognition) for detecting impending collisions. The sensing system 110 can combine the data received from the CAS with the extracted data, such as after a collision of the vehicle, and use the data to estimate the medical status of the occupants.

[0118] According to some embodiments, the sensing system 110 may be mounted at the cabin roof or roof in a manner that allows viewing of the entire cabin, such as in a front portion 115 of the roof of the vehicle. According to some embodiments, the sensing system 110 may be installed, mounted, integrated and / or embedded in a vehicle, specifically in the cabin of the vehicle, such that the scene is the cabin interior 120, and the objects present in the cabin may include, for example, one or more of the following: vehicle occupants (e.g., driver, passenger, pet, etc.), one or more objects associated with the cabin (e.g., seats, doors, windows, headrests, armrests, etc.), items associated with one or more vehicle occupants (e.g., baby seats, pet carriers, briefcases, toys, etc.), etc.

[0119] It is important to emphasize that one of the main advantages of mounting the system 110 in the roof or cabin roof of the vehicle is that in this location of the vehicle, the system 110 is less exposed to damage and strong impacts during an accident.

[0120] According to some embodiments, the systems and methods are configured to generate outputs for one or more occupants following an incident of a vehicle, such as one or more output signals containing medical information such as a real-time medical status.

[0121] According to some embodiments, the sensing system 110 may include, for example, one or more sensors of different types, such as 2D imaging sensors and / or 3D imaging sensors (e.g., stereo cameras) and / or RF imaging sensors and / or vibration (micro-vibration) sensors, structured light sensors, ultrasonic sensors, etc., to capture sensory data of the vehicle cabin, as will be described in detail in the following. Figure 3B Specifically, the 2D imaging sensor may capture images of, for example, a vehicle cabin from different angles and generate a raw visual image of the cabin. In an embodiment, the sensing system 110 may include an imaging sensor configured to capture 2D and 3D images of the vehicle cabin and at least one processor configured to analyze the images to generate a depth map of the cabin.

[0122] In another embodiment, system 110 may use one or more vibration sensors to detect vibrations (eg, micro-vibrations) of one or more objects in the cabin, and / or analyze captured 2D or 3D images to identify vibrations (eg, micro-vibrations) of objects.

[0123] According to another embodiment, the system 110 may also include a face detector sensor and / or a face detection and / or face recognition software module for analyzing the captured 2D and / or 3D images.

[0124] In an embodiment, system 110 may include or may be in communication with a computing module comprising one or more processors configured to receive sensory data captured by sensors of system 110 and analyze the data based on one or more of a computer vision algorithm and / or a machine learning algorithm to generate medical information including an estimate of a medical condition of one or more occupants in a vehicle cabin, as described below.

[0125] Specifically, according to an embodiment, one or more processors are configured to combine various types of sensory data of the vehicle cabin (e.g., 2D data (e.g., captured 2D images), 3D data (depth maps)) along a time period of, for example, a few seconds or minutes (e.g., 1, 2, 3, 4-100 seconds and more) before, during, and after an accident to generate medical information related to one or more occupants in the vehicle cabin.

[0126] Advantageously, the system 110 provides only minimal hardware, such as one or more sensors and imagers for capturing visual and depth images of the interior of the vehicle 110. In some cases, an interface to the system 110 can supply the necessary power and transmit the acquired data to the vehicle's computing and / or processing unit, such as the VCS 109 and / or the ACU 108, where all processing is performed using its computing power. Thus, according to some embodiments, installing the system 110 becomes very easy and uses off-the-shelf components.

[0127] According to some embodiments, sensing system 110 includes one or more illuminators, such as illuminator 103, and sensor 101. Illuminator 103 generates light, such as a pattern of light, which is schematically indicated here as light spots 102. In some cases, the generated light pattern can cover all or selected portions of the occupants of vehicle 100, such as passenger 203 and driver 207, as shown in FIG. Figure 2A shown.

[0128] According to some embodiments, the sensor 101 may be selected from the group consisting of: a time-of-flight (ToF) image sensor, a camera, an RF (radio frequency) device, a stereo camera, a structured light sensor, an ultrasonic sensor.

[0129] According to some embodiments, the sensor 101 may be an image sensor equipped with a fisheye lens allowing coverage of the entire cabin. In some cases, in order to cover all possible positions of occupants in the vehicle, a typical coverage of more than 150 degrees may be used. The sensor may be a CMOS sensor (Complementary Metal Oxide Semiconductor) or a CCD sensor (Charge Coupled Device) with a resolution of, for example, VGA, 2MP, 5MPMP, 8MP. The lens may function in the visible range or in the IR range.

[0130] According to some embodiments, illuminator 103 generates a light pattern that may cover one or more portions or the entire cabin 120. An example of such a pattern is a spot pattern. Figure 2A This example of a spot pattern 206 covering the front and rear seats in a standard passenger car is shown. The pattern can be a spot pattern such as Figure 2A lines, grids or any other pre-configured shapes as shown. Advantageously, light is concentrated in a small area, e.g. Figure 2A The light spot shown in can improve the signal-to-background ratio and thereby provide a clearer speckle signal.

[0131] Figure 2B 296, which illustrates steps for detecting an occupancy state according to one embodiment, including identifying objects, such as hidden objects in a vehicle cabin, and providing information about the identified objects. Figure 2B As shown, the sensing system 280 includes one or more lighting units such as an illuminator 274 and one or more sensors such as a sensor 276, wherein the one or more lighting units provide structured light with a specific lighting pattern (e.g., light spots or stripes or other patterns) to objects 271 and 272 located at, for example, a rear portion of a vehicle cabin 100, and the one or more sensors capture images of the objects 271 and 272 in the vehicle cabin 100.

[0132] According to one embodiment, the sensing system 280 may include one or more processors, such as the processor 252. The processor 252 may communicate with the device and other processors by wire or wireless communication. For example, the output from the processor 252 may trigger a process within the processor 252, or may be transmitted to another processor or device to activate a process of the other processor or device.

[0133] According to another embodiment, the processor 252 may be external to the sensing system 280 and may be embedded in the vehicle or may be part of the vehicle's processing unit.

[0134] In one embodiment, processor 252 may instruct illuminator 265 to illuminate specific areas within the vehicle cabin.

[0135] According to some embodiments, the sensing system 280 may further include an RF transmit-receive unit 275, for example, including: an RF transmit-receive unit 275 such as an RF transceiver, which is configured to generate an RF beam using an RF antenna 275 and direct it toward the objects 271 and 272, and receive the reflected RF beam to provide an RF image of the vehicle cabin 100 and the objects 271 and 272. The captured image including, for example, the RF signal and the reflection pattern image is provided to the processor 252 to generate a depth map representation 291 and 2D / 3D segmentation of the vehicle cabin 100 and / or the objects 271 and 272.

[0136] According to one embodiment, the sensing system may include a sound sensor 269, such as one or more ultrasonic sensors and / or directional microphones, configured and capable of detecting the presence and / or signs of vitality of a person, to locate, for example, the mouth of a person who is speaking, and to generate data input to detect the position of one or more objects in the vehicle. According to some embodiments, the processor 252 may also receive and / or generate additional data 279, including, for example, information about the vehicle state 278, speed, and acceleration 277 captured by the vehicle sensors 273. The sensory data 282 and the additional data 279 are analyzed by the processor 252.

[0137] According to some embodiments, sensory data 282 and additional data 279 are analyzed using a variety of computer vision algorithms and machine learning algorithms. These may include, but are not limited to, convolutional neural networks that detect people, networks that specialize in detecting faces, hands, torsos, and other body parts, networks that can segment images based on 2D and 3D images and specifically segment passengers in images, algorithms that can calculate the volume of objects and people, and algorithms that can determine if there is motion in a certain area of ​​the car.

[0138] The analysis outputs various types of data about objects in the vehicle cabin, such as information about occupants or inanimate objects, such as a box or bag of groceries or an empty child seat, including, for example. Specifically, in some cases, the data may include: body parts of object 289 and / or motion of object 288 and / or volume and / or occupancy status of object 287 detected based on deep learning 286 in the vehicle cabin.

[0139] According to some embodiments, the various types of data may include depth data 294, including, for example, one or more depth images obtained by sensor 276. The depth data may be specifically used to detect body parts of passengers and to segment bodies or body parts in the images. In some cases, the depth data may also be used to determine the posture of a person, such as sitting upright, leaning to the side, or leaning sideways.

[0140] According to some embodiments, various types of data may contain prior knowledge 292 .

[0141] Non-limiting examples of prior knowledge may include: information about vehicle units, such as door / window / seatbelt states and / or prior information about objects and / or passengers in the vehicle, and / or rules or assumptions, such as physical assumptions or seat switching rules related to the possibility of movement inside the vehicle, for example to exclude unlikely changes in the occupancy prediction or to confirm expected changes in the occupancy state in the vehicle instead. Specific examples of physical assumptions may include, for example, a high probability that the driver's seat is occupied in a driven vehicle (non-autonomous vehicle) and / or a low probability that a passenger can move from one seat to another within a predetermined short time or in a single frame.

[0142] At the following step 293, the multiple types of data are fused to determine the number of seats for detecting and / or identifying the seating situation and / or objects (e.g., passengers) at step 294, such as the position and attributes of objects 271 and 272 (e.g., whether the passenger is sitting upright, sideways, or leaning forward). For example, the detection may include identifying the complete body or body parts, such as the face or hands, of one or more passengers, such as the rear portion of the vehicle.

[0143] According to one embodiment, multiple types of data are fused by a fusion algorithm that outputs an optimal decision that takes into account each data input (e.g., the motion of object 288 and / or the volume of object 287 and / or the riding state based on deep learning 286, facial detection 290, prior knowledge 292).

[0144] Specifically, the fusion algorithm includes analyzing the fused data to generate a stochastic prediction model (e.g., a Markov chain, e.g., in the form of a Markov matrix) that predicts one or more riding state probabilities. The prediction model is used to continuously update the probability of one or more current riding states (e.g., a probability vector) over time to generate an updated riding state, such as determining the position of objects 271 and 272 and / or the number of objects in the vehicle cabin in real time. In some cases, the predicted state and the current state are combined by weighting their uncertainties using, for example, a linear time-invariant (LTI) method, such as an infinite impulse response (IIR) filter. Once a body or body part is detected, the object is tracked at step 295. For example, conventional trackers such as correlation trackers and edge trackers can be used to track objects in 2D images or depth images.

[0145] According to one embodiment, based on the determined occupant state, the processor 252 may output data or signals at step 295 that may be used to provide information and / or to control a device that may be remote or integrated with the vehicle, for example, an electronic device such as an alarm, sirens, or lighting that may sound an alarm when out of position and thus activate an occupant protection device (e.g., airbag) or other device. According to an embodiment, the device may be controlled by the signal output, such as being activated or modulated.

[0146] In some cases, the output signals may include a seat belt reminder (SBR), an out-of-position indication (OOP) such as for air bag disabling, and a driver monitoring system (DMS) for driver alerting.

[0147] Alternatively or in combination, once an object is identified using a neural network, the neural network is trained to output a number of points, such as a predetermined number of points corresponding to specific body parts or skeleton lines. At a subsequent step, a conventional tracking algorithm, such as a Kalman filter, can be used to track these points and lines in time. It should be emphasized that the tracker can estimate the position of these points even if tracking is lost, such as when the body is completely or partially occluded, i.e., by the front seat.

[0148] According to some embodiments, body parts of passengers can be detected using computer vision algorithms and / or neural networks that are trained to detect people, such as human body outlines, such as complete human bodies and / or specific human body parts. Specifically, the faces of people can be detected by well-known detection or recognition methods. Non-limiting examples of such methods include the Viola-Jones algorithm or the SSD neural network. Body parts can be detected by a neural network that is specifically trained to detect body postures. Non-limiting examples of such methods include the OpenPose or OpenPose Plus methods.

[0149] Figure 2C 1 shows a more detailed block diagram of the vehicle 100. It is worth noting that the vehicle 100 may include more than Figure 2B For example, the vehicle 100 may include a wired or wireless system interface, such as a USB interface or Wi-Fi, for connecting the sensing system 110 and the computing unit 205 of the vehicle with the vehicle systems 200, such as vehicle systems 202-208, the vehicle sensors 201, such as vehicle sensors 212-218, the seat positions 232-238, and the corresponding seat sensors 232'-238'. Figure 2BThe hardware architecture of represents one embodiment of a representative vehicle that includes a sensing system 110 configured to automatically detect and identify objects and areas in the vehicle, such as hidden objects. In this regard, Figure 2C The vehicle in the embodiment also implements a method for monitoring objects and / or passengers to identify attributes of the objects and / or passengers, even in situations where the objects and / or passengers may not be identified due to temporary low visibility or false detection. The identification can be based on various types of information received in real time from various types of sensors in the vehicle, such as dedicated sensors embedded in the sensing system 110 and existing vehicle sensors embedded in various locations in the vehicle 100.

[0150] like Figure 2C As shown, the vehicle 100 includes a vehicle computing unit 205 that controls the vehicle's systems and sensors and a sensing system 110 that includes a control board 250 that controls the sensing system 110 and may communicate with vehicle units and systems such as the vehicle computing unit 205 and the vehicle system 200. In some cases, the control board may be included in the computing unit 205. The vehicle 100 also includes passenger seat locations 232-238, which include a driver's seat location 232, a front passenger seat location 234, and left and right rear passenger seat locations 236 and 238. Although for illustration purposes, the control board 250 is not shown in FIG. Figure 2B Four seating positions are shown, however the invention is not so limited and may accommodate any number of seats in any arrangement within the vehicle. In one embodiment, each passenger seating position has automatically adjustable settings for seating comfort, including but not limited to seat height adjustment, fore and aft adjustment position, and seat back angle adjustment. Each passenger seating position may also include one or more sensors 232'-238', each individually configurable, for controlling the passenger's seat and windows, as well as environmental controls for heating, cooling, ventilation direction, and audio / video consoles as appropriate. In some cases, passengers may have communication devices 242-248, one at each passenger position, indicating that each passenger in the vehicle 100 carries a communication device. Although Figure 2C The exemplary embodiment shown in shows each passenger carrying a communication device, but various embodiments contemplate that not all passengers need to carry a device.

[0151] According to one embodiment, the sensing system 110 may include multiple sensors of different forms. For example, the sensing system 110 may include a vibration sensor 241 and / or an acceleration sensor 242 and / or a 3D sensor 243 and / or an RF sensor 244 and / or a camera 245 such as a 2D camera.

[0152] In some cases, the sensing system may include an image sensor and a light source, such as a coherent light source suitable for projecting a structured light pattern such as a multi-beam pattern on a vehicle cabin, the image sensor being used to map a speckle field generated by each light spot formed on a surface of the vehicle. According to an embodiment, a processor such as processor 252 is configured to process the speckle field information received by the image sensor and derive surface vibration information to identify one or more objects in the cabin, including, for example, motion and micro-vibrations of one or more detected objects.

[0153] The projected structured light pattern may be composed of multiple diffuse light elements, such as dots, lines, shapes, and / or combinations thereof, which may be reflected by one or more objects present in the scene and captured by an imaging sensor integrated into the integral imaging device.

[0154] According to some embodiments, the control unit 250 may be connected to the vehicle systems 202-208, the vehicle's sensors 110, the sensing system sensors, the seats 232-238, and / or the seat's sensors '232'-238' via one or more wired connections. Alternatively, the control unit 250 may be connected to the vehicle systems 202-208, the vehicle's sensors 110, the sensing system sensors, the seats 232-238, and / or the seat's sensors '232'-238' via a wireless interface via a wireless connection unit 252. The wireless connection 252 may be any wireless connection, including but not limited to Wifi (IEEE 802.11x), Bluetooth, or other known wireless protocols and / or.

[0155] According to an embodiment, the computing unit 205 is also preferably controllably connected to the vehicle systems 202-208, the vehicle's sensors 110, the sensing system sensors, the seats 232-238, and / or the seat's sensors '232'-238'. The vehicle systems 222-228 may be connected via wired connections as shown in FIG. 2, or by other means.

[0156] Vehicle systems may include, but are not limited to, engine tuning systems, engine limiting systems, lights, air conditioning, multimedia, GPS / navigation systems, etc.

[0157] Control board 250 may include one or more of processor 252, memory 254, and communication circuitry 256. The components of control board 150 may be configured to transmit, store, and / or analyze captured sensory data, as described in further detail herein.

[0158] The control unit may also be connected to a user interface 260. The user interface 260 may include an input device 261, an output device 263, and a software program configured to allow a user to interact with the control panel. Such input and output devices include, but are not limited to, a display 268, a speaker 266, a keyboard 264, a directional key, a directional knob, a microphone 265, a touch screen 262, etc.

[0159] Microphone 265 helps capture sound (e.g., voice commands) and converts the captured sound into electrical signals. In some cases, the electrical signals can be used by the onboard computer 104 that interfaces with various applications 352.

[0160] Processor 252 may include tangible media including computer program instructions; for example, the processor may include a digital signal processing unit that may be configured to analyze and fuse data such as sensory data received from various sensors using multiple types of detection methods. In some cases, the processed data output may then be transmitted to communication circuit system 256, which may include a data encryption / transmission component, such as Bluetooth TM. Once encrypted, the data output may be transmitted to a vehicle computing unit and / or a vehicle user interface via Bluetooth, and may be further presented to a driver on the vehicle. Alternatively or in combination, the output data may be transmitted to a monitoring unit interface.

[0161] Figure 3A A schematic diagram of a sensing system 310 according to an embodiment is shown, the sensing system being configured and capable of capturing sensory data of a scene such as a vehicle cabin 320 containing one or more occupants (e.g., a driver 311 and / or a passenger 312), and analyzing the sensory data to estimate a medical condition of the one or more occupants, such as after an accident of the vehicle. In some cases, the sensing system 310 may be Figure 1A and 1B system 110.

[0162] System 310 includes sensing module 300 and control module 315. The two modules may reside in the same package, or may be separated into two different physical modules. For example, for a split configuration, sensing module 300 may be located on the roof of the vehicle, while control module 315 may be located behind the dashboard. The two modules are connected via a communication line and / or may communicate electrically and / or wirelessly, for example, via a dedicated connection such as a USB connection, a wireless connection, or any connection known in the art.

[0163] In one embodiment, the sensing system 310 is connected to the vehicle's power source. In an alternative embodiment, the system 310 includes a battery, optionally a rechargeable battery, which allows operation even when the vehicle's power source is disconnected. This design will allow the system to continue to operate even if the vehicle's power source fails during an accident. In another embodiment, the battery can be charged from the car's battery or the car's alternator.

[0164] The sensing system 310 may also be equipped with or have an interface to a crash sensor 325 that is configured to detect an impending collision. A non-limiting example of such a crash sensor is a crash sensor that is present in an automobile and is responsible for airbag deployment. In this configuration, notification of the impact is transmitted to the system 310 via an electronic signal from the crash sensor. The electronic signal may be transmitted directly or via a communication system such as a CAN bus interface of the vehicle. In some cases, the crash sensor may be or may be included in a vehicle as described herein. Figure 1A In the CAS mentioned.

[0165] Alternatively or in combination, the sensing system 310 is equipped with a built-in impact sensor, such as an accelerometer 312. When an acceleration (or more precisely a deceleration) above a certain threshold is detected, the system 310 considers that an impact signal has been issued and a collision will occur soon.

[0166] In another embodiment, an impact may be determined by analyzing data captured from within the vehicle cabin, for example using sensing module 300. For example, system 310 may monitor video motion of the driver, passengers, and objects in the cabin. When rapid or hectic movement exceeding a predefined threshold is detected, the system may conclude that an impact is occurring. Such analysis may be based on computer vision algorithms, such as, but not limited to, optical flow or tracking.

[0167] According to an embodiment, the sensing module 300 includes an image sensor 301 and at least one illuminator 303, wherein the image sensor can be configured to capture sensory data including one or more images of a scene (e.g., a vehicle cabin) and further transmit visual data to extract visual data, a depth map (e.g., a density depth map) and vibrations (e.g., micro-vibration data) of the scene using a control module, as further described in detail herein.

[0168] Image sensor 301 is equipped with lens 302. The lens may be a fisheye lens that covers the entire cockpit. Illuminator 303 produces a light pattern that illuminates scene 320. The light pattern may optionally be designed to cover approximately the same field of view as the image sensor.

[0169] Image sensor 301 may be a CMOS or CCD sensor of various resolution formats. For example, it may be a VGA format (640x480 pixels) or a 2M pixel format (1920x1080).

[0170] To generate the illumination pattern, the illuminator 303 may include a light source 324, which may be a coherent laser light source as an example. The illuminator 303 may also be equipped with a pattern generating optical element, which may be a diffractive optical element (DOE) 326 as an example. Other examples for pattern generation may be a mask, or a separate mirror and a diffuser.

[0171] For example, light source 324 can include electromagnetic energy of wavelengths in the optical range or a portion of the electromagnetic spectrum, including wavelengths in the human visible light range or a portion thereof (e.g., about 390nm-750nm) and / or wavelengths in the near infrared (NIR) (e.g., about 750nm-1400nm) or infrared (e.g., about 750nm-1mm) portion and / or near ultraviolet (NUV) (e.g., about 400nm-300nm) or ultraviolet (e.g., about 400nm-122nm) portion of the electromagnetic spectrum. The specific wavelengths are exemplary and are not meant to be limiting. Other wavelengths of the electromagnetic range can be used. In some embodiments, the illuminator 303 wavelength can be any of about 830nm, about 840nm, about 850nm, or about 940nm.

[0172] In a specific embodiment, the illumination is performed in the near infrared (NIR, approximately 750 nm to 1400 nm) spectral range in order to prevent the pattern from being visible to the naked eye of the occupants.

[0173] In some cases, the image sensor 301 can be equipped with a bandpass spectral filter to prevent it from capturing light with wavelengths that do not match the wavelength of the illuminator. When this bandpass filter is added to the image sensor 301, the signal-to-background ratio is improved.

[0174] In some aspects of the embodiments, the image sensor 301 and the illuminator 303 (eg, the sensing module 300 ) are both connected to or can communicate with one or more processors, such as the processor 304 located in the control module 315 .

[0175] In several embodiments, the processor 304 can operate the illuminator 303 and the image sensor 301 according to the monitoring strategy. For example, the processor 304 can operate the image sensor 301 at a frame rate of 30 frames per second (FPS), and operate the illuminator 303 as a constant light source. As another example, the processor 304 can operate the image sensor 301 at a rate of 30 FPS, and alternatively the illuminator 303 can obtain a pattern only every other frame, while the remaining frames have no pattern. For example, an option can include capturing a clean frame (e.g., it does not include a reflection pattern) and then patterning the frame. Alternatively, capture 5 clean frames, then capture a pattern frame, etc. The latter mode may be desirable because it can be advantageously used to obtain alternative clean image frames for deep learning algorithms, as well as pattern frames that can be used for vibration monitoring and vital sign extraction. Other options can be used, which include different relationships between the number of pattern frames and the number of clean frames.

[0176] In an embodiment, processor 304 is configured to extract any one or more images or combinations of images, such as video images, depth data, and vibration data, from the video stream.

[0177] In an embodiment, extracting the video image is performed by selecting a video frame in which the illumination source 324 is off and no light pattern is projected onto the scene. In another embodiment, the video image may contain a light pattern produced by an illuminator.

[0178] In an embodiment, extracting depth data is performed by using a structural technique. The light pattern caused by the illuminator 303 is captured by the image sensor 301. The processor 304 is configured to receive the captured image and analyze the captured image to identify the displacement of each previously known light pattern element, thereby calculating the depth of the object at this position.

[0179] The depth of each pattern element can be estimated by using the formula in Equation 1 below:

[0180] Equation 1:

[0181]

[0182] where z represents the depth estimate, B represents the baseline distance between the image sensor and the pattern illuminator, f represents the focal length of the camera module lens, and Δ represents the disparity, i.e., the distance the pattern element is displaced on the image plane.

[0183] Alternatively or additionally, the depth can also be obtained by using a lookup table mechanism. In order to use this mechanism, the system needs to go through a calibration process in which a light pattern is projected onto a screen positioned at several predetermined distances. For each distance, the pattern is recorded and stored in a memory such as memory module 305. Then, for each small area containing some light pattern elements observed by the camera during operation, a correlation-based algorithm is used to evaluate at which screen distance the closest matching pattern element appears. The distance with the highest correlation is selected as the estimated distance for this area.

[0184] It should be noted that the number of extracted depth points does not need to match the number of pixels in the image. Instead, it is related to the light pattern caused by the illuminator. According to an embodiment, the number of depth points is defined by the number of different light pattern elements in the projected pattern. For example, this can be the number of spots in a pseudo-random spot pattern, or a smaller coefficient of said number resulting from using clusters of 2-3 points as different patterns.

[0185] In another exemplary embodiment, the processor 304 is further configured to extract vibration information, such as micro-vibration information, from the scene. The extraction of micro-vibration is performed by analyzing the speckle content of at least some light (e.g., light speckle) pattern elements in the captured image. Changes in the speckle pattern over time may indicate micro-vibrations, i.e., very small and subtle movements that may be too small to be detected by analyzing changes in depth data or video images that do not contain light pattern elements.

[0186] In some embodiments, the speckle pattern analysis may include detecting changes in the speckle pattern by measuring the temporal standard deviation of the intensity of corresponding reflected diffuse light elements over a plurality of consecutive captured images to identify a temporal distortion pattern. i is the grayscale intensity of a particular pixel depicting a light pattern element and / or a portion thereof in image number i, then one or more processors, such as processor 304, may calculate the temporal standard deviation according to Equation 2 below.

[0187] Equation 2:

[0188]

[0189] Where n represents the current image, and k represents the number of previous images.

[0190] The analysis may also include comparing the result of the time standard deviation to a predetermined threshold to determine whether micro-vibrations occur. In the case where the time standard deviation value exceeds the predetermined threshold, for example, the processor 304 determines that the amplitude of the micro-vibrations in the specific light pattern element increases. On the other hand, in the case where the time standard deviation value does not exceed the predetermined threshold, the processor 304 may determine that the amplitude of the micro-vibrations has not changed. In some embodiments, the predetermined threshold may be fixed and pre-set. Optionally, the predetermined threshold may be dynamically adjusted based on the value of the time standard deviation measured over time.

[0191] Optionally, to improve immunity to noise that may affect the intensity level of the speckle pattern and to increase the signal-to-noise ratio (SNR) of the intensity of the speckle pattern, the temporal standard deviation may be averaged over a number of pixels (eg 5x5 pixels) per spot.

[0192] Optionally, to improve immunity to noise that may affect the intensity level of the speckle pattern and to increase the signal-to-noise ratio (SNR) of the intensity of the speckle pattern, the temporal standard deviation may be averaged over multiple speckle patterns of diffuse light elements reflected from the same surface and depicted in the same area in the captured image.

[0193] According to another embodiment, a change in the speckle pattern may be detected, for example by the processor 304, by analyzing a lateral translation of the speckle pattern, which indicates a tilt of the reflective object relative to the image sensor. The tilt, which may be very small, for example on a micro-radian scale, may be derived from a translation speed of one or more speckle pattern points over time (consecutive frames). The lateral speckle pattern translation may be derived from an analysis of diffuse light pattern elements depicted in a plurality of consecutive captured images according to Equation 3 below.

[0194] Equation 3:

[0195]

[0196] Where I represents the intensity of a pixel in the captured image, and the gray level varies with respect to time t or position x.

[0197] In the captured image, the angular velocity of a certain pixel (i, j) relative to its neighboring pixels in the i direction can be expressed by the following equation 4:

[0198] Equation 4:

[0199]

[0200] The angular velocity of a pixel (i, j) can be expressed similarly in the j direction. The angular velocity result is expressed in pixels per frame.

[0201] Optionally, the intensity I of pixel (i, j) i,j The intensity I may be normalized over time, for example by processor 304, to compensate for the intensity I due to the spot intensity encapsulation effect. i,j For example, the intensity I i,j The intensity I of one or more pixels (i, j) in the consecutive captured images can be calculated by applying a sliding time window i,j The average is taken for normalization.

[0202] In another example, the processor 304 may i,j Apply an infinite impulse response to smooth the intensity I in the time domain i,j To produce smoothed intensity As shown in Equation 5 below.

[0203] Equation 5:

[0204]

[0205] where α is a small coefficient, such as 0.05.

[0206] The intensity I of one or more pixels (i, j) can be i,j Normalized by dividing by the average intensity measured over time in multiple consecutive captured images to produce a normalized intensity As shown in Equation 6 below.

[0207] Equation 6:

[0208]

[0209] Substituting the intensity in Equation 4 The expression of is replaced by the normalized intensity The angular velocity can be expressed by the following Equation 7.

[0210] Equation 7:

[0211]

[0212] In some embodiments, to further improve the stability of the measured intensity against noise effects, the processor 304 may also spatially average the intensities of multiple adjacent reflected diffuse light elements (e.g., points, spots, etc.) in the captured image. The processor may also apply temporal filtering to the spatially averaged intensity value to improve the resulting intensity signal.

[0213] Further details on speckle pattern analysis for detecting micro-vibrations can be found in U.S. Patent No. US10,345,137 entitled “System and Method for Detecting Surface Vibrations” and PCT International Application Publication No. WO2019012535 entitled “System and Method for Acquiring Information from the Environment”, each of which is incorporated herein by reference in its entirety.

[0214] It should be noted that, depending on the configuration of the processor 304, depth data and vibrations (eg, micro-vibrations) may be extracted from the captured images separately or together (eg, simultaneously).

[0215] In some cases, the processor 304 may also analyze the extracted depth data and vibrations (e.g., micro-vibrations) and the captured visual data to provide medical information including an analysis of a medical condition, such as an injury-related condition of one or more occupants in the vehicle. In some cases, the medical information is provided in real time or near real time. In some embodiments, the processor analyzes post-crash vital signs, posture, and injury severity, as described in further detail herein.

[0216] According to various aspects and embodiments, the processor 304 is connected to or can communicate with a memory module 305. The memory module is configured to store data during analysis, such as 2D data (e.g., visual images) and / or 3D data and / or vibrations. The memory module can also store pre-crash data for retrieval in the event of an accident. In some cases, the memory module 305 can receive data from an internal unit in the vehicle, such as a computer device and / or a memory unit of the vehicle, and / or from an external device such as a mobile phone device.

[0217] Processor 304 can be connected to communication module 306 or can communicate with it (e.g., wirelessly) to transmit analysis results (e.g., medical data) to the outside world. In some cases, the results can be transmitted to a cloud-based storage system 307 or a rescue team 309.

[0218] In many embodiments, the sensing system 310 can communicate 116 wirelessly with the cloud-based storage system 307. In some cases, the system can transmit data to the mobile device 350 using a communication module 306 having a communication link, such as a wireless serial communication link, such as Bluetooth. TM The handheld device can receive data from the system and transmit the data to the backend server of the cloud-based storage system 307 .

[0219] The transmitted data may be structured in the form of a message containing information such as the position or status of one or more occupants in the vehicle after the collision. In an embodiment, the transmitted data may also contain an image or video sequence formed by the sensor.

[0220] According to an embodiment, the estimation of the post-crash medical condition of the vehicle occupant may be performed based on any combination of pre-crash data, during-crash data, and post-crash data as described in further detail herein.

[0221] Figure 3B A schematic diagram of a sensing system 370 according to an embodiment is shown, which is configured and capable of capturing sensory data of a scene such as a vehicle cabin 320 containing one or more occupants (e.g., a driver 311 and / or a passenger 312), and analyzing the sensory data to estimate a medical condition of one or more occupants, such as after an accident with the vehicle.

[0222] System 370 exhibits all of the elements of system 310 described above, but also includes a plurality of sensors, such as an image sensor 372 , a depth sensor 374 , such as a stereo camera, and a micro-vibration sensor 376 .

[0223] Figure 4 A flow chart of a method 400 for capturing and processing in-cabin data including pre-crash data, during-crash data, and post-crash data to provide information such as medical information after or during a vehicle accident is shown according to an embodiment. At step 410, during normal vehicle operation, e.g. Figures 1A-1C System 110 or Figure 3A A system such as system 310 receives and records data such as sensory data (e.g., raw data—defined as basic data collected from one or more sensors, such as sensors in a sensing module) that has been captured within a vehicle cabin. According to an embodiment, the sensory data can be captured by sensing module 300 and can include any one or more of video images, depth data, and vibration data. In some cases, a buffer such as buffer 301 can be used to store the captured sensory data. In some cases, buffer 301 can be a circular buffer including a storage capacity of, for example, a length of 10 seconds, a length of 20 seconds, or a greater length. The term "loop" as used herein refers to a fixed amount of memory used to store data captured over a period of time, such that when the memory is filled, old data is overwritten by new data in a cyclic manner.

[0224] In some cases, the sensory data may include vehicle interior data obtained from vehicle units such as vehicle embedded sensors.

[0225] At step 420, a crash detection signal may be received, for example, at a control module. In some cases, the crash detection signal may be generated by the crash sensor 325 and / or the processor and / or the sensing module after an impending collision detection. In some cases, the crash detection is obtained from a vehicle embedded sensor such as a CAS or other sensing module.

[0226] A non-limiting example of such a collision sensor is a collision sensor present in a car and responsible for airbag deployment. In this configuration, notification of the collision is transmitted to the system via an electronic signal from the collision sensor. The electronic signal can be transmitted directly or via a communication system such as a CAN bus interface of the vehicle. In another embodiment, the system is equipped with a built-in collision sensor such as the accelerometer mentioned above. When an acceleration (or more precisely a deceleration) above a certain threshold is detected, the system considers that a collision signal has been issued. In another embodiment, the collision situation is analyzed based on system data. For example, the system can monitor the video movement of the driver, passengers and objects in the cockpit. When fast or hectic movement exceeding a predefined threshold is detected, the system will conclude that a collision is occurring. Such analysis can be based on computer vision algorithms, such as but not limited to optical flow or tracking.

[0227] In some cases, at step 430, upon receiving an impact detection signal, such as from the impact detection sensor 302, the system mode is changed from the "normal vehicle operating mode" to the "accident mode". According to an embodiment, as part of the "accident mode", the system operates one or more actions as follows: At step 432, recording on the circular buffer 301 is stopped, and the raw data (e.g., sensory data) contained in the buffer is "frozen" in the memory 303 for further analysis.

[0228] Alternatively or in combination, the sensory data may also be transmitted to an external source, such as a cloud-based memory unit 307 located at an emergency agency, such as the police, hospital, etc., for advanced collision investigation. The transmission may occur during the collision or at a later stage initiating the investigation.

[0229] In an embodiment, details of an impact, such as one identified by the processor, may also be recorded by the processor. Such details may include the magnitude and / or direction of the impact. These details may then be used to assess the outcome of the accident.

[0230] At step 440, according to an embodiment, the sensory data stored in the buffer is analyzed, for example, by a processor, to generate pre-crash data including, for example, an identification of the state of the vehicle occupants prior to the collision (e.g., pre-crash state). Such analysis may provide any combination of the occupant's position, body posture, physical attributes, such as weight, height, age, gender, and body shape. The analysis may also include objects present in the cabin that may pose a risk of injury during the accident.

[0231] The analysis described herein can be performed using computer vision algorithms and / or machine learning algorithms. The location of the occupants can be obtained by training a computer vision module to identify the presence of individuals within the vehicle. Body posture can be obtained by training a neural network system to estimate body posture from video images. Body attributes can be obtained by training a neural network to estimate attributes from video images.

[0232] In embodiments, the analysis may also be performed or enhanced by using a depth data layer, providing information about the volume of the individual and its physical dimensions. The depth information may also provide information about body posture.

[0233] In an embodiment, the processor is configured to analyze one or more pre-collision depth maps to extract body attributes, including Figure 2B and 2C Weight, height, width, volume, age and gender as shown in .

[0234] In an embodiment, said analyzing comprises using a combination of at least one image and at least one depth map.

[0235] At step 450 , according to an embodiment, following the identification of the pre-crash vehicle occupant status at step 440 , the system evaluates possible accident outcomes in terms of injuries and medical issues based on the analyzed pre-crash data contained in the identification of the occupant status prior to the accident.

[0236] In an embodiment, the system employs a human mechanical model that can use recorded pre-crash data to assess vehicle crash outcomes in terms of injuries and medical issues. Available pre-crash information may include physical attributes of one or more or all vehicle occupants, initial body position, weight, age, body size and gender, and crash parameters such as magnitude and direction.

[0237] In some cases, the human mechanical model may be a finite element model of the body.Alternatively or in combination, the human mechanical model may contain heuristic equations based on an existing injury database.

[0238] In an embodiment, the system is triggered to initiate analysis of the nature or severity of the injury by using a vehicle impact sensor. In various embodiments, a vehicle impact sensor such as sensor 325 comprises a micro switch located inside the vehicle bumper that is configured to detect a strong impact. In an embodiment, the system uses the same sensor as the vehicle's airbag deployment system.

[0239] To illustrate factors that may be considered in such an assessment, the head posture of the occupant 102 relative to the body prior to the collision is discussed. Figure 1B As shown, the whiplash effect in a car accident can cause various types of injuries to the neck of the occupant. Knowing the direction and magnitude of the impact, such as obtained from the impact sensor, and the position and orientation of the head relative to the body and the car, obtained and extracted by analyzing the captured images, the system can provide a prediction of the nature and severity of the injury. Specifically, the analysis may include identifying which body part was hit. Based on the nature of this body part and the estimated impact intensity, tissue damage, vascular damage, bone damage, etc. are predicted.

[0240] In an embodiment, the system is configured to store the pre-crash image. In a further embodiment, the system is further configured to send the pre-crash image and pre-crash data to a first responder team or upload the image to a cloud service for retrieval by a first responder team.

[0241] In an embodiment, the processor is configured to analyze one or more stored pre-collision images to extract pre-collision body posture. In further embodiments, the processor is configured to analyze the pre-collision depth map (e.g., body posture), optionally in conjunction with any available information about the physical parameters of the impact, to provide an assessment of the nature or severity of the injury.

[0242] In an embodiment, the processor is configured to analyze one or more pre-collision images to extract at least one physical attribute, including but not limited to weight, height, width, volume, age, and gender. In some embodiments, the processor is further configured to use the at least one physical attribute, optionally in conjunction with any available information about the physical parameters of the impact, to analyze and assess the nature or severity of the injury.

[0243] According to some embodiments, once a detection of an impact (e.g., of step 420) is received, for example, at the system (e.g., a processor), at step 460, a sensor of the system, such as a system sensing module, may be instructed to switch the operating mode of the sensor and / or camera to a high frame rate capture mode to collect high frame rate data during the incident. Non-limiting examples of high frame rates include about 500 to 1000 frames per second or more. Non-limiting examples of the duration of the incident include about 1-5 seconds or more.

[0244] At step 470, the system is configured to capture data such as sensory data obtained during an incident such as a high frame rate period (e.g., during-crash data) to follow the trajectories of occupants and objects within the vehicle. The system may use either the captured video images and / or depth data to analyze the trajectories using computer vision methods as known in the art. Trajectories are particularly important for detecting body impacts with structures and various surfaces of the vehicle, as well as detecting impacts of objects within the vehicle with occupants.

[0245] At step 480, the trajectory and impact data are used to provide a medical information output that includes an assessment and prediction of the nature and severity of the occupant's injuries. In some cases, the medical information output may be provided based on additional data, such as pre-crash data. For example, detecting an impact between an occupant's head and the vehicle structure may indicate a high probability of head trauma. More specifically, if Figure 1A As shown, data including the distance 'd' between the head of the passenger 112 and the front seat headrest 113 and the speed of the vehicle 100 during the "normal vehicle operating mode" can be analyzed and combined with the passenger trajectory 118 and the broken window 114 trajectory extracted from the data captured during the vehicle 100 collision.

[0246] In some cases, the medical information output may be transmitted as one or more output signals to an external unit, such as a first responder unit, or to an internal unit of a vehicle to activate one or more devices or systems, such as a device of the vehicle.

[0247] Reference now Figure 5A , which shows a high-level block diagram 500 of a vehicle post-crash data, during-crash data, and pre-crash data collection process according to an embodiment. The collection process can be performed by a sensing system such as the sensing system 110 of FIG. 1 or the sensing system 310 of FIG. 3. In some cases, the system can communicate with or include additional sensors. As schematically shown, the system 110 collects some or all of the information available in the vehicle cabin to generate medical information including an assessment of the status of the occupant's medical condition and the nature and severity of the injury.

[0248] According to an embodiment, each type of captured information may be converted into one or more signals containing relevant information.

[0249] In an embodiment, a system such as system 110 can collect the heartbeat 501 of an occupant by capturing and analyzing micro-vibrations of the body, such as one or more micro-vibration signals obtained by system 110 and analyzed by one or more processors. The micro-vibrations are captured using speckle time analysis as described in detail above. Due to the pulse, the heartbeat causes mechanical micro-vibrations of the skin, which can be detected by the system 110 configured to measure the micro-vibrations. The heart rate or lack of heartbeat provides important information about the status of the medical condition of one or more occupants in the vehicle and the nature and severity of their injuries.

[0250] In another embodiment, a system such as system 110 is configured to collect the breathing rate 502 of the occupant by analyzing micro-vibrations of the body, and specifically micro-vibrations of the chest area. The breathing motion causes micro-vibrations or vibrations that can be detected by the system 110 configured to measure micro-vibrations. The breathing rate or hypopnea provides important information about the medical status of one or more occupants and the nature and severity of their injuries.

[0251] According to an embodiment, the signals of heart rate and breathing rate can be obtained by analyzing the micro-vibration signal of the body after the collision. In order to distinguish the two signals, spectral analysis can be used. For example, the micro-vibration signal may contain a superposition of a stronger breathing signal and a weaker heart rate signal. The typical range of the heart signal may be 50 times per minute (BPM) or more (about 1-1.2Hz), while the typical range of the breathing signal may be 15-20 times per minute (about 3-4Hz). Therefore, performing spectral analysis at about 2Hz in this example and separating the signal into high-pass and low-pass will allow two separate signals to be obtained.

[0252] In addition, the same spectrum analysis can be used in another embodiment to distinguish vital sign signals from mechanical vibrations generated by the vehicle or environment. For example, in a post-crash situation where the engine of a car continues to run, it can be expected that there will be mechanical vibrations associated with the operation of the engine. Assuming the engine is idling at 600 RPM, this means that the typical fundamental frequency of mechanical vibrations caused by the engine is 10 Hz. By using a low pass below this frequency, the engine-induced vibrations can be removed from the captured signal.

[0253] In an embodiment, the system is also configured to detect post-collision body posture 503. The system 110 may use some or all of the captured images and depth data. The system then uses a skeletal model or other heuristic method to estimate the body posture. Body posture, and specifically the relative position and orientation of the head and limbs, can provide valuable information about the state of medical conditions, the nature and severity of injuries. In some cases, the system and method described in USP application No. 62 / 806,840, entitled "SYSTEM, DEVICE, AND METHODS FOR DETECTING AND OBTAINING INFORMATION ON OBJECTS IN A VEHICLE", which is incorporated herein by reference, can be used to identify the body posture of the occupant.

[0254] In another embodiment, the system is further configured to use image and / or depth data, such as video images, to detect body motion of the occupants 504. Detection may be performed by known computer vision algorithms, such as, but not limited to, optical flow and visual tracking methods. For example, the Lucas-Kanade optical flow algorithm in OpenCV. Detecting the motion or inactivity of one or more occupants in the vehicle may provide valuable information about the status of the medical condition, nature and severity of the injury of one or more occupants.

[0255] In some cases, the system is triggered to initiate the analysis by visually detecting rapid, frantic movement of occupants within the vehicle interior. An example of such a detection mechanism is based on an optical flow algorithm configured to analyze the relative displacement of an object between two or more consecutive frames.

[0256] In another embodiment, the system is further configured to detect visual signs of bodily wounds 505 of the one or more occupants. Examples of visually detectable wounds may be raptures or bone fractures. One way to configure the system to detect wounds is by training a deep learning classifier to detect broken skin or bleeding, for example, by showing an image of a wound and training the classifier to recognize the wound. In operation, an image of one or more injured occupants that has been captured is obtained at a processor that operates a deep neural network to make predictions and detect and identify the type of injury.

[0257] According to an embodiment, each type of captured information, such as heartbeat 501, breathing rate 502, etc., may be converted into one or more signals containing relevant information.

[0258] In various embodiments, the conversion of information into one or more signals can be performed locally (using a processor and software provided with the system) or remotely. For example, for more complex analysis, more intensive calculations can be performed remotely.

[0259] According to an embodiment, all or part of the above-mentioned signals and evaluations 510 are transmitted to a data evaluation module, such as a single post-crash data evaluation module 506. The collected data signals 501, 502, 503, 504, and 505 can be used together or separately. It should be emphasized that in some cases, only some signals can be used because some signals are not received or due to some other desired configuration. For example, in some cases, only signals 501 and 502 can be received.

[0260] According to an embodiment, the post-crash data assessment module 506 is configured to fuse the different received signals containing the captured information and process the received signals to generate post-crash assessment data 506 ′ containing the medical condition of one or more occupants of the vehicle.

[0261] As an example, the assessment may include using a checklist such as a binary checklist that includes a list of questions that include medical questions for assessing the occupant's medical condition. The checklist may include one or more of the following questions: Is there a pulse and breathing; and / or are there signs of consciousness; and / or are there suspicions of broken bones and which body parts are suspected of broken bones; and / or are there suspicions of internal injuries; and / or are there suspicions of head injuries; and / or are there suspicions of open wounds or blood loss, etc. It should be understood that embodiments of the present invention may use any other type of diagnostic checklist or other method to identify the occupant's medical condition based on the captured data.

[0262] According to an embodiment, the system also includes a pre-collision assessment module 508, which is configured to receive raw data (e.g., sensory data), such as stored pre-collision data 508', to analyze the raw data, for example at a bumper, to generate pre-collision assessment data 508". The pre-collision assessment data 508" includes an identification of a pre-collision state of one or more occupants of the vehicle. Such analysis may include any combination of the occupant's position, body posture, and physical attributes, such as weight, height, age, gender, and body shape. The analysis may also include objects present in the cabin that may pose a risk of injury during an accident.

[0263] According to further embodiments, the system includes a during-crash data assessment module 507, which is configured to receive, store and analyze sensory data, such as during-crash data 507' captured during a vehicle accident, to generate during-crash assessment data 507' containing information allowing the system to follow the trajectory of vehicle occupants and / or objects.

[0264] The during-collision assessment data 507″, post-collision assessment data 506″ and pre-collision assessment data extracted by modules 506, 507 and 508 respectively are transmitted to a fusion module 509, which is configured to combine and process the received data according to one or more fusion methods and logics to produce one or more final post-collision assessment results 520 related to the medical status of the occupant.

[0265] According to an embodiment, an example of fusion logic used by the fusion module 509 includes a scenario in which the system is configured to evaluate the possibility of head trauma of one or more passengers in the vehicle cabin after an accident. The evaluation process includes obtaining one or more types of post-collision data 510 at the post-collision data evaluation module 506, such as body posture 503, body movement 504, etc., and analyzing the acquired post-collision data 510 to check whether there are signs of injury to the passenger's head. In some cases, the system may then also use the during-collision data evaluation module 507 to identify whether there is an impact of an automobile element or other object on the head during the collision based on the received during-collision data. This impact will increase the possibility of head trauma. If no head impact is detected during the collision and no head obstruction occurs, the possibility of head trauma will be reduced. Finally, the system may also obtain pre-collision data 508' at the pre-collision evaluation data module 508 to analyze the position of the passenger's head relative to various automobile elements and generate pre-collision evaluation data. The pre-crash assessment data 508', the during-crash assessment data, and the post-crash assessment data 506' are then fused to estimate the risk of head trauma at a fusion and final assessment module 509. Specifically, the fused estimate can be based on the direction and force of the external impact vehicle, and using the body model, the module 509 can infer an increase or decrease in the risk of head trauma.

[0266] It should be emphasized that for injuries of the head trauma type, the analysis was mainly based on visual inspection and injury classification.

[0267] It is also emphasized that the fusion module 509 may provide a final assessment result 520 based on any portion and / or combination of available data from the pre-crash data 508 ′, the during-crash data 507 ′, and the post-crash data 510 .

[0268] The fusion module 509 may include one or more stochastic prediction models (eg, Markov chains, such as in the form of a Markov matrix) that predict state probabilities.

[0269] Figure 1C An example of a final evaluation result 122 constructed in the form of a message according to an embodiment is shown. The message may contain one or more of the following details:

[0270] Initial position of passenger: left rear side

[0271] Passenger gender: Female

[0272] Estimated age of passengers: 50-65

[0273] Estimated passenger weight: 60-75Kg

[0274] Suspected injuries: Neck trauma, ribs, lower back

[0275] Post-crash vital signs: HR: 110 BPM, respirations per minute: 70

[0276] Figure 5B A flow chart of a method 540 for detecting the heartbeat and / or respiration of, for example, one or more occupants in a vehicle by analyzing a plurality of captured images including reflected light patterns according to an embodiment is shown. According to one embodiment, the method includes the following steps: At step 550, a plurality of images are received, for example, from a sensing module at a processor. At step 560, one or more changes in one or more speckle patterns of at least one reflection of the light pattern are detected in at least some consecutive images of the plurality of images. At step 570, micro-vibrations of at least one occupant are identified based on the speckle pattern analysis. At step 580, the micro-vibrations are analyzed using a computer vision algorithm, such as, but not limited to, optical flow or tracking, to extract a breathing rate or heartbeat signal of the occupant. A specific example of an analysis method for extracting a breathing rate or heartbeat signal of an occupant is described in detail in the accompanying drawings. Figures 7A-7E and Figures 8A-8D In some embodiments, the breathing or heartbeat signal is used to assess the medical condition of the occupant, such as Figure 5A At step 590, the detected breathing or heartbeat signal is used to assess the medical condition of the occupant.

[0277] Figure 6 Two options according to an embodiment are shown, including for example Figure 5A The final post-crash assessment results 520 of the medical condition and injury assessment provided in the crash report may be transmitted to an external responder unit, such as a first rescue responder.

[0278] In a first option, the assessment results 520 are transmitted directly from the system 610 in the vehicle 600 to the first responder 603 via a direct communication link 605. For example, the system 610 may broadcast one or more transmission signals containing the assessment results 520 to the first responder 603 via a local wireless network. After receiving the signal, the first responder 603 may download the assessment results 520, which may contain, for example, medical information about the in-cabin status of one or more occupants of the vehicle, by logging into the local network.

[0279] In a second option, the system 610 can send data including, for example, the assessment results 520 to the cloud service 604 via the communication channel 606. The cloud service can be a virtual service that stores all the information from the system. The first responder 603 can then log into the cloud service and download the relevant information.

[0280] The main advantage of the second approach over the first is that the medical condition assessment data can be accessed even before arriving at the crash scene. This will allow the correct first responder tools and vehicles to be sent to the scene.

[0281] In some cases, the cloud service may also connect to treatment facilities such as hospitals to alert medical teams of incoming patients.

[0282] Another advantage of the cloud service is that even if the system 601 is not able to survive the collision, it can still receive some of the data. In an embodiment, the pre-crash data 508' and the during-crash data 507' can be sent to the cloud service as soon as they are generated. In this case, even if the system fails or may be inoperable due to the collision, it can still be evaluated based on the data available on the cloud.

[0283] Once an in-vehicle medical assessment is available to first responders, it can be used to prepare the correct rescue procedures and the immediate treatment required. The assessment can also help first responders determine rescue and treatment priorities in multi-casualty situations.

[0284] In another embodiment, the system may use Figure 4 of impact detection 302 to send automatic alerts to local first responder organizations, such as police, fire departments, and medical teams.

[0285] Reference now Fig. 7A , which shows an exemplary configuration used in accordance with an embodiment for capturing light pattern images and translating identified speckle patterns for measuring, for example, the heartbeat and / or breathing rhythm of one or more occupants' breathing while the vehicle is moving.

[0286] Assuming that the illuminator is nearly parallel to the image sensor optical axis, the translation of the speckle pattern of the CMOS is given by the following equation:

[0287]

[0288] in:

[0289] Φ Lens aperture λ wavelength M magnification <![CDATA[z2]]> Distance from the object to the imaging plane <![CDATA[z3]]> Distance from the imaging plane to the lens F focal length D Laser spot diameter on the surface α(t) Surface tilt T(t) Speckle shifting on CMOS S Defocused spot size on CMOS Δx CMOS pixel size <![CDATA[f s ]]> Camera sampling rate z Distance between CMOS and focal plane

[0290] To measure the speckle size on a captured image, the dominant speckle size can be reduced by the system magnification:

[0291]

[0292] The defocused spot size (confusion circle) is given by the lens aperture (z2>>z3) as follows:

[0293]

[0294] We found Does not depend on M:

[0295] This can be expressed as a typical value, which is calculated, for example, as follows:

[0296]

[0297]

[0298]

[0299] Figure 7B Exemplary captured images 750 according to an embodiment are shown. In these images, the displacement of each speckle pattern relative to the initial pattern is identified. Thereafter, using, for example, a sinusoidal fit, e.g. Figure 7C The sine fit graph 760 is used to extract the speckle translation amplitude in pixels.

[0300] Therefore, the rate is extracted directly from the translation measurement. The translation rate is linearly related to the spot size, such as Fig.7D 770 and 780. In addition, each measured M is calculated based on dT / dα and compared with M to form, for example Fig. 7E The spot size is shown in Figure 790.

[0301] In the next step, the desired measurement limits can be extracted, such as the maximum linear and angular velocities that allow speckle tracking. To avoid boiling, the surface linear velocity perpendicular to the beam should be:

[0302]

[0303] In some cases, the translation of the speckle field should be smaller than the size of the spot in the image:

[0304]

[0305] Therefore, by using typical values ​​to calculate the following inequality

[0306]

[0307]

[0308] Based on these inequalities, we expect that at f s At ~200 Hz, the occupant must be almost motionless to measure his vital signs.

[0309] According to an embodiment, a maximum exposure time may also be estimated. Assume that the spots will be slightly blurred during the exposure time if:

[0310]

[0311] in is the speckle translation speed in pixels / second, and τ is the exposure time. For typical values, we find:

[0312]

[0313] use We obtain the following values:

[0314]

[0315] τ will allow given The maximum value of For example, the system will be subject to f s Limitations:

[0316]

[0317] Experimental data

[0318] Fig. 8AAn exemplary enlarged view of a captured image 810 of an occupant's chest suitable for incorporation according to an embodiment is shown. The image includes reflected light using an illuminator including an 830 nm laser power to illuminate the occupant's chest region. The captured image 810 shows characteristic features, including changes in the detected speckle pattern. As mentioned above, by analyzing the speckle pattern for lateral translations that indicate the tilt of the reflective object relative to the image sensor, the occupant's heartbeat can be detected. The tilt, which may be very small, for example, on a micro-radian scale, can be derived from the speed of translation of one or more speckle pattern points over time (sequential frames). The lateral speckle pattern translation can be derived from an analysis of diffuse light pattern elements depicted in multiple consecutive captured images.

[0319] Figure 8B Graphs 820, 830, and 840 show the results of processing captured images of an occupant when the occupant is not breathing and only their heartbeat is detected, respectively, according to embodiments. Graph 840 shows an exemplary spectrum (fft) of the processed image. In this example, the following parameters are used:

[0320] f s =200Hz exposure =5000μs D =1mm S =20 pixels (M = 29)

[0321] Figure 8C 850, 860 and 870 respectively show the results of processing the captured image of the occupant in the vehicle to include the changes in the occupant's heartbeat when the occupant breathes normally, according to an embodiment. Graph 870 shows an exemplary spectrum (fft) of the processed image. Specifically, in the scene, the occupant's breathing is strong compared to the heartbeat, thus masking the heartbeat. In this example, the same Figure 8B The parameters are the same as the parameters of .

[0322] Fig.8D Results 880 and 890 of processing an image of an empty vehicle (without any occupants in the vehicle) captured according to an embodiment are shown, respectively, so no heartbeat or breathing is detected. In this example, the measurements are performed in a FIAT 500 car, and the sensing system is located on the dashboard and includes the following parameters: f s = 200 Hz exposure = 5000 μs Z2 = 77 cm to the driver's chest.

[0323] z2 = 96 cm to the driver's seat.

[0324] D = 1 mm, S = 20 pixels (M = 29)

[0325] In some embodiments, the sensing system does not analyze the collected data, and the sensing module relays the data to a remote processing and control unit, such as a back-end server. Alternatively or in combination, the sensing system may partially analyze the data before transmitting it to the remote processing and control unit. The remote processing and control unit may be a cloud-based system that may transmit the analysis data or results to a user. In some embodiments, a handheld device is configured to receive the analysis data and may be associated with the sensing system. For example, the association may be achieved through a physical connection or wireless communication.

[0326] In some embodiments, the sensing system is equipped with a memory and a microprocessor, wherein the memory has a database of data stored therein, and the microprocessor has analysis software programmed with instructions. In some embodiments, the sensing system communicates with a computer memory and a microprocessor, wherein the computer memory has a database stored therein, and the microprocessor has analysis software programmed therein. The memory may be volatile or non-volatile so that the measurements are stored in the memory. All or part of the database and / or analysis software may be stored remotely, and the sensing system may communicate with the remote memory via a network (e.g., a wireless network) by any suitable method.

[0327] In various embodiments of the present invention, the conversion of raw data into medical information can be performed locally (using a processor and software provided with the system) or remotely. For example, for more complex analysis, more arduous calculations can be performed remotely.

[0328] In further embodiments, the systems disclosed herein include a processing unit, which may be a digital processing device comprising one or more hardware central processing units (CPUs) that perform the functions of the device. In yet further embodiments, the digital processing device further comprises an operating system configured to execute executable instructions. In some embodiments, the digital processing device is optionally connected to a computer network. In further embodiments, the digital processing device is optionally connected to the Internet, so that the digital processing device can access the World Wide Web. In yet further embodiments, the digital processing device is optionally connected to a cloud computing infrastructure. In other embodiments, the digital processing device is optionally connected to an intranet. In other embodiments, the digital processing device is optionally connected to a data storage device.

[0329] According to the description herein, suitable digital processing devices include, as non-limiting examples: server computers, desktop computers, laptop computers, notebook computers, electronic handbook computers, netbook computers, notepad computers, set-top computers, handheld computers, Internet appliances, mobile smart phones, tablet computers, personal digital assistants, video game consoles and vehicles. Those skilled in the art will recognize that many smart phones are suitable for use in the system described herein. Those skilled in the art will also recognize that select televisions with optional computer network connectivity are suitable for use in the system described herein. Suitable tablet computers include tablet computers with brochure, tablet and convertible configurations known to those skilled in the art.

[0330] In some embodiments, the digital processing device includes an operating system configured to execute executable instructions. For example, an operating system is software including programs and data that manages the hardware of the device and provides services for the execution of application programs. Those skilled in the art will recognize that, as non-limiting examples, suitable server operating systems include FreeBSD, OpenBSD, Linux, Mac OS X Windows as well as Those skilled in the art will recognize that suitable personal computer operating systems include, by way of non-limiting example, Mac OS and for example In some embodiments, the operating system is provided by cloud computing. Those skilled in the art will also recognize that, as non-limiting examples, suitable mobile smartphone operating systems include OS, Research In BlackBerry Windows OS, Windows OS, as well as

[0331] In some embodiments, the device includes a storage device and / or a memory device. A storage and / or memory device is one or more physical devices for temporarily or permanently storing data or programs. In some embodiments, the device is a volatile memory and requires power to maintain the stored information. In some embodiments, the device is a non-volatile memory and retains the stored information when the digital processing device is not powered. In other embodiments, the non-volatile memory includes a flash memory. In some embodiments, the non-volatile memory includes a dynamic random access memory (DRAM). In some embodiments, the non-volatile memory includes a ferroelectric random access memory (FRAM). In some embodiments, the non-volatile memory includes a phase change random access memory (PRAM). In other embodiments, the device is a storage device, and as a non-limiting example, the storage device includes a CD-ROM, a DVD, a flash memory device, a disk drive, a tape drive, an optical drive, and a cloud computing-based storage device. In other embodiments, the storage device and / or the memory device is a combination of devices such as those disclosed herein.

[0332] In some embodiments, the digital processing device includes a display that sends visual information to a user. In some embodiments, the display is a cathode ray tube (CRT). In some embodiments, the display is a liquid crystal display (LCD). In other embodiments, the display is a thin film transistor liquid crystal display (TFT-LCD). In some embodiments, the display is an organic light emitting diode (OLED) display. In multiple other embodiments, the OLED display is a passive matrix OLED (PMOLED) or an active matrix OLED (AMOLED) display. In some embodiments, the display is a plasma display. In other embodiments, the display is a video projector. In yet other embodiments, the display is a combination of devices such as those disclosed herein.

[0333] In some embodiments, the digital processing device includes an input device that receives information from a user. In some embodiments, the input device is a keyboard. In some embodiments, the input device is a pointing device, which includes, as non-limiting examples, a mouse, a trackball, a track pad, a joystick, a game controller, or a stylus. In some embodiments, the input device is a touch screen or multiple touch screens. In other embodiments, the input device is a microphone for capturing voice or other sound input. In other embodiments, the input device is a camera that captures motion or visual input. In yet other embodiments, the input device is a combination of devices such as those disclosed herein.

[0334] In some embodiments, the systems disclosed herein include one or more non-transitory computer-readable storage media encoded with a program that includes instructions executable by an operating system of an optionally networked digital processing device. In other embodiments, the computer-readable storage medium is a tangible component of the digital processing device. In yet other embodiments, the computer-readable storage medium is optionally removable from the digital processing device.

[0335] In some embodiments, as non-limiting examples, computer-readable storage media include CD-ROMs, DVDs, flash memory devices, solid-state memories, disk drives, tape drives, optical drives, cloud computing systems and services, etc. In some cases, the programs and instructions are permanently, substantially permanently, semi-permanently, or non-temporarily encoded on the medium. In some embodiments, the system disclosed herein includes at least one computer program or its use. The computer program includes a sequence of instructions that can be executed in a CPU of a digital processing device and written to perform a specific task. Computer-readable instructions can be implemented as program modules, such as functions, objects, application programming interfaces (APIs), data structures, etc., which perform specific tasks or implement specific abstract data types. Based on the disclosure provided herein, those skilled in the art will recognize that computer programs can be written in various versions of various languages.

[0336] The functionality of the computer-readable instructions can be combined or distributed in various environments as needed. In some embodiments, the computer program includes an instruction sequence. In some embodiments, the computer program includes multiple instruction sequences. In some embodiments, the computer program is provided from one location. In other embodiments, the computer program is provided from multiple locations. In various embodiments, the computer program includes one or more software modules. In various embodiments, the computer program partially or entirely includes one or more network applications, one or more mobile applications, one or more stand-alone applications, one or more network browser plug-ins, extensions, add-ons or add-ons or a combination thereof. In some embodiments, the computer program includes a mobile application provided to a mobile digital processing device. In some embodiments, the mobile application is provided to the mobile digital processing device at the time of manufacture. In other embodiments, the mobile application is provided to the mobile digital processing device via the computer network described herein.

[0337] In view of the disclosure provided herein, mobile applications are generated by techniques known to those skilled in the art using hardware, languages, and development environments known to those skilled in the art. Those skilled in the art will recognize that mobile applications are written in a variety of languages. As non-limiting examples, suitable programming languages ​​include C, C++, C#, Objective-C, Java, and C++. TM,Javascript,Pascal,Object Pascal,Python TM , Ruby, VB.NET, WML, and XHTML / HTML with or without CSS or a combination thereof.

[0338] Suitable mobile application development environments are available from several sources. As non-limiting examples, commercially available development environments include AirplaySDK, alcheMo, Celsius, Bedrock, FlashLite, .NET Compact Framework, Rhomobile, and WorkLight mobile platforms. As non-limiting examples, other development environments are freely available, including but not limited to Lazarus, MobiFlex, MoSyn, and Phonegap. Also, as non-limiting examples, mobile device manufacturers distribute software development kits, including but not limited to iPhone and iPad (iOS) SDKs, Android TM SDK, SDK, BREW SDK, OS SDK, Symbian SDK, webOS SDK, and Mobile SDK.

[0339] Those skilled in the art will recognize that, as a non-limiting example, several commercial forums may be used to distribute mobile applications that include App Store, Android TM market, App World, App Store for handheld devices, App Catalog for webOS, App Store for mobile market Used for Special services of the device, Applications and shop.

[0340] In some embodiments, the system disclosed herein includes software, server and / or database modules, or their uses. In view of the disclosure provided herein, the software module is created by the techniques known to the technical personnel in the field, using machines, software and languages ​​known to the technical personnel in the field. The software modules disclosed herein are implemented in a variety of ways. In various embodiments, the software module includes a file, a code segment, a programming object, a programming structure or a combination thereof. In various other embodiments, the software module includes multiple files, multiple code segments, multiple programming objects, multiple programming structures or a combination thereof. In various embodiments, as a non-limiting example, one or more software modules include a network application, a mobile application and a stand-alone application. In some embodiments, the software module is in a computer program or application. In other embodiments, the software module is in more than one computer program or application. In some embodiments, the software module is hosted on a machine. In other embodiments, the software module is hosted on more than one machine. In other embodiments, the software module is hosted on a cloud computing platform. In some embodiments, the software module is hosted on one or more machines in one location. In other embodiments, the software module is hosted on one or more machines in more than one location.

[0341] In some embodiments, the system disclosed herein includes one or more databases or the purposes of the databases. In view of the disclosure provided herein, those skilled in the art will recognize that many databases are suitable for storing and retrieving information as described herein. In various embodiments, as non-limiting examples, suitable databases include relational databases, non-relational databases, object-oriented databases, object databases, entity relationship model databases, associated databases, and XML databases. In some embodiments, the database is based on the Internet. In other embodiments, the database is based on a network. In yet other embodiments, the database is based on cloud computing. In other embodiments, the database is based on one or more local computer storage devices.

[0342] In the above description, an embodiment is an example or implementation of the invention. The various appearances of "one embodiment," "an embodiment," or "some embodiments" do not necessarily all refer to the same embodiments.

[0343] Although various features of the invention may be described in the context of a single embodiment, the features may also be provided separately or in any suitable combination. Conversely, although for clarity, the invention may be described herein in the context of separate embodiments, the invention may also be implemented in a single embodiment.

[0344] Reference in the specification to "some embodiments," "an embodiment," "one embodiment," or "other embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least some embodiments of the invention, but not necessarily in all embodiments.

[0345] It is to be understood that the phraseology and terminology employed herein should not be interpreted as limiting and are for description purpose only.

[0346] The principles and uses of the present teachings may be better understood with reference to the accompanying description, drawings and examples.

[0347] It is to be understood that the details set forth herein are not to be construed as limitations on the application of the present invention.

[0348] Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and can be implemented in embodiments other than those outlined in the above description.

[0349] It should be understood that the terms "comprises", "comprising", "comprising" and grammatical variations thereof do not exclude the addition of one or more components, features, steps or integers or groups thereof, and the terms should be interpreted as specifying the components, features, steps or integers.

[0350] If the specification or claims refer to "an additional" element, that does not preclude there being more than one of the additional element.

[0351] It should be understood that where the claims or specification refer to "a" or "an" element, such reference should not be interpreted as there being only one of the elements. It should be understood that where the specification states that a component, feature, structure, or characteristic "may", "might", "can", or "can" be included, the specific component, feature, structure, or characteristic does not have to be included. Where applicable, although state diagrams, flow charts, or both may be used to describe embodiments, the present invention is not limited to those illustrations or corresponding descriptions. For example, the process does not need to move through each of the boxes or states shown, or in exactly the same order as shown and described. The method of the present invention can be implemented by performing or completing selected steps or tasks manually, automatically, or a combination thereof.

[0352] The descriptions, examples, methods and materials presented in the claims and specification should not be interpreted as limiting, but should be interpreted only as illustrative. Unless otherwise defined, the meanings of technical terms and scientific terms used herein are those commonly understood by those of ordinary skill in the art to which the invention belongs. The present invention can be implemented in testing or practice using methods and materials equivalent or similar to those described herein.

[0353] Although the present invention has been described with reference to a limited number of embodiments, these should not be interpreted as limitations on the scope of the invention, but as examples of some preferred embodiments. Other possible changes, modifications and applications are also within the scope of the present invention. Therefore, the scope of the present invention should not be limited by what has been described so far, but by the attached claims and their legal equivalents.

[0354] All publications, patents and patent applications mentioned in this specification are hereby incorporated by reference in their entirety into this specification, and similarly, each individual publication, patent or patent application is also specifically and individually indicated to be incorporated by reference herein. In addition, the citation or identification of any reference document in this application should not be construed as an admission that such reference document is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting.

Claims

1. A system for providing medical information of at least one occupant in a vehicle cabin, the system comprising: a sensing module comprising at least one sensor configured to capture sensory data of a cabin of the vehicle, wherein the sensing module is configured to detect an impending collision of the vehicle; as well as A control module comprising at least one processor configured to: receiving the sensory data from the sensor; and analyzing the sensory data, wherein the sensory data is captured before the detection of an impending collision, during the collision, and after the collision; wherein sensory data captured prior to said detection of an impending collision is analyzed to generate pre-collision assessment data including an identification of a state of one or more occupants prior to said collision, and a likely accident outcome is assessed based on said pre-collision assessment data; and wherein sensory data captured during said collision is analyzed to generate during-collision assessment data including body trajectories of said one or more occupants; Medical information of the one or more occupants after the collision is provided based on the during-collision assessment data and the pre-collision assessment data.

2. The system of claim 1, wherein the at least one sensor is an image sensor. 3 . The system of claim 2 , wherein the sensing module further comprises at least one illuminator comprising a light source configured to project a light pattern onto the vehicle cabin.

4. The system of claim 3, wherein the light source comprises one or more optical elements for separating the individual light beams generated by the light source, the one or more optical elements being selected from the group consisting of: DOE, splitter mirror and diffuser. The system of claim 1 , wherein the sensing module comprises a depth sensor.

6. The system of claim 5, wherein the depth sensor is configured to capture depth data by projecting a light pattern onto the vehicle cabin, and wherein the at least one processor is configured to analyze the depth data for locations of known light pattern elements.

7. The system of claim 6, wherein the posture of the at least one occupant is estimated by analyzing the depth data using computer vision methods. The system of claim 1 , wherein the sensing module comprises a micro-vibration sensor.

9. The system according to claim 8, wherein the micro-vibration sensor is configured to: projecting light onto a scene in the vehicle cabin; capturing a plurality of images, wherein each image in the plurality of images includes a reflective diffuse light element; and wherein the processor is configured to: receiving the captured image; and One or more temporal variations in the speckle pattern of at least one of the plurality of reflected diffuse light elements in at least some consecutive images of the plurality of images are analyzed to generate micro-vibration data.

10. The system of claim 1, wherein the sensing module comprises a combination of at least two sensors selected from the group consisting of: Image sensors, depth sensors, and micro-vibration sensors.

11. The system of claim 1 , wherein the sensory data comprises a plurality of images, and wherein the at least one processor is further configured to classify the at least one vehicle occupant using a computer vision method by visually analyzing at least one of the plurality of images captured by the at least one sensor.

12. The system of claim 1 , wherein the detection of an impending collision is performed by one or more of: Vehicle impact sensors, hectic movement detection sensors; and external sensing systems.

13. A system for estimating the medical status of one or more occupants in a vehicle cabin after an accident involving the vehicle, the system comprising: A sensing module comprising: an illuminator comprising one or more illumination sources configured to project light in a structured light pattern onto the vehicle cabin; at least one image sensor configured to capture sensory data before, during, and after a collision of the vehicle, the sensory data comprising a series of 2D (two-dimensional) images and 3D (three-dimensional) images of a cabin of the vehicle, wherein at least one of the 2D images includes reflections of the structured light pattern from the one or more occupants in the cabin; A control module comprising: Memory module; at least one processor configured to: receiving captured sensory data from the sensing module; receiving an impact detection signal of an impending collision of the vehicle; storing received sensory data captured prior to receipt of the impact detection in the memory module; analyzing the received sensory data using a computer vision algorithm or a machine learning algorithm to generate pre-crash assessment data including an identification of a state of the one or more occupants prior to the collision, and assessing a possible accident outcome based on the pre-crash assessment data; receiving sensory data captured during the collision from the sensing module; analyzing the received sensory data captured during the collision to generate during-collision assessment data including body trajectories of the one or more occupants; Medical information of the one or more occupants after the collision is provided based on the during-collision assessment data and the pre-collision assessment data.

14. The system of claim 13, wherein the at least one processor is further configured to: receiving sensory data captured subsequent to the collision; The received sensory data captured after the crash data is analyzed using a computer vision algorithm or a machine learning algorithm to generate one or more of the post-crash assessment data for the one or more occupants.

15. The system of claim 14, wherein the post-crash assessment data includes one or more of: heart beat, breathing rate, body posture, body movement, visible wounds.

16. The system of claim 14, wherein the processor is further configured to combine the post-crash assessment data with the during-crash assessment data and the pre-crash assessment data to generate the medical information of the one or more occupants after the collision.

17. The system of claim 15, wherein the heartbeat or respiration rate is identified by analyzing one or more changes in one or more speckle patterns of at least one of the reflections of the structured light pattern in at least some consecutive images of the plurality of images; and Vibrations of the one or more occupants are identified based on the speckle pattern analysis.

18. The system of claim 13, wherein the one or more of body posture, body movement, and visible wounds are detected by analyzing the series of 2D or 3D images of the vehicle cabin.

Images