Multi-modal in-vehicle violence detection for shared vehicles and self-driving shuttle buses
By integrating modules such as voice and non-voice audio, accelerometer, heart rate detection, and semantic scene recognition, and combining prior knowledge, this system identifies threatening behaviors of passengers in shared autonomous vehicles, solving the problem of passenger safety risks in shared autonomous vehicles and realizing effective threat assessment and response measures.
Patent Information
- Application Number
- CN202211346442.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-01
- Filing Date
- 2022-10-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-10-31
AI Technical Summary
In shared autonomous vehicles, passengers sharing a trip with strangers may pose risks to their personal safety and emotional well-being, and current technology lacks an effective in-vehicle multi-modal violence detection system.
The system employs a voice and non-voice audio event recognition module, an in-vehicle accelerometer, a heart rate and respiratory rate detection module, an in-vehicle semantic scene recognition module, and an image compilation module. By combining prior knowledge and external information, it analyzes the occupant's threatening behavior, generates a threat level, and takes corresponding measures.
Effectively identify and assess potential threats inside the vehicle, and ensure passenger safety and reduce safety risks through vehicle horns, warning lights, brake assist systems, and emergency assistance requests.
Smart Images

Figure CN117184098B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a system for identifying abnormal or threatening conditions of automobiles. Background Technology
[0002] The emergence of commercially available ride-sharing services, such as car-sharing, carpooling, ride-hailing, and ride-hailing, demonstrates the significant maturity of ride-sharing technology. Shared autonomous vehicles (SAVs) and pooled shared autonomous vehicles (PSAVs) are also rapidly gaining popularity. Sharing a trip with strangers in a shared autonomous vehicle (SAV) may pose risks to the user's personal safety and emotional well-being. Passengers may also experience feelings of crowding and invasion of personal space when sharing or booking rides with strangers.
[0003] Therefore, while current shared and partnered autonomous vehicles achieve their intended purposes, a new and improved in-vehicle multi-modal violence detection system for autonomous vehicles is needed. Summary of the Invention
[0004] According to several aspects, an in-vehicle multimodal violence detection system includes a voice and non-voice audio event recognition module that captures threatening verbal and non-voice audio events of vehicle occupants. Multiple in-vehicle accelerometers generate in-vehicle accelerometer data analyzed in a sway motion recognition module. A heart rate and respiratory rate detection module measures physiological changes in the occupants' heart rate and respiratory rhythm. An in-vehicle semantic scene recognition module captures and analyzes non-verbal interactions between occupants. One or more occupant threat indicators include auditory threat indicators generated by the voice and non-voice audio event recognition module, visual threat indicators generated by the in-vehicle semantic scene recognition module, physiological threat indicators generated by the heart rate and respiratory rate detection module, and vibration-based threat indicators generated by the sway motion recognition module.
[0005] In another aspect of this disclosure, prior knowledge of a limited history of violence is applied to determine the threat level.
[0006] In another aspect of this disclosure, external sources provide prior information.
[0007] In another aspect of this disclosure, prior information includes driving areas in which known violence has occurred, as well as past historical records of different “normal” relative “threatening” behavioral events.
[0008] In another aspect of this disclosure, the picture compilation module identifies a vector combining the output signals from the speech and non-speech audio event recognition module, the swaying motion recognition module, the heart rate and respiratory rate detection module, and the in-vehicle semantic scene recognition module, and assigns a predetermined threshold to distinguish between normal events and threatening events.
[0009] In another aspect of this disclosure, the threat assessment module uses auditory, visual, physiological, and vibration-based indicators, as well as prior knowledge including a history of violence and environmental information with location and time of day, to analyze the threat events to determine the threat level.
[0010] In another aspect of this disclosure, the threat assessment unit receives the threat level, confirms the existence of an active threat, and generates an active threat signal.
[0011] In another aspect of this disclosure, after the activity threat signal is generated, a threat confirmation request is generated and forwarded to the threat timer unit.
[0012] In another aspect of this disclosure, the confirmation request is presented visually and audibly to the occupants, and a predetermined time interval is allowed for at least one of the occupants to confirm the presence of the active threat in the vehicle. If the predetermined time interval is exceeded before a response is received from any of the occupants, a threat level determination unit is activated by a time interval timeout signal.
[0013] In another aspect of this disclosure, if any of the occupants confirms the presence of an active threat, an active threat signal is generated and forwarded to the threat level determination unit. The threat level determination unit identifies whether the active threat should be classified as one of "low threat," "medium threat," or "high threat," wherein: identifying a "low threat" activates the vehicle horn and vehicle warning lights; identifying a "medium threat" activates the vehicle horn and vehicle warning lights, activates the vehicle brake assist system, and notifies an external source of the medium threat; and identifying a "high threat" activates the vehicle horn and vehicle warning lights, activates the vehicle brake assist system, and forwards a request to the external source to initiate an emergency assistance request.
[0014] According to several aspects, a method for performing in-vehicle multimodal violence detection includes: using a voice and non-voice audio event recognition module to capture threatening verbal and non-voice audio events of vehicle occupants; collecting in-vehicle accelerometer data; measuring sudden physiological changes in the heart rate and respiratory rhythm of the occupants in a heart rate and respiratory rate detection module; using an in-vehicle semantic scene recognition module to capture and analyze non-verbal interactions between the occupants; and generating one or more threat indicators, including auditory, visual, physiological, and vibration-based indicators.
[0015] In another aspect of this disclosure, the method includes: applying prior knowledge, including a history of violence, to determine the threat level.
[0016] In another aspect of this disclosure, the method includes applying environmental information, including vehicle location and time of day, derived from the prior knowledge.
[0017] In another aspect of this disclosure, the method includes: combining ride information and passenger information from a reservation database, and analyzing the ride information and passenger information to distinguish differences between the passengers.
[0018] In another aspect of this disclosure, the method includes: using the in-vehicle accelerometer data to identify swaying movements of the occupants of the vehicle.
[0019] In another aspect of this disclosure, the method includes applying indistinct sounds and sounds accompanied by threatening or aberrant behavior, including shouting, screaming, crying, and glass breaking, as audio-based threat indicators.
[0020] In another aspect of this disclosure, the method includes: analyzing the one or more occupant threat indicators in a multimodal threat assessment module.
[0021] According to several aspects, a method for performing in-vehicle multimodal violence detection includes: using a voice and non-voice audio event recognition module to capture threatening verbal and non-voice audio events of occupants of an autonomous vehicle; analyzing in-vehicle accelerometer data in a sway motion recognition module; measuring sudden physiological changes in the occupants' heart rate and respiratory rhythm in a heart rate and respiratory rate detection module; using an in-vehicle semantic scene recognition module to capture and analyze non-verbal interactions between the occupants; generating one or more occupant threat indicators, including auditory, visual, physiological, and vibration-based indicators; and identifying a vector combining output signals from the voice and non-voice audio event recognition module, the sway motion recognition module, the heart rate and respiratory rate detection module, and the in-vehicle semantic scene recognition module in an image compilation module, and assigning predetermined thresholds to distinguish between normal events and threatening events.
[0022] In another aspect of this disclosure, the method includes: collecting output signals from a plurality of in-vehicle cameras located in the autonomous vehicle; identifying whether any one of the plurality of in-vehicle cameras is covered in a camera evaluation unit; and generating an audio pulse signal by the camera evaluation unit to notify the occupant of the autonomous vehicle to remove the cover from any of the covered in-vehicle cameras.
[0023] In another aspect of this disclosure, the method includes: identifying whether two or more occupants are present in the autonomous vehicle by receiving an occupant status signal forwarded to an occupant confirmation unit; initiating operation of a ride type identification unit, wherein: if the ride type identification unit generates a "ride-hailing type" signal, a first procedural assumption is made that all the occupants present in the autonomous vehicle know each other and that no threat will subsequently occur; or if the ride type identification unit generates a "shared ride type" signal, a second procedural assumption is made that at least one of the occupants present in the autonomous vehicle was not previously known to the other occupants, and therefore, there is a possibility that the threat event may develop during subsequent operation of the autonomous vehicle.
[0024] The present invention also includes the following technical solutions.
[0025] Option 1. A multi-mode in-vehicle violence detection system, comprising: A voice and non-voice audio event recognition module that captures threatening speech and non-voice audio events of vehicle occupants; Multiple in-vehicle accelerometers generate in-vehicle accelerometer data that is analyzed in the sway motion recognition module; A heart rate and respiratory rate detection module that measures the physiological changes in the occupant's heart rate and respiratory rhythm; The in-vehicle semantic scene recognition module captures and analyzes nonverbal interactions between the occupants; and One or more occupant threat indicators, including auditory threat indicators generated by the voice and non-voice audio event recognition module, visual threat indicators generated by the in-vehicle semantic scene recognition module, physiological threat indicators generated by the heart rate and respiratory rate detection module, and vibration-based threat indicators generated by the swaying motion recognition module.
[0026] Option 2. The in-vehicle multi-mode violence detection system according to Option 1, including prior knowledge of violence history used to determine the threat level.
[0027] Option 3. The in-vehicle multi-mode violence detection system according to Option 2 includes an external source that provides prior information.
[0028] Option 4. The in-vehicle multi-mode violence detection system according to Option 2, wherein the prior information includes driving areas where known violence has occurred and past historical records of different “normal” relative “threat” events.
[0029] Option 5. The in-vehicle multi-mode violence detection system according to Option 1 includes an image compilation module, which identifies and combines vectors of output signals from the voice and non-voice audio event recognition module, the shaking and movement recognition module, the heart rate and respiratory rate detection module, and the in-vehicle semantic scene recognition module, and assigns predetermined thresholds to distinguish between normal events and threatening events.
[0030] Option 6. The in-vehicle multi-mode violence detection system according to Option 5 includes a threat assessment module, which uses auditory, visual, physiological and vibration-based indicators, as well as prior knowledge including violence history records and environmental information with location and time of day, to analyze the threat events to determine the threat level.
[0031] Solution 7. The in-vehicle multi-mode violence detection system according to Solution 6 includes a threat assessment unit, which receives the threat level, confirms whether there is an active threat, and generates an active threat signal.
[0032] Solution 8. The in-vehicle multi-mode violence detection system according to Solution 7, wherein after generating the active threat signal, a threat confirmation request is generated and forwarded to the threat timer unit.
[0033] Option 9. The in-vehicle multi-mode violence detection system according to Option 8 includes: A confirmation request is set, which is presented visually and audibly to the occupants, and a predetermined time interval is set, allowing at least one of the occupants to confirm the presence of the active threat in the vehicle; and The threat level determination unit is activated by a time interval timeout signal if a predetermined time interval is exceeded before a response is received from any of the occupants.
[0034] Option 10. The in-vehicle multi-mode violence detection system according to Option 9 includes: If any of the occupants confirms the existence of an active threat, the active threat signal is generated and forwarded to the threat level determination unit; The threat level determination unit identifies whether the active threat should be classified as one of "low threat," "medium threat," or "high threat," wherein: Identifying the “low threat” will activate the vehicle horn and vehicle warning lights; Recognizing the "moderate threat" will activate the vehicle horn and warning lights, activate the vehicle's brake assist system, and notify the external source of the moderate threat; and Identifying the “high threat” will activate the vehicle horn and the vehicle warning lights, activate the vehicle brake assist system, and forward a request to the external source to initiate an emergency assistance request.
[0035] Option 11. A method for performing multi-mode brute-force detection inside a vehicle, comprising: Use a voice and non-voice audio event recognition module to capture threatening speech and non-voice audio events of vehicle occupants; Analyze in-vehicle accelerometer data in the sway and motion recognition module; The heart rate and respiratory rate detection module identifies sudden physiological changes in the occupant's heart rate and respiratory rhythm. The in-vehicle semantic scene recognition module is used to capture and analyze non-verbal interactions between the occupants; and Generate one or more occupant threat indicators, including auditory, visual, physiological, and vibration-based indicators.
[0036] Option 12. The method according to Option 11 further includes applying prior knowledge, including a history of violence, to determine the threat level.
[0037] Option 13. The method according to Option 12 further includes applying environmental information, including vehicle location and time of day, derived from the prior knowledge.
[0038] Option 14. The method according to Option 11 further includes combining ride information and passenger information from the reservation database, and analyzing the ride information and passenger information to distinguish differences between the passengers.
[0039] Option 15. The method according to Option 11 further includes using the in-vehicle accelerometer data to identify the swaying movement of the occupants of the vehicle.
[0040] Option 16. The method according to Option 11 further includes applying indistinct sounds and sounds accompanied by threatening or aberrant behavior, including shouting, screaming, crying, and glass breaking, as audio-based threat indicators.
[0041] Option 17. The method according to Option 11 further includes analyzing the one or more occupant threat indicators in a multi-modal threat assessment module.
[0042] Option 18. A method for performing multi-mode in-vehicle brute-force detection, comprising: Use a voice and non-voice audio event recognition module to capture threatening speech and non-voice audio events of occupants in autonomous vehicles; Analyze in-vehicle accelerometer data in the sway and motion recognition module; The sudden physiological changes in the occupant's heart rate and respiratory rhythm are measured in the heart rate and respiratory rate detection module; The in-vehicle semantic scene recognition module is used to capture and analyze the non-verbal interactions between the occupants. Generate one or more occupant threat indicators, including auditory, visual, physiological, and vibration-based indicators; and In the image compilation module, a vector combining the output signals from the voice and non-voice audio event recognition module, the shaking and movement recognition module, the heart rate and respiratory rate detection module, and the in-vehicle semantic scene recognition module is identified, and a predetermined threshold for distinguishing between normal events and threatening events is assigned.
[0043] Option 19. The method according to Option 18 further includes: Collect output signals generated by multiple in-vehicle cameras located in the autonomous vehicle; Identify whether any of the plurality of in-vehicle cameras is covered in the camera evaluation unit; and The camera evaluation unit generates an audio pulse signal to notify the occupants of the autonomous vehicle to remove the cover from any of the plurality of in-vehicle cameras that are covered.
[0044] Option 20. The method according to Option 18 further includes: The presence of two or more occupants in the autonomous vehicle is identified by receiving occupant status signals forwarded to the occupant confirmation unit. The operation of activating the passenger type recognition unit includes: If the ride type identification unit generates a "ride-hailing type" signal, it makes a first procedural assumption that all the occupants in the autonomous vehicle know each other and that no threat will subsequently occur; or If the ride type identification unit generates a "shared ride type" signal, it makes a second procedural assumption that at least one of the occupants in the autonomous vehicle was not previously known to the other occupants, and therefore, there is a possibility that the threat event may develop during subsequent operation of the autonomous vehicle.
[0045] Other applicable areas will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description
[0046] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.
[0047] Figure 1 This is a system diagram of an in-vehicle multi-mode violence detection system based on an exemplary aspect; Figure 2 yes Figure 1 A flowchart illustrating the method, steps, and components of a multi-mode in-vehicle violence detection system; and Figure 3 yes Figure 1 A flowchart of the audio-based event recognition component of an in-vehicle multi-mode violence detection system. Detailed Implementation
[0048] The following description is exemplary in nature and is not intended to limit this disclosure, its application or use.
[0049] refer to Figure 1 The in-vehicle multi-modal violence detection system 10 is equipped with a vehicle 11, which includes, but is not limited to, autonomous vehicles, including, but not limited to, shared autonomous vehicles (SAVs) and partner-shared autonomous vehicles (PSAVs). The in-vehicle multi-modal violence detection system 10 provides multiple means for identifying threats, including audio threats, vibration-based threats, visual threats, and physiological threats that may affect vehicle occupants 12 such as driver or operator occupant 12A, front passenger occupant 12B, and one or more rear passenger occupants 12C. As will be discussed below, identified threats can be characterized as low, medium, and high threats, and can be further differentiated based on the operating area of the vehicle 11.
[0050] The in-vehicle multi-mode violence detection system 10 includes multiple in-vehicle microphones 14 that detect occupant voice and direct signals from the microphones 14 to a voice and audio event recognition module 16. The system also includes multiple in-vehicle accelerometers 18 that detect occupant movements, including hand and head movements, and direct signals and data from the accelerometers 18 to a sway motion module 20. The sway motion module 20 uses the accelerometer data to identify sway motion, defined as vibrations, oscillations, or sudden movements that define instantaneous or immediate motion.
[0051] The in-vehicle multi-mode violence detection system 10 also includes a reference Figure 2Multiple in-vehicle cameras, discussed in more detail, generate in-vehicle camera data 22 related to the occupant's position and movement over time, and direct signals from the in-vehicle camera data 22 to the in-vehicle semantic scene recognition module 24. The in-vehicle semantic scene recognition module 24 captures and analyzes nonverbal interactions between vehicle occupants 12 to generate occupant threat indicators.
[0052] The in-vehicle multi-mode violence detection system 10 also includes one or more in-vehicle radar devices 26, which detect the heart rate and respiratory rate of the occupant 12 and direct signals from the in-vehicle radar devices 26 to the heart rate and respiratory rate detection module 28. The heart rate and respiratory rate detection module 28 analyzes sudden changes in the occupant's heart rate and respiratory rhythm.
[0053] The voice and audio event recognition module 16 analyzes the detected occupant voice patterns, volume, etc., and generates an audio-based threat indicator 30. The swaying motion module 20 analyzes the acceleration, amplitude, and direction of occupant movement and generates a vibration-based threat indicator 32. The in-vehicle semantic scene recognition module 24 analyzes changes in occupant position or head or limb position over time as a changing semantic scene and generates a visual-based threat indicator 34. The heart rate and respiratory rate detection module 28 analyzes the data generated by the in-vehicle radar device 26, and its output is used to generate a physiological-based threat indicator 36 for the occupant.
[0054] Audio-based threat indicators 30, vibration-based threat indicators 32, visual threat indicators 34, and physiological threat indicators 36 are forwarded to image compilation module 38. Image compilation module 38 identifies input vectors, such as those combining visual and auditory signals, and assigns predetermined thresholds to each input to distinguish “normal” events, where there will be some increased sound level, or possibly some increased vibration or motion, which are differentiated from “threat” events by one or more applied thresholds. Image compilation module 38 compiles one or more observation vectors 40 of the input data and thresholds for transmission to threat assessment module 42. Threat assessment module 42 analyzes the threat situation using auditory, visual, and physiological and vibration-based indicators, and uses prior knowledge such as historical records of violence and environmental information such as location and time of day to determine the threat level.
[0055] The threat assessment module 42 receives the observation vector 40 and augments it with additional information to help distinguish between "normal" and "threat" events. For example, an external source 43, such as a cloud-based system, can provide prior information, such as driving areas where known violence has occurred and historical records of different "normal" relative "threat" behavioral events, for comparison with the current observation vector 40. Environmental information can also be received from the external source 43, which may include data distinguishing different areas the vehicle 11 will travel through and the ratio or different types of violence that may occur at different times of day or night. For example, a high-crime area may have an unusually high robbery rate after 10:00 PM, or multiple areas may historically be considered safe during the early morning or afternoon hours. The external source 43 may also provide occupant information from the SAV booking database, such as when one of the occupants 12 makes a carpooling or ride-hailing request. The threat assessment module 42 uses the above-described input, stored, and collected information to identify potential threats for further analysis and confirmation.
[0056] Note that external source 43 may also include monitoring and reporting systems such as OnStar®, which may further lead to the forwarding of threat assessments, along with communications of emergency contact and identification information related to vehicle 11, to emergency services such as 911 operators.
[0057] Threat assessment module 42 forwards potential threat information to threat assessment unit 44. If threat assessment unit 44 confirms that no active threat exists, it generates a first no-threat signal 46 and stores it in the "no anomaly" block 48, effectively ending the threat assessment of the potential threat information received by threat assessment module 42. If threat assessment unit 44 generates an active threat signal 50 based on the data received from threat assessment module 42, it generates a threat confirmation request 52 and forwards it to threat timer unit 54. The confirmation request is presented to occupant 12 visually and audibly, allowing a predetermined time interval for one of occupants 12 to confirm the presence of an active threat in vehicle 11. If the predetermined time interval is not exceeded before a response is received from any occupant 12, a time interval not exceeded signal 56 is generated, which initiates the operation of confirmation unit 58. If confirmation unit 58 receives confirmation from any occupant 12 that no threat exists, it generates and stores a second no-threat signal 60 in the "no anomaly" block 48, effectively ending the threat assessment of the potential threat information received by confirmation unit 58.
[0058] If the predetermined time interval is exceeded before a response is received from any occupant 12, a time interval timeout signal 62 is generated, and the operation of the threat level determination unit 64 is initiated. If the confirmation unit 58 receives confirmation of the presence of a threat from any occupant 12, an active threat signal 66 is generated and forwarded to the threat level determination unit 64. The threat level determination unit 64 applies logic, for example from Tables 1 and 2 below, to identify whether the active threat should be classified as one of “low threat” 68, “medium threat” 70, or “high threat” 72. Identification of “low threat” 68 results in the generation of an activation command 74, which activates the vehicle horn and vehicle warning lights. Identification of “medium threat” 70 results in a second activation command 76, which activates the vehicle horn and vehicle warning lights, activates the vehicle brake assist system, and notifies the external source 43 of the medium threat 70. Identification of “high threat” 72 results in a third activation command 78, which activates the vehicle horn and vehicle warning lights, activates the vehicle brake assist system, and forwards the request to the external source 43 to initiate a 911 emergency assistance request.
[0059] refer to Figure 2 And refer again Figure 1 ,Apart from Figure 1 In addition to the identified features, the in-vehicle multi-mode violence detection system 10 initially collects occupant information in the occupant information request 80. The occupant confirmation unit 82 identifies the presence of only a single occupant 12 or passenger in the vehicle 11 by receiving the occupant status signal 84 forwarded to it; this requires no further operation from the in-vehicle multi-mode violence detection system 10, as no threat is deemed to exist. If the occupant confirmation unit 82 generates a multi-occupant status signal 86, it confirms the presence of two or more occupants 12 or passengers in the vehicle 11, which may require further operation from the in-vehicle multi-mode violence detection system 10. The generation of the multi-occupant status signal 86 initiates the operation of the passenger type identification unit 88.
[0060] If the ride type identification unit 88 generates a "ride-hailing type" signal 90, it makes the procedural assumption that all occupants 12 in the vehicle 11 know each other and that no further threats requiring the in-vehicle multi-modal violence detection system 10 will occur. If the ride type identification unit 88 generates a "shared ride type" signal 92, it makes the procedural assumption that at least one of the occupants 12 in the vehicle 11 was not previously known to the other occupants 12, and therefore there is a possibility of a threat developing during subsequent operation of the vehicle 11.
[0061] The shared ride type signal 92 is forwarded to the voice and audio event recognition module 16 and the shaking motion movement module 20. The voice and audio event recognition module 16 receives the signal from the reference... Figure 1One or more of the in-vehicle microphones 14 and microphones 94 operate to receive occupant voice signals. These occupant voice signals are evaluated against stored data to identify whether the occupant's voice pattern indicates an abnormal audio event. Abnormal audio events may include, but are not limited to, verbal threats to the health or safety of one of the occupants or volumes exceeding a predetermined maximum volume threshold. An abnormal audio trigger evaluation 96 is performed, and if the audio trigger event 98 is negative, the program returns to the voice and audio event recognition module 16 for continued audio monitoring. If an audio trigger event 100 is detected and therefore positive, the program continues to the image compilation module 38 and the heart rate and respiratory rate detection module 28.
[0062] If a "shared ride type" signal 92 is generated, the shaking motion module 20 operates in parallel with the voice and audio event recognition module 16 from the reference. Figure 1 One or more in-vehicle accelerometers 18 receive accelerometer signals 102. The accelerometer signals 102 are evaluated against stored data to identify whether the accelerometer data indicates an abnormal motion event. Abnormal motion events may include, but are not limited to, rapid or violent acceleration or movement of one of the occupants 12, or an acceleration level exceeding a predetermined maximum accelerometer threshold. An abnormal motion trigger evaluation 104 is performed, and if the motion trigger event 106 is negative, the program returns to the shaking motion module 20 for continued motion monitoring. If a motion trigger event 108 is detected and therefore positive, the program continues to the image compilation module 38 and the heart rate and respiratory rate detection module 28.
[0063] Heart rate and respiratory rate detection module 28 receives output radar data 110 from one or more in-vehicle radar devices 26, and its output data is compared with reference data. Figure 1 The discussion relates to the heart rate and respiratory rate of occupant 12. The stored data is evaluated against output radar data 110 to identify whether the radar data indicates an abnormal occupant heart rate or respiratory rate event. This event may include, but is not limited to, a rapid or dramatic increase in the heart rate or respiratory rate of one of the occupants 12, or a heart rate or respiratory rate level exceeding a predetermined maximum heart rate or respiratory rate threshold. An abnormal occupant heart rate or respiratory rate trigger evaluation 112 is performed, and if the heart rate or respiratory rate trigger event 114 is negative, the program returns to the heart rate and respiratory rate detection module 28 for continued heart rate and respiratory rate monitoring. If a heart rate or respiratory rate trigger event 116 is detected and therefore positive, the program continues to the image compilation module 38 and the in-vehicle semantic scene recognition module 24.
[0064] The in-vehicle semantic scene recognition module 24 receives camera signals 118 from multiple in-vehicle cameras 119 as a reference in the image quality assessment module 120. Figure 1The image quality assessment module 120 reviews and adjusts the image quality of the received camera signals 118, based on the in-vehicle camera data 22. If one or more of the in-vehicle cameras 119 are covered by the camera assessment unit 122, an initial assessment is performed. If a camera coverage signal 124 is generated, one or more of the in-vehicle cameras 119 are covered and therefore unable to receive images, and the camera assessment unit 122 generates an audio pulse signal to notify the occupants 12 of the vehicle 11 to remove the cover from one of the covered in-vehicle cameras 119, and the process moves to the image compilation module 38. If a camera uncovered signal 127 is generated, at least one or more of the in-vehicle cameras 119 are uncovered and therefore able to receive images, and the process moves to the in-vehicle semantic scene recognition module 24.
[0065] The in-vehicle semantic scene recognition module 24 compares image data received from the in-vehicle camera 119 with a database of images representing "normal" scenes, such as occupant 12 sitting and moving normally during conversation, leaving, or entering the vehicle 11, in comparator 128. If the in-vehicle camera data 22 generates a no-abnormal-scene signal 130 compared to the images in the stored database, the program returns to the in-vehicle semantic scene recognition module 24 to continue monitoring. If the in-vehicle camera data 22 generates a visual-based threat indicator 34 indicating the presence of an abnormal scene compared to the images in the stored database, the program moves to the image compilation module 38. (As mentioned above...) Figure 1 The image compilation module 38 communicates with the threat assessment module 42 to identify potential threats for further analysis and confirmation.
[0066] refer to Figure 3 And refer again Figure 1 and Figure 2 According to several aspects, the in-vehicle multi-mode violence detection system 10 includes an audio-based event recognition component 132, which is related to a reference... Figure 1 The described voice and audio event recognition module 16 operates in conjunction. When occupant 12's voice is detected, the audio-based event recognition section 132 provides the communication of an audio signal 134 generated by the in-vehicle microphone 14 to the voice activity detection module 136. The voice activity detection module 136 confirms that the acoustic signal is related to the acoustic vocalization of occupant 12, which is converted into an acoustic vocalization signal 138. The acoustic vocalization signal 138 is forwarded to the feature extraction module 140, which extracts a feature vector 142, including Mel-frequency cepstral coefficients (MFCC), filter bank energy, the logarithm of the filter bank energy, and the spectral sub-band centroid, for example, using a Fourier transform equation.
[0067] Feature vector 142 and similar features from feature extraction module 140 are forwarded to speech and non-speech audio event recognition module 144. Speech and non-speech audio event recognition module 144 accesses memory 146 with an exemplary audio data file 148 representing known normal and threatening data stored in memory 146. Examples of threatening data stored in audio data file 148 include, but are not limited to, screams, shouts, cries, profanities, violent threats, glass breaking, gunshots, explosions, etc. Feature vector 142 is compared with the data stored as audio data file 148 to identify audio patterns 150 representing threatening audio events. If a threatening audio event is identified, the above reference is generated. Figure 1 Describe one or more audio-based threat indicators 30.
[0068] Table 1 below identifies exemplary fusion threat estimates, assuming that ATI is an audio-based threat indicator, VTI is a visual threat indicator, XTI is a vibration-based threat indicator, PTI is a physiological threat indicator, and FTE is a fusion threat estimate developed from threat indicators.
[0069] Table 1: <![CDATA[ Audio-based threat indicators (ATI) ]]> <![CDATA[ Visual Threat Indicator (VTI) ]]> <![CDATA[ Vibration-based threat indicator XTI ]]> <![CDATA[ Physiological Threat Indicator (PTI) ]]> <![CDATA[ Fusion Threat Estimation (FTE) ]]> No threat No threat No threat No threat No threat No threat No threat No threat threaten Low threat No threat No threat threaten No threat Low threat No threat No threat threaten threaten Low threat No threat threaten No threat No threat Medium threat No threat threaten No threat threaten High threat No threat threaten threaten No threat High threat No threat threaten threaten threaten High threat threaten No threat No threat No threat Medium threat threaten No threat No threat threaten High threat threaten No threat threaten No threat High threat threaten No threat threaten threaten High threat threaten threaten No threat No threat High threat threaten threaten No threat threaten High threat threaten threaten threaten No threat High threat threaten threaten threaten threaten High threat .
[0070] Other methods can be used to fuse these four indicators, such as a weighted scheme that combines the four threat indicators using inversely proportional weights, where the uncertainty of each threat indicator is quantified using its variance. Another approach is to use Dempster's combination rule, which takes into account the credibility of each threat indicator.
[0071] Table 2 below identifies the merged threat estimate (CTE) generated using the merged threat estimate, along with environmental information, including the vehicle's location and time of day.
[0072] Table 2: .
[0073] Methods for operating the in-vehicle multimodal violence detection system 10 may include the following: A voice and non-voice audio event recognition module 144 identifies multiple profanities and threatening speech, as well as non-voice audio events, such as indistinct sounds and sounds accompanied by threatening or abnormal behavior, such as shouting, screaming, crying, and glass breaking, as audio-based threat indicators. Audio events include, but are not limited to, crying, screaming, shouting, glass breaking, gunshots, and explosions. A swaying motion movement module 20 uses in-vehicle accelerometer data to identify swaying movements of the occupant 12. A heart rate and respiratory rate detection module 28 analyzes sudden physiological changes in the heart rate and respiratory rhythm of the occupant 12. An in-vehicle semantic scene recognition module 24 captures and analyzes non-verbal interactions between the occupants 12 and generates one or more occupant threat indicators if a threat situation exists. A multimodal threat assessment module 42 analyzes the situation using auditory, visual, physiological, and vibration-based indicators and applies prior knowledge, including violence history records and environmental information such as vehicle location and time of day, to determine the threat level.
[0074] The in-vehicle multi-mode harassment detection system 10 provides detection of one or more covered in-vehicle cameras 119 and, for example, uses audible alerts to remind occupants 12 inside the vehicle 11 to remove the covers from the one or more covered in-vehicle cameras 119. The in-vehicle multi-mode harassment detection system 10 automatically detects changes in occupant heart rate and heart rate, and applies this data as an occupant threat indicator when necessary. Different actions are recommended based on the threat level.
[0075] This disclosed in-vehicle multi-modal violence detection system 10 provides a system and method for assessing potential threats to the driver or occupants of an autonomous vehicle, or between unfamiliar occupants or passengers, during ride-sharing in autonomous vehicles. The system includes a non-voice audio event recognition module, an in-vehicle accelerometer-based sway and movement detection module, an in-vehicle semantic scene understanding module, and a threat level estimation module. For ride-sharing applications, the system and method of this disclosure combine and analyze passenger and ride-hailing information from a shared autonomous vehicle reservation database to identify threat scenarios between strangers.
[0076] The in-vehicle multimodal violence detection system 10 disclosed herein offers several advantages. These advantages include a system and method for assessing potential threats to a vehicle driver or occupant 12 or to unfamiliar passengers 11 within a limited shared autonomous vehicle 11. It includes a non-voice audio event recognition module, an in-vehicle accelerometer-based sway motion detection module, an in-vehicle semantic scene understanding module, and a threat level estimation module. In ride-sharing applications, the system and method of this disclosure combine and analyze ride and occupant information from a SAV booking database to identify scenarios between strangers.
[0077] The description in this disclosure is exemplary in nature only, and variations thereof that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.
Claims
1. A multi-mode in-vehicle violence detection system, comprising: A voice and non-voice audio event recognition module that captures threatening speech and non-voice audio events of vehicle occupants; Multiple in-vehicle accelerometers generate in-vehicle accelerometer data that is analyzed in the sway motion recognition module; One or more in-vehicle radar devices detect the heart rate and respiratory rate of vehicle occupants and direct signals from the in-vehicle radar devices to a heart rate and respiratory rate detection module, which measures the physiological changes in the heart rate and respiratory rhythm of the occupants and analyzes sudden changes in heart rate and respiratory rhythm. One or more in-vehicle cameras generate in-vehicle camera data related to the occupant's position and movement over time, and direct signals from the in-vehicle camera data to an in-vehicle semantic scene recognition module, which captures and analyzes non-verbal interactions between the occupants. One or more occupant threat indicators, including auditory threat indicators generated by the voice and non-voice audio event recognition module, visual threat indicators generated by the in-vehicle semantic scene recognition module, physiological threat indicators generated by the heart rate and respiratory rate detection module, and vibration-based threat indicators generated by the swaying motion recognition module; The threat level determination unit is activated by a time interval timeout signal if a predetermined time interval is exceeded before a response is received from any of the occupants. If any of the occupants confirms the existence of an active threat, the active threat signal is generated and forwarded to the threat level determination unit; The threat level determination unit identifies whether the active threat should be classified as low, medium, or high threat, wherein: Identifying the low threat level will activate the vehicle horn and vehicle warning lights; Recognizing the moderate threat will activate the vehicle horn and the vehicle warning lights, activate the vehicle brake assist system, and notify the external source of the moderate threat; as well as Identifying the high threat will activate the vehicle horn and warning lights, activate the vehicle brake assist system, and forward a request to the external source to initiate an emergency assistance request.
2. The in-vehicle multi-mode violence detection system according to claim 1, including prior knowledge of violence history records used to determine the threat level.
3. The in-vehicle multi-mode violence detection system according to claim 2, comprising an external source providing prior information.
4. The in-vehicle multi-mode violence detection system according to claim 2, wherein, Prior information includes driving areas where known violence has occurred, as well as past historical records of different “normal” relative “threat” events.
5. The in-vehicle multi-mode violence detection system according to claim 1, comprising an image compilation module, wherein the image compilation module identifies and combines vectors of output signals from the voice and non-voice audio event recognition module, the shaking and movement recognition module, the heart rate and respiratory rate detection module and the in-vehicle semantic scene recognition module, and assigns a predetermined threshold for distinguishing between normal events and threatening events.
6. The in-vehicle multi-mode violence detection system according to claim 5, comprising a threat assessment module, the threat assessment module using auditory, visual, physiological and vibration-based indicators, as well as prior knowledge including violence history records and environmental information having location and time of day, to analyze the threat events to determine the threat level.
7. The in-vehicle multi-mode violence detection system according to claim 6, comprising a threat assessment unit, wherein the threat assessment unit receives the threat level, confirms whether there is an active threat, and generates an active threat signal.
8. The in-vehicle multi-mode violence detection system according to claim 7, wherein, After generating the active threat signal, a threat confirmation request is generated and forwarded to the threat timer unit.
9. The in-vehicle multi-mode violence detection system according to claim 8, comprising: A confirmation request is set, which is presented visually and audibly to the occupants, and a predetermined time interval is set, allowing at least one of the occupants to confirm whether the active threat exists in the vehicle.
10. A method for performing multi-mode in-vehicle brute-force detection, comprising: Use a voice and non-voice audio event recognition module to capture threatening speech and non-voice audio events of vehicle occupants; Analyze in-vehicle accelerometer data in the sway and motion recognition module; The heart rate and respiratory rate of vehicle occupants are detected using one or more in-vehicle radar devices, and the signals from the in-vehicle radar devices are directed to the heart rate and respiratory rate detection module, in which sudden physiological changes in the heart rate and respiratory rate rhythm of the occupants are identified. Use one or more in-vehicle cameras to generate in-vehicle camera data related to the occupant's position and movement over time; The signal from the in-vehicle camera data is directed to the in-vehicle semantic scene recognition module; The in-vehicle semantic scene recognition module is used to analyze the non-verbal interactions between the occupants. as well as Generate one or more occupant threat indicators, including auditory, visual, physiological, and vibration-based indicators, including: If a predetermined time interval is exceeded before a response is received from any of the occupants, the threat level determination unit is activated by the time interval timeout signal; If any of the occupants confirms the existence of an active threat, the active threat signal is generated and forwarded to the threat level determination unit; The threat level determination unit identifies whether the active threat should be classified as low, medium, or high threat, and: Activate the vehicle horn and vehicle warning lights when the low threat is identified; Upon detection of the moderate threat, activate the vehicle horn and warning lights, activate the vehicle brake assist system, and notify an external source of the moderate threat; and Upon identification of the high threat, the vehicle horn and warning lights are activated, the vehicle brake assist system is activated, and a request is forwarded to the external source to initiate an emergency assistance request.
11. The method of claim 10, further comprising applying prior knowledge, including a history of violence, to determine the threat level.
12. The method of claim 11, further comprising applying environmental information, including vehicle location and time of day, derived from the prior knowledge.
13. The method of claim 10 further comprises combining ride information and passenger information from a reservation database, and analyzing the ride information and passenger information to distinguish differences between the passengers.
14. The method of claim 10, further comprising using the in-vehicle accelerometer data to identify swaying movements of the occupants of the vehicle.
15. The method of claim 10, further comprising applying indistinct sounds and sounds accompanied by threatening or aberrant behavior, including shouting, screaming, crying, and glass breaking, as audio-based threat indicators.
16. The method of claim 10, further comprising analyzing the one or more occupant threat indicators in a multi-modal threat assessment module.
17. A method for performing multi-mode in-vehicle brute-force detection, comprising: Use a voice and non-voice audio event recognition module to capture threatening speech and non-voice audio events of occupants in autonomous vehicles; Analyze in-vehicle accelerometer data in the sway and motion recognition module; The heart rate and respiratory rate of a vehicle occupant are detected using one or more in-vehicle radar devices, and the signals from the one or more in-vehicle radar devices are directed to the heart rate and respiratory rate detection module, in which sudden physiological changes in the heart rate and respiratory rhythm of the occupant are measured. Use one or more in-vehicle cameras to generate in-vehicle camera data related to the occupant's position and movement over time; The signal from the in-vehicle camera data is directed to the in-vehicle semantic scene recognition module; The in-vehicle semantic scene recognition module is used to analyze the non-verbal interactions between the occupants. Generate one or more occupant threat indicators, including auditory, visual, physiological, and vibration-based indicators; and In the image compilation module, a vector combining the output signals from the speech and non-speech audio event recognition module, the swaying motion recognition module, the heart rate and respiratory rate detection module, and the in-vehicle semantic scene recognition module is identified, and a predetermined threshold for distinguishing between normal and threatening events is assigned, including: If a predetermined time interval is exceeded before a response is received from any of the occupants, the threat level determination unit is activated by the time interval timeout signal; If any of the occupants confirms the existence of an active threat, the active threat signal is generated and forwarded to the threat level determination unit; The threat level determination unit identifies whether the active threat should be classified as low, medium, or high threat, and: Activate the vehicle horn and vehicle warning lights when the low threat is identified; Upon detection of the moderate threat, activate the vehicle horn and warning lights, activate the vehicle brake assist system, and notify an external source of the moderate threat; and Upon identification of the high threat, the vehicle horn and warning lights are activated, the vehicle brake assist system is activated, and a request is forwarded to the external source to initiate an emergency assistance request.
18. The method of claim 17, further comprising: Collect output signals generated by multiple in-vehicle cameras located in the autonomous vehicle; Identify whether any of the plurality of in-vehicle cameras is covered in the camera evaluation unit; as well as The camera evaluation unit generates an audio pulse signal to notify the occupants of the autonomous vehicle to remove the cover from any of the plurality of in-vehicle cameras that are covered.
19. The method of claim 17, further comprising: The presence of two or more occupants in the autonomous vehicle is identified by receiving occupant status signals forwarded to the occupant confirmation unit. The operation of activating the passenger type recognition unit includes: If the ride type identification unit generates a "ride-hailing type" signal, it makes a first procedural assumption that all the occupants in the autonomous vehicle know each other and that no threat will subsequently occur. or If the ride type identification unit generates a "shared ride type" signal, it makes a second procedural assumption that at least one of the occupants in the autonomous vehicle was not previously known to the other occupants, and therefore, there is a possibility that the threat event may develop during subsequent operation of the autonomous vehicle.
Citation Information
Patent Citations
Technologies for determining a threat assessment based on fear responses
US20160189517A1
Systems and methods for monitoring the situational awareness of a vehicle according to reactions of a vehicle occupant
US20210009140A1
Method for detecting blocking state of image acquisition device, electronic device and storage medium
US20210250470A1
Detection and mitigation of inappropriate behaviors of autonomous vehicle passengers
US20210397858A1
