Fusion-based method for detecting in-vehicle violence using a video camera

Abstract

An in-vehicle violence detection and fusion system includes a standard vehicle camera positioned within a vehicle that captures images of at least one occupant of the vehicle. An event camera positioned within the vehicle captures motion images of the at least one occupant. Region of interest (ROI) signals based on the standard vehicle camera and ROI signals based on the event camera are received and fused by a ROI fusion module to produce a fused ROI signal. A threat assessment module receives the fused ROI signal and augments the fused ROI signal with saved data to facilitate distinguishing between normal events and threat events. A violence indicator based on the event camera indicates a threat event to the at least one occupant of the vehicle.

Description

Technical Field

[0001] This disclosure relates to a vehicle abnormality or threat situation recognition system. Background Technology

[0002] The technology for shared mobility is already quite mature, as evidenced by the emergence of commercially available shared mobility services such as car sharing, ridesharing, ride-hailing, and ride-sourcing. Shared autonomous vehicles (SAVs) and pooled shared autonomous vehicles (PSAVs) are also rapidly appearing. Sharing a ride with a stranger in a shared autonomous vehicle (SAV) poses risks to the user's personal physical and emotional safety. Passengers may also experience feelings of crowding and invasion of personal space when sharing a ride or vehicle with strangers.

[0003] Therefore, although current shared and co-shared autonomous vehicles have achieved their intended purpose, new and improved in-vehicle violence detection and fusion systems are still needed. Summary of the Invention

[0004] According to some aspects, an in-vehicle violence detection and fusion system includes a standard vehicle camera positioned inside the vehicle, which acquires images of at least one occupant of the vehicle. An event camera positioned inside the vehicle acquires motion images of at least one occupant. A ROI fusion module receives and fuses the Region of Interest (ROI) signals based on the standard vehicle camera and the ROI signals based on the event camera to generate a fused ROI signal. A threat assessment module receives the fused ROI signal and enhances it with stored data to help distinguish “normal” events from violent or “threatening” events. A violence indicator based on the event camera indicates a threatening event to at least one occupant of the vehicle.

[0005] In another aspect of this disclosure, a privacy masking module applies privacy masking to ensure the privacy of at least one occupant of the vehicle. The privacy masking module receives the output of a standard vehicle camera and generates a region of interest (ROI) signal based on the standard vehicle camera.

[0006] In another aspect of this disclosure, the recommendation module based on the event camera device receives the output signal from the event camera device and outputs the ROI signal based on the event camera device.

[0007] In another aspect of this disclosure, the body skeleton detection and tracking module receives images from the privacy masking module and distinguishes human anatomy within the vehicle based on stored data retrieved from a database in the computer's memory, which stores images of human shape and movement.

[0008] In another aspect of this disclosure, the violent object detection module receives an image from the privacy masking module and distinguishes dangerous moving objects detected inside the vehicle.

[0009] In another aspect of this disclosure, an abnormal exposure detector receives output from an event camera device. The abnormal exposure detector distinguishes between exposures with normal light and contrast and exposures with high-contrast exposures. If the abnormal exposure detector determines that the output from the event camera device exceeds a predetermined event camera device output threshold, it generates an abnormal exposure signal.

[0010] In another aspect of this disclosure, the exposure adjustment module of the standard vehicle camera receives an abnormal exposure signal and generates an exposure change signal and sends the exposure change signal to the standard vehicle camera.

[0011] In another aspect of this disclosure, a person recognition module receives data output from a standard vehicle camera device to distinguish human-based camera image motion. A motion evaluation module, communicating with an event camera device, provides event camera image recognition on the evaluated image output. An action recognition module, communicating with both the person recognition module and the motion evaluation module, combines the outputs of the motion evaluation module and the person recognition module to compare the camera image data with stored data containing previously acquired and stored human action images.

[0012] In another aspect of this disclosure, the event camera device has a high speed limit of less than 1 ms. Standard vehicle camera devices operate at a low speed limit of less than 60 fps.

[0013] In another aspect of this disclosure, the event camera device has a high dynamic range defined as greater than 140 dB. The standard vehicle camera device has a low dynamic range defined as less than 100 dB.

[0014] According to some aspects, a method for detecting and fusing in-vehicle violence using camera equipment includes: positioning a standard vehicle camera that captures images of at least one occupant of the vehicle inside the vehicle; positioning a high-speed event camera that captures motion images of at least one occupant inside the vehicle; dividing the camera images into a grid distinguishing between positive and negative events, the grid having multiple grid cells, each of the multiple grid cells having an event count, the event count indicating the number of changing events recorded in the grid cell representing the camera images within a time interval; creating a circular buffer to track the change in the count of each of the multiple grid cells within the time interval; and updating the count for each time interval Δ. t The number of events, of which And fill the next grid with multiple grid cells; move the pointer to the next grid cell in the grid cells of the next grid and update the next grid cell of the grid cell; and use a ring buffer to indicate the change count of the next grid cell of the grid cell.

[0015] In another aspect of this disclosure, the method further includes indicating a positive event grid by a count of the overall increase in events in the grid cells, the count of the overall increase in events in the grid cells indicating the amount of increase in motion events within the time interval.

[0016] In another aspect of this disclosure, the method further includes indicating a negative event grid by counting the overall decrease in events in the grid cells, the overall decrease in events in the grid cells indicating the amount of reduction in motion events within the time interval.

[0017] In another aspect of this disclosure, the method further includes: identifying when an overall positive (+) event count occurs in most of the grid cells of a grid in a grid, resulting in an increase in overall scene brightness; and reducing the exposure of a standard vehicle camera.

[0018] In another aspect of this disclosure, the method further includes: identifying when an overall negative (-) event count occurs in most of the grid cells of a grid in a grid, resulting in a reduction in overall scene brightness; and increasing the exposure for a standard vehicle camera.

[0019] In another aspect of this disclosure, the method further includes deleting expired event counts by filling a quantity of zero (0) in one of the grid cells with the event counts that have not changed since the previous event count if no change in the counts is recorded.

[0020] In another aspect of this disclosure, the method further includes: applying an event camera device to optimally provide regions of interest (ROIs) with fast motion; using a standard vehicle camera device to provide ROIs with people; combining the ROIs from the event camera device and the ROIs from the standard vehicle camera device; and calculating the weights (w) of each grid based on + / - event counts, wherein: And determine the ROI region with the highest weight to fit the downstream receptive field.

[0021] According to some aspects, methods for detecting and fusing in-vehicle violence using camera equipment include: positioning a standard vehicle camera that captures images of at least one occupant inside the vehicle; positioning an event camera that captures moving images of at least one occupant inside the vehicle; generating a region of interest (ROI) signal based on the standard vehicle camera; generating an ROI signal based on the event camera; applying an ROI fusion module to receive and fuse the ROI signals based on the event camera and the standard vehicle camera and generate a fused ROI signal; sending the fused ROI signal to a threat assessment module to enhance the fused ROI signal with stored data to help distinguish normal events from violent or threatening events; and using a violence indicator based on the event camera to indicate a threatening event against at least one occupant of the vehicle.

[0022] In another aspect of this disclosure, the method further includes applying correlated information, including vehicle location and time of day, to reinforce the violence indicator.

[0023] In another aspect of this disclosure, the method further includes: applying privacy masking to ensure the privacy of at least one occupant of the vehicle; and forwarding the output of a standard vehicle camera to the privacy masking module to generate a region of interest (ROI) signal based on the standard vehicle camera.

[0024] In other words, this disclosure provides an in-vehicle violence detection and fusion system, comprising: a standard vehicle camera positioned inside the vehicle for acquiring images of at least one occupant of the vehicle; an event camera positioned inside the vehicle for acquiring motion images of the at least one occupant; a region of interest (ROI) signal based on the standard vehicle camera; an ROI signal based on the event camera; an ROI fusion module that receives and fuses the ROI signal based on the event camera and the ROI signal based on the standard vehicle camera to generate a fused ROI signal; a threat assessment module that receives the fused ROI signal and enhances the fused ROI signal with stored data to help distinguish between normal events and threatening events; and a violence index based on the event camera, indicating a threatening event to the at least one occupant of the vehicle. The in-vehicle violence detection and fusion system further includes: a privacy masking module that applies privacy masking to ensure the privacy of the at least one occupant of the vehicle; and the privacy masking module receives the output of the standard vehicle camera and generates the region of interest (ROI) signal based on the standard vehicle camera. The in-vehicle violence detection and fusion system includes a suggestion module based on an event camera device, which receives the output signal from the event camera device and outputs the ROI signal based on the event camera device. The system also includes a body skeleton detection and tracking module, which receives images from the privacy masking module and distinguishes human anatomy within the vehicle based on stored data retrieved from a database in a computer's memory, the database storing images of human shapes and movement. Furthermore, the system includes a violent object detection module, which receives images from the privacy masking module and distinguishes dangerous moving objects detected within the vehicle. Finally, the system includes an abnormal exposure identifier, which receives the output from the event camera device and distinguishes between normal light and contrast exposures and high contrast exposures. An abnormal exposure signal is generated if the abnormal exposure identifier determines that the output from the event camera device exceeds a predetermined event camera device output threshold. The in-vehicle violence detection and fusion system includes a standard vehicle camera exposure adjustment module, which receives the abnormal exposure signal, generates an exposure change signal, and sends the exposure change signal to the standard vehicle camera.The in-vehicle violence detection and fusion system includes: a human recognition module that receives data output from the standard vehicle camera device to distinguish human-based camera image motion; a motion evaluation module communicating with the event camera device that provides event camera image recognition for the evaluated image output; and an action recognition module that communicates with the human recognition module and the motion evaluation module and combines the outputs of the motion evaluation module and the human recognition module to compare the camera image data with stored data having previously acquired and stored human action images. Wherein: the event camera device has a high speed of less than 1 ms; and the standard vehicle camera device operates at a low speed of less than 60 fps. Wherein: the event camera device has a high dynamic range defined as greater than 140 dB; and the standard vehicle camera device has a low dynamic range defined as less than 100 dB. This disclosure also provides a method for detecting and fusing in-vehicle violence using a camera device, comprising: positioning a standard vehicle camera device inside the vehicle to acquire images of at least one occupant; positioning a high-speed event camera device inside the vehicle to acquire motion images of the at least one occupant; dividing the camera images into a grid distinguishing between positive and negative events, the grid having a plurality of grid cells, each of the plurality of grid cells having an event count, the event count indicating the number of changing events recorded in the grid cell representing the camera images within a time interval; creating a circular buffer to track changes in the number of individual grid cells in the plurality of grid cells within the time interval; and updating the event count for each time interval Δt, wherein... The method further includes: filling the next grid with multiple grid cells; moving the pointer to the next grid cell in the next grid cell and updating the next grid cell of the grid cell; and using a ring buffer to indicate the change count of the next grid cell of the grid cell. The method further includes indicating a positive event grid by indicating the increase count of events in the grid cell by indicating the number of increases in motion events during the time interval. The method further includes indicating a negative event grid by indicating the decrease count of events in the grid cell by indicating the number of decreases in motion events during the time interval. The method further includes: identifying when an overall positive (+) event count occurs in most grid cells of the grid cell of a grid in the grid, resulting in an increase in overall scene brightness; and reducing the exposure of the standard vehicle camera. The method further includes identifying when an overall negative (-) event count occurs in most grid cells of the grid cell of a grid in the grid, resulting in a decrease in overall scene brightness; and increasing the exposure of the standard vehicle camera. The method further includes deleting expired event counts by filling in a quantity of zero (0) into the event count of one grid cell that has remained unchanged since the previous event count if no change is recorded in the count. The method further includes: applying the event camera device to optimally provide regions of interest (ROIs) with fast movement; using the standard vehicle camera device to provide ROIs with people; combining the ROIs from the event camera device and the ROIs from the standard vehicle camera device; and calculating the weights (w) of each grid cell based on the + / - event counts, wherein: The method also provides a method for detecting and fusing in-vehicle violence using camera equipment, comprising: positioning a standard vehicle camera that acquires images of at least one occupant of the vehicle within the vehicle; positioning an event camera that acquires motion images of the at least one occupant within the vehicle; generating a region of interest (ROI) signal based on the standard vehicle camera; generating an ROI signal based on the event camera; applying an ROI fusion module to receive and fuse the event camera-based ROI signal and the standard vehicle camera-based ROI signal to generate a fused ROI signal; sending the fused ROI signal to a threat assessment module to enhance the fused ROI signal with stored data to help distinguish normal events from violent or threatening events; and using a violence index based on the event camera to indicate the threatening event to the at least one occupant of the vehicle. The method further includes applying correlation information including vehicle location and time of day to reinforce the violence index based on the event camera. The method further includes: applying privacy masking to ensure the privacy of the at least one occupant of the vehicle; and forwarding the output of the standard vehicle camera to the privacy masking module to generate the region of interest (ROI) signal based on the standard vehicle camera.

[0025] Other applicable areas will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are intended to be illustrative only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0026] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0027] Figure 1 This is a system diagram illustrating an in-vehicle violence detection and fusion system based on an exemplary aspect; Figure 2 It is a limitation Figure 1 A vision-based scene flowchart that is part of the system; Figure 3 It is by Figure 1 The grid of positive and negative events output by the system; Figure 4 It displays updated data. Figure 3 Positive event grid; Figure 5 It displays updated data. Figure 4 Positive event grid; Figure 6 It is used for fusion Figure 1 The grid of calculated weights for the system's data; Figure 7 It is a grid with final weight values; Figure 8 This is a system diagram showing the generation of data from standard vehicle camera equipment based on violence indicators from standard vehicle camera equipment; Figure 9 It is shown by Figure 1 The system generates an overlapping skeleton segment, a frontal view of the rear of an exemplary vehicle; and Figure 10 It is shown by Figure 1 The system generates a frontal view of a bounded, threatening object facing the rear of an exemplary vehicle. Detailed Implementation

[0028] The following description is exemplary only and is not intended to limit this disclosure, application, or purpose.

[0029] Reference Figure 1 The in-vehicle violence detection and fusion system 10 includes a vehicle 12, which includes, but is not limited to, autonomous vehicles, including, but not limited to, shared autonomous vehicles (SAVs), shared-shared autonomous vehicles (PSAVs), and shuttle buses. The in-vehicle violence detection and fusion system 10 provides multiple means for identifying violence, including visual threats that can affect vehicle occupants 14, such as a driver or operator (first occupant 14A) and one or more front or rear passenger (second occupant 14B). The in-vehicle violence detection and fusion system 10 identifies and classifies the level of violence. As will be discussed below, the identified level of violence can be described as low, medium, and high, and the level of violence can be further differentiated based on the operating area of ​​the vehicle 12.

[0030] Control of the in-vehicle violence detection and fusion system 10 can be performed using a single device such as computer 16, software components, or an on-board computer (collectively referred to as computer 16). Computer 16 can be deployed in a sensing vehicle 12. Computer 16 is a non-general-purpose electronic control device having: a pre-programmed digital controller or processor; memory or non-transitory computer-readable medium for storing data such as control logic, software applications, instructions, computer code, data, lookup tables, etc.; and transceivers or input / output ports. Computer-readable medium includes any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, compact disc (CD), digital video disc (DVD), or any other type of memory. Non-transitory computer-readable medium does not include wired, wireless, optical, or other communication links that transmit transient electrical signals or other signals. Non-transitory computer-readable medium includes media in which data can be permanently stored and media in which data can be stored and rewritten later, such as rewritable optical discs or erasable storage devices. Computer code includes any type of program code, which includes source code, object code, and executable code.

[0031] The in-vehicle violence detection and fusion system 10 also includes multiple in-vehicle cameras that generate in-vehicle camera data related to occupant position, occupant movement, and objects detected over time. The multiple in-vehicle cameras include at least one standard vehicle camera 18. The standard vehicle camera 18 is a camera similar to those provided in many different vehicle designs and provides optimized appearance detail, but is not optimized for fast motion resolution. The standard vehicle camera 18 operates at a low speed, defined herein as less than 60 fps (<60 fps), has a low dynamic range defined as less than 100 dB, and requires very high computational power to analyze the camera output data in real time. To provide high-speed camera data, the multiple in-vehicle cameras also include at least one event camera 20. The event camera 20 is optimized for detecting fast motion, but not for providing appearance detail. The event camera 20 provides a high dynamic range defined as greater than 140 dB and a high speed defined as less than 1 ms. The event camera device 20 does not produce polarity and intensity (-1, +1), but only produces a single pixel output (x, y, t, p) and a low resolution that limits the resolution to less than 1m pixels.

[0032] The output of image signals from one or more standard vehicle camera devices 18 is forwarded to a privacy masking module 22, which generates and forwards a masked image sequence 24 to a standard camera device ROI fusion module 26. The standard camera device ROI fusion module 26 generates a standard vehicle camera-based ROI signal 28, which is forwarded to an ROI fusion module 30. The output of the privacy masking module 22 is also forwarded in parallel to a body skeleton detection and tracking module 32 and a violent object detection module 34. The body skeleton detection and tracking module 32 uses a comparison algorithm to distinguish human anatomy within the vehicle 12 based on data retrieved from a database in the memory of the computer 16—which stores images of human shapes and movements. The violent object detection module 34 uses a comparison algorithm to distinguish objects detected within the vehicle 12 based on data retrieved from the database in the memory of the computer 16 into objects associated with dangerous activities, such as guns and knives, and normally existing objects, such as bags, wallets, and clothing.

[0033] The output signal from the event camera device 20 is forwarded to the event camera-based ROI suggestion module 36, which outputs an event camera-based ROI signal 38 to the ROI fusion module 30. The ROI signal 28 based on the standard vehicle camera and the event camera-based ROI signal 38 are fused by the ROI fusion module 30 to generate a fused ROI signal 40, which is forwarded to the threat assessment module 42. The body skeleton detection and tracking module 32 and the violent object detection module 34 also forward their outputs to the threat assessment module 42 in parallel.

[0034] The output of the high-speed operating event camera 20 can also be used to enhance the capabilities of the standard vehicle camera 18 during light-changing events or high-contrast events, such as when a vehicle enters a tunnel from sunlight. The exposure of the standard vehicle camera 18 can be adjusted during these events. For example, the output from the event camera 20 can be forwarded to an aberration exposure identifier 44, which operates in parallel with the ROI fusion module 30. The aberration exposure identifier 44 distinguishes between exposures with normal light and contrast and exposures with high-contrast exposures. If the aberration exposure identifier 44 determines that the output from the event camera 20 does not exceed a predetermined event camera output threshold, no aberration exposure signal 46 is generated and forwarded to the event camera-based ROI recommendation module 36 for further evaluation. If the aberration exposure identifier 44 determines that the output from the event camera 20 exceeds a predetermined event camera output threshold, an aberration exposure signal 48 is generated and forwarded to the standard vehicle camera exposure adjustment module 50. The standard vehicle camera exposure adjustment module 50 generates an exposure change signal and sends it to one or more of the standard vehicle cameras 18.

[0035] The threat assessment module 42 receives fused ROI signals 40 from the ROI fusion module 30, as well as outputs from the body skeleton detection and tracking module 32 and the violent object detection module 34, and enhances this information with additional data to help distinguish normal events from violent or threatening events. For example, external sources 52, such as cloud-based systems, can provide prior information, such as travel areas where known violent events have occurred, and historical records of different normal-to-threatening behavior events for comparative purposes. Correlation information can also be received from external sources 52, which may include data distinguishing different areas the vehicle 12 will travel through and the rate of violence or different types of violence that may occur at different times of day or night. For example, a high-crime area may have an unusually high robbery rate after 10 p.m., or an area may have historically been considered safe during the early morning or afternoon. External sources 52 may also provide occupant information from the SAV reservation database when one of the first occupants 14A or the second occupant 14B requests a ride or shared vehicle. The threat assessment module 42 uses the above inputs plus the stored and collected information to generate a threat index 54 based on the event camera device that indicates a threat to the occupants 14 of the vehicle 12, so as to be further analyzed and confirmed by the violence threat estimation unit 56.

[0036] It should be noted that external source 52 may further include monitoring and reporting systems, such as OnStar®, which may further enable violence estimation communications, along with emergency contact and identification information related to vehicle 12, to be forwarded to emergency services, such as 911 operators.

[0037] The threat assessment module 42 forwards potential threat information, including threat indicators 54, to the threat estimation unit 56. If the threat estimation unit 56 confirms that no actual threat exists, a no-threat signal is generated, which effectively terminates the threat estimation of the potential threat information received by the threat assessment module 42. If an actual threat signal is generated by the threat estimation unit 56 from the data received from the threat assessment module 42, a threat confirmation request is generated.

[0038] Reference Figure 2 And refer to again Figure 1 The vision-based scene flowchart 58 illustrates how data from the standard vehicle camera 18 and the event camera 20 are used by the in-vehicle violence detection and fusion system 10. The event camera 20 and the standard vehicle camera 18 communicate with a calibrator 60, which combines the camera images and identifies the pixel correspondences between the standard vehicle camera 18 and the event camera 20 to associate the images of the standard vehicle camera 18 and the event camera 20. Data output from the standard vehicle camera 18 is forwarded to a human recognition module 62, which distinguishes human-based motion in the camera images. The event camera 20 communicates with a motion evaluation module 64, which allows for event image recognition and provides evaluated image output. The human recognition module 62 and the motion evaluation module 64 together communicate with a motion recognition module 66 to combine the data and signal outputs of the motion evaluation module 64 and the human recognition module 62 to compare the image data with a database containing previously acquired and stored images of human motion. The output of the action recognition module 66, defining, for example, detected human motion signals, is forwarded to the event detection module 68. The event detection module 68 communicates with the external source 52 to retrieve external source data and combines the external source data with a threshold from the event detection module 68 indicating a certain level of threat in human motion. The output 70 of the event detection module 68, similar to a threat indicator 54, can be forwarded to a reference... Figure 1 The threat estimation unit 56 is described.

[0039] The output data from the standard vehicle camera 18 and the event camera 20 can be used as combined inputs to the ultra-high frame rate video reconstruction module 72 to perform a second fusion of threat estimation data. The ultra-high frame rate video reconstruction module 72 is capable of distinguishing occupant motion rates that are less than a predetermined motion rate threshold indicating normal human motion and greater than a predetermined motion rate threshold indicating threatening human motion, such as rapid movement from one occupant of vehicle 12 towards another occupant. The output of the ultra-high frame rate video reconstruction module 72 is also forwarded to the action recognition module 66.

[0040] Reference Figure 3 And refer to again Figure 1 and Figure 2 The in-vehicle violence detection and fusion system 10 disclosed herein operates as follows: Camera images are divided into grids distinguishing between positive and negative events, such as a first positive event grid 74A and a first negative event grid 76 defined at a first time interval t. Each grid cell 78 has a count quantity 80, which indicates the number of events at a time interval t. k The number of change events recorded in that portion or unit of the internal camera image. The positive event grid is defined by the total increase in the count of events in the grid and indicates the time interval t. k The number of increases in internal motion events. The negative event grid is defined by the total decrease in the number of events in the grid and indicates the time interval t. k The number of reduced internal motion events. For each grid, a ring buffer 82 is created to count or track the number of events in each individual grid cell or time interval t of grid cell 78. k The change in quantity within.

[0041] Reference Figure 4 And refer to again Figure 3 For each time interval Δt, where The event count is updated, and a new or next grid, such as the first grid 74B with a second time limit, is filled. For each grid, pointer 84 is then moved to the next cell in cell 86, and cell 86 is updated with the new event count. If no change in the count is recorded, a negative count is not recorded; however, expired event counts are deleted by filling the event count, which has not changed since the previous event count, in the appropriate cell, such as exemplary cell 88. A ring buffer 90 indicates the change in the count of cell 86.

[0042] If the overall positive (+) event count increases in most grids, the overall scene becomes brighter, which would necessitate reducing the exposure of the standard vehicle camera 18, as shown in the reference. Figure 1 As discussed below, if the overall negative (-) event count increases in most grids, the overall scene becomes darker. This would require increasing the exposure of the standard vehicle camera 18.

[0043] Reference Figure 5 And refer to again Figure 3 and Figure 4 The event count is updated again, and a new or next grid, such as the modified first grid 74C in the third time-limited grid, is filled. The overall positive (+) event count has increased, indicated by the value in cell 86 or an increase in count 92 from [previous data]. Figure 4The 44 in the modified first grid 74B shown increases to 67 in the modified first grid 74C. The ring buffer 94 tracks the increase in the count of unit 86. The overall positive increase (+) is further marked as the increase value or count 96 in unit 98, from... Figure 4 The 12 in the modified first grid 74B shown is increased to 34 in the modified first grid 74C. The ring buffer 100 tracks the increased count of the unit 98.

[0044] In such and Figure 4 compared to Figure 5 As indicated by the symbols, as the overall positive (+) event count increases in most grids, the overall scene becomes brighter, which allows for a reduction in the exposure of the standard vehicle camera 18. The event camera 20 provides guidance for the exposure control of the standard vehicle camera 18. The event camera 20 is selected to optimally provide ROI areas with fast movement. The standard vehicle camera 18 is best suited to provide ROI areas with people. One or more ROI areas from the event camera 20 and the standard vehicle camera 18 are combined, and weights are added. The ROI area with the highest weight is then determined, which is capable of fitting into the downstream receptive field. Optionally, areas downstream of the receptive field are selected and combined with the ROIs.

[0045] The exposure adjustment amount can be calculated using the following equation 1: Equation 1: .

[0046] Then, based on the + / - event count, the weight of each grid is calculated using the following Equation 2: Equation 2 : , in: c + and c - It is the event count of positive and negative events for each grid; w + and w - It is a weighted parameter for the + event count and the - event count; and f(c) It is a function used to convert event counts into appropriate weights.

[0047] Then, a standard vehicle camera 18 is used to detect and identify people in the scene. The standard vehicle camera weights are calculated for each grid cell based on the detected person's location, for example, by applying Gaussian weighting using Equation 3 below: Equation 3: , Where d is the distance from the grid to the centroid of the detected person; and σ is the parameter for Gaussian weighting.

[0048] Then, the weights of the meshes from the event camera device 20 and one or more standard vehicle camera devices 18 are fused using the following Equation 4: Equation 4: , in: α is a predetermined event value; and β is a predetermined standard camera device or conventional value.

[0049] Reference Figure 6 And refer to again Figures 3 to 5 To find candidate regions for downstream processing, the following steps are used. First, a table or grid 102 is generated, having weight values ​​calculated above, which are filled into multiple rows, such as exemplary rows 104 and 106, and multiple columns, such as exemplary columns 108 and 110, defining multiple grids 112. A window 114 with the same side 116 as the downstream receptive field is found. Second, the window 114 slides on the grid 112 in a first direction 118 or a second direction 120, and the sum of weights in the consecutive grids 112 of the window 114 is determined. Third, the position of the window 114 with the highest total weight is selected.

[0050] Reference Figure 7 And refer to again Figures 3 to 6 In the fourth step, a window 122 is created that contains a portion of the downstream receptive field, for example, one-quarter the size of window 114. In the fifth step, window 122 is slid across the grid, and the sum of the weights of the individual consecutive windows is determined. In the sixth step, the window with the highest total weight is selected. In the seventh step, the selected window is tiled to fill the receptive field.

[0051] Table 1 below illustrates a method for demonstrating the Fused Threat Estimate (FTE): Table 1: .

[0052] Table 2 below illustrates a method for consolidating violence estimation (CVE): Table 2: .

[0053] Other methods can be used to combine the two indicators, such as: 1) using a weighting scheme that combines the two indicators with weights that are inversely proportional to the uncertainty of each violent indicator as quantified by the variance of each violent indicator; or 2) using Dempster's rule of combination that takes into account the confidence level of each violent indicator.

[0054] Reference Figure 8 And refer to again Figures 1 to 7 The event camera device 20 identifies areas of rapid movement and combines these areas with human areas identified by the standard vehicle camera device 18. The downstream threat assessment module 42 only needs to focus on the areas identified in the previous steps, significantly reducing computational costs and improving accuracy. The body skeleton detection and tracking module 32 forwards data to the overlapping skeleton segment or boundary module 126, which integrates overlapping skeleton segments or boundaries into a violence indicator. The output of the overlapping skeleton segment or boundary module 126 is directed to the intersection query unit 128. If a no-intersection signal 130 is generated, the program returns to the overlapping skeleton segment or boundary module 126. If a data intersection signal 132 is generated, the program moves to the threat assessment module 42.

[0055] Information fusion is performed between threat indicator 54 based on event camera equipment and violence indicator 140 based on standard vehicle camera equipment using combinational logic (OR or AND), or by using a weighting scheme that combines the two indicators with weights inversely proportional to the uncertainty of each violence indicator quantified by the variance of each violence indicator, or by using Dempster's combination rule that considers the confidence level of each violence indicator. Violence indicators are reinforced using relevant information such as location and time of day. Different actions can be recommended based on the identified threat level.

[0056] Reference Figure 9 And refer to again Figures 1 to 8 Privacy masking based on standard vehicle camera equipment 142 uses reference Figure 1 The privacy masking module 22 is shown to ensure the privacy of occupants 14 of vehicle 12, including autonomous vehicles, shared vehicles, and shuttle buses. Violence detection using standard vehicle camera equipment 18 uses a body skeleton detection and tracking module 32 and an overlapping skeleton segment or boundary module 126 to generate overlapping skeleton segments 144, 146.

[0057] Reference Figure 10And refer to again Figure 8 and Figure 9 The program uses a violent object detection module 34 to detect threatening objects, such as weapons 148, which include sharp objects including knives and swords, guns and firearms, or martial arts and self-defense items. If the violent object detection module 34 generates a no-threat signal 136, the program returns to the violent object detection module 34 for further monitoring. If the violent object detection module 34 generates a threat signal 138, the program moves to the threat assessment module 42.

[0058] The in-vehicle violence detection and fusion system 10 of this disclosure applies an algorithm to generate Regions of Interest (ROIs) from an event camera device 20. The event camera device 20 is used to guide the exposure control of a standard vehicle camera device 18. A method is provided to fuse ROIs from the event camera device 20 and the standard vehicle camera device 18. The algorithm generates candidate regions for downstream processing. Violence detection based on the standard vehicle camera device is integrated into a violence index using overlapping skeleton segments or boundaries and used to detect threatening objects. Information fusion is performed between a threat index 54 based on the event camera device and a violence index 140 based on the standard vehicle camera device. Violence indices are consolidated using correlated information such as location, time of day, etc.

[0059] The in-vehicle violence detection and fusion system 10 disclosed herein offers several advantages. These advantages include a system and method for detecting violence inside a vehicle cabin or in a vehicle using a standard vehicle camera device 18. The fusion method utilizes the complementary properties of an event camera device 20 with high dynamic range and high speed, and an in-vehicle camera device with high resolution and rich detail, to detect violence inside a vehicle cabin or in a vehicle. More specifically, the event camera device 20 is used to identify areas of rapid movement, which are combined with human areas identified by the standard vehicle camera device 18. The downstream threat assessment module 42 only needs to focus on the areas identified in the previous steps, which significantly reduces computational costs and improves the accuracy of violence detection.

[0060] The description in this disclosure is merely exemplary and variations thereof are possible without departing from the spirit and scope of this disclosure. These variations should not be construed as departing from the spirit and scope of this disclosure.

Claims

1. A method for detecting and fusing in-vehicle violence using camera equipment, comprising: A standard vehicle camera device is provided inside the vehicle for capturing images of at least one occupant of the vehicle; An event camera device is provided in the vehicle for capturing motion images of the at least one occupant; Generate region of interest (ROI) signals based on standard vehicle camera equipment; Generate ROI signals based on event camera devices; The generation of ROI signals based on event camera devices includes: The motion image from the event camera device is divided into a grid that distinguishes between positive and negative events. Each grid has multiple grid cells, and each grid cell has an event count, which indicates the number of change events recorded in the grid cell that represent the motion image within a time interval. A circular buffer is created to track the change in the number of individual grid cells in the plurality of grid cells over the time interval; Update for each time interval Δ t The number of events counted; The event count for each grid cell within the grid is updated by moving the pointer sequentially to the next grid cell within the grid in the circular buffer, while the circular buffer is used to identify the change count for each grid cell. An ROI fusion module is applied to receive and fuse the ROI signal from the event-based camera device and the ROI signal from the standard vehicle camera device to generate a fused ROI signal. An application threat assessment module receives the fused ROI signal and enhances the fused ROI signal with stored data to help distinguish between normal events and threat events; and Violence indicators based on event camera devices are used to identify threat events to at least one occupant of the vehicle.

2. The method according to claim 1, comprising: A privacy masking module is applied, which applies privacy masking to ensure the privacy of at least one occupant of the vehicle; and The privacy masking module receives the output of the standard vehicle camera and generates the region of interest (ROI) signal based on the standard vehicle camera.

3. The method of claim 2, further comprising an application of an event-based camera device recommendation module, which receives an output signal from the event-based camera device and outputs an event-based camera device ROI signal.

4. The method of claim 2, further comprising applying a body skeleton detection and tracking module that receives images from the privacy masking module and distinguishes human anatomy within the vehicle based on stored data retrieved from a database in a computer's memory, the database storing images of human shapes and movements.

5. The method of claim 2, further comprising applying a violent object detection module that receives an image from the privacy masking module, the violent object detection module distinguishing dangerous moving objects detected inside the vehicle.

6. The method of claim 1, further comprising applying an abnormal exposure detector that receives output from the event camera device, the abnormal exposure detector distinguishing between normal light and contrast exposures and high contrast exposures, wherein an abnormal exposure signal is generated if the abnormal exposure detector determines that the output from the event camera device exceeds a predetermined event camera device output threshold.

7. The method according to claim 6, further comprising an exposure adjustment module for a standard vehicle camera, which receives the abnormal exposure signal, generates an exposure change signal, and sends the exposure change signal to the standard vehicle camera.

8. The method according to claim 1, comprising: The application includes a human recognition module that receives data output from the standard vehicle camera device to distinguish human-based motion in the camera images. An application communication motion evaluation module is provided with the event camera device, which provides event camera image recognition of the evaluated image output; and An application motion recognition module communicates with the human recognition module and the motion evaluation module and combines the outputs of the motion evaluation module and the human recognition module to compare the camera image data with stored data containing previously acquired and stored human motion images.

9. The method according to claim 1, wherein: The event camera device has a high speed of less than 1ms; and The standard vehicle camera operates at a low speed of less than 60fps.

10. The method according to claim 9, wherein: The event camera device has a high dynamic range with a defined range greater than 140 dB; and The standard vehicle camera equipment has a low dynamic range limited to less than 100dB.

Images