Ultra-low power consumption monitoring device and method

Abstract

The present invention discloses an ultra-low power consumption monitoring device and method. In order to solve the technical problem of high power consumption of monitoring equipment, the monitoring device disclosed in the present invention includes at least a first video camera and a first event imaging device, as well as a pulse neural network processor and a monitoring logic control unit; the first event imaging device is used to detect moving objects in the field of view and output a pulse sequence; the pulse neural network processor is used to perform inference on the pulse sequence output by the first event imaging device to obtain an inference result; based on the inference result, the monitoring logic control unit decides whether to wake up the first video camera. The present invention solves the problem of excessive power consumption and difficulty in long-term battery life in the monitoring field, and achieves technical effects such as adaptive adjustment of monitoring strategies according to the environment. The present invention is applicable to the field of video surveillance.

Description

[0001] This invention is a divisional application of Chinese invention patent application No. 202210518666.0 (filing date May 13, 2022, entitled "Monitoring device, method, and driving recorder"). All technical solutions described in that application are hereby incorporated by reference into this invention. Technical Field

[0002] The present invention relates to an ultra-low power consumption monitoring device and method, and in particular to a device, method and driving recorder for realizing monitoring with ultra-low power consumption. Background Art

[0003] A dashcam is a common video surveillance device in cars, providing detailed driving data and a crucial tool for protecting the rights and interests of drivers. In addition to recording data while driving, users or device providers often require dashcams to monitor vehicle parking. Parking monitoring can effectively record incidents such as theft, scratches, and other accidents.

[0004] However, when the car is stopped, there are generally two ways to power the dashcam: 1. The dashcam's built-in battery or / and a power bank (2-12Wh); 2. The car battery (approximately 700Wh) provides constant power. Constant power sources include the cigarette lighter, OBD port, ACC fuse box, and reading light fuse box. The former, due to the limited space in the dashcam, provides limited power, resulting in a shorter effective parking monitoring period. High temperatures and other environmental factors can easily cause the battery to swell or even explode, posing a safety hazard. With the latter, although the battery has a high capacity, continuous long-term video recording will drain it. Once the battery is discharged, the car will not start, creating a very bad experience for the driver. Specialized step-down circuits exist to cut off power before the battery is discharged, but these are only workarounds and do not fundamentally solve the problem. Furthermore, even if the dashcam triggers a low-voltage power outage, other in-car electrical devices will continue to consume power, which can also cause the car to not start.

[0005] The ultimate goal of this field is to effectively implement parking monitoring with ultra-low power consumption. In order to reduce the power consumption of surveillance video, the current solutions mainly include:

[0006] (1) When the vibration detection sensor detects that the vehicle has collided, the video recording function will be activated;

[0007] (2) Reduce the frame rate and record the video in time-lapse mode, for example, one frame per second;

[0008] (3) When motion is detected, the recording function is automatically activated.

[0009] With the first option, if the recording is activated only after the collision occurs, the vehicle may have already left the scene due to startup delays and the vehicle's high speed. The recorded footage will only show the vehicle's back as it drives away, or other vehicles passing by. It won't capture the moment the vehicle struck the driver's vehicle, meaning the moment before the accident. This monitoring solution makes it difficult to determine whether the vehicle in the recorded video is the driver or just a passing vehicle. This makes it difficult for traffic police to determine liability and thus hinders the protection of the driver's rights.

[0010] In addition, for some minor scratches, if the vibration sensor or its settings are not sensitive enough, the necessary surveillance video will be missed; and if the sensor or its settings are too sensitive, vibrations such as those caused by wind and rain will also trigger wake-up.

[0011] The second solution, due to the time lag, is likely to miss crucial footage. Furthermore, since a significant amount of circuitry is still operating while waiting for the next shot, this solution doesn't significantly reduce power consumption. For example, for time-lapse recording at 5 frames per second, power consumption remains around 2 watts (48 Wh / 24 hours). Furthermore, prolonged rewriting of the memory also shortens its lifespan.

[0012] As for the third option, motion detection is a more ideal solution. However, although current motion detection solutions can only operate during the effective time period, detecting movement through image processing technology still requires a lot of computing and power consumption. Furthermore, even the mere passing of pedestrians (such as in open-air parking lots with heavy traffic) will trigger recording. However, in certain situations, such as streets with a large number of pedestrians, this will trigger a large amount of recording footage, which is usually unnecessary.

[0013] Currently, there is also a solution that uses microwave sensors / microwave eyes to achieve motion detection. That is, once a moving object is detected within the set range of the vehicle, the recording is awakened. Compared with the time-lapse recording solution (about 50Wh / 24 hours), it can achieve a lower power consumption of 8Wh / 24 hours, but it is still difficult to meet the needs of ultra-long-term (several months) parking monitoring, and there is still a risk of power outage, especially when meeting multi-angle monitoring needs.

[0014] In addition, there is also a solution that replaces the sleep mode without shooting with a one-second shooting mode, and then combines the first or third solution as a solution to wake up the normal shooting mode, but these solutions do not significantly improve power consumption.

[0015] Prior art 1: US2021 / 0185265A1.

[0016] Prior art 1 discloses a video surveillance solution using an event-based sensor (EBS), which utilizes the event-triggered feature of EBS. Compared to traditional frame image sensors that need to capture a frame image periodically according to the frame rate, this sensor event-triggered feature reduces the power consumption of the image perception link to a certain extent. However, on the one hand, this solution needs to compress the EBS sensor data before data processing (to obtain a frame image similar to the output of the frame image sensor), and on the other hand, the processing link still uses the high-power information processing mode of traditional artificial neural networks (devices that deploy artificial neural networks are usually high-power devices). Therefore, as far as the system as a whole is concerned, it is still a high-power monitoring solution.

[0017] So far, with the exception of the first solution (which suffers from the aforementioned inherent drawbacks), all other solutions or combinations have shown relatively high power consumption. Currently, mainstream brands of parking monitoring systems have measured power consumption of approximately 8 to 50 Wh per 24 hours, failing to achieve truly ultra-low power consumption and ultra-long battery life. Many users inevitably require long-term parking. Therefore, a technical solution that can achieve ultra-low power consumption and comprehensive, effective parking monitoring is urgently needed. Summary of the Invention

[0018] In order to solve or alleviate some or all of the above technical problems, the present invention is achieved through the following technical solutions:

[0019] A monitoring device comprises at least a first recording camera and a first event imaging device, as well as a pulse neural network processor and a monitoring logic control unit, wherein: the first event imaging device is used to detect moving objects in a field of view and output a pulse sequence; the pulse neural network processor is used to perform inference on the pulse sequence output by the first event imaging device to obtain an inference result; and based on the inference result, the monitoring logic control unit decides whether to wake up the first recording camera.

[0020] In a certain class of embodiments, the first event imaging device is based on a pulse sequence obtained in any one of the following ways: (a) DVS; (b) DAVIS; (c) a pulse sequence obtained based on a frame image sensor difference frame.

[0021] In some embodiments, the monitoring logic control unit further receives auxiliary information and uses the information to determine whether to wake up the first recording camera.

[0022] In a certain embodiment, the monitoring device is a parking monitoring device; the auxiliary information includes one or more of the following: GPS signal quality information or location information obtained based on GPS, microwave sensor output information, user's vehicle usage history data, the time since the last time the first recording camera was awakened, the number of times the first recording camera was awakened within the set time, vibration sensor information, sound detection information, and current time information.

[0023] In a certain embodiment, the monitoring device is a parking monitoring device; the control instructions issued by the monitoring logic control unit include one or more of the following: normal recording instructions, time-lapse recording instructions, no recording instructions, deletion instructions, and lock instructions; if the monitoring logic control unit issues a no recording instruction or does not send an instruction, the operation of waking up the first recording camera will not be executed.

[0024] In a certain embodiment, the monitoring device is a parking monitoring device; the instructions issued by the monitoring logic control unit also include one or more of the following: a warning sound broadcast instruction, a user notification instruction, a door unlocking instruction, a window opening instruction, a user and vehicle call establishment instruction, and an air conditioning turn-on instruction.

[0025] In a certain embodiment, the monitoring device is a parking monitoring device; the monitoring device has at least a high-sensitivity monitoring strategy and a low-sensitivity monitoring strategy that can be applied; the high-sensitivity monitoring strategy and the low-sensitivity monitoring strategy differ at least in whether the wake-up can be triggered for people and / or cars within the field of view.

[0026] In a certain embodiment, the monitoring device is a parking monitoring device; if the monitoring device detects a vibration signal indicating that the vehicle body is vibrating, the monitoring logic control unit issues a locking instruction, which indicates that the video recorded before and after the vibration signal appears cannot be automatically overwritten by the monitoring device.

[0027] In certain embodiments, the recorded video includes video recorded by an event imaging device or obtained through processing.

[0028] In some embodiments, the monitoring logic control unit further receives auxiliary information, and determines which sensitivity monitoring strategy to apply based on the auxiliary information, and decides whether to wake up the first recording camera according to the determined sensitivity monitoring strategy.

[0029] In some embodiments, the monitoring device is a parking monitoring device; whether a pedestrian in the field of view can trigger wake-up is determined based on one or more of the length of stay, whether contact with the vehicle body occurs, and the direction of the pedestrian's travel.

[0030] In some embodiments, the monitoring device is a parking monitoring device; the pulse neural network processor and / or the monitoring logic control unit does not output the operation to trigger the wake-up for ordinary pedestrians and / or ordinary vehicles.

[0031] In some embodiments, the first recording camera and the first event imaging device have different fields of view; and compared with the field of view of the first recording camera, the field of view of the first event imaging device is conducive to reducing unnecessary wake-up operations.

[0032] In a certain embodiment, the monitoring device also includes at least a second recording camera and a second event imaging device, wherein the second event imaging device is used to provide a basis for whether to wake up the second recording camera; and the field of view of the second recording camera and the second event imaging device is different from the field of view of the first recording camera and the first event imaging device.

[0033] In some embodiments, the monitoring device is a parking monitoring device; the pulse neural network processor is configured to identify visual information patterns of weather conditions and / or light strobing in the field of view; or / and, the pulse neural network processor is configured to identify visual information patterns of vehicles entering / exiting adjacent parking spaces in the field of view; or / and, the pulse neural network processor is configured to identify visual information patterns of vehicles in adjacent parking spaces opening their doors in the field of view.

[0034] In a certain embodiment, the monitoring device is a parking monitoring device; the video camera and the associated event imaging device of the monitoring device are installed at one or more of the following locations: left rearview mirror, right rearview mirror, rear of the car, front of the cockpit, and front of the vehicle body; for the solution of installing video cameras and associated event imaging devices in multiple locations, if a wake-up video signal is detected, all video cameras are turned on at the same time.

[0035] In certain embodiments, when an object moves in front of the event imaging device, a pulse event or video output by the event imaging device is recorded.

[0036] In a certain embodiment, the monitoring device is a parking monitoring device; after detecting a vibration signal, the pulse event or video output by the event imaging device and the video recorded by the video camera are merged into one video, and the merged video displays a time stamp corresponding to the on-site information.

[0037] In certain embodiments, the resolution size of the video output by the event imaging device or the video generated via pulse event conversion is enlarged.

[0038] In certain embodiments, when the videos recorded by the event imaging device and the video recording camera overlap in time, the video output by the event imaging device or the video generated by the pulse event conversion is embedded in the video recorded by the video recording camera in a picture-in-picture manner.

[0039] In some embodiments, the field of view of the first event imaging device is lower than the field of view of the first video camera.

[0040] A monitoring device, which includes at least a first recording camera and a first event imaging device, and a monitoring logic control unit, wherein: the first event imaging device is used to detect moving objects in the field of view and output a pulse sequence or a differential frame image; based on the pulse sequence or the differential frame image, it is determined whether the change in visual information captured by the first event imaging device exceeds a preset value, and if so, the pulse sequence, differential frame image, or frame image before the differential frame output by the first event imaging device is stored; if the monitoring logic control unit receives a vibration signal, the first recording camera is awakened and a recording video is recorded; based on the recording video and the pulse sequence, differential frame image, or frame image before the differential frame output by the first event imaging device, a final target video is generated.

[0041] In certain embodiments, the pulse sequence outputted by the first event imaging device, or the difference frame image, or the frame image before the difference frame is stored in a buffer, and the data is stored in an overwritable manner.

[0042] A monitoring device comprises at least a first recording camera and a first event imaging device, wherein: the first event imaging device is used to detect moving objects in a field of view and output a pulse sequence or a differential frame image; based on the pulse sequence or the differential frame image, it is determined whether the change in visual information captured by the first event imaging device exceeds a preset value; if so, a pulse neural network processor performs inference on the pulse sequence or the pulse sequence converted from the differential frame image to determine whether it is a case where awakening is not required; if it is not a case where awakening is not required, the first recording camera is awakened.

[0043] A monitoring method is applied to a driving recorder, which includes at least a first recording camera and a first event imaging device, as well as a pulse neural network processor and a monitoring logic control unit, wherein: the first event imaging device is used to detect moving objects in the field of view and output a pulse sequence; the pulse neural network processor is used to perform inference on the pulse sequence output by the first event imaging device to obtain an inference result; and based on the inference result, the monitoring logic control unit decides whether to wake up the first recording camera.

[0044] In a certain type of embodiment, in different scenarios, there are at least a high-sensitivity monitoring strategy and a low-sensitivity monitoring strategy to choose from; the high-sensitivity monitoring strategy and the low-sensitivity monitoring strategy differ at least in whether the wake-up can be triggered for people and / or cars within the field of view.

[0045] A driving recorder comprises the monitoring device as described in any one of the preceding items, or applies the monitoring method as described in any one of the preceding items.

[0046] Some or all of the embodiments of the present invention have the following beneficial technical effects:

[0047] 1) Ultra-low power parking monitoring. Compared to existing technologies that typically require a maximum of several days to two weeks for parking monitoring, this invention achieves sub-milliwatt standby power consumption, enabling ultra-long-term parking monitoring. This meets the industry's demand for extreme endurance and addresses the industry's pain point of battery depletion and subsequent ignition failure caused by high power consumption.

[0048] 2) Environmental adaptability. Based on external environmental information and / or internal historical data, the system can adaptively apply monitoring strategies of varying sensitivity, striking a balance between power consumption and user value (whether anomalies can be recorded promptly). Compared to traditional motion detection solutions, this adjustable wake-up sensitivity can further reduce unnecessary wake-ups, thereby reducing overall system power consumption.

[0049] 3) It can provide a complete chain of evidence. Traditional solutions that detect vibrations and then record video can only provide invalid evidence after the fact. However, this invention can also provide strong video evidence before the accident occurred. The chain of evidence before, during, and after the accident provides a complete chain of evidence for subsequent accountability and claims, improving users' experience with the uselessness of traditional parking monitoring.

[0050] 4) Solve the problem of side scratches, which are difficult to capture with traditional parking monitoring. The power consumption of a single recording camera in traditional monitoring solutions is already high enough, making it difficult to solve the problem, and it is even more difficult to handle recording angles such as side scratches.

[0051] 5) The additional in-vehicle liveness detection capability enriches the parking monitoring function, preventing users from more accidents or even irreparable losses.

[0052] More beneficial effects will be further introduced in the preferred embodiments.

[0053] The technical solutions / features disclosed above are intended to summarize the technical solutions and features described in the detailed description, and therefore the scope of the disclosure may not be exactly the same. However, the new technical solutions disclosed in this section are also part of the numerous technical solutions disclosed in this invention document. The technical features disclosed in this section, together with the technical features disclosed in the subsequent detailed description and portions of the drawings not explicitly described in the specification, can be reasonably combined to disclose further technical solutions.

[0054] The technical solution formed by combining all the technical features disclosed anywhere in the present invention is used to support the summary of the technical solution, the modification of the patent document, and the disclosure of the technical solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of the structure of the sensing side of the driving recorder;

[0056] Figure 2 This is a schematic diagram of the structure on the opposite side of the driving recorder sensor;

[0057] Figure 3 It is a structural diagram of the monitoring device in the present invention;

[0058] Figure 4 It is an example of applying different sensitivity monitoring strategies in real scenarios;

[0059] Figure 5 It is a schematic diagram of the parking monitoring scenario;

[0060] Figure 6 This is a schematic diagram of the foreground field of view for parking monitoring;

[0061] Figure 7 It is a schematic diagram of 360 panoramic monitoring mode;

[0062] Figure 8 It is a schematic diagram of the front and rear dual camera mode;

[0063] Figure 9 This is a schematic diagram of controlling the number of wake-up times through the field of view;

[0064] Figure 10 This is a schematic diagram of a detection process for a collision when a car leaves a garage in a certain embodiment;

[0065] Figure 11 2. It is a monitoring diagram from not waking up to waking up in one embodiment;

[0066] Figure 12 It is a schematic diagram of the composition of surveillance video;

[0067] Figure 13 This is a comparison chart based on SNN processor solutions. DETAILED DESCRIPTION

[0068] Because it is impossible to exhaustively describe various alternative solutions, the following will clearly and completely describe the key points of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Other technical solutions and details not disclosed in detail below generally fall within the technical objectives or technical features that can be achieved through conventional means in the art. Due to space limitations, the present invention will not describe them in detail.

[0069] Unless used in the context of division, " / " in any position in this disclosure represents a logical "OR." Sequential numbers such as "first" and "second" in any position in this disclosure are used solely as descriptive markers and do not imply an absolute temporal or spatial order, nor do they imply that the term prefixed with such a number necessarily refers to a different thing than the same term prefixed with another modifier.

[0070] This disclosure describes various key points that can be combined into various specific embodiments, which will be incorporated into various methods and products. In this disclosure, even if a key point is described only when introducing a method / product solution, it means that the corresponding product / method solution also explicitly includes the technical features.

[0071] When a step, module, or feature is described as existing or included at any point in the present invention, it does not imply that such existence is exclusive and unique. Those skilled in the art can obtain other embodiments based on the technical solutions disclosed in the present invention and supplemented by other technical means. Based on the key points described in the specific embodiments of the present invention, those skilled in the art can replace, delete, add, combine, or reorder certain technical features to obtain a technical solution that still complies with the concept of the present invention. Such solutions that do not deviate from the technical concept of the present invention are also within the scope of protection of the present invention.

[0072] Explanation of terms:

[0073] 1. Video camera: It is a device used to record optical information. It can be any camera suitable for monitoring, such as a traditional optical device that generates RGB images or videos. Its sensor is generally a frame image sensor.

[0074] 2. Event imaging device: A device that can generate pulse events, such as a dynamic vision sensor (DVS), a DAVIS sensor, or a device that generates pulse events based on the pixel value conversion output by a traditional optical sensor (such as RGB, CCD, grayscale sensor).

[0075] The present invention is applicable to any scenario requiring low power consumption monitoring, such as indoor monitoring, community security monitoring, etc. The present invention takes parking monitoring as an example and generalizes it to the aforementioned scenarios, without being limited to any specific scenario.

[0076] refer to Figure 1 , which shows a schematic structural diagram of the sensing side of the driving recorder shown in the present invention. Optionally, the driving recorder includes a connection portion, such as an axis connection device with a universal joint or multiple degrees of freedom, to connect the driving recorder to the driving recorder installation vehicle (hereinafter referred to as the user vehicle), such as the windshield, center console, or A-pillar. Optionally, the driving recorder includes a microphone for recording surrounding sound information. In certain embodiments, command recognition is performed on the sound information recorded by the microphone to execute certain set functions.

[0077] The driving recorder also includes a first recording camera, which is a core component for recording surrounding optical information and is also the main component for the driving recorder to achieve monitoring purposes. This solution can be any known or suitable optical module.

[0078] The driving recorder also includes a first event imaging device. Common event imaging devices, such as DVS, only capture dynamic visual information and generate a pulse event sequence (referred to as a pulse sequence). In addition, the pixel values ​​obtained by the traditional RGB sensor (such as the pixel values ​​obtained after differentiating two frames of images) are converted into pulse events, which can also be the event imaging device here. Preferably, the (target) pulse sequence is obtained by the difference between the images of several adjacent frames (including the situation of two adjacent frames before and after) output by the frame image sensor, or after the noise filtering, the entire difference frame image is taken as a whole, and the generated pulse sequence is randomized.

[0079] refer to Figure 2 , which shows a schematic structural diagram of the user side of the driving recorder shown in a certain embodiment of the present invention. In addition to the aforementioned connection part, it optionally also includes a display screen and buttons. Preferably, in order to achieve in-vehicle monitoring and / or side (both sides and rear of the vehicle) monitoring, in addition to the aforementioned first recording camera, the driving recorder also includes at least a second recording camera, which is mainly used to record areas outside the field of view of the first recording camera, such as the interior of the vehicle and / or the side windows and rear window of the seat.

[0080] Preferably, the driving recorder further includes a second event imaging device, which is essentially the same as the first event imaging device and will not be described in detail.

[0081] For each video camera that needs to be awakened, there is an associated event imaging device with the same or approximately the same field of view. The specific implementation of the event imaging device and the video camera can adopt a variety of different schemes. For example, whether a set of optical modules are shared, whether a single sensor integrates DVS and frame image sensor, whether the frame image sensor uses differential frame to obtain the (target) pulse sequence, etc., the present invention is not limited to any specific form.

[0082] In some embodiments, the first video camera and the first event imaging device share the same set of optical modules (lenses, sensors, etc.). To achieve this purpose, a sensor called DAVIS (Dynamic and Active-Pixel Vision Sensor) can be used, which integrates traditional APS pixels and DVS pixels on the same sensor, so that both traditional images and pulse events can be obtained.

[0083] Similarly, in certain embodiments, the second video camera and the second event imaging device share the same optical module and use a DAVIS sensor. In the present invention, for the event imaging device and the video camera having (approximately) the same field of view, a DAVIS sensor is optionally used.

[0084] In certain embodiments, the first video camera and the first event imaging device, or / and the second video camera and the second event imaging device, share the same optical module and use a conventional frame image sensor. In other words, the event imaging device can be a (target) pulse sequence obtained by subtracting the different frame images output by the frame image sensor. For how to obtain the target pulse sequence, especially the randomized pulse sequence, reference can be made to Chinese Patent No. 202210387836.6, the entire contents of which are incorporated into this application by reference.

[0085] Furthermore, in order to fully reduce noise, if the number of all impulse events corresponding to a difference frame is less than a preset value, all impulse events corresponding to the difference frame are discarded.

[0086] Furthermore, the frame image sensor (for example, with a resolution of 1920×1080) is downsampled to obtain a frame image or difference frame with a resolution of 128×128 or 64×64, which can reduce the amount of subsequent processing calculations and reduce power consumption.

[0087] refer to Figure 3, which gives the framework diagram of the monitoring device of the present invention. First, the dynamic visual information is captured by the event imaging device, and the generated pulse sequence is sent to the Spiking Neural Network (SNN) processor (abbreviated as SNN processor) to obtain the inference result. The SNN processor here is also called brain-like chip or neuromorphic chip, such as SPECK TM Series chips or DYNAP-CNN TM The chip can achieve sub-milliwatt static power consumption and milliwatt average power consumption.

[0088] The pulse sequence here can be a pulse sequence directly generated and output by the DVS, or it can be a pulse sequence converted from the difference frames between the images output by the aforementioned frame image sensor (in this case, the frame image sensor and the software / hardware device that performs the difference frame and pulse sequence conversion can be regarded as part of the event imaging device).

[0089] The inference result here can be a specific classification result, such as any of the following: pedestrians (which can be further subdivided into whether they are holding an umbrella); pedestrians on bicycles or motorcycles; opening a car door; cars (which can be further subdivided into sedans, SUVs, trucks, tricycles, and cars entering a warehouse); weather conditions such as wind (shaking tree trunks, flying garbage, etc.), snow, and rain; and flashing lights. Pedestrians can also be subdivided into ordinary pedestrians or suspicious persons. In certain embodiments, if the SNN processor receives the pedestrian inference result repeatedly or continuously over a certain period of time, the monitoring logic control unit can determine that the person is a suspicious person.

[0090] The monitoring logic control unit issues an instruction to activate the recording camera based on the inference result. The monitoring logic control unit can determine whether to activate recording or time-lapse recording based on the inference result. For example, if the SNN processor receives the result of pedestrian inference multiple times over a certain period of time, it will generate recording control instructions / logic related to the suspicious person. In certain embodiments, recording is always triggered for suspicious persons.

[0091] In a certain type of embodiment, if the SNN processor inference result is a pedestrian or / and a vehicle (including bicycles, motorcycles, cars, etc.), an instruction to wake up the video camera to record is sent. Normally, the resolution (or resolution after downsampling) of the event imaging device is relatively low, such as 128×128. If the pedestrian or vehicle is far away from the user's vehicle, the imaging pixels of the pedestrian or vehicle are relatively small. Due to the blurred imaging, the SNN usually does not identify it as a pedestrian or a vehicle (it can even be concluded that pedestrians or vehicles at a long distance are not pedestrians or vehicles when training the SNN network). In this embodiment, although a complex distance judgment mechanism is not designed, based on the low-resolution imaging effect, the function of waking up the video only when a close target object approaches is objectively achieved.

[0092] Furthermore, the solution can also incorporate microwave sensors. Even if the SNN processor identifies a moving pedestrian or vehicle, if the microwave sensor does not detect the pedestrian or vehicle within a preset range, the recording function will not be activated. Preferably, the microwave sensor is activated and receives its output information only when the SNN processor's output requires further action.

[0093] In some embodiments, pedestrians passing by vehicles are common, especially in commercial spaces or environments. Frequently waking up the system based on pedestrians passing by will increase system power consumption. Therefore, in some embodiments, recording is only activated when a pedestrian is determined to be suspicious.

[0094] Preferably, the monitoring logic control unit can also make more reasonable control instructions based on other auxiliary information. For example, for the same inference result of the SNN processor, if other auxiliary information is different, different monitoring strategies may be affected.

[0095] In certain embodiments, during the day, if the SNN processor outputs an inference result for a pedestrian and the pedestrian is not identified as a suspicious person, the monitoring logic control unit outputs a control instruction not to record the video. However, at night, for example, after midnight, even if the pedestrian is an ordinary pedestrian and not identified as a suspicious person, the monitoring logic control unit will issue a control instruction to record the pedestrian based on the time information. This adaptive adjustment strategy helps reduce unnecessary recording while increasing the likelihood of capturing valuable footage.

[0096] In certain embodiments, if the GPS signal quality is good, the vehicle is more likely to be in an open-air parking lot. Therefore, if the SNN processor infers a pedestrian, the monitoring logic control unit can output a command not to record the vehicle. Conversely, if the GPS signal quality is poor, a control command to record the vehicle is issued, which is most likely in an underground parking lot. The former may apply to situations such as shopping malls, where frequent traffic easily triggers recording, so blocking this type of recording helps reduce system power consumption. The latter, on the other hand, applies to underground parking lots, where recording is typically not triggered frequently. Increasing the false alarm probability helps preserve valuable footage.

[0097] In certain embodiments, the vehicle's map coordinates can also be obtained based on GPS signals (networked or otherwise). If the area near these map coordinates is determined to be a location with a high concentration of pedestrians or vehicles, such as a shopping mall, street, or school, the monitoring logic control unit can apply a low-sensitivity monitoring strategy to reduce the probability of false alarms and unnecessary recording. In locations with a low concentration of moving objects, such as residential parking lots, a high-sensitivity monitoring strategy can be applied. The primary difference between high- and low-sensitivity monitoring strategies lies in whether or not they filter out very common pedestrians and / or vehicles passing by.

[0098] In certain embodiments, triggering a high-sensitivity or low-sensitivity monitoring strategy can also be determined based on the user's vehicle usage history. For example, if the user uses their vehicle regularly, frequently on weekdays but only occasionally or not on weekends, then based on this type of historical data, the high-sensitivity monitoring strategy can be applied, as frequent vehicle use often requires battery charging, reducing the risk of battery drain. On the other hand, if the user hasn't used the vehicle for an extended period, such as over a week, a low-sensitivity monitoring strategy can be applied, reducing the frequency of recording triggers and extending parking monitoring duration.

[0099] In certain embodiments, if the time since the last wake-up operation is long, such as more than a day, a high-sensitivity monitoring strategy can be applied. This long period of no wake-up triggering indicates that the parking location is unlikely to experience frequent wake-up events over a long period of time. Therefore, increasing monitoring sensitivity can help catch occasional unexpected situations. For example, if the car has not been started for more than a week, and historical wake-up data shows that it has not been woken up in two days, a high-sensitivity monitoring strategy can be applied.

[0100] In certain embodiments, the decision to trigger a high-sensitivity or low-sensitivity monitoring strategy is based on the number of wake-up times (which can be considered the wake-up frequency) of the dashcam within a set period of time. For example, if a user parks in a commercial parking lot and pedestrians or vehicles frequently trigger wake-ups, such as more than 10 wake-ups during recording in the past hour, the low-sensitivity monitoring strategy can be applied. If, for example, the dashcam has not triggered a wake-up more than once during the past hour, the high-sensitivity monitoring strategy can be applied.

[0101] In certain embodiments, the decision to issue a wake-up command and what other control commands to send can be made based on information from the vibration sensor. For example, if a vibration is detected, the monitoring logic control unit can, in addition to issuing a recording command, also issue a command to perform the following functions: storing the recordings from a certain period before the vibration, and from a certain period after the vibration, and preferably locking these recordings to prevent them from being overwritten by looped recordings. Optionally, the next time the user starts the vehicle, a prompt message (such as a voice message or a pop-up message requiring confirmation) can be issued to remind the user to handle potential accidents.

[0102] Furthermore, if a vibration is detected, video footage from a certain period before the vibration is stored. This can include compressed DVS video, preferably with a timestamp, or traditional frame-based video output from an event imaging device, preferably with a timestamp. This allows for the preservation of pre-collision visual information, which can serve as a crucial factual basis for identifying the responsible party.

[0103] In certain embodiments, the probability of triggering monitoring wake-up due to weather changes such as wind, snow, and rain is reduced. Strong winds often cause branches, tree trunks, garbage bags, etc. to move, and natural phenomena such as rain and snow can also trigger the event imaging device to output a large number of target pulse sequences. However, the movement of these objects is usually not the data to be recorded by the monitoring. Once these moving objects appear in the parking environment, they may trigger long-term monitoring wake-up, and these videos are usually meaningless. To this end, the SNN network can be trained to recognize these environments and not trigger wake-up when only the corresponding category is detected (for example, if a suspicious person is detected, wake-up will still be triggered).

[0104] Preferably, abnormal sound monitoring can be used to further ensure wake-up recording. For example, a dedicated intelligent voice chip can be used to identify abnormal sounds. For example, if the sound of window breaking is detected, the monitoring logic control unit receives auxiliary information including the sound detection result. In addition to ensuring wake-up monitoring recording, an alarm message can also be issued.

[0105] For example, the control instructions herein include one or more of the following: The control instructions issued by the monitoring logic control unit include one or more of the following: a normal recording instruction, a time-lapse recording instruction, a disable recording instruction, a delete instruction, and a lock instruction. Upon receiving a disable recording instruction or not sending the instruction, the entire system enters a low-power standby / sleep / hibernation state, or a partially operational state.

[0106] In addition to the aforementioned wake-up recording, the monitoring logic control unit can also send other instructions. For example, in the preset mode, the sound output unit is controlled to emit a warning sound (such as a simple click). If someone is found wandering outside the car late at night, the warning sound can dispel the suspicious person's bad intentions. After detecting the sound of broken windows, the sound output unit is controlled to emit a harsh warning sound, etc.; when it is detected that the traffic police has issued a ticket (the SNN network must be trained to learn this mode), the user is notified in time by sending a text message notification (or a dedicated APP notification for the driving recorder, a WeChat public account message, etc.).

[0107] For example, it is also possible to detect moving targets in the car, such as children, pets, etc., through a second event imaging device. Preferably, visual and sound information is processed by a chip with multimodal information processing capabilities (Chinese invention patent 202210277287.7). For example, when the presence of a child crying, an animal calling, etc. is detected, the user can be notified that the child or animal may be forgotten in the car, the door or window can be automatically unlocked, the user can shout / talk to the car, or the air conditioner can be turned on to prevent accidents.

[0108] It should be noted that the above embodiments can be combined and progressed to form an optimal embodiment. For example, after the user parked in the underground garage of the community at 8 pm on Friday, the GPS signal was weak, and a high-sensitivity monitoring strategy was applied; 8-9 pm: a total of 11 wake-ups were triggered; 9-10 pm: a low-sensitivity monitoring strategy was applied, and a total of 2 wake-ups were triggered, one of which also detected a vibration signal and recorded and locked the video before and after the vibration. 10-11 pm: a low-sensitivity monitoring strategy was applied, with a total of 0 wake-ups. 11-12 pm: since there was no more than 1 wake-up in the previous hour, a high-sensitivity monitoring strategy was applied, with a total of 0 wake-ups. 0-6 am: since it is a high-risk time period, a high-sensitivity monitoring strategy was applied, with a total of 1 wake-up. For more examples, please refer to Figure 4 More combinations can be applied to actual specific cases. The above examples are sufficient to inspire those skilled in the art to implement a monitoring strategy that is adaptively adjusted according to environmental changes. In the present invention, there is no limitation to a specific combination.

[0109] refer to Figure 5 , which discloses common parking monitoring scenarios. Pedestrians are the most common mobile targets in parking monitoring. They may be individuals who need to be monitored, or they may simply be passing by. In some scenarios, there may be few pedestrians, and monitoring them will not significantly increase system power consumption. However, in other scenarios (such as shopping mall ground parking lots and roadside parking), there are many pedestrians, and monitoring them is obviously a waste of energy. The aforementioned environment-adaptive monitoring strategy can effectively reduce system power consumption.

[0110] The key difference between a pedestrian passing by and a suspicious person is the length of time they stay, and / or whether they come into contact with a vehicle, and / or the direction of their travel. If a person is determined to be suspicious, the recording function should be activated.

[0111] For example, if the SNN identifies a pedestrian classification and the duration of this classification (a certain inference result) output reaches a threshold, the person is considered suspicious and the recording should be activated. This threshold can be determined based on the time it takes for an average pedestrian to pass through the field of view of the event imaging device. The threshold for determining a person as suspicious can be set to different values ​​for the first and second event imaging devices based on actual circumstances.

[0112] For example, based on the different human silhouettes projected onto the event imaging device, the direction of a pedestrian's movement can be determined. For example, a person with a front view can be determined to be walking towards the user's vehicle and be a suspicious person, while a person with a side view can be determined to be an ordinary pedestrian.

[0113] For example, if a pedestrian classification is detected (even if it is an ordinary pedestrian) and the vibration sensor also detects a vibration signal, then the pedestrian should generally be judged as a suspicious person, so the wake-up recording should be triggered in time.

[0114] Figure 6 This section illustrates various possible pedestrian situations surrounding a parking vehicle. This is particularly true for roadside parking, where various pedestrian and vehicle movements are present. For more distant pedestrians, such as pedestrian F, the first event imaging device has very few pixels, perhaps tens to one or two hundred. While motion can be detected, it's difficult to consistently classify them as valid categories (or as a default classification that incorporates various normal scenarios). Therefore, these pedestrians can be excluded as triggering arousal. Pedestrians C, D, and E are located on either side of the field of view, with smaller effective imaging pixels. Optionally, specific training can be performed on these types of samples, assuming they are normal pedestrians. In another embodiment, C through F can all be classified into the same category. Pedestrian B, effectively imaged by the event imaging device, is the most common pedestrian potentially causing arousal. This pedestrian's most distinctive feature is its profile. Therefore, this pedestrian can be considered a separate category, potentially activating the camera or not under different sensitivity monitoring strategies. Pedestrian A is walking towards the vehicle and is generally considered suspicious, thus falling into the arousable category. By inferring and classifying various pedestrians through SNNs, the greatest significance of this part is to distinguish between those who need to be captured and ordinary pedestrians who do not need to be captured, thereby reducing the number of unnecessary wake-ups of the dashcam and thus reducing power consumption. For monitoring equipment installed in other locations, the processing method is similar, that is, filtering out situations that do not require wake-up.

[0115] Continue to refer Figure 5, cars entering / exiting adjacent parking spaces (reversing into a garage or / and parallel parking) are the most likely scenarios for scratches. Preferably, for the SNN processor, the SNN network deployed thereon should learn the scenarios of cars entering / exiting the garage. If a car is identified as entering / exiting the garage, and this scenario does not usually occur frequently, preferably, video monitoring should be awakened regardless of whether a vibration signal is detected or what sensitivity monitoring strategy is applied. Optionally, when the SNN processor detects a car entering / exiting the garage, in addition to awakening the video, the video duration should be longer (such as 2 minutes) or continue until there is no non-noise output pulse sequence in the event imaging device. If a vibration signal is also detected, the video recording of a certain length before and after the vibration signal should be locked, and the user should be reminded by means such as sound and image when starting the next time.

[0116] Scratches caused by opening the door of an adjacent parking space, rear-end collisions or side scratches, or malicious scratches on the car are also pain points for users. Common driving recorders only have one first recording camera, but its field of view is limited. If you want to detect such as door scratches, you need at least one camera that can cover the field of view that the first recording camera cannot cover. This solution can be applied to more high-end driving recorders. Figure 2 An example of such a layout is provided in , but this is not the only example. The benefit of this embodiment is that it can have both the front field of view and the rear field of view in a single device and minimize blind spots. Figure 2 An alternative to the represented solution may be to use two sets of video cameras and event imaging devices (not shown), one set primarily for monitoring the left window and the other for monitoring the right window, which solution trades off hardware costs.

[0117] refer to Figure 7 , which shows a scenario with multiple video cameras in a certain embodiment. The video cameras and their associated event imaging devices (which may or may not share a set of optical modules, as mentioned above) can also be deployed at other locations on the vehicle body.

[0118] In the prior art, a car equipped with 360-degree panoramic imaging has multiple recording cameras deployed on its body, such as front, rear, left, and right recording cameras. For example, in certain embodiments, some or all of these recording cameras are equipped with corresponding event imaging devices.

[0119] For example, for the left / right rearview mirror equipped with a recording camera and an associated event imaging device, it is easy to detect the opening of the door of the adjacent parking space, scratches on the side, malicious scratches on the car, etc., and it also has a certain detection capability for rear-end collisions (the possibility of rear-end collisions that do not occur in the blind spot of the camera at the left / right rearview mirror is low). However, if it is equipped with a rear camera and an associated event imaging device, it can easily deal with rear-end collisions from directly behind.

[0120] For example, Figure 8 As shown in the figure, for the front and rear dual-camera driving monitoring solution, video cameras and associated event imaging devices are set up in front of the cockpit and at the rear of the car. Compared with the conventional single-field monitoring solution, it is more suitable for monitoring the rear of the car, especially rear-end collision accidents. A better solution should also include Figure 2 A second video camera and a second event imaging device are shown.

[0121] In the present invention, the above are merely examples of possible application positions of video cameras and associated event imaging devices. Any suitable position, angle and number can be put into practical application by those skilled in the art, and the present invention does not impose any restrictions on this.

[0122] Preferably, in some embodiments, if wake-up recording is detected, all recording cameras or some cameras (such as the direction in which the moving object appears) can be turned on at the same time.

[0123] More preferably, in certain embodiments, if a vibration signal is also detected, all recording cameras can be activated simultaneously to record more comprehensive and detailed on-site video data. Advantageously, due to the angle of some cameras, although they may be able to capture the vehicle or person involved in the accident, they may not be able to capture the license plate or face at a good angle. Therefore, if all recording cameras are activated, it is less likely to miss the best opportunity to capture important information such as the license plate number and face.

[0124] In certain embodiments, the field of view of the video camera and the event imaging device may be slightly different, for example, by adjusting the field of view range or angle. Figure 9 The field of view of the event imaging device is slightly lower than that of the video camera. This has the advantage of optically blocking pedestrians and vehicles at a distance, allowing only those closer to be captured by the event imaging device and identified by the SNN processor. Side scratches can also be handled by adjusting the angle of the event imaging device.

[0125] refer to Figure 10 The figure illustrates the parking monitoring process in a specific scenario. In an underground parking lot, the first-event imaging device of a user's vehicle (the gray block in the figure represents the front of the vehicle) captures an approaching pedestrian. Based on this dynamic information, the SNN processor infers the corresponding classification (for example, pedestrian or suspicious person). It then optionally checks the monitoring strategy and determines that the first recording camera needs to be awakened. It then captures a video of the pedestrian passing by and records the time in the video.

[0126] For example, the second event imaging device detects the movement of a vehicle on the left side of the user's vehicle, and optionally checks the monitoring strategy to determine that the second recording camera needs to be awakened, so it takes a video of the left vehicle leaving the parking space and records the time information in the video.

[0127] After the vehicle turns right, it enters the field of view of the first event imaging device / first video camera. The SNN processor corresponding to the first event imaging device should preferably learn the scene of the vehicle turning left or / and right when leaving the warehouse, and after the scene occurs, it should preferably immediately wake up the first video camera to record the dangerous scene. Of course, it is also possible that the first video camera may capture the vehicle turning right when leaving the warehouse during the subsequent continuous recording process (for example, at least 30 seconds of recording after waking up) after being woken up by the aforementioned pedestrian.

[0128] If a collision occurs while the vehicle is turning right while leaving the garage, and the vibration sensor inside the vehicle detects the collision, the video footage (which may include but is not limited to footage from the first and / or second video cameras) before and after the collision should be locked to alert the user of the detected anomaly the next time the vehicle is used. Optionally, a set prompt can be added to the video, such as "Vibration detected, intensity level: 2," to remind the user of the specific time and intensity of the collision while watching the video.

[0129] In certain embodiments, when an object moves in front of the event imaging device (determined based on, but not limited to, the inference results of the SNN processor, the number of pulse events output by the sensor, or the number of valid pixels in the difference frame), the video output by the event imaging device is recorded. Preferably, the video includes time information.

[0130] For example, a DVS can be converted into a video recording the outline of an object's motion by compressing frames. For event imaging devices based on traditional frame image sensors, both differential frame videos and frame image videos can be stored. Due to their low resolution, these videos contribute less to power consumption. This embodiment has the advantage of recording visual information before the camera wakes up, at a certain power cost. Although these videos have lower image resolution, they are temporally continuous or substantially continuous with the high-definition videos recorded after the camera wakes up, forming a complete video logic chain.

[0131] This embodiment is particularly suitable for low-sensitivity monitoring strategies (the present invention is not limited to this). If the pedestrian / vehicle pair is determined not to need to be awakened at the time, but a vibration signal is detected later (such as a scratch or malicious car scrape), the recording camera will be awakened. In this way, there is a low-resolution video recording (with a time stamp) before the accident, and a high-definition video recording at the time and after the accident. The two are temporally coherent, recording the events in the same space in different forms, forming an undeniable chain of evidence.

[0132] refer to Figure 11 , which shows a schematic diagram of a certain type of solution mentioned above. If the event imaging device detects the movement of an object, but does not perform a wake-up operation. For example, a low-sensitivity strategy is applied, and it is considered that it is just an ordinary pedestrian, so the video camera is not woken up. At this time, the data recorded by the event imaging device is stored in the form of a video, such as the aforementioned compressed frame video, difference frame video, and frame image video. If no vibration signal occurs, it is considered that the conclusion just made as an ordinary pedestrian is correct and no special processing is required. However, if a vibration is detected, the video camera is immediately woken up to record the on-site data. Preferably, the video recorded by the aforementioned event imaging device and the video recorded by the video camera are merged into the same video, and the video has a time stamp showing the corresponding on-site information and is locked. Since on-site data is also recorded before the vibration signal appears, and high-definition video is recorded after the vibration, the two are continuous in time, forming favorable evidence.

[0133] In certain embodiments, after determining the occurrence of a vibration signal, the resolution of the video recorded by the event imaging device is enlarged. For example, the original resolution of 64×64 can be enlarged to 640×640 for easier display. Furthermore, the enlarged resolution video is combined with the high-definition recorded video to form the final locked video.

[0134] In certain embodiments, reference Figure 12 The video recorded by the event imaging device can continue to record for a period of time after the video camera wakes up to maintain the continuity of its imaging. Preferably, when the videos recorded by the event imaging device and the video camera overlap in time, the video recorded by the former is embedded in the high-definition video recorded by the latter in a picture-in-picture manner (for example, the video output by the event imaging device is increased in resolution to 320×320).

[0135] In a certain type of embodiment, dynamic images are captured only by an event imaging device. If a large change in the image is detected, such as the number of effective changed pixels (preferably, counted based on coordinates to shield the influence of DVS hot pixels) exceeds a preset value, the subsequent image or pulse event sequence is written into a memory (such as a cache). If a vibration signal is also detected, the data in the memory is solidified into a non-volatile memory such as a flash memory, and the video camera is awakened to record. Similar to the aforementioned embodiment, the data recorded by the event imaging device and the data recorded by the video camera are used as the final output target video (which can be a single video or multiple single videos). This type of embodiment can greatly reduce the number of times the video camera is awakened and minimize power consumption, but the disadvantage is that it lacks intelligence and there will definitely be no high-definition video before the accident.

[0136] In certain embodiments, the aforementioned parking monitoring mechanism may also trigger the recording of the user's own video before and after normal use of the vehicle, but not all users wish to do so (users may be asked whether they wish to save their own recorded videos during initial installation). To protect user privacy, if the user does not consent to recording themselves or the legal user of the vehicle, after the vehicle is started / unlocked, the dashcam will check to see if any video has been recorded within a set time (e.g., 1 minute). If so, the newly recorded unlocked video will be automatically deleted. If the video is a locked video, the user will be asked whether to delete it (in case the user arrives and unlocks the vehicle within 1 minute after an accident occurs, automatically deleting the video that should be retained).

[0137] In certain embodiments, if vehicle locking information is detected, a search is performed to determine whether any locking video was recorded between the start of parking monitoring and the locking of the vehicle. If so, the locking video is automatically deleted. Preferably, the aforementioned locking videos are automatically deleted if their total duration (a single video or multiple independent consecutive videos) exceeds a set duration (for example, greater than or equal to 2 minutes). Furthermore, the automatically deleted locking videos are videos recorded by a second recording camera that may violate the user's privacy.

[0138] Figure 13 A schematic diagram compares a solution that uses only an event imaging device and one that also incorporates an SNN processor. In the former, the event imaging device captures object movement and immediately triggers recording once a sufficient number of events are detected. This solution has the advantages of high sensitivity and is less likely to miss events. However, its disadvantages include frequent wake-up calls for recording, high power consumption, and a lack of intelligence.

[0139] In the latter embodiment, an event imaging device is used to capture the movement of objects. After detecting whether the change in visual information exceeds a preset value (enough events occur), the SNN processor is handed over to perform classification. In order to ensure the recording of valid events, the SNN processor is configured to identify situations where wake-up is clearly not required. After obtaining a certain type of video that does not require wake-up, the video is not woken up; while other situations wake up the video. This type of embodiment takes into account the limited actual classification capabilities of the SNN processor and the complexity of the real scene. The clever combination of triggering the number of events and reverse exclusion not only ensures the recording of valid events, but also effectively reduces the wake-up frequency / system power consumption. The situations where wake-up is clearly not required here include but are not limited to: ordinary passers-by (including cyclists), ordinary passing vehicles, rain, snow, hail and other weather phenomena, and light strobing. The other technical features of this type of embodiment are the same as those of the previous embodiment. They are incorporated into this embodiment by reference and will not be repeated here.

[0140] The changes in visual information captured here can preferably be determined by the following: the number of pulse events with different coordinates received within a set time period exceeds a set value, or the sum of the pixel values ​​of the difference frame images exceeds a set value. The pulse events and pixel values ​​here can also be the result of noise filtering. Achieving this technical goal is readily apparent to those skilled in the art, and the present invention is not limited to a specific method for characterizing changes in visual information.

[0141] Although the present invention has been described with reference to specific features and embodiments thereof, various modifications, combinations, and substitutions may be made thereto without departing from the present invention. The scope of protection of the present invention is not intended to be limited to the specific embodiments of the processes, machines, manufactures, compositions of matter, devices, methods, and steps described in the specification, and such methods and modules may also be implemented in one or more related, interdependent, cooperative, or preceding or subsequent products and methods.

[0142] Therefore, the specification and drawings should be simply regarded as an introduction to some embodiments of the technical solution defined by the appended claims. The appended claims should be interpreted according to the principle of maximum reasonable interpretation, and are intended to cover as much as possible all modifications, changes, combinations or equivalents within the scope of the present invention, while avoiding unreasonable interpretations.

[0143] To achieve better technical effects or meet the needs of certain applications, those skilled in the art may make further improvements to the technical solution based on the present invention. However, even if such improvements / designs are creative and / or progressive, as long as they rely on the technical concept of the present invention and cover the technical features defined in the claims, such technical solutions should also fall within the scope of protection of the present invention.

[0144] Some of the technical features mentioned in the attached claims may have alternative technical features, or the order of certain technical processes or the order of material organization may be reorganized. After becoming aware of the present invention, a person of ordinary skill in the art will easily conceive of such alternative means, or change the order of the technical processes or the order of material organization, and then adopt substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the claims explicitly define the aforementioned means and / or order, such modifications, changes, and substitutions should fall within the scope of protection of the claims in accordance with the doctrine of equivalents.

[0145] In conjunction with the various method steps or modules described in the embodiments disclosed herein, they can be implemented in hardware, software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application or design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered outside the scope of protection claimed by the present invention.

Claims

1. A monitoring device, wherein the monitoring device is a parking monitoring device, characterized in that: The monitoring device comprises at least a first video camera and a first event imaging device, a spiking neural network processor and a monitoring logic control unit, wherein: The first event imaging device is used to detect moving objects in the field of view and output a pulse sequence; The pulse neural network processor is used to perform inference on the pulse sequence output by the first event imaging device to obtain an inference result; According to the inference result, the monitoring logic control unit determines whether to wake up the first recording camera; When an object moves in front of the first event imaging device, recording the pulse event output by the first event imaging device or the processed video; If the monitoring device detects a vibration signal indicating that the vehicle body is vibrating, the monitoring logic control unit issues a locking instruction, which indicates that the videos recorded before and after the vibration signal cannot be automatically overwritten by the monitoring device.

2. The monitoring device according to claim 1, wherein: The first event imaging device is a pulse sequence obtained based on any one of the following methods: (a) DVS; (b) DAVIS; (c) a pulse sequence obtained based on a frame image sensor difference frame.

3. The monitoring device according to claim 1, wherein: The monitoring logic control unit also receives auxiliary information and uses it to decide whether to wake up the first video camera.

4. The monitoring device according to claim 3, wherein: The auxiliary information includes one or more of the following: GPS signal quality information or location information obtained based on GPS, microwave sensor output information, user's vehicle usage history data, the time since the last time the first recording camera was awakened, the number of times the first recording camera was awakened within the set time, vibration sensor information, sound detection information, and current time information.

5. The monitoring device according to claim 1, wherein: The control instructions issued by the monitoring logic control unit include one or more of the following: normal recording instructions, time-lapse recording instructions, no recording instructions, deletion instructions, and lock instructions; if the monitoring logic control unit issues a no recording instruction or does not send an instruction, the operation of waking up the first recording camera will not be executed.

6. The monitoring device according to claim 1, characterized in that: The instructions issued by the monitoring logic control unit also include one or more of the following: warning sound broadcast instructions, user notification instructions, door unlocking instructions, window opening instructions, user and vehicle call establishment instructions, and air conditioning turn-on instructions.

7. The monitoring device according to claim 1, wherein: The monitoring device has at least a high-sensitivity monitoring strategy and a low-sensitivity monitoring strategy that can be applied.

8. The monitoring device according to claim 7, characterized in that: The high-sensitivity monitoring strategy differs from the low-sensitivity monitoring strategy at least in terms of whether the wake-up can be triggered for a person or / and a vehicle within the field of view.

9. The monitoring device according to claim 7, characterized in that: The monitoring logic control unit also receives auxiliary information and determines which sensitivity monitoring strategy to apply based on the auxiliary information.

10. The monitoring device according to claim 7, characterized in that: The monitoring logic control unit determines whether to wake up the first video camera according to the determined sensitivity monitoring strategy.

11. The monitoring device according to claim 1, characterized in that: Whether a pedestrian in the field of view can trigger wake-up is determined based on one or more of the following factors: the length of stay, whether there is contact with the vehicle body, and the direction of the pedestrian's travel.

12. The monitoring device according to claim 1, wherein: The pulse neural network processor and / or monitoring logic control unit does not output the operation that triggers the awakening for ordinary pedestrians and / or ordinary vehicles.

13. The monitoring device according to claim 1, wherein: The first video camera and the first event imaging device have different fields of view.

14. The monitoring device according to claim 1, wherein: The monitoring device also includes at least a second recording camera and a second event imaging device, wherein the second event imaging device is used to provide a basis for whether to wake up the second recording camera; and the field of view of the second recording camera and the second event imaging device is different from the field of view of the first recording camera and the first event imaging device.

15. The monitoring device according to claim 1, wherein: The pulse neural network processor is configured to recognize visual information patterns of weather environment and / or light strobing in the field of view; or / and, the pulse neural network processor is configured to recognize visual information patterns of vehicles entering / exiting adjacent parking spaces in the field of view; or / and, the pulse neural network processor is configured to recognize visual information patterns of vehicles opening doors in adjacent parking spaces in the field of view.

16. The monitoring device according to claim 1, characterized in that: The video cameras and associated event imaging devices of the monitoring device are installed in one or more of the following locations: the left rearview mirror, the right rearview mirror, the rear of the car, the front of the cockpit, and the front of the vehicle body; for the solution of installing video cameras and associated event imaging devices in multiple locations, if a wake-up video signal is detected, all video cameras will be turned on at the same time.

17. The monitoring device according to claim 1, characterized in that: After the vibration signal is detected, the pulse event or video output by the first event imaging device and the video recorded by the first video camera are merged into one video, and the merged video displays the time stamp of the corresponding scene information.

18. The monitoring device according to claim 17, characterized in that: The resolution size of the video output by the first event imaging device or the video generated via impulse event conversion is enlarged.

19. The monitoring device according to claim 17 or 18, characterized in that: When the videos recorded by the first event imaging device and the first recording camera overlap in time, the video output by the first event imaging device or the video generated by pulse event conversion is embedded in the video recorded by the first recording camera in a picture-in-picture manner.

20. The monitoring device according to claim 13, wherein: The field of view of the first event imaging device is lower than the field of view of the first video camera.

21. A monitoring method, applied to a parking monitoring device, characterized in that: The monitoring device comprises at least a first video camera and a first event imaging device, a spiking neural network processor and a monitoring logic control unit, wherein: The first event imaging device is used to detect moving objects in the field of view and output a pulse sequence; The pulse neural network processor is used to perform inference on the pulse sequence output by the first event imaging device to obtain an inference result; According to the inference result, the monitoring logic control unit determines whether to wake up the first recording camera; When an object moves in front of the first event imaging device, recording the pulse event output by the first event imaging device or the processed video; If the monitoring device detects a vibration signal indicating that the vehicle body is vibrating, the monitoring logic control unit issues a locking instruction, which indicates that the videos recorded before and after the vibration signal cannot be automatically overwritten by the monitoring device.

22. The monitoring method according to claim 21, characterized in that: In different scenarios, there are at least high-sensitivity monitoring strategies and low-sensitivity monitoring strategies to choose from; The high-sensitivity monitoring strategy differs from the low-sensitivity monitoring strategy at least in whether the wake-up can be triggered for a person or / and a car within the field of view.

23. A driving recorder, characterized in that: The driving recorder includes the monitoring device described in any one of claims 1-20, or applies the monitoring method described in any one of claims 21-22.

Images