Security detection method, device and storage medium

Abstract

The application provides a security detection method and device and a storage medium. The method comprises the following steps: determining that an event camera detects an event occurring in a target place; calculating the activity of the event in pixel coordinates according to a timestamp and pixel coordinates, and filtering events with an activity less than an activity threshold; calculating the confidence of the event belonging to a motion edge according to the activity and polarity information, and filtering events with a confidence less than a confidence threshold; calculating the aggregation degree of the event on a motion trajectory according to the confidence at multiple space-time scales; calculating the likelihood of the event belonging to a foreground according to the aggregation degree, and filtering events with a likelihood less than a likelihood threshold; calculating the activation value of the event on motion consistency according to the likelihood; if the activation value is greater than an activation threshold, waking up a main camera to collect main image data of the target place, projecting the event into event image data; and detecting security information of the target place according to the main image data and the event image data. The power consumption of the overall security monitoring is effectively reduced.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision, and in particular relates to a security detection method, device and storage medium. Background Technology

[0002] RGB (red, green, blue) cameras, CMOS (complementary metal-oxide-semiconductor) cameras, CCD (charge-coupled device) cameras and other cameras can provide rich visual information. By combining them with various types of neural networks according to business needs, a variety of security monitoring functions can be realized, such as personnel behavior analysis, fire monitoring, and icing detection.

[0003] Because cameras consume a lot of resources, resulting in high power consumption, these cameras are usually equipped with low-power modes. One type of low-power mode is the timed sleep mode, in which the camera is in deep sleep most of the time. The built-in timer wakes the camera up periodically (e.g., every 5 seconds, 10 seconds) to collect a frame of image data and detect whether there are people or moving objects. If so, it continues to record for security checks. If not, it immediately enters deep sleep. Another type of low-power mode is the low frame rate guard mode, in which the camera does not sleep and runs at a low frame rate (e.g., 0.5 FPS, 1 FPS, etc.) with low power consumption (partially shutting down the ISP (Image Preferred Processor)). When the scene changes, it switches to a high frame rate (e.g., 25 FPS, 30 FPS, etc.) for high-definition recording for security checks.

[0004] Whether in timed sleep mode or low frame rate monitoring mode, the camera runs for a long time, and its power consumption is still high in business scenarios such as battery power supply and 24 / 7 monitoring. Summary of the Invention

[0005] In view of this, the present invention provides a security detection method, device and storage medium to reduce the power consumption of cameras for security monitoring.

[0006] A first aspect of the present invention provides a security detection method, comprising: The event camera detects an event occurring at the target location; the event has pixel coordinates, a timestamp, and polarity information. The activity level of the event at the pixel coordinates is calculated based on the timestamp and the pixel coordinates, and events with an activity level less than the activity threshold are filtered out. Based on the activity level and the polarity information, calculate the confidence level that the event belongs to the motion edge, and filter out the events whose confidence level is less than the confidence threshold; The aggregation degree of the event on the motion trajectory is calculated based on the confidence level at multiple spatiotemporal scales. Calculate the likelihood that the event belongs to the foreground based on the aggregation degree, and filter out events whose likelihood is less than the likelihood threshold; Calculate the activation value of the event in terms of motion consistency based on the likelihood. If the average value of the activation values ​​is greater than the activation threshold, the main camera is activated to collect main image data of the target location, and the event is projected into the event image data. Security information of the target location is detected based on the main image data and the event image data.

[0007] A second aspect of the present invention provides a security detection device, comprising: The event determination module is used to determine events detected by the event camera that occur in the target location; the event has pixel coordinates, timestamps, and polarity information; The point processing module is used to calculate the activity level of the event in the pixel coordinates based on the timestamp and the pixel coordinates, and filter the events whose activity level is less than the activity threshold. The line processing module is used to calculate the confidence level of the event belonging to the motion edge based on the activity level and the polarity information, and filter the events whose confidence level is less than the confidence threshold. The surface processing module is used to calculate the aggregation degree of the event on the motion trajectory based on the confidence level at multiple spatiotemporal scales; The foreground processing module is used to calculate the likelihood that the event belongs to the foreground based on the aggregation degree, and filter the events whose likelihood is less than the likelihood threshold; An activation value calculation module is used to calculate the activation value of the event in terms of motion consistency based on the likelihood. The wake-up module is used to wake up the main camera to collect main image data of the target location if the average value of the activation value is greater than the activation threshold, and to project the event into the event image data. The security information detection module is used to detect security information of the target location based on the main image data and the event image data.

[0008] A third aspect of the present invention provides a camera device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the security detection method as described in the first aspect above.

[0009] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the security detection method as described in the first aspect above.

[0010] A fifth aspect of the present invention provides a computer program product that, when run on a computer, causes the computer to perform the security detection method as described in the first aspect above.

[0011] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment, the event camera detects an event occurring in the target location; the event has pixel coordinates, a timestamp, and polarity information; the activity level of the event in the pixel coordinates is calculated based on the timestamp and pixel coordinates, and events with activity levels less than an activity threshold are filtered out; the confidence level of the event belonging to the motion edge is calculated based on the activity level and polarity information, and events with confidence levels less than a confidence threshold are filtered out; the aggregation degree of the event on the motion trajectory is calculated based on the confidence level at multiple spatiotemporal scales; the likelihood of the event belonging to the foreground is calculated based on the aggregation degree, and events with likelihood levels less than a likelihood threshold are filtered out; the activation value of the event in motion consistency is calculated based on the likelihood; if the activation value is greater than the activation threshold, the main camera is activated to acquire main image data of the target location, and the event is projected into the event image data; the security information of the target location is detected based on the main image data and the event image data. This embodiment uses an event camera as a secondary device to detect external events, thereby waking up the main camera for security detection. This keeps the main camera in a dormant state for extended periods, effectively reducing the overall power consumption of security monitoring. Furthermore, it employs event-driven layer-by-layer filtering at the point, line, surface, foreground, and optical flow levels. The filtering mechanism progresses from coarse to fine, filtering noise and reducing the number of events layer by layer, thereby reducing computational load. While ensuring the accuracy of waking up the main camera, it minimizes the resource consumption of running the event camera, making it suitable for application in resource-constrained devices. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a security detection method provided in an embodiment of the present invention; Figure 2 This is a diagram illustrating the operation mechanism of a main camera and an event camera according to an embodiment of the present invention. Figure 3 This is a schematic diagram of a hierarchical filtering event provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a security detection network provided in an embodiment of the present invention; Figure 5This is a schematic diagram of a security detection device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a camera device provided in an embodiment of the present invention. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the present invention. However, those skilled in the art will recognize that the present application may be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail that could obscure the description of the present application.

[0015] The technical solution of the present invention will be illustrated below through specific embodiments.

[0016] Reference Figure 1 The diagram illustrates a security detection method provided by an embodiment of the present invention, which may specifically include the following steps: Step 101: Determine if the camera has detected an event occurring at the target location.

[0017] In this embodiment, security monitoring equipment (such as surveillance cameras, robots, etc.) can be deployed at target locations (such as residential buildings, factories, roads, etc.). Figure 2 As shown, the camera device is equipped with two cameras, one of which is the main camera, such as an RGB camera, CMOS camera, CCD camera, etc. The specific type of the main camera can be selected according to the requirements of the monitoring business. The other camera is the auxiliary camera, which is an event camera.

[0018] The principle of an event camera is that after the cumulative change in brightness of a pixel reaches a certain trigger threshold, it outputs an event. The event has pixel coordinates, timestamp, and polarity information. The polarity information includes On information indicating that the increase in brightness exceeds the trigger threshold and Off information indicating that the decrease in brightness exceeds the trigger threshold.

[0019] Event cameras output changing data with very little bandwidth. At the same time, because event cameras are better at capturing changes in brightness, they can output effective data in both dark and bright scenes, giving them characteristics such as low latency, high dynamic range, and extremely low power consumption.

[0020] In this embodiment, the main camera is in a sleep state, while the event camera is in a working state. When the event camera is in a working state, it can generate a corresponding event if it detects a change in brightness in the target area.

[0021] Within a certain time frame, the event camera can generate a tuple stream E of asynchronous discrete events. raw ={e i |e i =(x i ,y i ,t i ,p i )}, where E raw For a tuple stream of events, e i For the i-th event, (x i ,y i ) represents pixel coordinates, t i For timestamps, p i For polarity information, p i =+1 indicates enabling information, p i =-1 indicates that the information is off.

[0022] Step 102: Calculate the activity level of the event in the pixel coordinates based on the timestamp and pixel coordinates, and filter out events with an activity level less than the activity threshold.

[0023] Events generated by real motion exhibit spatiotemporal causal relationships (the edge of a moving object sweeping across a pixel triggers subsequent events in its neighborhood). That is, when the edge of a moving object sweeps across the pixel array of the event camera, the event generation is not random and independent, but rather has a clear spatiotemporal causality—the event of the previous pixel will excite subsequent events in adjacent pixels. Noise events, on the other hand, are spatiotemporally isolated points with no causal relationship. Therefore, as... Figure 3 As shown, at the point level, the spatiotemporal causality can be reflected by the timestamp of the event and the pixel coordinates, thereby calculating the activity of the event in the pixel coordinates.

[0024] Real-world events are continuously stimulated by historical events in the neighborhood, and their activity level accumulates to a high level; isolated noise events, on the other hand, have no stimulating effect from historical events in the neighborhood, and their activity level remains at a low level.

[0025] Therefore, events with activity levels below the activity threshold can be filtered out, while events with activity levels greater than or equal to the activity threshold can be retained, isolated noise can be suppressed, and discrete pulses from the camera can be transformed into coded events with continuous activity characteristics.

[0026] In the specific implementation, an activity level A can be configured for each pixel coordinate. When the event camera is initialized, the activity level is set to 0, that is, A(x,y,0)=0, which means that the activity level of the pixel coordinate (x,y) is 0 at the initial time (i.e., time 0).

[0027] The events are traversed in a set order (such as the order of timestamps). When the current event is reached, the first neighborhood of the current event is determined. The first neighborhood includes other events that are adjacent to the current event in time and space. That is, the absolute value of the difference between the timestamp of other events and the timestamp of the current event is less than or equal to a certain threshold, and the distance between the pixel coordinates of other events and the pixel coordinates of the current event is less than or equal to a certain threshold.

[0028] On the one hand, within the first neighborhood, the time decay amount is determined; wherein, the time decay amount is based on a natural number and exponent is the negative of the ratio between the event interval duration and the time decay coefficient; the event interval duration is the difference between the timestamp of the current event and the timestamp of the previous event in the first neighborhood.

[0029] The time decay can simulate the physical characteristic that the influence of a moving object gradually disappears with the event. After a moving object leaves a certain pixel, the motion activity at that position will gradually return to zero, which matches the motion continuity of the moving object. At the same time, the time dimension filters out isolated noise events.

[0030] When there are no events, activity level will naturally decay to 0 over time. When a new event arrives, activation energy is replenished and activity level is updated.

[0031] On the other hand, within the first neighborhood, the spatial diffusion is determined; wherein the spatial diffusion is based on a natural number and has an exponent that is the negative of the ratio between the pixel distance and the square of the standard deviation of spatial diffusion; the pixel distance is the average distance between the pixel coordinates of the current event and the pixel coordinates of other events in the first neighborhood.

[0032] Real-world motion events generate excitation effects within the spatiotemporal neighborhood. Therefore, when a new event arrives, it not only updates the activity level of its own pixels but also diffuses activation energy into the surrounding neighborhood through space, forming a continuous motion activation heatmap. This strengthens the spatiotemporal correlation of real motion and further distinguishes effective events from noise.

[0033] The product of the previous activity level at the pixel coordinate and the time decay, and the product of the activation gain and the spatial diffusion, are added together to obtain the activity level of the current event at the pixel coordinate.

[0034] Therefore, the activity level of the current event in pixel coordinates can be represented as: ; Where, A(x) i ,y i ,t i To generate the current time t i When the event occurs, the pixel coordinates (x) i ,yi The activity of A(x) i ,y i ,t last To generate the previous time t last When the event occurs, the pixel coordinates (x) i ,y i The activity level of ) is given by r, the time decay coefficient (e.g., 10ms~100ms), a, the activation gain, σ, and the spatial diffusion standard deviation (e.g., 1pixel~5pixel). ij Let t be the current time. i The pixel coordinates (x) of the event i ,y i ) and the previous time t last The pixel coordinates (x) of the event j ,y j The distance between ) and D ij =(x i -x j ) 2 +(y i -y j ) 2 , j∈n, where n is the number of events in the first neighborhood.

[0035] Weights such as time decay coefficient and spatial diffusion standard deviation can be pre-stored in local registers. When an event arrives, the table can be looked up directly for calculation, without relying on real-time floating-point calculation. During hardware implementation, local registers are accessed, without relying on global memory read and write, resulting in low power consumption.

[0036] In this embodiment, the native asynchronous event stream of the event camera is used as input. The spatiotemporal causal relationship of real motion is modeled through a spatiotemporal self-excited kernel. Each event is assigned a continuous activation weight to achieve soft filtering of noise and continuous feature encoding of discrete events, while maintaining the low power consumption characteristics of event-driven operation throughout.

[0037] Step 103: Calculate the confidence score of an event belonging to the edge of motion based on activity and polarity information, and filter out events with confidence scores less than the confidence threshold.

[0038] Real rigid body motion produces paired, spatially ordered polar edges. For example, when a moving object moves from left to right, the pixel brightness increases when the forward edge (right side) sweeps across a pixel, triggering the On polarity, and the pixel brightness decreases when the backward edge (left side) sweeps across a pixel, triggering the Off polarity. However, global light abrupt changes are almost all of the same polarity (single On or single Off), without paired On and Off. Random thermal noise, dynamic background jitter, etc., are randomly distributed polarities without spatial order.

[0039] Therefore, as Figure 3As shown, at the online level, the confidence level of an event belonging to the edge of motion can be calculated based on activity and polarity information. Events with confidence levels less than the confidence threshold are filtered out, while events with confidence levels greater than or equal to the confidence threshold are retained. Polarity purification of events is then performed to further filter out invalid events.

[0040] In the specific implementation, the remaining events after filtering in step 102 are traversed in a set order (such as the order of timestamps). When the current event is reached, the second neighborhood of the current event is determined. The second neighborhood includes other events that are adjacent to the current event in time and space (i.e., the absolute value of the difference between the timestamp of other events and the timestamp of the current event is less than or equal to a certain threshold, and the distance between the pixel coordinates of other events and the pixel coordinates of the current event is less than or equal to a certain threshold). In addition, the activity of other events is greater than or equal to a specified proportion of the activity of the current event (such as 0.6).

[0041] Construct a polarity co-occurrence matrix for the second neighborhood; wherein the elements in the polarity co-occurrence matrix include the number of open information pairs based on open information, the number of closed information pairs based on open information, the number of open information pairs based on closed information, and the number of closed information pairs based on closed information.

[0042] The so-called pairing refers to taking an event in the second neighborhood as a reference and forming event pairs with all other events in the second neighborhood except the reference one by one.

[0043] Therefore, the polar co-occurrence matrix can be expressed as: ; Where C is the polar co-occurrence matrix, N On-On The number of open information pairs is based on the open information. This value tends to be larger when the global light intensity increases, but its proportion is low during actual motion. N On-Off Based on the number of open and closed information pairs, real motion edges will generate a large number of such event information pairs, N Off-On To determine the number of open and closed information pairs based on closed information, this value is related to N during actual motion. On-Off The difference between the two is large when they are close together, or when there are changes in noise or light. Off-Off The value of this parameter is based on the number of parameters paired with the parameter information. It tends to be larger when the global light intensity decreases, and it has a lower proportion during actual motion.

[0044] The information entropy of the polarity co-occurrence matrix is ​​calculated. Information entropy is an indicator that quantifies the degree of disorder in the distribution. The more ordered the polarity information distribution (real motion), the lower the information entropy. The more disordered the polarity information distribution (such as noise, dynamic background, etc.), the higher the information entropy.

[0045] The degree of order is obtained by calculating the difference between the information entropy of 1 and the polar co-occurrence matrix.

[0046] Calculate the spatial displacement vector between the first centroid corresponding to all open information and the second centroid corresponding to all closed information within the second neighborhood; wherein, for all events corresponding to open information, the first centroid is the sum of the products between pixel coordinates and activity level divided by the sum of activity levels, and for all events corresponding to closed information, the second centroid is the sum of the products between pixel coordinates and activity level divided by the sum of activity levels.

[0047] Then, the first mass (x) On ,y On ) can be represented as x On =sum j=1~m (A j x j ) / sum j=1~m (A j ), y On =sum j=1~m (A j y j ) / sum j=1~m (A j ), where (x j ,y j ) represents the pixel coordinates of the event to which the open information belongs, j∈m, m is the number of events to which the open information belongs, and sum is the summation function.

[0048] Accordingly, the second mass center (x Off ,y Off ) can be represented as x Off =sum j=1~n (A j x j ) / sum j=1~n (A j ), y Off =sum j=1~n (A j y j ) / sum j=1~n (A j ), where (x j ,y j ) represents the pixel coordinates of the event to which the relevant information belongs, j∈n, and n is the number of events to which the relevant information belongs.

[0049] Therefore, the spatial displacement vector d can be expressed as d=(x On -x Off , y On -y Off ).

[0050] The events are traversed in a set order (such as the order of timestamps). When the current event is reached, the product of activity, orderliness and the magnitude of the spatial displacement vector is calculated to obtain the confidence that the current event belongs to the motion edge.

[0051] Therefore, the confidence that the current event belongs to the edge of motion can be represented as M. i =A i ×(1-H norm (C))×|d|, where M i Let A be the confidence score for the i-th event to belong to the motion edge. i H represents the activity level of the i-th event. norm (C) is the information entropy of the polar co-occurrence matrix C, and |d| is the magnitude of the spatial displacement vector d.

[0052] In this embodiment, global light mutations, random thermal noise, dynamic background jitter, etc. can be filtered out. Real motion edges and false events are distinguished from each other at the semantic level. Furthermore, the operation is performed in the local neighborhood without global frame processing. The hardware implementation mainly uses registers, without external memory access, resulting in low power consumption.

[0053] Step 104: Calculate the aggregation degree of the event on the motion trajectory based on the confidence level at multiple spatiotemporal scales.

[0054] Moving objects can be categorized into fast-moving small targets, conventionally moving targets, and slow-moving large targets, such as flying insects, rustling leaves, walking people, and instantaneous flashes of light. Real rigid body motion exhibits cross-scale continuity in the spatiotemporal domain, while disturbances possess single-scale discreteness. In this embodiment, for example... Figure 3 As shown, at the surface level, the aggregation degree of the remaining events after filtering in step 103 on the motion trajectory can be calculated based on confidence at multiple spatiotemporal scales. For example, a walking person will generate dense edge events (clothing texture, body outline) at a fine scale, aggregate into a complete human motion area at a medium scale, and aggregate into a continuous motion trajectory at a coarse scale. Interference such as flying insects and leaf shaking will generate discrete activations at a single scale, and will accumulate low weights in the progressive aggregation and will be filtered layer by layer.

[0055] In a specific implementation, multiple cascaded aggregation layers can be determined. Cascaded means that the output of the previous aggregation layer is the input of the next aggregation layer, and the input of the top-ranked aggregation layer is the remaining events filtered in step 103.

[0056] Each aggregation layer is configured with a spatial window, a temporal window, and an aggregation threshold. The scale of the spatial window and the scale of the temporal window are positively correlated with the depth of the aggregation layer. That is, the shallower the aggregation layer, the smaller the scale of the spatial window and the smaller the scale of the temporal window; the deeper the aggregation layer, the larger the scale of the spatial window and the larger the scale of the temporal window. Thus, from top to bottom, the scale of the spatial window and the scale of the temporal window continuously increase, achieving a filtering process from fine to coarse.

[0057] Traverse each aggregation layer in sequence. When traversing to the current aggregation layer, add a spatial window and a time window to the current event in the current aggregation layer. For the current event, determine other events whose pixel coordinates are located in the spatial window and whose timestamps are located in the time window (i.e., the absolute value of the difference between the timestamps of other events and the timestamps of the current event is less than or equal to the scale of the spatial window, and the distance between the pixel coordinates of other events and the pixel coordinates of the current event is less than or equal to the scale of the spatial window), and regard them as the third neighborhood.

[0058] In the third neighborhood, the confidence scores of other events are merged into the confidence score of the current event through linear (such as weighted summation) or nonlinear methods.

[0059] In the current aggregation layer, events with confidence levels less than the aggregation threshold are filtered out, while events with confidence levels greater than or equal to the aggregation threshold are retained.

[0060] If the current aggregation layer is not the last in the sorting, the remaining events will be filtered and output to the next aggregation layer. The process will iterate from the current aggregation layer to the next aggregation layer until the next aggregation layer becomes the current aggregation layer. Then, the process will return to the current aggregation layer and execute the steps for determining other events in the current aggregation layer whose pixel coordinates are located in the spatial window and whose timestamps are located in the time window, as the third neighborhood.

[0061] If the current aggregation layer is at the bottom of the sort, the confidence of the event in each aggregation layer is merged into the aggregation degree of the event on the motion trajectory through a linear (such as weighted summation) or non-linear method.

[0062] Taking three aggregation layers as an example, the spatial window size of the first-ranked aggregation layer is 3×3 and the time window size is 50ms; the spatial window size of the second-ranked aggregation layer is 15×15 and the time window size is 200ms; and the spatial window size of the third-ranked aggregation layer is 50×50 and the time window size is 1000ms.

[0063] The first-ranked aggregation layer captures fast-moving, small-sized objects (such as intruders at a distance or small objects moving at high speeds), retains fine motion edge features based on confidence level S1, and filters out isolated noise from single pixels.

[0064] The second-ranked aggregation layer captures moving objects of normal size and speed (such as people walking in the park or non-motorized vehicles traveling on the road), aggregates the fine edge features output by the previous aggregation layer into a complete motion region, enhances the global features of real intrusion targets, and filters out transient interference that is only activated at the fine scale (such as flying insects) based on the confidence S2.

[0065] The third-ranked aggregation layer captures slow-moving, large-sized moving objects (such as slowly moving camouflaged intruders or large-scale wall climbing behavior), aggregates the motion region features output by the previous aggregation layer into a complete motion trajectory, and filters out transient interference that only occurs at the mesoscale (such as leaf swaying or instantaneous flashes of light) based on the confidence level S3.

[0066] At this point, the aggregation degree S = S1 + S2 + S3 preserves the full-scale features of weakly moving objects. For example, for slow-moving intrusion targets under low light at night, the confidence of the three aggregation layers is not high, but after accumulation, an effective aggregation degree may still be obtained, and it will not be filtered out, thus reducing the false negative rate. However, for single-scale interference events, only the confidence of one aggregation layer is high, while the confidence of the other aggregation layers is low. The aggregation degree after accumulation is low and will be filtered out.

[0067] Step 105: Calculate the likelihood of an event belonging to the foreground based on the aggregation degree, and filter out events with a likelihood less than the likelihood threshold.

[0068] In different target locations, there may be some dynamic backgrounds (such as swaying leaves, slight camera shake, raindrops, etc.), therefore, such as Figure 3 As shown, at the foreground level, a background event occurrence rate model can be constructed for each pixel. Based on its aggregation degree, the likelihood of the remaining events after filtering in step 104 belonging to the foreground is calculated, thereby distinguishing background events from potential foreground events, filtering events with a likelihood less than the likelihood threshold, and filtering dynamic background interference.

[0069] In a specific implementation, a first separation threshold and a second separation threshold can be set, with the first separation threshold being less than the second separation threshold.

[0070] A background probability model is constructed for the target location based on methods such as the Poisson model. The background probability model describes the probability that each pixel coordinate belongs to the background in the target location, representing the average number of times the event occurs per unit time under static and dynamic backgrounds.

[0071] If the aggregation degree is less than the first separation threshold, the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is weighted and summed with the unit event occurrence rate using the first update rate to obtain the probability that the pixel coordinates belong to the background in the target location when the current event occurs.

[0072] The unit event occurrence rate is the ratio between the aggregation degree and the first conversion coefficient (e.g., 1000ms), ensuring that the unit event occurrence rate is consistent with the dimension of the probability that the pixel coordinates belong to the background in the target location.

[0073] At this point, the process of updating the background probability model is: P(x i ,y i ,t i )=(1-w1)×P(x i ,y i ,t last )+w1×S i / Δt1, where P(x i ,y i ,t i (t) represents the current time. i The pixel coordinates (x) when the event occurs i ,y i The probability that x belongs to the background in the target location, P(x) i ,y i ,t last (t) represents the previous time step. last The pixel coordinates (x) when the event occurs i ,y i The probability that a given location belongs to the background in the target location, where w1 is the first update rate (e.g., 0.001), and S i Δt1 represents the aggregation degree of the event, and Δt1 represents the first conversion coefficient.

[0074] When the aggregation degree is less than the first separation threshold, the events are usually isolated noise or very weak background activation. In this case, adapting to the slow changes in the scene (such as the gradual change of light throughout the day or the background change caused by seasonal changes) with a slow learning rate (i.e. the first update rate) ensures that the noise will not pollute the background baseline.

[0075] If the aggregation degree is greater than or equal to the first separation threshold and less than the second separation threshold, then the second update rate is used to weight and sum the probability that the pixel coordinates belong to the background in the target location when the previous event occurred with the unit event occurrence rate, so as to obtain the probability that the pixel coordinates belong to the background in the target location when the current event occurs.

[0076] The second update rate is greater than the first update rate.

[0077] At this point, the process of updating the background probability model is: P(x i ,y i ,t i )=(1-w2)×P(x i ,y i ,t last )+w2×S i / Δt1, where P(xi ,y i ,t i (t) represents the current time. i The pixel coordinates (x) when the event occurs i ,y i The probability that x belongs to the background in the target location, P(x) i ,y i ,t last (t) represents the previous time step. last The pixel coordinates (x) when the event occurs i ,y i The probability that a given location belongs to the background in the target location, w2 is the second update rate (e.g., 0.01), S i Δt1 represents the aggregation degree of the event, and Δt1 represents the first conversion coefficient.

[0078] When the aggregation degree is between the first separation threshold and the second separation threshold, the event is usually a valid candidate event that has been verified. However, the aggregation degree has not reached the foreground threshold and belongs to the dynamic background event (such as swaying leaves, water ripples, etc.). The fast learning rate (i.e. the second update rate) enables the background probability model to quickly adapt to the event occurrence rate changes of the dynamic background. The probability that the corresponding pixel coordinate belongs to the background in the target location increases, avoiding misjudging the dynamic background as the foreground.

[0079] If the aggregation degree is greater than or equal to the second separation threshold, then the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is assigned as the probability that the pixel coordinates belong to the background in the target location when the current event occurs.

[0080] When the degree of aggregation is greater than or equal to the second separation threshold, the event is a potential intrusion target. Updating the background probability model is prohibited to avoid the high probability of foreground events raising the background baseline, which could lead to missed detections after the background probability model is contaminated.

[0081] For cases where the degree of aggregation is greater than or equal to the second separation threshold, the product between the degree of aggregation and the significance factor is calculated to obtain the likelihood that the event belongs to the prospect.

[0082] The significance factor is based on a specified constant (such as 2), and the ratio between the observed event occurrence rate and the probability that the pixel coordinates at the time of the current event belong to the background in the target location is a true number.

[0083] The observed event occurrence rate is the ratio between the aggregation degree and the second conversion coefficient (e.g., 100ms). The second conversion coefficient is matched with the time window of the aggregation layer at the specified level (e.g., the second-ranked aggregation layer) to ensure that the observed event occurrence rate is consistent with the dimension of the probability that the pixel coordinates belong to the background in the target location.

[0084] Therefore, the likelihood that an event belongs to the prospect can be expressed as: Li =S i ×log a (S i / Δt2 / P(x i ,y i ,t i ), where L i S represents the likelihood that an event belongs to the prospect. i Let a be the degree of aggregation of the event, Δt2 be a constant, and P(x) be the second transformation coefficient. i ,y i ,t i (t) represents the current time. i The pixel coordinates (x) when the event occurs i ,y i The probability that a location belongs to the background in the target location.

[0085] In this embodiment, the log-likelihood ratio is used to amplify the difference between the foreground and the background, making it more sensitive to weak foreground targets with low contrast and slow intrusion.

[0086] Step 106: Calculate the activation value of the event in motion consistency based on the likelihood.

[0087] The actual intrusion target is a rigid body in motion, and its foreground events exhibit highly consistent optical flow vectors. However, the remaining background jitter and abrupt changes in light will generate optical flow vectors with random directions. In this embodiment, for example... Figure 3 As shown, at the optical flow level, the motion consistency characteristics of an event are calculated based on the likelihood that the event belongs to the foreground, and a global motion activation value is output accordingly for the main camera to trigger the judgment.

[0088] In the specific implementation, the remaining events after filtering in step 105 are traversed in a set order (such as the order of timestamps). When the current event is traversed, the fourth neighborhood of the current event is determined. The fourth neighborhood includes other events that are adjacent to the current event in time and space. That is, the absolute value of the difference between the timestamp of other events and the timestamp of the current event is less than or equal to a certain threshold, and the distance between the pixel coordinates of other events and the pixel coordinates of the current event is less than or equal to a certain threshold.

[0089] The motion edges of the event camera satisfy the constant brightness assumption (which holds strictly for edge events). The essence of the event is to find the derivative with respect to the timestamp. When the gradient of the velocity of a local event is less than 0, the velocity gradient can be ignored. The event spatiotemporal consistency is used to fit the optical flow to form an asynchronous optical flow.

[0090] Therefore, motion vectors in each direction are determined in the fourth neighborhood; where the motion vector in each direction is the ratio between the total coordinate offset in each direction and the sum of the likelihoods of other events; the total coordinate offset in each direction is the sum of the products of the unit coordinate offset in each direction and the likelihoods of other events; the unit coordinate offset in each direction is the ratio between the coordinate deviation value in each direction and the time offset; the coordinate deviation value in each direction is the distance between the pixel coordinates in each direction of the current event and the pixel coordinates in each direction of other events; and the time offset is the difference between the timestamp of the current event and the timestamps of other events.

[0091] So, in event e i motion vector (u) i ,v i In ), the horizontal motion vector u i =sum j=1~n (L j ×(x j -x i ) / (t j -t i )) / sum j=1~n L j The vertical motion vector v i =sum j=1~n (L j ×(y j -y i ) / (t j -t i )) / sum j=1~n L j Among them, event e i The pixel coordinates are (x i ,y i The timestamp is t. i Event e j The pixel coordinates are (x j ,y j The timestamp is t. j L j For event e j The likelihood of j∈n, where n is the number of other events in the fourth neighborhood.

[0092] Furthermore, the degree of dispersion in each direction is determined in the fourth neighborhood; wherein, the degree of dispersion in each direction is the ratio between the total vector offset in each direction and the sum of the likelihoods of other events; the total vector offset in each direction is the sum of the products of the squares of the vector offset in each direction and the likelihoods of other events; and the vector offset in each direction is the difference between the motion vectors in each direction of the current event and the motion vectors in each direction of other events.

[0093] So, in event e i The degree of dispersion in the horizontal direction σ u =sum j=1~n (L j ×(u j -u i ) 2 ) / sum j=1~n L j The degree of dispersion σ in the vertical direction v =sum j=1~n (L j ×(v j -v i ) 2 ) / sum j=1~n L j Among them, event e i The motion vector is (u i ,v i ), event e j The motion vector is (u j ,u j ), L j For event e j The likelihood of j∈n, where n is the number of other events in the fourth neighborhood.

[0094] The dispersion in each direction is normalized to the activation value of the event in motion consistency. For example, the dispersion in each direction is added together to obtain the consistency index. An exponential function is constructed with e as the base (i.e., the exp function) and the negative of the product between the consistency index σ and the scaling factor a (e.g., 0.1~0.3) as the exponent. The exponential function C is calculated (i.e., C=exp(-a×σ)) to obtain the activation value of the event in motion consistency.

[0095] Step 107: If the average value of the activation values ​​is greater than the activation threshold, the main camera is activated to collect main image data of the target location and the event is projected into the event image data.

[0096] In this embodiment, the average value of the activation values ​​of each event within a certain time range can be calculated, and the average value of the activation values ​​can be compared with a preset activation threshold.

[0097] like Figure 2 As shown, if the average value of the activation values ​​is greater than the activation threshold, the main camera can be woken up. The main camera switches from the sleep state to the working state and continuously collects main image data of the target location in the working state.

[0098] In addition, the polarity information of the remaining events after filtering in step 105 can be projected into the event image data according to the pixel coordinates. At this time, the event image data contains brightness change information over a historical period of time.

[0099] At this point, a sleep operation can be performed on the event camera, switching the event camera from working state to sleep state.

[0100] Step 108: Detect security information of the target location based on the main image data and event image data.

[0101] In this embodiment, there is a spatiotemporal continuity between the event image data and the main image data. The event image data can be used as preliminary information. By combining the main image data and the event image data, security detection can be performed to obtain the security information of the target location.

[0102] In different target locations, security information can be set according to business requirements. For example, if the target location is a residential building, the security information includes pedestrians, etc., and if the target location is a warehouse, the security information includes smoke, fire, etc.

[0103] In one embodiment of the present invention, step 108 may include the following steps: Step 1081: Load the security detection network.

[0104] In this embodiment, as Figure 4 As shown, a security detection network can be built and trained for the main camera. The types of security detection networks include classification networks (including binary classification networks and multi-classification networks) and object detection networks. The structure of the security detection network includes an encoder, an attention fusion module, a first decoder, and a second decoder.

[0105] If the main camera is equipped with a neural network for security detection, then logically the neural network can be divided into an encoder and a second decoder.

[0106] For example, if the neural network is YOLO (object detection network), the backbone can be divided into encoders, and the neck and head can be divided into second decoders.

[0107] For example, if the neural network is a ResNet (classification network), the first to third convolutional blocks can be classified as encoders based on the number of layers in the ResNet, and the remaining convolutional blocks and the Head (including pooling layers, fully connected layers, etc.) can be classified as second decoders.

[0108] The encoder, attention fusion module, and first decoder are treated as another complete model (classification network or object detection network) and trained. During training, the parameters of the encoder are kept unchanged, while the parameters of the attention fusion module and the first decoder are updated.

[0109] Step 1082: Input the main image data into the encoder to encode the main image features.

[0110] In this embodiment, as Figure 4 As shown, the main image data is input into the encoder for encoding to obtain the main image features.

[0111] Step 1083: Before the delay time point, input the brightness image data and event image data from the main image data into the attention fusion module to calculate the attention weights.

[0112] A delay time can be set when waking up the main camera. Let T0 be the time to wake up the main camera, and T be the delay time. d If the time interval between the delay point and the time point of waking up the main camera is a fixed duration ΔT, then T d =T0+ΔT.

[0113] The main camera is in a cold start state. Before the delay point, there is a lack of sufficient security information for smoothing, which may result in a certain false detection rate. In this embodiment, before the delay point, the event image data is used as the context information of the main image data to reduce the false detection rate.

[0114] In specific implementations, such as Figure 4 As shown, the brightness values ​​of each pixel coordinate can be separated from the main image data to obtain brightness image data. Brightness image data is more sensitive to the spatiotemporal response of event image data and can be used as a transition between event image data and main image data.

[0115] Taking an RGBD camera as an example, the brightness value can be expressed as: L=0.299R+0.587G+0.114B, where L is the brightness value, R is the red value, G is the green value, and B is the blue value.

[0116] The brightness image data and event image data in the main image data are input into the attention fusion module, and the attention weights are calculated using the brightness image data and event image data in the main image data according to the attention mechanism.

[0117] In a design, such as Figure 4 As shown, the attention fusion module includes a first feature layer, a second feature layer, and an attention layer. The first feature layer and the second feature layer can both be convolutional layers or other structures, and the first feature layer and the second feature layer share parameters.

[0118] On the one hand, the brightness image data in the main image data is input into the first feature layer to extract features and obtain brightness image features. On the other hand, the event image data is input into the second feature layer to extract features and obtain event image features. The symmetric structure with shared parameters can ensure that the brightness image features and event image features are in the same feature space, thus eliminating modal differences for attention calculation.

[0119] In the attention layer, a query matrix (Query) is constructed based on brightness image features, and a key matrix (Key) and a value matrix (Value) are constructed based on event image features. These are then substituted into the attention mechanism's calculation formula (Softmax(QK)). T / sqr(d k )) V, where Q is the query matrix, K is the key matrix, V is the value matrix, and d k The attention weights are calculated in the matrix (where T is the dimension of the key matrix, T is the transpose matrix, Softmax is the normalization function, and sqr is the square root function). Here, the attention weights represent the degree of matching between the event image features and the brightness image features. The higher the degree of matching, the more likely the brightness region corresponds to the occurrence of the event, and the greater the attention weight.

[0120] Step 1084: Integrate the attention weights into the main image features to obtain multimodal image features.

[0121] In this embodiment, as Figure 4 As shown, attention weights can be fused into the main image features to obtain multimodal image features.

[0122] For example, the main image features can be multiplied element-wise with the attention weights to obtain weighted image features. Then, based on the main image features, the product between the intensity coefficient and the weighted image features can be added to obtain multimodal image features.

[0123] Therefore, the multimodal image features can be represented as: F fusion =F RGB +a(F RGB ×A faet ), where F fusion For multimodal image features, F RGB Main image features, A faet denoted as attention weight, and 'a' as intensity coefficient (e.g., 0.8~1.0).

[0124] Step 1085: Input the multimodal image features into the first decoder to detect the security information of the target location.

[0125] In this embodiment, the first decoder includes structures such as pooling layers, fully connected layers, and activation functions. Its structure can be customized, or the structure of the decoding part of a third-party network (such as YOLO, ResNet, etc.) can be reused to make it the same as the structure of the second decoder. This embodiment does not impose any restrictions on this.

[0126] like Figure 4 As shown, the multimodal image features are input into the first decoder to perform security detection (such as classification, target detection, etc.) to obtain security information of the target location.

[0127] Generally, the dimensions of the security information output by the first decoder are consistent with those of the security information output by the second decoder, so as to facilitate subsequent normal access and smooth processing and other business operations.

[0128] Step 1086: After the delay time point, input the main image features into the second decoder to detect the security information of the target location.

[0129] After the delay point, a certain amount of safety information undergoes smoothing processing. At this time, such as... Figure 4 As shown, the main image features can be input into the second decoder to perform security detection (such as classification, object detection, etc.) to obtain security information of the target location.

[0130] In this embodiment, the event camera detects an event occurring in the target location; the event has pixel coordinates, a timestamp, and polarity information; the activity level of the event in the pixel coordinates is calculated based on the timestamp and pixel coordinates, and events with activity levels less than an activity threshold are filtered out; the confidence level of the event belonging to the motion edge is calculated based on the activity level and polarity information, and events with confidence levels less than a confidence threshold are filtered out; the aggregation degree of the event on the motion trajectory is calculated based on the confidence level at multiple spatiotemporal scales; the likelihood of the event belonging to the foreground is calculated based on the aggregation degree, and events with likelihood levels less than a likelihood threshold are filtered out; the activation value of the event in motion consistency is calculated based on the likelihood; if the activation value is greater than the activation threshold, the main camera is activated to acquire main image data of the target location, and the event is projected into the event image data; the security information of the target location is detected based on the main image data and the event image data. This embodiment uses an event camera as a secondary device to detect external events, thereby waking up the main camera for security detection. This keeps the main camera in a dormant state for extended periods, effectively reducing the overall power consumption of security monitoring. Furthermore, it employs event-driven layer-by-layer filtering at the point, line, surface, foreground, and optical flow levels. The filtering mechanism progresses from coarse to fine, filtering noise and reducing the number of events layer by layer, thereby reducing computational load. While ensuring the accuracy of waking up the main camera, it minimizes the resource consumption of running the event camera, making it suitable for application in resource-constrained devices.

[0131] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0132] Reference Figure 5 The diagram illustrates a security detection device provided by an embodiment of the present invention, which may specifically include the following modules: The event determination module 501 is used to determine an event detected by the event camera that occurs in the target location; the event has pixel coordinates, timestamp, and polarity information. Point processing module 502 is used to calculate the activity level of the event in the pixel coordinates based on the timestamp and the pixel coordinates, and filter the events whose activity level is less than the activity threshold; The line processing module 503 is used to calculate the confidence level of the event belonging to the motion edge based on the activity level and the polarity information, and filter the events whose confidence level is less than the confidence threshold. Surface processing module 504 is used to calculate the aggregation degree of the event on the motion trajectory based on the confidence level at multiple spatiotemporal scales; The foreground processing module 505 is used to calculate the likelihood that the event belongs to the foreground based on the aggregation degree, and filter the events whose likelihood is less than the likelihood threshold. Activation value calculation module 506 is used to calculate the activation value of the event in motion consistency based on the likelihood. The wake-up module 507 is used to wake up the main camera to collect main image data of the target location if the average value of the activation value is greater than the activation threshold, and to project the event into the event image data. The security information detection module 508 is used to detect the security information of the target location based on the main image data and the event image data.

[0133] In one embodiment of the present invention, the point processing module 502 is further configured to: Determine a first neighborhood of the current event; the first neighborhood includes other events that are temporally and spatially adjacent to the current event. Determine the time decay amount; the time decay amount is based on a natural number and has an exponent of the negative of the ratio between the event interval duration and the time decay coefficient; the event interval duration is the difference between the timestamp of the current event and the timestamp of the previous event in the first neighborhood. Determine the spatial diffusion amount; the spatial diffusion amount is based on a natural number and has an exponent that is the inverse of the ratio between the pixel distance and a multiple of the square of the standard deviation of spatial diffusion; the pixel distance is the average distance between the pixel coordinates of the current event and the pixel coordinates of other events in the first neighborhood; The product of the previous activity level and the time decay amount at the pixel coordinate, and the product of the activation gain and the spatial diffusion amount, are added together to obtain the activity level of the current event at the pixel coordinate.

[0134] In one embodiment of the present invention, the polarity information includes on information indicating that the brightness increase exceeds the trigger threshold and off information indicating that the brightness decrease exceeds the trigger threshold; The line processing module 503 is also used for: A second neighborhood is determined for the current event; the second neighborhood includes other events that are temporally and spatially adjacent to the current event, and the activity of the other events is greater than or equal to a specified proportion of the activity of the current event; A polarity co-occurrence matrix is ​​constructed for the second neighborhood; the elements in the polarity co-occurrence matrix include the number of open information pairs based on the open information, the number of closed information pairs based on the open information, the number of open information pairs based on the closed information, and the number of closed information pairs based on the closed information. Calculate the difference between 1 and the information entropy of the polarity co-occurrence matrix to obtain the degree of order. Calculate the spatial displacement vector between the first centroid corresponding to all the open information and the second centroid corresponding to all the closed information within the second neighborhood; the first centroid is the sum of the products of the pixel coordinates and the activity level divided by the sum of the activity levels, and the second centroid is the sum of the products of the pixel coordinates and the activity level divided by the sum of the activity levels. For the current event, the product of the activity level, the orderliness, and the magnitude of the spatial displacement vector is calculated to obtain the confidence level that the current event belongs to the motion edge.

[0135] In one embodiment of the present invention, the surface processing module 504 is further configured to: Multiple cascaded aggregation layers are defined; each aggregation layer is configured with a spatial window, a temporal window, and an aggregation threshold, wherein the scale of the spatial window and the scale of the temporal window are both positively correlated with the depth of the aggregation layer. In the current aggregation layer, other events whose pixel coordinates are located in the spatial window and whose timestamps are located in the time window for the current event are identified as the third neighborhood. In the third neighborhood, the confidence scores of the other events are merged into the confidence score of the current event; In the current aggregation layer, events with a confidence level less than the aggregation threshold are filtered out; If the current aggregation layer is not the last in the sorting, the event is output to the next aggregation layer; then, the process returns to execute the other events in the current aggregation layer that determine the pixel coordinates of the current event to be located in the spatial window and the timestamp to be located in the time window, which are then used as the third neighborhood. If the current aggregation layer is at the bottom of the sort, then the confidence level of the event in each aggregation layer is merged into the aggregation level of the event on the motion trajectory.

[0136] In one embodiment of the present invention, the foreground processing module 505 is further configured to: If the aggregation degree is less than the first separation threshold, then the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is weighted and summed with the unit event occurrence rate using the first update rate to obtain the probability that the pixel coordinates belong to the background in the target location when the current event occurs. If the aggregation degree is greater than or equal to the first separation threshold and less than the second separation threshold, then the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is weighted and summed with the unit event occurrence rate using the second update rate to obtain the probability that the pixel coordinates belong to the background in the target location when the current event occurs; the second update rate is greater than the first update rate; the unit event occurrence rate is the ratio between the aggregation degree and the first conversion coefficient; If the degree of aggregation is greater than or equal to the second separation threshold, then the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is assigned as the probability that the pixel coordinates belong to the background in the target location when the current event occurs. The product of the aggregation degree and the saliency factor is calculated to obtain the likelihood that the event belongs to the foreground; the saliency factor is a true number, which is the ratio of the observed event occurrence rate to the probability that the pixel coordinates belong to the background in the target location when the event occurs, with a specified constant as the base; the observed event occurrence rate is the ratio between the aggregation degree and the second transformation coefficient.

[0137] In one embodiment of the present invention, the activation value calculation module 506 is further configured to: Determine the fourth neighborhood of the current event; the fourth neighborhood includes other events that are temporally and spatially adjacent to the current event. A motion vector is determined in the fourth neighborhood; the motion vector is the ratio between the total coordinate offset and the sum of the likelihoods of the other events; the total coordinate offset is the sum of the products of the unit coordinate offset and the likelihoods of the other events; the unit coordinate offset is the ratio between the coordinate deviation value and the time offset; the coordinate deviation value is the distance between the pixel coordinates of the current event and the pixel coordinates of the other events; the time offset is the difference between the timestamp of the current event and the timestamps of the other events. The degree of dispersion is determined in the fourth neighborhood; the degree of dispersion is the ratio between the total vector offset and the sum of the likelihoods of the other events; the total vector offset is the sum of the squares of the vector offset and the products of the likelihoods of the other events; the vector offset is the difference between the motion vector of the current event and the motion vectors of the other events. The degree of dispersion is normalized to the activation value of the event in terms of motion consistency.

[0138] In one embodiment of the present invention, the security information detection module 508 is further configured to: A security detection network is loaded; the security detection network includes an encoder, an attention fusion module, a first decoder, and a second decoder. The main image data is input into the encoder and encoded into main image features; Before the delay point, the brightness image data in the main image data and the event image data are input into the attention fusion module to calculate the attention weight; The attention weights are fused into the main image features to obtain multimodal image features; The multimodal image features are input into the first decoder to detect the security information of the target location; After the delay point, the main image features are input into the second decoder to detect the security information of the target location.

[0139] In one embodiment of the present invention, the attention fusion module includes a first feature layer, a second feature layer and an attention layer, wherein the first feature layer and the second feature layer share parameters; The security information detection module 508 is also used for: The brightness image data in the main image data is input into the first feature layer to extract the brightness image features; The event image data is input into the second feature layer to extract event image features; In the attention layer, a query matrix is ​​constructed based on the brightness image features, and a key matrix and a value matrix are constructed based on the event image features. The attention weights are then calculated using the query matrix, the key matrix, and the value matrix. The security information detection module 508 is also used for: The main image features are multiplied element-wise with the attention weights to obtain the weighted image features; Based on the main image features, the product between the intensity coefficient and the weighted image features is added to obtain multimodal image features.

[0140] The present invention provides a security detection device, which can be used to implement the steps in the aforementioned security detection method embodiments.

[0141] It should be noted that the module division in the various security detection devices provided in the above embodiments is illustrative and only represents one logical functional division. In actual implementation, other division methods may also be used. Furthermore, the functional modules in the various embodiments of this invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0142] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the embodiments of the present invention can be embodied in the form of a computer program product, which is stored in a computer storage medium and includes several instructions to cause a camera device or processor to execute all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned computer storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] Furthermore, the security detection device and security detection method provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0144] Reference Figure 6 The diagram illustrates a camera device provided in an embodiment of the present invention. Figure 6 As shown, the camera device in this embodiment of the invention includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above-described security detection method embodiment. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described security detection device embodiment.

[0145] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which can be used to describe the execution process of the computer program in the camera device.

[0146] The camera device can be a desktop computer, cloud server, or other computing device. The camera device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 6 This is merely one example of a camera device and does not constitute a limitation on the camera device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the camera device may also include input / output devices, network access devices, buses, etc.

[0147] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0148] The memory can be an internal storage unit of the camera device, such as the hard drive or RAM of the camera device. The memory can also be an external storage device of the camera device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the camera device. Furthermore, the memory can include both internal and external storage units of the camera device. The memory is used to store the computer program and other programs and data required by the camera device. The memory can also be used to temporarily store data that has been output or will be output.

[0149] This invention also discloses a camera device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the security detection method as described in the foregoing embodiments.

[0150] This invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the security detection method as described in the foregoing embodiments.

[0151] This invention also discloses a computer program product that, when run on a computer, causes the computer to execute the security detection methods described in the foregoing embodiments.

[0152] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A security detection method, characterized in that, include: The event camera detects an event occurring at the target location; The event has pixel coordinates, timestamp, and polarity information; The activity level of the event at the pixel coordinates is calculated based on the timestamp and the pixel coordinates, and events with an activity level less than the activity threshold are filtered out. Based on the activity level and the polarity information, calculate the confidence level that the event belongs to the motion edge, and filter out the events whose confidence level is less than the confidence threshold; The aggregation degree of the event on the motion trajectory is calculated based on the confidence level at multiple spatiotemporal scales. Calculate the likelihood that the event belongs to the foreground based on the aggregation degree, and filter out events whose likelihood is less than the likelihood threshold; Calculate the activation value of the event in terms of motion consistency based on the likelihood. If the average value of the activation values ​​is greater than the activation threshold, the main camera is activated to collect main image data of the target location, and the event is projected into the event image data. Security information of the target location is detected based on the main image data and the event image data.

2. The method according to claim 1, characterized in that, The calculation of the activity level of the event at the pixel coordinates based on the timestamp and the pixel coordinates includes: Determine a first neighborhood of the current event; the first neighborhood includes other events that are temporally and spatially adjacent to the current event. Determine the time decay amount; the time decay amount is based on a natural number and has an exponent of the negative of the ratio between the event interval duration and the time decay coefficient; the event interval duration is the difference between the timestamp of the current event and the timestamp of the previous event in the first neighborhood. Determine the spatial diffusion amount; the spatial diffusion amount is based on a natural number and has an exponent that is the inverse of the ratio between the pixel distance and a multiple of the square of the standard deviation of spatial diffusion; the pixel distance is the average distance between the pixel coordinates of the current event and the pixel coordinates of other events in the first neighborhood; The product of the previous activity level and the time decay amount at the pixel coordinate, and the product of the activation gain and the spatial diffusion amount, are added together to obtain the activity level of the current event at the pixel coordinate.

3. The method according to claim 1, characterized in that, The polarity information includes on information indicating that the brightness increase exceeds the trigger threshold and off information indicating that the brightness decrease exceeds the trigger threshold; The calculation of the confidence level that the event belongs to the motion edge based on the activity and polarity information includes: A second neighborhood is determined for the current event; the second neighborhood includes other events that are temporally and spatially adjacent to the current event, and the activity of the other events is greater than or equal to a specified proportion of the activity of the current event; A polarity co-occurrence matrix is ​​constructed for the second neighborhood; the elements in the polarity co-occurrence matrix include the number of open information pairs based on the open information, the number of closed information pairs based on the open information, the number of open information pairs based on the closed information, and the number of closed information pairs based on the closed information. Calculate the difference between 1 and the information entropy of the polarity co-occurrence matrix to obtain the degree of order. Calculate the spatial displacement vector between the first centroid corresponding to all the open information and the second centroid corresponding to all the closed information within the second neighborhood; the first centroid is the sum of the products of the pixel coordinates and the activity level divided by the sum of the activity levels, and the second centroid is the sum of the products of the pixel coordinates and the activity level divided by the sum of the activity levels. For the current event, the product of the activity level, the orderliness, and the magnitude of the spatial displacement vector is calculated to obtain the confidence level that the current event belongs to the motion edge.

4. The method according to claim 1, characterized in that, The calculation of the aggregation degree of the event on the motion trajectory based on the confidence level at multiple spatiotemporal scales includes: Multiple cascaded aggregation layers are defined; each aggregation layer is configured with a spatial window, a temporal window, and an aggregation threshold, wherein the scale of the spatial window and the scale of the temporal window are both positively correlated with the depth of the aggregation layer. In the current aggregation layer, other events whose pixel coordinates are located in the spatial window and whose timestamps are located in the time window for the current event are identified as the third neighborhood. In the third neighborhood, the confidence scores of the other events are merged into the confidence score of the current event; In the current aggregation layer, events with a confidence level less than the aggregation threshold are filtered out; If the current aggregation layer is not the last in the sorting, the event is output to the next aggregation layer; then, the process returns to execute the other events in the current aggregation layer that determine the pixel coordinates of the current event to be located in the spatial window and the timestamp to be located in the time window, which are then used as the third neighborhood. If the current aggregation layer is at the bottom of the sort, then the confidence level of the event in each aggregation layer is merged into the aggregation level of the event on the motion trajectory.

5. The method according to claim 1, characterized in that, The calculation of the likelihood that the event belongs to the foreground based on the aggregation degree includes: If the aggregation degree is less than the first separation threshold, then the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is weighted and summed with the unit event occurrence rate using the first update rate to obtain the probability that the pixel coordinates belong to the background in the target location when the current event occurs. If the aggregation degree is greater than or equal to the first separation threshold and less than the second separation threshold, then the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is weighted and summed with the unit event occurrence rate using the second update rate to obtain the probability that the pixel coordinates belong to the background in the target location when the current event occurs; the second update rate is greater than the first update rate; the unit event occurrence rate is the ratio between the aggregation degree and the first conversion coefficient; If the degree of aggregation is greater than or equal to the second separation threshold, then the probability that the pixel coordinates belong to the background in the target location when the previous event occurred is assigned as the probability that the pixel coordinates belong to the background in the target location when the current event occurs. The product of the aggregation degree and the saliency factor is calculated to obtain the likelihood that the event belongs to the foreground; the saliency factor is a true number, which is the ratio of the observed event occurrence rate to the probability that the pixel coordinates belong to the background in the target location when the event occurs, with a specified constant as the base; the observed event occurrence rate is the ratio between the aggregation degree and the second transformation coefficient.

6. The method according to claim 1, characterized in that, The calculation of the activation value of the event in motion consistency based on the likelihood includes: Determine the fourth neighborhood of the current event; the fourth neighborhood includes other events that are temporally and spatially adjacent to the current event. A motion vector is determined in the fourth neighborhood; the motion vector is the ratio between the total coordinate offset and the sum of the likelihoods of the other events; the total coordinate offset is the sum of the products of the unit coordinate offset and the likelihoods of the other events; the unit coordinate offset is the ratio between the coordinate deviation value and the time offset; the coordinate deviation value is the distance between the pixel coordinates of the current event and the pixel coordinates of the other events; the time offset is the difference between the timestamp of the current event and the timestamps of the other events. The degree of dispersion is determined in the fourth neighborhood; the degree of dispersion is the ratio between the total vector offset and the sum of the likelihoods of the other events; the total vector offset is the sum of the squares of the vector offset and the products of the likelihoods of the other events; the vector offset is the difference between the motion vector of the current event and the motion vectors of the other events. The degree of dispersion is normalized to the activation value of the event in terms of motion consistency.

7. The method according to any one of claims 1-6, characterized in that, The step of detecting security information of the target location based on the main image data and the event image data includes: A security detection network is loaded; the security detection network includes an encoder, an attention fusion module, a first decoder, and a second decoder. The main image data is input into the encoder and encoded into main image features; Before the delay point, the brightness image data in the main image data and the event image data are input into the attention fusion module to calculate the attention weight; The attention weights are fused into the main image features to obtain multimodal image features; The multimodal image features are input into the first decoder to detect the security information of the target location; After the delay point, the main image features are input into the second decoder to detect the security information of the target location.

8. The method according to claim 7, characterized in that, The attention fusion module includes a first feature layer, a second feature layer, and an attention layer, wherein the first feature layer and the second feature layer share parameters. The step of inputting the brightness image data from the main image data and the event image data into the attention fusion module to calculate the attention weights includes: The brightness image data in the main image data is input into the first feature layer to extract the brightness image features; The event image data is input into the second feature layer to extract event image features; In the attention layer, a query matrix is ​​constructed based on the brightness image features, and a key matrix and a value matrix are constructed based on the event image features. The attention weights are then calculated using the query matrix, the key matrix, and the value matrix. The process of fusing the attention weights into the main image features to obtain multimodal image features includes: The main image features are multiplied element-wise with the attention weights to obtain the weighted image features; Based on the main image features, the product between the intensity coefficient and the weighted image features is added to obtain multimodal image features.

9. A camera device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the security detection method as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the security detection method as described in any one of claims 1-8.

Images