A foreign object falling detection method, device, equipment, medium and product

Abstract

The present invention discloses a foreign object falling detection method, device, equipment, medium and product. The method includes: performing real-time detection on a target area through an event camera to capture an event data stream; performing event frame compression on the event data in the current time window spatially and determining an area of interest; performing principal component analysis on the event data in the area of interest, and reconstructing the event data in the current time window according to the principal component analysis result; if an object in the current time window meets the falling trigger condition, determining the current image position of the object according to the reconstructed event data in the current time window; determining the historical image position of the object according to the reconstructed event data in the historical time window; and judging whether an object has fallen in the target area according to the current image position and the historical image position. The embodiments of the present invention can improve the accuracy of foreign object falling detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a foreign object falling detection method, device, equipment, medium and product. Background Art

[0002] Foreign object falling detection can eliminate potential safety hazards, prevent personal injury or property loss caused by foreign object falling, and ensure the safety of production and living environments. Traditional foreign object falling detection methods mainly focus on filtering and frame-based analysis methods. However, filtering methods have limited performance in dealing with non-linear dynamic scenes and are prone to losing important motion information; while frame-based analysis methods cannot fully utilize the time advantage of event cameras. Summary of the Invention

[0003] The present invention provides a foreign object falling detection method, device, equipment, medium and product to solve the problem of inaccurate existing foreign object falling detection methods.

[0004] According to one aspect of the present invention, a foreign object falling detection method is provided, including:

[0005] Performing real-time detection on a target area through an event camera to capture an event data stream;

[0006] Performing event frame compression on the event data in the current time window spatially and determining an area of interest;

[0007] Performing principal component analysis on the event data in the area of interest and reconstructing the event data in the current time window according to the principal component analysis result;

[0008] If it is determined that an object in the current time window meets the falling trigger condition according to the reconstructed event data and the principal component analysis result in the current time window, determining the current image position of the object according to the reconstructed event data in the current time window;

[0009] Determining the historical image position of the object according to the reconstructed event data in the historical time window;

[0010] Judging whether an object has fallen in the target area according to the current image position and the historical image position.

[0011] According to another aspect of the present invention, a foreign object falling detection device is provided, including:

[0012] An event data acquisition module, configured to perform real-time detection on a target area through an event camera to capture an event data stream;

[0013] An area of interest determination module, configured to perform event frame compression on the event data in the current time window spatially and determine an area of interest;

[0014] A data reconstruction module for performing principal component analysis on the event data in the region of interest and reconstructing the event data in the current time window according to the results of the principal component analysis;

[0015] An object falling judgment module for determining that if the object in the current time window meets the falling trigger condition according to the reconstructed event data and the results of the principal component analysis in the current time window, determining the current image position of the object according to the reconstructed event data in the current time window; determining the historical image position of the object according to the reconstructed event data in the historical time window; and judging whether an object has fallen in the target region according to the current image position and the historical image position.

[0016] According to another aspect of the present invention, there is provided a computer program product including a computer program which, when executed by a processor, implements the foreign object falling detection method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, there is provided an electronic device, which includes:

[0018] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the foreign object falling detection method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the foreign object falling detection method according to any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, there is provided a computer program product including a computer program / instructions which, when executed by a processor, implement the foreign object falling detection method according to any embodiment of the present invention.

[0021] In the embodiments of the present invention, by determining the region of interest in the target region and reconstructing the event data in the time window based on the principal component analysis result of the event data in the region of interest, unnecessary global processing is reduced, the overall processing efficiency is improved, the consumption of computing resources is reduced, and the real-time performance of foreign object fall detection is improved. Excluding the isolated points in the region of interest can focus more accurately on the key information, improving the accuracy of foreign object fall detection. And it can track the movement direction of the object. If the object in the current time window meets the fall trigger condition, by accurately positioning the position of the object in the current and historical images, it can quickly and accurately determine whether an object has fallen in the target region, greatly improving the sensitivity and reliability of fall event detection. Applying it to the detection of foreign object falls in the refueling water pool can accurately identify the fall of a tiny object with a volume of only 1 cubic centimeter within a range of 5 meters, send an alarm signal, and mark the position of the fallen object in the picture. It can track the movement trajectory of the target object in real time, and effectively track the movement state of the target by detecting the edge position and movement direction of the object.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 is the first flowchart of a foreign object fall detection method provided by an embodiment of the present invention;

[0025] Figure 2 is the second flowchart of a foreign object fall detection method provided by an embodiment of the present invention;

[0026] Figure 3 is the structural schematic diagram of a foreign object fall detection device provided by an embodiment of the present invention;

[0027] Figure 4 is the structural schematic diagram of an electronic device for implementing an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0030] Figure 1 FIG. 1 is a first flowchart of a foreign object falling detection method provided by an embodiment of the present invention. This embodiment is applicable to the case of foreign object falling detection based on an event camera. This method can be executed by a foreign object falling detection device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device with corresponding data processing capabilities. As Figure 1 shown, the method includes:

[0031] S110. Real-time detect a target area through an event camera to capture an event data stream; perform event frame compression on the event data in the current time window in space and determine a region of interest.

[0032] An event camera is a new type of vision sensor, also known as a dynamic vision sensor (DVS) or DAVIS (Dynamic and Active-Pixel Vision Sensor). Compared with a traditional camera, the event camera has significant differences in the data acquisition method. Instead of capturing images at a fixed frame rate, the event camera detects changes in the brightness of the scene and records these changes as "events". Each event includes a timestamp, pixel coordinates, and a polarity representing the brightness change.

[0033] The working principle of an event camera is based on detecting changes in pixel brightness. When the pixel brightness change exceeds a preset threshold, an event is generated. This mechanism enables the event camera to work under high dynamic range, while reducing the complexity and latency of data processing. The output of an event camera is an asynchronous event stream, and each event is independently marked with a timestamp, thus having extremely low latency. In addition, the event camera only responds to changing pixels, greatly reducing the output of invalid information and lowering the data bandwidth and processing requirements.

[0034] The target area is detected in real time by the event camera to capture the event data stream. Each event data includes spatial coordinates, polarity, and timestamp. The spatial coordinates are pixel coordinates, and the polarity is used to describe the direction of pixel brightness change. Specifically, the polarity represents whether the brightness changes from low to high or from high to low. In the output data of the event camera, each event will contain a polarity value. When the brightness increases, the polarity is usually marked as positive (e.g., 1 or +1), indicating a positive event. On the contrary, when the brightness decreases, the polarity is marked as negative (e.g., 0 or -1), indicating a negative event.

[0035] Within a preset time window (such as 50 milliseconds), collect the event data generated by the event camera, preprocess the event data, select the spatial coordinates and timestamps of the event data with positive polarity, and organize the event data into a three-dimensional array such as (x, y, t), only retaining the spatial and time information. Standardize the event data so that the data in each dimension satisfies zero mean and unit variance. The processed event data includes the first spatial dimension, the second spatial dimension, and the time dimension.

[0036] Accumulate the event data in the current time window to generate an event frame. Perform spatial event frame compression on the event data in the current time window, such as reducing the data volume by aggregating adjacent events or reducing the resolution, thereby reducing the complexity of data processing and improving the processing efficiency. Image processing techniques such as deep learning can be used to automatically identify the regions of interest in the event frame. The regions of interest are the areas where events are concentrated. Optionally, the event data in the current time window can also be clustered according to the first spatial dimension and the second spatial dimension, and the area within the preset range of the cluster center is used as the region of interest.

[0037] Furthermore, determine the distance between each event data in the region of interest and the center point of the region of interest according to the second spatial dimension and the time dimension, and exclude the event data whose distance exceeds the preset distance.

[0038] By determining the region of interest, subsequent processing can focus on the data within the region of interest, reducing unnecessary global processing, improving the overall processing efficiency, reducing the consumption of computing resources, and enhancing the real-time performance of foreign object fall detection. Excluding isolated points in the region of interest can more accurately focus on key information and improve the accuracy of foreign object fall detection.

[0039] S120. Perform principal component analysis on the event data in the region of interest, and reconstruct the event data in the current time window according to the results of the principal component analysis.

[0040] Calculate the covariance matrix based on the event data in the region of interest, perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues and eigenvectors; the eigenvalues represent the importance (i.e., variance size) of each principal component, and the eigenvectors represent the directions of the principal components. Determine the eigenvectors corresponding to the eigenvalues as the first principal component, the second principal component, and the third principal component in sequence according to the magnitudes of the eigenvalues, and the eigenvector corresponding to the largest eigenvalue is the first principal component. Reconstruct the event data in the current time window in the principal component space. Determine the principal component space based on the event data in the region of interest and reconstruct the event data in the current time window in this principal component space. By mapping the event data in the current time window to the new three-dimensional space after principal component analysis, the key features of the data can be efficiently extracted, and the main change trends of the data can be highlighted in the new space.

[0041] S130. If it is determined that the object in the current time window meets the fall trigger condition according to the reconstructed event data and the results of the principal component analysis in the current time window, determine the current image position of the object according to the reconstructed event data in the current time window.

[0042] Considering that within a very short time, the movement trajectory of the falling object presents a linear mathematical model, that is, a straight line. Therefore, its principal components show a phenomenon of one large and two small; at the same time, its displacement in the horizontal direction is small, and it shows a phenomenon of rapidly increasing with time in the vertical direction, so the angles of its eigenvalues also show a phenomenon of one large and two small. Specific thresholds can be determined through a large number of experiments to obtain the fall trigger condition.

[0043] If it is confirmed that the object in the current time window meets the preset fall trigger condition according to the eigenvalues in the results of the principal component analysis of the event data in the region of interest and the reconstructed event data in the current time window, determine the current image position of the object according to the event data coordinates at the edge position in the reconstructed event data in the current time window.

[0044] S140. Determine the historical image position of the object according to the reconstructed event data in the historical time window.

[0045] Optionally, the historical time window is the nearest time window that satisfies the fall trigger condition before the current time window within a preset time range.

[0046] Within a preset time range, for example, within 1 second, the nearest time window before the current time window that satisfies the fall trigger condition is used as the historical time window, and the historical image position of the object is determined according to the reconstructed event data in the historical time window.

[0047] S150. Determine whether an object has fallen in the target area according to the current image position and the historical image position.

[0048] Compare and analyze the current image position of the object in the target area with the historical image position of the object in the target area, and judge whether the object has fallen in the target area based on the change of the object position. Here, the image position is the edge position of the object in the target area. If the current image position and the historical image position in the target area are vertically aligned within a certain error range after precise comparison, and the current image position has moved downward relative to the historical image position, it indicates that an object has fallen in the target area. Rely on the precise measurement and analysis of the change of the object position to reliably identify and confirm the phenomenon of object fall.

[0049] In the embodiment of the present invention, by determining the region of interest in the target area and reconstructing the event data in the time window based on the principal component analysis result of the event data in the region of interest, unnecessary global processing is reduced, the overall processing efficiency is improved, the consumption of computing resources is reduced, and the real-time performance of foreign object fall detection is improved. Excluding the isolated points in the region of interest can focus more accurately on the key information and improve the accuracy of foreign object fall detection. And it can track the movement direction of the object. If the object satisfies the fall trigger condition in the current time window, by accurately positioning the position of the object in the current and historical images, it can quickly and accurately judge whether an object has fallen in the target area, greatly improving the sensitivity and reliability of fall event detection. Applying it to the detection of foreign object fall in the refueling water pool can accurately identify the fall of a tiny object with a volume of only 1 cubic centimeter within a range of 5 meters, send an alarm signal, and mark the position of the fallen object in the picture. It can track the movement trajectory of the target object in real time, and effectively track the movement state of the target by detecting the edge position and movement direction of the object.

[0050] In an optional implementation manner, the determining whether an object has fallen in the target area according to the current image position and the historical image position includes: if the current image position overlaps with the historical image position in the first direction, and the current image position satisfies an increase between the historical image position in the second direction, determine that a foreign object fall event has occurred in the target area.

[0051] Specifically, the second direction is the forward movement direction of the object. According to the minimum position coordinate and the maximum position coordinate of the current image position in the first direction, the current occupancy interval of the current image position in the first direction is determined. Similarly, according to the minimum position coordinate and the maximum position coordinate of the historical image position in the first direction, the historical occupancy interval of the historical image position in the first direction is determined. If the overlap degree between the current occupancy interval and the historical occupancy interval is greater than a preset value, there is an overlap between the current image position in the first direction and the historical image position in the first direction. If the minimum position coordinate of the current image position in the second direction is greater than the minimum position coordinate of the historical image position in the second direction, and the maximum position coordinate of the current image position in the second direction is greater than the maximum position coordinate of the historical image position in the second direction, then an increase is satisfied between the current image position in the second direction and the historical image position in the second direction. When there is an overlap between the current image position in the first direction and the historical image position in the first direction, and an increase is satisfied between the current image position in the second direction and the historical image position in the second direction, it is determined that a foreign object falling event has occurred in the target area.

[0052] By accurately judging whether there is a foreign object falling event in the target area according to the occupancy intervals and relative changes of the current image position and the historical image position in the first and second directions, the accuracy of event detection is improved, the movement trend of the object is effectively identified, a more reliable decision-making basis is provided for the real-time monitoring and warning system, and it helps to respond to and handle potential safety hazards in a timely manner.

[0053] Figure 2 It is the second flowchart of a foreign object falling detection method provided by an embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. As Figure 2 shown, the method includes:

[0054] S210. Perform real-time detection on the target area through an event camera to capture an event data stream; perform event frame compression on the event data in the current time window in space and determine the region of interest.

[0055] In an optional implementation manner, the performing event frame compression on the event data in the current time window in space and determining the region of interest includes: performing spatial compression on the event data in the current time window to obtain an event data set; using the coordinate with the largest number of event points in the event data set as the center point, and using the area within a preset range of the center point as the region of interest.

[0056] Specifically, event data is accumulated within the current time window to generate an event frame. In the event frame, each spatial location (i.e., pixel location) corresponds to an event counter, which records the number of events that occurred at that location within the current time window. Event frames are compressed to reduce data volume and improve processing efficiency. Compression methods include removing redundant information, merging adjacent events, and reducing resolution. The location with the largest number of events in the compressed event frame is designated as the center of the region of interest (ROI), and the area within a preset range of the center is designated as the ROI.

[0057] By compressing event frames and identifying regions of interest (ROIs), the amount of data required for subsequent processing can be significantly reduced, thereby improving processing efficiency and enhancing the real-time performance of falling foreign object detection. Determining ROIs helps remove background noise and irrelevant information, enhancing the robustness and accuracy of the method.

[0058] S220: Perform principal component analysis on the event data in the region of interest, and reconstruct the event data in the current time window according to the principal component analysis result.

[0059] Specifically, the event data is reconstructed through inverse operations after discarding points with large errors.

[0060] S230 , determining a relative eigenvalue based on the maximum eigenvalue and the second largest eigenvalue in the principal component analysis result; and determining a motion direction of the object in the current time window based on the reconstructed event data.

[0061] The difference between the maximum eigenvalue and the second largest eigenvalue in the principal component analysis result is used as the relative eigenvalue; the direction of movement of the object in the current time window is determined based on the reconstructed event data. Optionally, the method of determining the direction of movement of the object in the current time window based on the reconstructed event data includes: linearly fitting the reconstructed event data using the least squares method to obtain a linear equation, and determining the direction of movement of the object in the current time window based on the slope of the linear equation. Since movement in three-dimensional space involves multiple directions, a linear equation is obtained by linear fitting each direction separately, and the slope of the linear equation is the direction of movement of the object. If the slope of the linear equation obtained by fitting in any direction is greater than zero, it indicates that the object is moving forward in that direction.

[0062] S240: If the relative characteristic value is greater than a preset threshold and the object is moving forward along the movement direction, the object in the current time window meets the falling trigger condition.

[0063] The relative eigenvalue reflects the degree of dispersion of event data in the direction of the principal component. When the relative eigenvalue is large, it indicates that the change of event data in the direction of the first principal component is relatively significant, and the main motion trend of the object can be identified. By accurately using the maximum eigenvalue and the second largest eigenvalue of the principal component analysis to determine the relative eigenvalue, and combining the reconstructed event data to efficiently identify the object falling state and accurately judge the object motion direction. Once a moving object appears in the target area, the falling warning mechanism can be immediately triggered, which not only ensures the accuracy of the judgment but also greatly improves the response speed.

[0064] S250. If the object in the current time window meets the falling trigger condition, determine the current image position of the object in the current image according to the reconstructed event data in the current time window.

[0065] S260. Determine the historical image position of the object according to the reconstructed event data in the historical time window.

[0066] S270. Judge whether an object has fallen in the target area according to the current image position and the historical image position.

[0067] When the object in the current time window meets the falling trigger condition, by comparing the image positions of the object in the current and historical time windows, it is possible to quickly and accurately determine whether an object falling event has actually occurred in the target area. Significantly improve the sensitivity and accuracy of falling event detection, determine the direction of object fall, and track the object fall trajectory, showing great advantages and application values in safety monitoring and accident prevention.

[0068] In the embodiments of the present invention, the maximum eigenvalue and the second largest eigenvalue of the principal component analysis are accurately used to determine the relative eigenvalue, and the reconstructed event data is combined to efficiently identify the object falling state and accurately judge the object motion direction. Once a moving object appears in the target area, the falling warning mechanism can be immediately triggered, which not only ensures the accuracy of the judgment but also greatly improves the response speed. When the object in the current time window meets the falling trigger condition, by comparing the image positions of the object in the current and historical time windows, it is possible to quickly and accurately determine whether an object falling event has actually occurred in the target area. Significantly improve the sensitivity and accuracy of falling event detection, determine the direction of object fall, and track the object fall trajectory, showing great advantages and application values in safety monitoring and accident prevention.

[0069] Figure 3 It is a schematic structural diagram of a foreign object falling detection device provided by an embodiment of the present invention. As Figure 3 shown, the device includes:

[0070] An event data acquisition module 310, configured to perform real-time detection on a target area through an event camera and capture an event data stream;

[0071] An interested region determination module 320 is configured to perform event frame compression on event data in a current time window spatially and determine an interested region;

[0072] A data reconstruction module 330 is configured to perform principal component analysis on the event data in the interested region and reconstruct the event data in the current time window according to the principal component analysis result;

[0073] An object falling judgment module 340 is configured to, if it is determined that an object in the current time window meets a falling trigger condition according to the reconstructed event data and the principal component analysis result in the current time window, determine the current image position of the object according to the reconstructed event data in the current time window; determine the historical image position of the object according to the reconstructed event data in a historical time window; and judge whether an object has fallen in the target region according to the current image position and the historical image position.

[0074] The foreign object falling detection device provided by an embodiment of the present invention can execute the foreign object falling detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0075] Optionally, the object falling judgment module includes: a falling trigger condition unit configured to determine a relative eigenvalue according to the maximum eigenvalue and the second largest eigenvalue in the principal component analysis result; determine the moving direction of the object in the current time window according to the reconstructed event data; and if the relative eigenvalue is greater than a preset threshold and the object moves forward along the moving direction, the object in the current time window meets the falling trigger condition.

[0076] Optionally, the falling trigger condition unit includes: a moving direction determination sub-unit configured to perform linear fitting on the reconstructed event data by using the least squares method to obtain a linear equation, and determine the moving direction of the object in the current time window according to the slope of the linear equation.

[0077] Optionally, the object falling judgment module includes: an object falling judgment unit configured to determine that a foreign object falling event has occurred in the target region if the current image position overlaps with the historical image position in a first direction and the current image position satisfies an increase between the historical image position in a second direction.

[0078] Optionally, the historical time window is the nearest time window that meets the falling trigger condition before the current time window within a preset time range.

[0079] Optionally, the region of interest determination module includes a first unit for region of interest determination, which is configured to perform spatial compression on the event data in the current time window to obtain an event data set, and use the coordinate with the largest number of event points in the event data set as the center point, and use the region within the preset range of the center point as the region of interest.

[0080] The foreign object falling detection device described in further detail can also execute the foreign object falling detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0081] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0082] Figure 4 FIG. shows a schematic structural diagram of an electronic device 40 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0083] As Figure 4 shown, the electronic device 40 includes at least one processor 41, and a memory communicatively connected to at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc. Among them, the memory stores a computer program executable by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. The input / output (I / O) interface 45 is also connected to the bus 44.

[0084] Multiple components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disc, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0085] The processor 41 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the foreign object falling detection method.

[0086] In some embodiments, the foreign object falling detection method can be implemented as a computer program, which is tangibly included in a computer-readable storage medium, such as the storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the foreign object falling detection method described above can be executed. Alternatively, in other embodiments, the processor 41 can be configured to execute the foreign object falling detection method by any other suitable means (e.g., by means of firmware).

[0087] The various embodiments of the systems and technologies described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs, the one or more computer programs can be executed and / or interpreted on a programmable system including at least one programmable processor, the programmable processor can be a dedicated or general-purpose programmable processor, can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0088] A computer program for implementing the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing device, such that when the computer programs are executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer programs may be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0089] In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0091] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend, middleware, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0092] A computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0093] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0094] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for detecting falling foreign objects, characterized in that: The method comprises: Use event cameras to detect target areas in real time and capture event data streams; Compress the event data in the current time window spatially into event frames and determine the region of interest; Performing principal component analysis on the event data in the region of interest, and reconstructing the event data in the current time window according to the principal component analysis result; If it is determined that the object in the current time window meets the fall trigger condition based on the event data reconstructed in the current time window and the principal component analysis result, the current image position of the object is determined based on the event data reconstructed in the current time window; Determine the historical image location of the object based on the event data reconstructed in the historical time window; determining whether an object has fallen in the target area according to the current image position and the historical image position; The step of determining whether an object in the current time window meets a fall trigger condition based on the reconstructed event data in the current time window and the principal component analysis result includes: Determine the relative eigenvalue based on the largest eigenvalue and the second largest eigenvalue in the principal component analysis results; Determine the motion direction of the object in the current time window based on the reconstructed event data; If the relative characteristic value is greater than a preset threshold and the object moves forward along the movement direction, the object in the current time window meets the falling trigger condition.

2. The method according to claim 1, characterized in that Determining the movement direction of the object in the current time window according to the reconstructed event data includes: The reconstructed event data is linearly fitted using the least squares method to obtain a linear equation, and the motion direction of the object in the current time window is determined according to the slope of the linear equation.

3. The method according to claim 1, characterized in that The determining whether an object has fallen in the target area according to the current image position and the historical image position includes: If the current image position overlaps with the historical image position in the first direction, and the current image position and the historical image position in the second direction satisfy an increasing relationship, it is determined that a foreign object falling event occurs in the target area.

4. The method according to claim 1, wherein The historical time window is the most recent time window that meets the fall trigger condition before the current time window within a preset time range.

5. The method according to claim 1, wherein The spatially compressing the event frames of the event data in the current time window and determining the region of interest includes: Perform spatial compression on the event data in the current time window to obtain an event dataset; The coordinates with the largest number of event points in the event dataset are taken as the center point, and the area within the preset range of the center point is taken as the area of interest.

6. A foreign body falling detection device, characterized in that: The device comprises: The event data acquisition module is used to detect the target area in real time through the event camera and capture the event data stream; An area of interest determination module is used to spatially compress event frames of event data in a current time window and determine an area of interest; a data reconstruction module, configured to perform principal component analysis on the event data in the region of interest, and reconstruct the event data in the current time window according to the principal component analysis result; An object fall judgment module is configured to, if it is determined based on the event data reconstructed in the current time window and the principal component analysis results that an object in the current time window meets the fall trigger condition, determine the current image position of the object based on the event data reconstructed in the current time window; determine the historical image position of the object based on the event data reconstructed in the historical time window; and judge whether an object has fallen in the target area based on the current image position and the historical image position; The object fall judgment module includes: a fall trigger condition unit, which is used to determine a relative eigenvalue based on the maximum eigenvalue and the second largest eigenvalue in the principal component analysis result; determine the movement direction of the object in the current time window based on the reconstructed event data; if the relative eigenvalue is greater than a preset threshold and the object moves forward along the movement direction, then the object in the current time window meets the fall trigger condition.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the foreign object falling detection method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the foreign object falling detection method according to any one of claims 1 to 5 when executed.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for detecting falling foreign objects according to any one of claims 1 to 5.

Images