Foreign matter falling detection method, device, equipment, medium and product

Abstract

The invention discloses a foreign matter falling detection method, device and equipment, a medium and a product. The method comprises the following steps: detecting a target area in real time through an event camera, and capturing an event data stream; performing event frame compression on the event data in the current time window in space and determining a region of interest; performing principal component analysis on the event data in the region of interest, and reconstructing the event data in the current time window according to a principal component analysis result; if the object in the current time window meets the falling triggering condition, the current image position where the object is located is determined according to the event data reconstructed in the current time window; determining a historical image position where the object is located according to the event data reconstructed in the historical time window; and judging whether an object falls in the target area according to the current image position and the historical image position. According to the embodiment of the invention, the accuracy of foreign matter falling detection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to a foreign object falling detection method, device, equipment, medium and product. Background Art

[0002] Foreign object falling detection can eliminate potential safety hazards, prevent personal injuries or property damage caused by foreign object falling, and ensure the safety of the production and living environment. Traditional foreign object falling detection methods mainly focus on filtering and frame-based analysis methods. However, the filtering method has limited performance in dealing with non-linear dynamic scenes and is prone to losing important motion information; while the frame-based analysis method cannot fully utilize the time advantage of the event camera. Summary of the Invention

[0003] The present invention provides a foreign object falling detection method, device, equipment, medium and product to solve the problem of inaccurate existing foreign object falling detection methods.

[0004] According to one aspect of the present invention, there is provided a foreign object falling detection method, including:

[0005] Performing real-time detection on a target area through an event camera to capture an event data stream;

[0006] Performing event frame compression on the event data in the current time window spatially and determining a region of interest;

[0007] Performing principal component analysis on the event data in the region of interest, and reconstructing the event data in the current time window according to the principal component analysis result;

[0008] If it is determined that an object in the current time window meets the falling trigger condition according to the reconstructed event data and the principal component analysis result in the current time window, determining the current image position of the object according to the reconstructed event data in the current time window;

[0009] Determining the historical image position of the object according to the reconstructed event data in the historical time window;

[0010] Judging whether an object falls in the target area according to the current image position and the historical image position.

[0011] According to another aspect of the present invention, there is provided a foreign object falling detection device, including:

[0012] An event data acquisition module, configured to perform real-time detection on a target area through an event camera to capture an event data stream;

[0013] A region of interest determination module, configured to perform event frame compression on the event data in the current time window spatially and determine a region of interest;

[0014] A data reconstruction module, configured to perform principal component analysis on the event data in the region of interest, and reconstruct the event data in the current time window according to the results of the principal component analysis;

[0015] An object falling judgment module, configured to, if it is determined according to the reconstructed event data and the results of the principal component analysis in the current time window that an object in the current time window meets the falling trigger condition, determine the current image position of the object according to the reconstructed event data in the current time window; determine the historical image position of the object according to the reconstructed event data in the historical time window; and determine whether an object has fallen in the target region according to the current image position and the historical image position.

[0016] According to another aspect of the present invention, there is provided a computer program product, including a computer program, where the computer program, when executed by a processor, implements the foreign object falling detection method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, there is provided an electronic device, where the electronic device includes:

[0018] At least one processor; and a memory communicatively connected to the at least one processor; where the memory stores a computer program executable by the at least one processor, and the computer program, when executed by the at least one processor, enables the at least one processor to execute the foreign object falling detection method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, there is provided a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions, when executed by a processor, implement the foreign object falling detection method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, there is provided a computer program product, including computer program / instructions, where the computer program / instructions, when executed by a processor, implement the foreign object falling detection method according to any embodiment of the present invention.

[0021] In the embodiments of the present invention, by determining the region of interest in the target area and reconstructing the event data in the time window based on the principal component analysis result of the event data in the region of interest, unnecessary global processing is reduced, the overall processing efficiency is improved, the consumption of computing resources is reduced, and the real-time performance of foreign object fall detection is improved. Excluding the isolated points in the region of interest can focus more accurately on the key information, improving the accuracy of foreign object fall detection. And it can track the movement direction of the object. If the object in the current time window meets the fall trigger condition, by accurately positioning the position of the object in the current and historical images, it can quickly and accurately determine whether an object has fallen in the target area, greatly improving the sensitivity and reliability of fall event detection. Applying it to the detection of foreign object falls in the refueling water pool can accurately identify the fall of a tiny object with a volume of only 1 cubic centimeter within a range of 5 meters, send an alarm signal, and mark the position of the fallen object in the picture. It can track the movement trajectory of the target object in real time, and effectively track the movement state of the target by detecting the edge position and movement direction of the object.

[0022] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 is the first flowchart of a foreign object fall detection method provided by the embodiments of the present invention;

[0025] Figure 2 is the second flowchart of a foreign object fall detection method provided by the embodiments of the present invention;

[0026] Figure 3 is the structural schematic diagram of a foreign object fall detection device provided by the embodiments of the present invention;

[0027] Figure 4 is the structural schematic diagram of an electronic device for implementing the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solution in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0030] Figure 1 is the first flowchart of a foreign object falling detection method provided by an embodiment of the present invention. This embodiment is applicable to the case of foreign object falling detection based on an event camera. This method can be executed by a foreign object falling detection device, which can be implemented in the form of hardware and / or software, and the device can be configured in an electronic device with corresponding data processing capabilities. As Figure 1 shown, this method includes:

[0031] S110. Real-time detect a target area through an event camera to capture an event data stream; perform event frame compression on the event data in the current time window in space and determine a region of interest.

[0032] An event camera is a new type of visual sensor, also known as a dynamic vision sensor (DVS) or DAVIS (Dynamic and Active-Pixel Vision Sensor). Compared with a traditional camera, the event camera has significant differences in the data acquisition method. The event camera does not capture images at a fixed frame rate, but detects the brightness changes in the scene and records these changes as "events". Each event contains a timestamp, pixel coordinates, and a polarity indicating the brightness change.

[0033] The working principle of an event camera is based on detecting changes in pixel brightness. When the change in pixel brightness exceeds a preset threshold, an event is generated. This mechanism enables the event camera to work in a high dynamic range while reducing the complexity and latency of data processing. The output of the event camera is an asynchronous event stream, and each event is independently marked with a timestamp, thus having extremely low latency. In addition, the event camera only responds to changing pixels, greatly reducing the output of invalid information and lowering the data bandwidth and processing requirements.

[0034] The target area is detected in real time by the event camera to capture the event data stream. Each event data includes spatial coordinates, polarity, and timestamp. The spatial coordinates are pixel coordinates, and the polarity is used to describe the direction of the change in pixel brightness. Specifically, the polarity represents whether the brightness changes from low to high or from high to low. In the output data of the event camera, each event will contain a polarity value. When the brightness increases, the polarity is usually marked as positive (e.g., 1 or +1), indicating a positive event. On the contrary, when the brightness decreases, the polarity is marked as negative (e.g., 0 or -1), indicating a negative event.

[0035] Within a preset time window (such as 50 milliseconds), the event data generated by the event camera is collected, preprocessed, the spatial coordinates and timestamps of the event data with positive polarity are selected, and the event data is organized into a three-dimensional array such as (x, y, t), only retaining the spatial and time information. The event data is normalized so that the data in each dimension satisfies zero mean and unit variance. The processed event data includes the first spatial dimension, the second spatial dimension, and the time dimension.

[0036] The event data is accumulated in the current time window to generate an event frame. The event data in the current time window is compressed spatially for the event frame, for example, reducing the data volume by aggregating adjacent events or reducing the resolution, thereby reducing the complexity of data processing and improving the processing efficiency. Image processing techniques such as deep learning can be used to automatically identify the region of interest in the event frame. The region of interest is the area where events are concentrated. Optionally, the event data in the current time window can also be clustered according to the first spatial dimension and the second spatial dimension, and the area within the preset range of the cluster center is used as the region of interest.

[0037] Furthermore, the distance between each event data in the region of interest and the center point of the region of interest is determined according to the second spatial dimension and the time dimension, and the event data with a distance exceeding the preset distance is excluded.

[0038] By determining the region of interest, subsequent processing can focus on the data within the region of interest, reducing unnecessary global processing, improving the overall processing efficiency, reducing the consumption of computing resources, and enhancing the real-time performance of foreign object fall detection. Excluding isolated points in the region of interest can more accurately focus on key information and improve the accuracy of foreign object fall detection.

[0039] S120. Perform principal component analysis on the event data in the region of interest, and reconstruct the event data in the current time window according to the results of the principal component analysis.

[0040] Calculate the covariance matrix based on the event data in the region of interest, perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues and eigenvectors; the eigenvalues represent the importance of each principal component (i.e., the variance size), and the eigenvectors represent the directions of the principal components. Determine the eigenvectors corresponding to the eigenvalues as the first principal component, the second principal component, and the third principal component in order according to the magnitudes of the eigenvalues, and the eigenvector corresponding to the largest eigenvalue is the first principal component. Reconstruct the event data in the current time window in the principal component space. Determine the principal component space based on the event data in the region of interest, and reconstruct the event data in the current time window in this principal component space. By mapping the event data in the current time window to the new three-dimensional space after principal component analysis, the key features of the data can be efficiently extracted, and the main change trends of the data can be highlighted in the new space.

[0041] S130. If it is determined that the object in the current time window meets the fall trigger condition according to the reconstructed event data and the results of the principal component analysis in the current time window, then determine the current image position of the object according to the reconstructed event data in the current time window.

[0042] Considering that within a very short time, the motion trajectory of the falling object presents a linear mathematical model, that is, a straight line. Therefore, its principal components show a phenomenon of one large and two small; at the same time, its displacement in the horizontal direction is small, and it shows a phenomenon of rapidly increasing in the vertical direction with time, so the angles of its eigenvalues also show a phenomenon of one large and two small. Specific thresholds can be determined through a large number of experiments to obtain the fall trigger condition.

[0043] If it is confirmed that the object in the current time window meets the preset fall trigger condition according to the eigenvalues in the results of the principal component analysis of the event data in the region of interest and the reconstructed event data in the current time window, then determine the current image position of the object according to the event data coordinates at the edge position in the reconstructed event data in the current time window.

[0044] S140. Determine the historical image position of the object according to the reconstructed event data in the historical time window.

[0045] Optionally, the historical time window is the nearest time window that satisfies the fall trigger condition before the current time window within a preset time range.

[0046] Within a preset time range, for example, within 1 second, the nearest time window before the current time window that satisfies the fall trigger condition is used as the historical time window, and the historical image position of the object is determined according to the event data reconstructed in the historical time window.

[0047] S150. Determine whether an object has fallen in the target area according to the current image position and the historical image position.

[0048] Compare and analyze the current image position of the object in the target area with the historical image position of the object in the target area, and judge whether the object has fallen in the target area based on the change in the object position. Among them, the image position is the edge position of the object in the target area. If the current image position in the target area is vertically aligned within a certain error range with the historical image position after precise comparison, and the current image position has moved downward relative to the historical image position, it indicates that an object has fallen in the target area. Relying on the precise measurement and analysis of the change in the object position, the phenomenon of object fall can be reliably identified and confirmed.

[0049] In the embodiment of the present invention, by determining the region of interest in the target area and reconstructing the event data in the time window based on the principal component analysis result of the event data in the region of interest, unnecessary global processing is reduced, the overall processing efficiency is improved, the consumption of computing resources is reduced, and the real-time performance of foreign object fall detection is improved. Excluding the isolated points in the region of interest can focus more accurately on the key information and improve the accuracy of foreign object fall detection. And it can track the movement direction of the object. If the object satisfies the fall trigger condition in the current time window, by accurately positioning the position of the object in the current and historical images, it can quickly and accurately judge whether an object has fallen in the target area, greatly improving the sensitivity and reliability of fall event detection. Applying it to the detection of foreign object fall in the refueling water pool can accurately identify the fall of a tiny object with a volume of only 1 cubic centimeter within a range of 5 meters, send an alarm signal, and mark the position of the fallen object in the picture at the same time. It can track the movement trajectory of the target object in real time, and effectively track the movement state of the target by detecting the edge position and movement direction of the object.

[0050] In an alternative embodiment, the determining whether an object has fallen in the target area according to the current image position and the historical image position includes: if the current image position overlaps with the historical image position in the first direction, and the current image position satisfies an increase between the historical image position in the second direction, it is determined that a foreign object fall event has occurred in the target area.

[0051] Specifically, the second direction is the forward movement direction of the object. According to the minimum position coordinate and the maximum position coordinate of the current image position in the first direction, the current occupancy interval of the current image position in the first direction is determined. Similarly, according to the minimum position coordinate and the maximum position coordinate of the historical image position in the first direction, the historical occupancy interval of the historical image position in the first direction is determined. If the overlap degree between the current occupancy interval and the historical occupancy interval is greater than a preset value, there is an overlap between the current image position in the first direction and the historical image position in the first direction. If the minimum position coordinate of the current image position in the second direction is greater than the minimum position coordinate of the historical image position in the second direction, and the maximum position coordinate of the current image position in the second direction is greater than the maximum position coordinate of the historical image position in the second direction, then there is an increase between the current image position in the second direction and the historical image position in the second direction. When there is an overlap between the current image position in the first direction and the historical image position in the first direction, and there is an increase between the current image position in the second direction and the historical image position in the second direction, it is determined that a foreign object falling event has occurred in the target area.

[0052] By accurately judging whether there is a foreign object falling event in the target area according to the occupancy intervals and relative changes of the current image position and the historical image position in the first and second directions, the accuracy of event detection is improved, the movement trend of the object is effectively identified, a more reliable decision-making basis is provided for the real-time monitoring and warning system, and it helps to respond to and handle potential safety hazards in a timely manner.

[0053] Figure 2 It is the second flowchart of a foreign object falling detection method provided by an embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. As Figure 2 shown, the method includes:

[0054] S210. Perform real-time detection on the target area through an event camera to capture an event data stream; perform event frame compression on the event data in the current time window in space and determine the region of interest.

[0055] In an alternative embodiment, the performing event frame compression on the event data in the current time window in space and determining the region of interest includes: performing spatial compression on the event data in the current time window to obtain an event data set; using the coordinate with the largest number of event points in the event data set as the center point, and using the region within a preset range of the center point as the region of interest.

[0056] Specifically, event data is accumulated in the current time window to generate an event frame. In the event frame, each spatial position, i.e., pixel position, corresponds to an event counter for recording the number of events occurring at that position within the current time window. The event frame is compressed to reduce the data volume and improve processing efficiency. The compression methods include: removing redundant information, merging adjacent events, reducing resolution, etc. The position with the largest number of events in the compressed event frame is used as the central position of the region of interest, and the region within a preset range of the central position is used as the region of interest.

[0057] By compressing the event frame and determining the region of interest, the data volume for subsequent processing can be significantly reduced, thereby improving the processing efficiency and enhancing the real-time performance of foreign object fall detection. Determining the region of interest helps to remove background noise and irrelevant information, enhancing the robustness and accuracy of the method.

[0058] S220. Perform principal component analysis on the event data in the region of interest, and reconstruct the event data in the current time window according to the results of the principal component analysis.

[0059] Specifically, after discarding the points with large errors, the event data is reconstructed through inverse operation.

[0060] S230. Determine the relative eigenvalue according to the largest eigenvalue and the second largest eigenvalue in the results of the principal component analysis; determine the movement direction of the object in the current time window according to the reconstructed event data.

[0061] The difference between the largest eigenvalue and the second largest eigenvalue in the results of the principal component analysis is used as the relative eigenvalue; determine the movement direction of the object in the current time window according to the reconstructed event data. Optionally, the determining the movement direction of the object in the current time window according to the reconstructed event data includes: performing linear fitting on the reconstructed event data using the least squares method to obtain a linear equation, and determining the movement direction of the object in the current time window according to the slope of the linear equation. Since the movement in three-dimensional space involves multiple directions, linear fitting is performed on each direction separately to obtain a linear equation, and the slope of the linear equation is the movement direction of the object. If the slope of the linear equation obtained by fitting any direction is greater than zero, it means that the object is moving forward in that direction.

[0062] S240. If the relative eigenvalue is greater than the preset threshold and the object is moving forward in the movement direction, then the object in the current time window satisfies the fall trigger condition.

[0063] The relative eigenvalue reflects the degree of dispersion of event data in the principal component direction. When the relative eigenvalue is large, it indicates that the change of event data in the first principal component direction is relatively significant, and the main motion trend of the object can be identified. By accurately using the maximum eigenvalue and the second largest eigenvalue of the principal component analysis to determine the relative eigenvalue, and combining the reconstructed event data to efficiently identify the object falling state and accurately judge the object motion direction. Once a moving object appears in the target area, the falling warning mechanism can be triggered immediately, which not only ensures the accuracy of the judgment but also greatly improves the response speed.

[0064] S250. If the object in the current time window meets the falling trigger condition, determine the current image position of the object in the current image according to the reconstructed event data in the current time window.

[0065] S260. Determine the historical image position of the object according to the reconstructed event data in the historical time window.

[0066] S270. Judge whether an object has fallen in the target area according to the current image position and the historical image position.

[0067] When the object in the current time window meets the falling trigger condition, by comparing the image positions of the object in the current and historical time windows, it is possible to quickly and accurately determine whether an object falling event has actually occurred in the target area. Significantly improve the sensitivity and accuracy of falling event detection, determine the object falling direction, track the object falling trajectory, and show great advantages and application value in safety monitoring and accident prevention.

[0068] In the embodiments of the present invention, the maximum eigenvalue and the second largest eigenvalue of the principal component analysis are accurately used to determine the relative eigenvalue, and combined with the reconstructed event data to efficiently identify the object falling state and accurately judge the object motion direction. Once a moving object appears in the target area, the falling warning mechanism can be triggered immediately, which not only ensures the accuracy of the judgment but also greatly improves the response speed. When the object in the current time window meets the falling trigger condition, by comparing the image positions of the object in the current and historical time windows, it is possible to quickly and accurately determine whether an object falling event has actually occurred in the target area. Significantly improve the sensitivity and accuracy of falling event detection, determine the object falling direction, track the object falling trajectory, and show great advantages and application value in safety monitoring and accident prevention.

[0069] Figure 3 It is a schematic structural diagram of a foreign object falling detection device provided by an embodiment of the present invention. As Figure 3 shown, the device includes:

[0070] An event data acquisition module 310, configured to perform real-time detection on a target area through an event camera and capture an event data stream;

[0071] An interested region determination module 320, configured to perform event frame compression on event data in a current time window spatially and determine an interested region;

[0072] A data reconstruction module 330, configured to perform principal component analysis on event data in the interested region, and reconstruct event data in the current time window according to the principal component analysis result;

[0073] An object falling judgment module 340, configured to, if it is determined that an object in the current time window meets a falling trigger condition according to the reconstructed event data and the principal component analysis result in the current time window, determine the current image position of the object in the current time window according to the reconstructed event data in the current time window; determine the historical image position of the object according to the reconstructed event data in a historical time window; and determine whether an object has fallen in the target region according to the current image position and the historical image position.

[0074] The foreign object falling detection device provided by an embodiment of the present invention can execute the foreign object falling detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0075] Optionally, the object falling judgment module includes: a falling trigger condition unit, configured to determine a relative eigenvalue according to the maximum eigenvalue and the second largest eigenvalue in the principal component analysis result; determine the movement direction of the object in the current time window according to the reconstructed event data; and if the relative eigenvalue is greater than a preset threshold and the object moves forward along the movement direction, the object in the current time window meets the falling trigger condition.

[0076] Optionally, the falling trigger condition unit includes: a movement direction determination sub-unit, configured to perform linear fitting on the reconstructed event data by using the least square method to obtain a linear equation, and determine the movement direction of the object in the current time window according to the slope of the linear equation.

[0077] Optionally, the object falling judgment module includes: an object falling judgment unit, configured to determine that a foreign object falling event has occurred in the target region if the current image position overlaps with the historical image position in a first direction and the current image position increases between the historical image position in a second direction in the second direction.

[0078] Optionally, the historical time window is the nearest time window that meets the falling trigger condition before the current time window within a preset time range.

[0079] Optionally, the region of interest determination module includes a first unit for determining the region of interest, which spatially compresses the event data in the current time window to obtain an event data set; and uses the coordinate with the largest number of event points in the event data set as the center point, and takes the region within the preset range of the center point as the region of interest.

[0080] The foreign object falling detection device described in further detail can also execute the foreign object falling detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0081] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0082] Figure 4 FIG. shows a schematic structural diagram of an electronic device 40 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0083] As Figure 4 shown, the electronic device 40 includes at least one processor 41, and a memory communicatively connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc. The memory stores a computer program executable by the at least one processor. The processor 41 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. The input / output (I / O) interface 45 is also connected to the bus 44.

[0084] Multiple components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disc, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0085] The processor 41 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the foreign object fall detection method.

[0086] In some embodiments, the foreign object fall detection method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the foreign object fall detection method described above can be executed. Alternatively, in other embodiments, the processor 41 can be configured to execute the foreign object fall detection method by any other suitable means (e.g., by means of firmware).

[0087] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0088] A computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0089] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0091] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected with each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0092] A computing system can include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0093] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0094] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for detecting a falling foreign object, characterized in that: The method comprises: Real-time detection of target areas is performed through event cameras to capture event data streams; Compress the event data in the current time window spatially into event frames and determine the region of interest; Performing principal component analysis on the event data in the region of interest, and reconstructing the event data in the current time window according to the principal component analysis result; If it is determined that the object in the current time window meets the fall triggering condition according to the event data reconstructed in the current time window and the principal component analysis result, the current image position of the object is determined according to the event data reconstructed in the current time window; Determine the historical image location of the object based on the event data reconstructed in the historical time window; Whether an object falls in the target area is determined according to the current image position and the historical image position.

2. The method according to claim 1, characterized in that The determining, based on the event data reconstructed in the current time window and the principal component analysis result, that the object in the current time window meets the fall triggering condition includes: Determine the relative eigenvalue according to the largest eigenvalue and the second largest eigenvalue in the principal component analysis result; Determine the direction of motion of the object in the current time window based on the reconstructed event data; If the relative characteristic value is greater than a preset threshold and the object is moving forward along the moving direction, the object in the current time window meets the falling trigger condition.

3. The method according to claim 2, characterized in that Determining the moving direction of the object in the current time window according to the reconstructed event data includes: The least square method is used to perform linear fitting on the reconstructed event data to obtain a linear equation, and the movement direction of the object in the current time window is determined according to the slope of the linear equation.

4. The method according to claim 1, characterized in that: The determining whether an object falls in the target area according to the current image position and the historical image position includes: If the current image position overlaps with the historical image position in the first direction, and the current image position and the historical image position in the second direction satisfy an increasing relationship, it is determined that a foreign object falling event occurs in the target area.

5. The method according to claim 1, characterized in that The historical time window is the most recent time window that meets the fall trigger condition before the current time window within a preset time range.

6. The method according to claim 1, characterized in that The spatially compressing the event frame of the event data in the current time window and determining the region of interest includes: Perform spatial compression on the event data in the current time window to obtain an event data set; The coordinates with the largest number of event points in the event dataset are taken as the center point, and the area within the preset range of the center point is taken as the area of ​​interest.

7. A foreign body falling detection device, characterized in that: The device comprises: An event data acquisition module is used to detect the target area in real time through an event camera and capture the event data stream; An area of ​​interest determination module, used for spatially compressing event frames of event data in a current time window and determining an area of ​​interest; A data reconstruction module, used to perform principal component analysis on the event data in the region of interest, and reconstruct the event data in the current time window according to the principal component analysis result; The object falling judgment module is used to determine the current image position of the object according to the event data reconstructed in the current time window; determine the historical image position of the object according to the event data reconstructed in the historical time window; and judge whether an object falls in the target area according to the current image position and the historical image position, if it is determined that the object in the current time window meets the falling trigger condition according to the event data reconstructed in the current time window and the principal component analysis result.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the foreign object falling detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the foreign object falling detection method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for detecting a falling foreign object according to any one of claims 1 to 6 is implemented.

Images