A high-altitude falling object detection method and system based on machine vision

Abstract

The present application relates to the field of image processing, and particularly to a high-altitude falling object detection method and system based on machine vision. The high-altitude falling object detection method based on machine vision provided by the present application comprises the following steps: determining a visual area and acquiring a visual picture of the visual area, wherein the visual picture is captured by at least one imaging device; identifying a first falling object in the visual picture and evaluating the risk degree of the first falling object; identifying a second falling object in the visual picture according to the risk degree of the first falling object, wherein the risk degree of the second falling object exceeds a risk degree threshold; predicting a falling point range of the second falling object, and triggering a high-altitude falling object warning according to the number of personnel in the falling point range. The present application can monitor and warn high-altitude falling objects in complex environments such as construction sites, thereby effectively improving personnel and property safety protection, reducing the occurrence of safety accidents, and making up for the shortcomings of traditional safety measures.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a high-altitude falling object detection method and system based on machine vision. BACKGROUND

[0002] In the field of construction engineering, high-altitude falling objects are one of the common safety hazards on construction sites, which can cause serious personal injury and property loss. Due to the complex and variable environment of construction sites, frequent and uncertain high-altitude operations, traditional safety measures can provide some degree of protection, but there are many limitations in practical application, such as physical barriers that cannot cover all risk areas, personnel monitoring limited by visual angle, reaction time, and fatigue. SUMMARY

[0003] The present application provides a high-altitude falling object detection method and system based on machine vision, aiming to solve the shortcomings of traditional safety measures in the field of construction engineering and improve the detection and early warning capabilities of high-altitude falling objects.

[0004] In a first aspect, the present application provides a high-altitude falling object detection method based on machine vision, comprising the following steps:

[0005] Determine the visual area and obtain the visual picture of the visual area, which is captured by at least one imaging device;

[0006] Identify the first falling object in the visual picture and evaluate the risk level of the first falling object;

[0007] According to the risk level of the first falling object, identify the second falling object in the visual picture, and the risk level of the second falling object exceeds the risk level threshold;

[0008] Predict the landing point range of the second falling object, and trigger a high-altitude falling object warning according to the number of personnel in the landing point range.

[0009] Further, the step of obtaining the visual picture of the visual area comprises the following steps:

[0010] Divide the visual area to obtain a plurality of visual sub-areas, and the projections of the plurality of visual sub-areas on the vertical plane completely cover the projection of the falling object space on the vertical plane;

[0011] Correspond one-to-one between the plurality of imaging devices and the plurality of visual sub-areas, and respectively use the imaging devices to obtain the visual sub-picture of the corresponding visual sub-area, and the projection of any visual sub-picture on the vertical plane covers part of the projection of the falling object space on the vertical plane.

[0012] Further, the step of identifying the first falling object in the visual picture comprises the following steps:

[0013] identifying the first falling object in each visual sub-picture respectively, wherein the identification of the first falling object in any visual sub-picture comprises the following steps:

[0014] obtaining a first visual sub-picture, the first visual sub-picture being a previous visual sub-picture in two adjacent visual sub-pictures;

[0015] obtaining a second visual sub-picture, the second visual sub-picture being a next visual sub-picture in the two adjacent visual sub-pictures;

[0016] obtaining a difference image based on the pixel distribution of the first visual sub-picture and the second visual sub-picture, wherein a pixel value at any position in the difference image is a pixel difference value at the corresponding position between the first visual sub-picture and the second visual sub-picture;

[0017] dividing a motion region and a static region in the visual sub-picture according to the pixel distribution in the binarized difference image, and marking the first falling object based on the edge of the motion region.

[0018] Further, the marking of the first falling object based on the edge of the motion region comprises the following steps:

[0019] extracting an edge pixel in the difference image, and generating a minimum enclosing rectangle based on the edge pixel, wherein the minimum enclosing rectangle is used to mark the corresponding first falling object.

[0020] Further, the first visual sub-picture, the second visual sub-picture, and the third visual sub-picture are obtained by the same imaging device.

[0021] Further, the frame number corresponding to the second visual sub-picture is continuous with the frame number corresponding to the third visual sub-picture.

[0022] Further, the triggering of the high-altitude falling object warning according to the number of personnel in the landing point range comprises the following triggering rules:

[0023] When the number of personnel in the landing point range is greater than zero, the high-altitude falling object warning is triggered.

[0024] In a second aspect, the present application provides a high-altitude falling object detection system based on machine vision, comprising an input device, a processor, a memory and an output device, wherein the input device, the processor, the memory and the output device are connected to each other, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the high-altitude falling object detection method based on machine vision in the first aspect.

[0025] The machine vision-based high-altitude falling object detection method and system provided by the application has the advantages of:

[0026] The application realizes automatic identification and tracking of high-altitude falling objects through machine vision technology, can predict the falling point range, and can timely warn according to the personnel distribution in the falling point area, thereby improving the safety management efficiency of the construction site and reducing the risk of personnel injury and property loss. Compared with traditional safety measures, the application can monitor and warn high-altitude falling objects in complex environments such as construction sites, thereby effectively improving personnel and property safety protection, reducing the occurrence of safety accidents, and making up for the shortcomings of traditional safety measures. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The machine vision-based high-altitude falling object detection method flowchart provided by the embodiment of the application;

[0028] Figure 2 The visual area division schematic diagram provided by the embodiment of the application;

[0029] Figure 3 The machine vision-based high-altitude falling object detection system schematic diagram provided by the embodiment of the application. DETAILED DESCRIPTION

[0030] In the following description, specific details are set forth such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the application. However, techniques and / or apparatuses have been described and / or claimed herein with or without these specific details.

[0031] However, those skilled in the art should understand that the application can be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details that hinder the description of the application.

[0032] It should be understood that when used in the specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or collections.

[0033] For the sake of simplicity of the drawings, only the parts related to the application are shown in the drawings, which do not represent the actual structure of the product.

[0034] In addition, in some drawings, only one of the components with response structure or function is shown schematically, or only one of them is marked; in this document, "one" not only means "only one", but also means "more than one" situation.

[0035] It should also be understood that the term "and / or" as used herein refers to a combination of one or more of the associated listed items, and all possible combinations, and includes these combinations.

[0036] In the embodiments shown in the drawings, the indications of directions, such as up, down, left, right, front and back, are used to explain the structure and movement of various components of the present application, and are not absolute but relative. These descriptions are appropriate when the components are in the positions shown in the drawings. If the positions of the components change, the indications of the directions also change accordingly.

[0037] In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0038] In order to more clearly illustrate the embodiments of the present application or the prior art, the specific embodiments of the present application will be described below with reference to the drawings. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without creative labor, and other embodiments can also be obtained.

[0039] In one embodiment, please refer to Figure 1 , Figure 1 The flow chart of the high-altitude falling object detection method based on machine vision provided by the embodiments of the present application.

[0040] As Figure 1 shown, the high-altitude falling object detection method based on machine vision provided by the embodiments of the present application includes the following steps:

[0041] S01, determine the visual area, and acquire the visual picture of the visual area, the visual picture is captured by at least one imaging device.

[0042] In this embodiment, the visual area refers to the spatial range for high-altitude falling object detection by machine vision in the construction site, which covers at least the area with high-altitude falling object risk.

[0043] It can be understood that the spatial range of the visual area needs to be determined according to the actual situation of the construction site to ensure the all-around monitoring of the high-altitude falling object.

[0044] Further, to realize the comprehensive monitoring of the area with high-altitude falling object risk in the visual area, the acquisition of the visual picture of the visual area in step S01 includes the following steps:

[0045] S011. Divide the visual area to obtain multiple visual sub-areas, where projections of the multiple visual sub-areas on a vertical plane completely cover the projection of the falling object space on the vertical plane.

[0046] It should be noted that the vertical plane described in this embodiment refers to a two-dimensional plane perpendicular to the horizontal ground, which is used to represent the vertical movement projection of falling objects from high altitude in three-dimensional space, such as a vertical plane with the outer wall of a high-rise building as a reference in a high-rise construction project.

[0047] It should also be noted that the falling object space described in this embodiment refers to the three-dimensional spatial area within the construction site that falling objects from high altitudes may pass through or fall into. It indicates the range that potential falling objects from high altitudes may occupy during the falling process and is the key area that the visual picture needs to cover.

[0048] S012. Match the multiple imaging devices to the multiple visual sub-areas one by one, and use the imaging devices to obtain visual sub-pictures of the corresponding visual sub-areas respectively, and the projection of any visual sub-picture on the vertical plane covers the partial projection of the falling object space on the vertical plane.

[0049] In this example, see Figure 2 , Figure 2 A schematic diagram of visual area division provided by an embodiment of the present invention. Figure 2 The solid rectangle in the middle represents the building, and the black dots represent the imaging equipment installed on the exterior wall of the building. The visible range on the side of the exterior wall where the imaging equipment is installed belongs to the visual area of ​​this example. The rectangular area framed by the dotted lines and the solid lines of the exterior wall represents the falling object space. The spatial coordinate system XYZ is constructed based on the layout position of the imaging equipment O, with its Y axis parallel to the direction of gravity, and the XOZ plane perpendicular to the Y axis. It is usually used to represent the horizontal ground in actual applications.

[0050] like Figure 2 As shown, the projection of the visual area corresponding to the building exterior wall on the vertical plane XOZ can completely cover the projection of the falling object space on the vertical plane XOZ; further, the visual picture corresponding to the visual area can be divided into multiple visual sub-pictures, and any visual sub-picture can cover the partial projection of the falling object space on the vertical plane.

[0051] It should be noted that for a visual area, each imaging device deployed to capture its visual images has a fixed shooting angle during the process of capturing the corresponding visual images. It should also be noted that for two visual areas, the number, position, shooting angle, shooting focal length and other parameters of the imaging devices deployed to capture the corresponding visual images may be the same or different. The specific parameters need to be determined based on the actual situation at the construction site to ensure all-round monitoring of falling objects from high altitude.

[0052] S02, identify the first falling object in the visual picture, and evaluate the risk level of the first falling object.

[0053] In this embodiment, the first falling object in step S02 refers to an object with obvious falling characteristics that is first detected in the visual picture in the high-altitude environment of the construction site, which usually starts from a high place such as a scaffold, a basket, a floor edge, a tower crane arm, etc.

[0054] Further, the step of identifying the first falling object in the visual picture in step S02 includes the following steps: identifying the first falling object in each visual sub-picture respectively.

[0055] Further, the identification of the first falling object in any visual sub-picture includes the following steps:

[0056] S021, obtain a first visual sub-picture, which is a previous visual sub-picture in two adjacent visual sub-pictures.

[0057] S022, obtain a second visual sub-picture, which is a next visual sub-picture in two adjacent visual sub-pictures.

[0058] It should be noted that the next visual sub-picture mentioned here is the current visual sub-picture.

[0059] S023, obtain a difference image through the pixel distribution of the first visual sub-picture and the second visual sub-picture, wherein the pixel value of any position in the difference image is the pixel difference value of the corresponding position of the first visual sub-picture and the second visual sub-picture.

[0060] It should be noted that for the convenience of data calculation, the pixel difference value calculation of the difference image in step S023 is based on the pixel value distribution of the first visual sub-picture and the second visual sub-picture after grayscale.

[0061] S024, divide the motion area and the static area in the visual sub-picture according to the pixel distribution in the binary difference image, and mark the first falling object based on the edge of the motion area.

[0062] In this embodiment, the dynamic area refers to an area where the pixel value changes significantly in two consecutive visual sub-pictures, which is formed by the moving track of the first falling object; the static area refers to an area where the pixel value is basically unchanged in two consecutive visual sub-pictures, such as the background environment.

[0063] Further, in the step of binarizing the difference image in step S024, the binarization grayscale threshold is set according to the specific scene.

[0064] Further, the marking the first falling object based on the edge of the motion region in step S024 comprises the following steps:

[0065] The edge pixels in the difference image are extracted by using an edge detection algorithm, and a minimum enclosing rectangle is generated based on the edge pixels, the minimum enclosing rectangle being used to mark the first falling object.

[0066] In the embodiment, the risk degree of the first falling object in step S02 is a preliminary assessment of the potential threat degree caused by the first falling object falling to the ground to the on-site personnel based on the position feature, two-dimensional feature and motion feature of the first falling object.

[0067] Specifically, the risk degree of any first falling object satisfies the following assessment model: wherein E represents the risk degree, S c represents the minimum enclosing rectangle area of the first falling object, Ah represents the displacement amount of the center of the minimum enclosing rectangle of the first falling object in the vertical direction between the second visual sub-picture and the third visual sub-picture, At represents the recording time difference between the second visual sub-picture and the third visual sub-picture, g represents the gravity acceleration, and H represents the height of the imaging device and the horizontal ground when the second visual sub-picture and the third visual sub-picture are collected.

[0068] Further, Ah = y a -y b wherein y a represents the center point height of the minimum enclosing rectangle of the first falling object in the vertical direction in the second visual sub-picture, and y b represents the center point height of the minimum enclosing rectangle of the first falling object in the vertical direction in the third visual sub-picture.

[0069] It is noted that the third visual sub-picture is another frame of visual sub-picture collected by the corresponding imaging device after the second visual sub-picture marking the first falling object, and the frame number thereof can be continuous or discontinuous with the frame number of the second visual sub-picture.

[0070] For example, if the second visual sub-picture is counted as the second frame of visual sub-picture, the third visual sub-picture can be any one of the third frame, the fourth frame,..., and the nth frame of visual sub-picture.

[0071] It is further noted that the first visual sub-picture, the second visual sub-picture and the third visual sub-picture are obtained by the same imaging device.

[0072] S03, identifying the second falling object in the visual picture according to the risk degree of the first falling object, the risk degree of the second falling object exceeding a risk degree threshold.

[0073] In the embodiment, the second falling object refers to a high-risk falling object selected from the multiple first falling objects through a screening mechanism after the multiple first falling objects are preliminarily detected.

[0074] Further, the risk degree threshold is a critical value for screening the high-risk falling object. The multiple target detection results are filtered effectively to eliminate other irrelevant image interference, so as to ensure the accuracy and effectiveness of the multiple target detection, such as the first falling object with a minimum enclosing rectangle area and in a motion state but not continuously falling. Specifically, the risk degree threshold can be set by a fixed, dynamic or statistical method to ensure the accuracy and effectiveness of the multiple target detection.

[0075] S04, predicting a falling point range of the second falling object, and triggering a high-altitude falling object warning according to the number of personnel in the falling point range.

[0076] In the embodiment, the falling point range refers to a range of an area where the second falling object may finally contact the ground or a building structure during falling, which is usually represented as a circular, elliptical or rectangular area on a horizontal ground surface perpendicular to the vertical plane, for covering all possible landing positions of the falling object.

[0077] Further, the falling point range S E satisfies the following model:

[0078] (x-x0) 2 +(z-z0) 2 ≤R 2 , wherein, (x, z) represents a point in the falling point range S E , (x0, z0) represents a projection point of a center of a minimum enclosing rectangle of the second falling object on a horizontal ground surface perpendicular to the vertical plane, the origin of the horizontal ground surface is a projection point of a layout position of an imaging device of the second visual sub-picture on the horizontal ground surface, and R is a risk radius.

[0079] Further, the risk radius R can be fixedly set by an empirical value or a safety specification, such as a safety boundary of 1-2 meters usually set in a construction site as the risk radius.

[0080] In the embodiment, the triggering of the high-altitude falling object warning according to the number of personnel in the falling point range in step S04 includes the following triggering rule: the high-altitude falling object warning is triggered when the number of personnel in the falling point range is greater than zero.

[0081] Further, the personnel in the falling point range can be counted by a machine vision algorithm (such as YOLO), and the personnel in the falling point range can be notified by a visual, sound or communication method to remind the personnel in the falling point range to pay attention and quickly take protective measures.

[0082] In one embodiment, please refer to Figure 3 , Figure 3 a schematic diagram of a high-altitude falling object detection system based on machine vision provided by the embodiments of the present application.

[0083] As Figure 3 shown, the high-altitude falling object detection system based on machine vision provided by the present application comprises an input device, a processor, a memory and an output device, which are connected to each other.

[0084] Further, the input device of the present embodiment is used to acquire visual pictures of the visual area and transmit these pictures to the processor for further processing.

[0085] Generally, the input device is an industrial camera or other types of imaging devices; in some embodiments, it can include multiple industrial cameras to cover a large range of monitoring areas.

[0086] Further, the processor of the present embodiment is the computing and analysis core of the system, which is used to perform image processing, target detection, risk assessment and early warning triggering, etc. It performs the detection method of high-altitude falling objects by calling program instructions in the memory.

[0087] It can be understood that the processor needs to have high-performance computing capability to process a large amount of image data acquired by the input device in real time; in some embodiments, in order to reduce the computing pressure of the central processor, some edge processors are also provided to preliminarily process the image data.

[0088] Further, the memory is used to store computer programs, which include program instructions, and the processor is configured to call the program instructions to execute the detection method of high-altitude falling objects based on machine vision provided by the present application.

[0089] Specifically, the memory includes short-term storage (such as RAM) and long-term storage (such as SSD); in some embodiments, the memory also includes cloud storage, which can synchronize a large amount of image data to the cloud for storage.

[0090] Further, the output device is responsible for transmitting the detection results, risk assessment and early warning signals to relevant personnel or systems to realize the visualization of information and the communication of early warning.

[0091] In the present embodiment, the output device supports multiple early warning methods, including visual output, sound output and communication output. Specifically, the output device includes a display screen, a loudspeaker, a mobile terminal (such as a smart phone, a smart watch, etc.) which is signal connected to the above-mentioned processor through a wireless network (such as Wi-Fi, 4G / 5G).

[0092] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0093] It should be noted that the above embodiments can be freely combined as needed. The above is only the preferred embodiment of the present application; it should be noted that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can also be made, which should be considered as the protection scope of the present application.

Claims

1. A machine vision-based high altitude falling object detection method, characterized by, The method comprises the following steps: determining a visual area and obtaining a visual picture of the visual area, the visual picture being captured by at least one imaging device; identifying a first falling object in the visual picture and evaluating a risk level of the first falling object; identifying a second falling object in the visual picture according to the risk level of the first falling object, the risk level of the second falling object exceeding a risk level threshold; predicting a falling point range of the second falling object and triggering a high-altitude falling object warning according to a number of personnel in the falling point range; the risk level is a preliminary evaluation of a potential threat level caused by the first falling object falling to the ground to on-site personnel based on position characteristics, two-dimensional characteristics and motion characteristics of the first falling object; the risk level of any first falling object satisfies the following evaluation model: ， wherein E represents the risk degree, S c represents the minimum enclosing rectangular area of the first falling object, Δh represents the displacement amount of the center of the minimum enclosing rectangle of the first falling object in the vertical direction between the second visual sub-picture and the third visual sub-picture, Δt represents the recording time difference between the second visual sub-picture and the third visual sub-picture, g represents the acceleration of gravity, and H represents the height of the imaging device collecting the second visual sub-picture and the third visual sub-picture from the horizontal ground. Δh = ya - yb, wherein ya represents a center point height of a minimum enclosing rectangle of the first falling object in a second visual sub-picture in a vertical direction, and yb represents a center point height of a minimum enclosing rectangle of the first falling object in a third visual sub-picture in the vertical direction; the obtaining of the visual picture of the visual area comprises the following steps: dividing the visual area to obtain a plurality of visual sub-areas, projections of the plurality of visual sub-areas on a vertical plane completely covering a projection of a falling object space on the vertical plane; corresponding the plurality of imaging devices to the plurality of visual sub-areas one by one, and respectively obtaining visual sub-pictures of the corresponding visual sub-areas by using the imaging devices, a projection of any visual sub-picture on the vertical plane covering a part of a projection of the falling object space on the vertical plane; constructing a spatial coordinate system XYZ based on the arrangement positions of the imaging devices, the Y axis being parallel to the direction of gravity, and the XOZ plane being perpendicular to the Y axis; the falling point range SE of the second falling object satisfies the following model: (x - x0) 2 +(z - z0) 2 ≤ R 2 , wherein (x, z) represents a point in the falling point range SE, (x0, z0) represents a projection point of a center of a minimum enclosing rectangle of the second falling object on a horizontal ground surface perpendicular to the vertical plane, the origin of the horizontal ground surface being a projection point of an arrangement position of an imaging device of the second visual sub-picture on the horizontal ground surface, and R being a risk radius; for a visual area, the shooting angle of each imaging device arranged to obtain a visual picture thereof is fixed during the obtaining of the corresponding visual picture.

2. The machine vision-based high-object-fall detection method according to claim 1, wherein, the identifying of the first falling object in the visual picture comprises the following steps: respectively identifying the first falling object in each visual sub-picture, wherein the identification of the first falling object in any visual sub-picture comprises the following steps: obtaining a first visual sub-picture, the first visual sub-picture being a previous frame visual sub-picture in adjacent two frame visual sub-pictures. 3.The machine vision-based high-falling object detection method according to claim 2, wherein, the identifying of the first falling object in the visual picture comprises the following steps: obtaining a second visual sub-picture, the second visual sub-picture being a next frame visual sub-picture in adjacent two frame visual sub-pictures; obtaining a difference image through pixel distribution of the first visual sub-picture and the second visual sub-picture, a pixel value of any position in the difference image being a pixel difference value of the first visual sub-picture and the second visual sub-picture at the corresponding position; According to the pixel distribution in the difference image after binarization, the motion area and the static area in the visual sub-picture are divided, and a first falling object is marked based on the edge of the motion area. 4.The machine vision based high-rise falling object detection method according to claim 3, wherein, The marking of the first falling object based on the edge of the motion area comprises the following steps: Edge pixels in the difference image are extracted, and a minimum enclosing rectangle is generated based on the edge pixels, the minimum enclosing rectangle being used to mark the corresponding first falling object. 5.The machine vision-based high-falling object detection method according to claim 4, wherein, The first visual sub-picture, the second visual sub-picture and the third visual sub-picture are acquired by the same imaging device. 6.The machine vision-based high-falling object detection method according to claim 5, wherein, The frame number corresponding to the second visual sub-picture is continuous with the frame number corresponding to the third visual sub-picture.

7. The machine vision-based high-rise falling object detection method according to any one of claims 1-6, wherein, The triggering of the high-altitude falling object warning according to the number of personnel in the falling point range comprises the following triggering rules: When the number of personnel in the falling point range is greater than zero, the high-altitude falling object warning is triggered.

8. A machine vision based high altitude object detection system, characterized in that, The device comprises an input device, a processor, a memory and an output device, which are connected to each other, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to invoke the program instructions to execute the machine vision-based high-altitude falling object detection method according to any one of claims 1 to 7.

Images