System for detecting human-vehicle invasion and material accumulation at material port in coal warehouse area

Abstract

The invention relates to a system for detecting human-vehicle invasion and material accumulation at a material port in a coal warehouse area, and belongs to the technical field of intelligent traffic safety monitoring of a coal warehouse of an iron making plant. The system comprises a material port accumulated material detection module, a man-vehicle intrusion detection module, an alarm prompt module, a warehouse area camera and a material port camera. The material port accumulated material detection module is composed of a point cloud detection sub-module and an image detection sub-module; the man-vehicle intrusion detection module is composed of an alarm event sub-module and a grab bucket detection sub-module. The method is based on the intelligent driving equipment, and no extra hardware cost needs to be increased; the safety risk of automatic operation of the intelligent travelling crane can be effectively reduced, the attention anxiety of operators in a central control room is released, one person can supervise operation of multiple intelligent travelling cranes, and the labor cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent driving safety monitoring of coal bunkers in iron and steel plants, and in particular relates to a system for detecting human and vehicle intrusion into a coal bunker area and material accumulation at a material opening. Background Art

[0002] After the intelligent crane transformation of the coal depot, the driver's cab at the crane site is unmanned and is replaced by semi-automatic or fully automatic control from a remote central control room. Since it is difficult for remote operators to pay attention to video surveillance throughout the entire process, when a manned vehicle enters the depot area or the material port area during the automatic crane operation, an alarm needs to be issued in a timely manner to remind the operator to pay attention to the video surveillance and intervene in the route of the automatic crane operation according to the safety situation to avoid safety accidents.

[0003] When the material at the automatic feeding port of the driving crane accumulates to a certain amount, it is necessary to remind the operator to intervene in the driving crane to stop feeding and perform material port cleaning to avoid the material port being blocked due to overfilling of the silo. The intelligent driving crane can operate in the 1~3# storage area and the corresponding 1~3# material ports, and usually only operates in one storage area at the same time. (The material port of the 1# storage area is the 1# material port, the material port of the 2# storage area is the 2# material port, and the material port of the 3# storage area is the 3# material port).

[0004] At present, there are few technical means for automatic monitoring of intelligent driving safety, and even if there are, it is difficult to achieve comprehensive safety detection. The means of safety monitoring are single, or only image detection or only point cloud detection is used, and the detection reliability and robustness are not high. For example, for material accumulation detection at the material mouth, if only image detection is used, the position of the accumulated material is difficult to accurately locate, and if only point cloud detection is used, the data of the non-operating area of ​​intelligent driving cannot be updated in time.

[0005] Therefore, how to overcome the shortcomings of the existing technology is an urgent problem to be solved in the field of intelligent driving safety monitoring technology in coal storages of iron and steel plants. Summary of the invention

[0006] The purpose of the present invention is to solve the deficiencies of the prior art and to provide a system for detecting the intrusion of people and vehicles into a coal storage area and the accumulation of materials at a material opening.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A system and method for detecting human and vehicle intrusion and material accumulation at a coal storage area, comprising a material accumulation detection module at a material storage area, a human and vehicle intrusion detection module, an alarm prompt module, a storage area camera and a material storage area camera; The warehouse area camera is used to obtain images of the warehouse area and detect intrusion of personnel, trucks and forklifts in the warehouse area; The material inlet camera is used to obtain images of the material inlet and detect intrusion of personnel in the material inlet area; The material inlet accumulation detection module is composed of a point cloud detection submodule and an image detection submodule; The point cloud detection submodule is connected to the laser radar and 3D modeling system, the storage area camera, and the material inlet camera respectively. It is used to read the storage area point cloud data from the laser radar and 3D modeling system, extract the point cloud of the material inlet, calculate the volume of the material surface accumulated in the material inlet area through gridding, and compare it with the material accumulation threshold to determine the material accumulation status at the material inlet. The image detection submodule is connected to the material inlet camera and is used to capture the flow in real time from the material inlet camera and detect the material inlet logistics accumulation state through image template matching analysis; The human-vehicle intrusion detection module is composed of an alarm event submodule and a grab bucket detection submodule; The alarm event submodule is connected to the storage area camera and the material port camera respectively. Through the built-in regional intrusion detection function of the storage area camera and the material port camera, the alarm triggering events of the storage area camera and the material port camera are read to analyze the entry of people and vehicles into the storage area or the intrusion of personnel at the material port. The grab detection submodule is connected to the storage area camera and is used to obtain pictures from the storage area camera in real time and match them with the image template of the typical position of the grab to detect whether the grab is in the alarm area; The alarm prompt module is connected to the material inlet accumulation detection module and the human and vehicle intrusion detection module respectively; The alarm prompt module includes an audio device, an alarm channel shielding submodule and an alarm information display screen; When receiving alarm messages from the human and vehicle intrusion detection module and the material accumulation detection module, the alarm channel shielding submodule is used to shield the alarm channels that are not related to the current operation task from the alarm message; then the remaining alarm channels are given a language alarm through the audio equipment, and the corresponding alarm information is displayed on the alarm information display screen.

[0008] Furthermore, preferably, the alarm prompt module is deployed in the front-end machine of the central control room.

[0009] Furthermore, preferably, if the material surface volume calculated by the point cloud detection submodule remains unchanged for five consecutive times, it is determined that the point cloud data is not updated.

[0010] Furthermore, preferably, if the image detection is inconsistent with the point cloud detection result, no alarm will be temporarily given; after five point cloud detections, if the five results are still inconsistent, the point cloud detection result will be used as the basis for the current operating material opening area of ​​the intelligent driving, and the image detection result will be used as the basis for the non-operating material opening area.

[0011] Further, preferably, the point cloud detection submodule reads the json file of the warehouse model generated by the laser radar and the 3D modeling system, parses and converts it into a three-dimensional point cloud of the warehouse area, and then extracts each material port area point cloud from the warehouse area point cloud; by dividing the xy axis of each material port area point cloud into a plane grid with M×N rows and columns, counting the points (x, y, z) falling within the range of each xy grid to form each grid point set, calculating the z coordinate average value of each grid point set, denoted as Zavg, the grid midpoint coordinates denoted as Xmid, Ymid, Zavg), the grid area denoted as Sxy, and the midpoint coordinates of the grid with the largest Zavg value in the material port area are found and denoted as (Xmid_zm, Ymid_zm, Zavg_zm), the volume of a single grid is calculated by the grid area Sxy×Zavg, the volume of the material surface accumulated at the material port is calculated by accumulating the volumes of each grid, and the material surface volume of the material port is compared with the accumulation threshold to determine whether there is material accumulation at the material port; When the material surface volume of the material inlet calculated for multiple times (for example, 5 times) does not change, it is determined that the material inlet point cloud data has not been updated, and the material accumulation status of the material inlet is marked as no data update; if the calculated material surface volume of the material inlet changes and exceeds the material accumulation threshold, it is marked as too much material accumulation at the material inlet, otherwise it is marked as too little material accumulation at the material inlet; The material accumulation detection module at the material inlet and the vehicle intrusion detection module are also connected to the intelligent vehicle dispatching system; Finally, the material accumulation detection results and the grid coordinates with the largest Zavg value in the material port area are uploaded to the alarm prompt module and the intelligent driving dispatching system as the material port cleaning points (Xmid_zm, Ymid_zm, Zavg_zm).

[0012] Further, preferably, the material opening accumulation image detection submodule divides the empty opening area image into multiple slices to make a template image, and rotates the template image to make the opening rotated straight, and numbers them in order of position from the left side to the right side of the opening, and the weight of the template from the left side of the opening to the center line is from 1 to 7, and the weight of the template from the right side of the opening to the center line is from -1 to -7; The material inlet ROI area is extracted from the images periodically collected by the material inlet camera, matched with each template image one by one and normalized, and then the best matching position is located. The corresponding position of the template image relative to the collected image is checked to be within the threshold range. Three points in the neighborhood of the best matching position are randomly selected to check that the matching degree is greater than the matching degree threshold. The template can be determined to be successfully matched and marked. At the same time, the absolute values ​​of the weights of the successfully matched templates are accumulated to calculate the total matching weight, and the absolute maximum weight, number and position number of the successfully matched templates are sorted and searched to obtain the absolute maximum weight and number of the successfully matched templates; finally, the material accumulation status of the material in the material inlet is determined by analyzing the total matching weight, maximum weight-number-position number of the successfully matched templates.

[0013] Further, preferably, the alarm channel includes human and vehicle intrusion alarms of all storage areas to be detected from the human and vehicle intrusion detection module, all personnel intrusion alarms of material ports to be detected, and all material accumulation alarms of material ports to be detected from the material port accumulation detection module.

[0014] In the present invention, image detection is used as a supplement to point cloud detection, and the accuracy of material accumulation detection at the material mouth can be improved by combining the results of the two.

[0015] In the present invention, the points (x, y, z) within the grid range, where x is the position of the grab midpoint in the storage area in the direction of the driving truck, which is detected by the Gray busbar of the truck and obtained by coordinate conversion; y is the position of the grab midpoint in the storage area in the direction of the driving car, which is detected by the Gray busbar of the car and obtained by coordinate conversion; z is the vertical distance from the laser emission point of the laser radar on the driving car to the ground of the storage area minus the vertical distance from the laser emission point to the material surface. The above information is already included in the json file, and the converted three-dimensional point cloud also includes the above information.

[0016] In the present invention, (Xmid, Ymid) is the coordinate of the midpoint of each xy grid in the material inlet area, and Zavg is to find the average value of the z values ​​of all points falling within the xy grid range, and use it as the z value of the grid midpoint, that is, the coordinate of the grid midpoint is (Xmid, Ymid, Zavg). The coordinates (Xmid_zm, Ymid_zm, Zavg_zm) indicate that a point with the largest Zavg value among all grid midpoints in the material inlet area is found, and the coordinates of this point are marked as (Xmid_zm, Ymid_zm, Zavg_zm), indicating the point with the highest material accumulation in the material inlet area, and the grab bucket will grab this point first when cleaning the material inlet. That is, Xmid_zm represents the x-axis coordinate (i.e., the traveling direction of the trolley) of the grid where the highest point of material surface accumulation in the material port area is located, Ymid_zm represents the coordinate of the grid (i.e., the traveling direction of the trolley) where the highest point of material surface accumulation in the material port area is located, and Zavg_zm represents the z-axis coordinate (i.e., the height of the highest point of the accumulated material surface) of the grid where the highest point of material surface accumulation in the material port area is located.

[0017] In the present invention, the accumulation threshold is preferably 9.0.

[0018] Point cloud detection is preferably performed every 100 seconds; the material inlet camera periodically collects images, and the periodicity is preferably once every 50 seconds.

[0019] The best matching position is located in the present invention. The best matching position is the position where the template image has the highest matching degree in the newly acquired image, which is usually near the original position of the template image. The blank opening area image is sliced ​​into a group of template images of equal size. The position of each slice, i.e., the template image in the entire blank opening area image is fixed and known (referred to as the original position). The best matching position of the template image in the newly acquired image is usually and required to be near the original position of the template image.

[0020] Then check that the corresponding position of the template image relative to the captured image should be within the threshold range; template matching detection requires that it is meaningful only when it is near the original position range of the template image. Although the matching degree is high beyond this range, it can no longer be considered that the new captured image at the template image position is very similar to the original empty material port area image at the template image position, that is, it is matched; the template image area of ​​the new captured image is either empty, that is, it is matched, or there is accumulated material, that is, it is not matched. For the threshold range: Assume that the starting point of the template image is (x1, y1) and the end point is (x2, y2). For example, if the threshold is set to 4 pixels, the threshold range expands 4 pixels in four directions in the template image position size, that is, the starting coordinates of the threshold range are (x1-4, y1-4) and the end coordinates are (x2+4, y2+4).

[0021] Randomly select 3 points in the neighborhood of the best matching position to check the matching degree to be greater than the matching degree threshold, that is, randomly select points in the threshold range starting point (x1-4, y1-4) and end point (x2+4, y2+4) for 3 image matching. The matching area is the randomly selected area starting point and end point within the threshold range, but the area size template image size must be consistent, that is, randomly move within the threshold range to check the matching degree, and the 3 matches cannot differ too much. The matching degree threshold here is preferably 0.9 or 90%.

[0022] In the present invention, the absolute maximum weight, number and position number of the successfully matched template are obtained by sorting and searching. The sorting is bubble sorted according to the absolute value of the weight of the matched template image to obtain the absolute maximum weight of the matched template, and then the absolute maximum weight is divided into positive maximum weight and negative maximum weight to find the position number of the successfully matched template image, and the number is counted. The sorting operation is to use the template image number position and number corresponding to the maximum weight to simply evaluate the position of the area surrounded by which template positions the material at the material port is accumulated, and to infer an approximate position for cleaning the material port (but this position is not as accurate as the highest point of the accumulated material given by the point cloud detection, and is only used as a backup plan), and also verify the accumulation result judged by the total weight.

[0023] The present invention first determines whether the material accumulation at the material port is large or small by using the total matching weight and threshold (e.g. 47), the maximum weight and threshold (e.g. 4), and the number and threshold of the maximum weight template (e.g. 2). Secondly, the maximum weight-number-position number of the successfully matched template is used to evaluate which template images of the material accumulation are within the boundaries of the material port, and infer the approximate material accumulation location point. For the sorting of template image position numbers, please refer to Figure 5 , the red number in the figure is the sorting position number of the template image, and the blue number is the weight value. Rotate the material opening in the image clockwise and sort it from left to right along the horizontal center line of the material opening. The sorting sequence number (that is, the position number of the template image or the template image number) corresponds to the template image file name. The program can extract the template image number from the file name. In addition, the template image sorting number list maps a fixed weight value list, and the weight value can be found by the template image number. The weight value of the template image indicates that the closer to the horizontal center line of the material opening, the larger it is. The positive and negative signs of the weight indicate that the left side of the center line is positive and the right side is negative. A high matching degree indicates that the material piled up in the area where the matching template image is located is empty.

[0024] (1) Low implementation cost; (2) Effectively reduce the safety risks and regulatory pressure of automatic operation of intelligent driving; (3) It is feasible and can relieve the attention anxiety of operators in the central control room, allowing one person to supervise the operations of multiple intelligent vehicles and reduce labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural schematic diagram of a system for detecting human and vehicle intrusion into a coal storage area and material accumulation at a material port according to the present invention; Figure 2 The figure is a schematic diagram of the camera installation position in the application example. In the figure, the numbers 1, 3, and 5 are the camera installation positions of the three storage areas, 2, 4, and 6 are the camera installation positions of the three material ports, 7, 8, and 9 correspond to the 1, 2, and 3# material ports, 10, 11, and 12 correspond to the 1, 2, and 3# storage areas, 13 is the coal unloading aisle for cars, and 14 is the driving track platform on one side of the plant; Figure 3 Schematic diagram of the position of the laser radar system; in the figure, reference numeral 21 represents 1# laser radar scanner, 22 represents 2# laser radar scanner, 23 represents wide platform of the traveling trolley, 24 represents narrow platform of the traveling trolley, 25 represents traveling beam, 26 represents traveling trolley, and 27 represents traveling end beam; Figure 4The effect diagram of typical material accumulation detection at the 2# material port is shown in the figure, and the ROI area of ​​the material port is shown in the figure; the grid in Figure (a) shows all the template images of the 2# empty material port, the template position number is marked in the upper left corner of the grid, and the template weight is marked in the middle of the grid; (b) (c) (e) (f) are the template images, position numbers and weights that are successfully matched under different material accumulation states at the material port; (f) is the template image that is successfully matched when the grab bucket is unloading; Figure 5 Schematic diagram of template image position number and weight. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below in conjunction with embodiments.

[0027] Those skilled in the art will appreciate that the following examples are only used to illustrate the present invention and should not be considered to limit the scope of the present invention. If no specific techniques or conditions are specified in the examples, the techniques or conditions described in the literature in the art or the product specifications are used. If the manufacturer of the materials or equipment used is not specified, they are all conventional products that can be purchased. Example

[0028] like Figure 1 As shown, a system and method for detecting human and vehicle intrusion and material accumulation at the coal storage area, including a material accumulation detection module at the material storage area, a human and vehicle intrusion detection module, an alarm prompt module, a storage area camera and a material storage area camera; The warehouse area camera is used to obtain images of the warehouse area and detect intrusion of personnel, trucks and forklifts in the warehouse area; The material inlet camera is used to obtain images of the material inlet and detect intrusion of personnel in the material inlet area; The material inlet accumulation detection module is composed of a point cloud detection submodule and an image detection submodule; The point cloud detection submodule is connected to the laser radar and 3D modeling system, the storage area camera, and the material inlet camera respectively. It is used to read the storage area point cloud data from the laser radar and 3D modeling system, extract the point cloud of the material inlet, calculate the volume of the material surface accumulated in the material inlet area through gridding, and compare it with the material accumulation threshold to determine the material accumulation status at the material inlet. The image detection submodule is connected to the material inlet camera and is used to capture the flow in real time from the material inlet camera and detect the material inlet logistics accumulation state through image template matching analysis; The human-vehicle intrusion detection module is composed of an alarm event submodule and a grab bucket detection submodule; The alarm event submodule is connected to the storage area camera and the material port camera respectively. Through the built-in regional intrusion detection function of the storage area camera and the material port camera, the alarm triggering events of the storage area camera and the material port camera are read to analyze the entry of people and vehicles into the storage area or the intrusion of personnel at the material port. The grab detection submodule is connected to the storage area camera and is used to obtain pictures from the storage area camera in real time and match them with the image template of the typical position of the grab to detect whether the grab is in the alarm area; The alarm prompt module is connected to the material inlet accumulation detection module and the human and vehicle intrusion detection module respectively; The alarm prompt module includes an audio device, an alarm channel shielding submodule and an alarm information display screen; When receiving alarm messages from the human and vehicle intrusion detection module and the material accumulation detection module, the alarm channel shielding submodule is used to shield the alarm channels that are not related to the current operation task from the alarm message; then the remaining alarm channels are given a language alarm through the audio equipment, and the corresponding alarm information is displayed on the alarm information display screen. Example

[0029] like Figure 1 As shown, a system and method for detecting human and vehicle intrusion and material accumulation at the coal storage area, including a material accumulation detection module at the material storage area, a human and vehicle intrusion detection module, an alarm prompt module, a storage area camera and a material storage area camera; The warehouse area camera is used to obtain images of the warehouse area and detect intrusion of personnel, trucks and forklifts in the warehouse area; The material inlet camera is used to obtain images of the material inlet and detect intrusion of personnel in the material inlet area; The material inlet accumulation detection module is composed of a point cloud detection submodule and an image detection submodule; The point cloud detection submodule is connected to the laser radar and 3D modeling system, the storage area camera, and the material inlet camera respectively. It is used to read the storage area point cloud data from the laser radar and 3D modeling system, extract the point cloud of the material inlet, calculate the volume of the material surface accumulated in the material inlet area through gridding, and compare it with the material accumulation threshold to determine the material accumulation status at the material inlet. The image detection submodule is connected to the material inlet camera and is used to capture the flow in real time from the material inlet camera and detect the material inlet logistics accumulation state through image template matching analysis; The human-vehicle intrusion detection module is composed of an alarm event submodule and a grab bucket detection submodule; The alarm event submodule is connected to the storage area camera and the material port camera respectively. Through the built-in regional intrusion detection function of the storage area camera and the material port camera, the alarm triggering events of the storage area camera and the material port camera are read to analyze the entry of people and vehicles into the storage area or the intrusion of personnel at the material port. The grab detection submodule is connected to the storage area camera and is used to obtain pictures from the storage area camera in real time and match them with the image template of the typical position of the grab to detect whether the grab is in the alarm area; The alarm prompt module is connected to the material inlet accumulation detection module and the human and vehicle intrusion detection module respectively; The alarm prompt module includes an audio device, an alarm channel shielding submodule and an alarm information display screen; When receiving alarm messages from the human and vehicle intrusion detection module and the material accumulation detection module, the alarm channel shielding submodule is used to shield the alarm channels that are not related to the current operation task from the alarm message; then the remaining alarm channels are given a language alarm through the audio equipment, and the corresponding alarm information is displayed on the alarm information display screen.

[0030] The alarm prompt module is deployed in the front-end machine of the central control room.

[0031] If the material surface volume calculated by the point cloud detection submodule remains unchanged for five consecutive times, it is judged that the point cloud data has not been updated.

[0032] If the image detection is inconsistent with the point cloud detection result, no alarm will be given for the time being; after five point cloud detections, if the five results are still inconsistent, the intelligent driving will use the point cloud detection result as the standard for the current operating material opening area and the image detection result as the standard for the non-operating material opening area.

[0033] The point cloud detection submodule reads the json file of the warehouse model generated by the laser radar and 3D modeling system, parses it into a three-dimensional point cloud of the warehouse area, and then extracts the point cloud of each material port area from the warehouse area point cloud; by dividing the xy axis of each material port area point cloud into a plane grid with M×N rows and columns, the points (x, y, z) falling within the range of each xy grid are counted to form each grid point set, and the z coordinate average value of each grid point set is calculated and recorded as Zavg, the grid midpoint coordinates are recorded as Xmid, Ymid, Zavg), the grid area is recorded as Sxy, and the midpoint coordinates of the grid with the largest Zavg value in the material port area are found and recorded as (Xmid_zm, Ymid_zm, Zavg_zm). The volume of a single grid is calculated by the grid area Sxy×Zavg, and the volume of the material surface accumulated at the material port is calculated by accumulating the volumes of each grid, and the material surface volume of the material port is compared with the accumulation threshold to determine whether there is material accumulation on the material port; When the material surface volume of the material port calculated for multiple times does not change, it is determined that the material port point cloud data has not been updated, and the material port accumulation status is marked as no data update; if the calculated material surface volume of the material port changes and exceeds the accumulation threshold, it is marked as too much material accumulation at the material port, otherwise it is marked as too little material accumulation at the material port; The material accumulation detection module at the material inlet and the vehicle intrusion detection module are also connected to the intelligent vehicle dispatching system; Finally, the material accumulation detection results and the grid coordinates with the largest Zavg value in the material port area are uploaded to the alarm prompt module and the intelligent driving dispatching system as the material port cleaning points (Xmid_zm, Ymid_zm, Zavg_zm).

[0034] The material opening accumulation image detection submodule divides the empty opening area image into multiple slices to make a template image, and rotates the template image to make the opening correct. The templates are numbered in order from the left to the right of the opening. The weight of the template from the left to the center line of the opening is from 1 to 7, and the weight of the template from the right to the center line of the opening is from -1 to -7. The material inlet ROI area is extracted from the images periodically collected by the material inlet camera, matched with each template image one by one and normalized, and then the best matching position is located. The corresponding position of the template image relative to the collected image is checked to be within the threshold range. Three points in the neighborhood of the best matching position are randomly selected to check that the matching degree is greater than the matching degree threshold. The template can be determined to be successfully matched and marked. At the same time, the absolute values ​​of the weights of the successfully matched templates are accumulated to calculate the total matching weight, and the absolute maximum weight, number and position number of the successfully matched templates are sorted and searched to obtain the absolute maximum weight and number of the successfully matched templates; finally, the material accumulation status of the material in the material inlet is determined by analyzing the total matching weight, maximum weight-number-position number of the successfully matched templates.

[0035] The alarm channel includes the human and vehicle intrusion alarms of all storage areas to be detected from the human and vehicle intrusion detection module, the intrusion alarms of all personnel at the material openings to be detected, and the material accumulation alarms of all material openings to be detected from the material opening accumulation detection module. Example

[0036] like Figure 1 As shown, a system and method for detecting human and vehicle intrusion and material accumulation at the coal storage area, including a material accumulation detection module at the material storage area, a human and vehicle intrusion detection module, an alarm prompt module, a storage area camera and a material storage area camera; The system obtains point cloud data of the warehouse area from the laser radar and 3D modeling system. The detection results of the system trigger a voice alarm and upload it to the intelligent driving dispatching system.

[0037] The material inlet accumulation detection module is composed of a point cloud detection submodule and an image detection submodule; The point cloud detection submodule reads the point cloud data of the warehouse area from the lidar and 3D modeling system, extracts the point cloud of the material opening, calculates the volume of the material surface accumulated in the material opening area through gridding, compares it with the material accumulation threshold, and determines the material accumulation status at the material opening; if the calculated material surface volume by the point cloud detection submodule remains unchanged for five consecutive times, it is judged that the point cloud data has not been updated.

[0038] The image detection submodule captures the flow of the material inlet camera in real time, and detects the accumulation status of the material flow at the material inlet through image template matching analysis; image detection is a supplement to point cloud detection, and the combination of the two results can improve the accuracy of material accumulation detection at the material inlet. If the image detection is inconsistent with the point cloud detection result, no alarm will be given for the time being; after five point cloud detections, the intelligent driving will give priority to the point cloud detection result in the next operating material inlet area (when the point cloud detection result is that there is little material accumulation at the material inlet or a lot of material accumulation at the material inlet - alarm), and give priority to the image detection result in the non-operating material inlet area (when the point cloud detection result is that the data is not updated).

[0039] The warehouse area camera is used to obtain images of the warehouse area and detect intrusion of personnel, trucks and forklifts in the warehouse area; The material port camera is used to obtain the material port protrusion and detect intrusion of personnel in the material port area; The human-vehicle intrusion detection module is composed of an alarm event submodule and a grab bucket detection submodule; The alarm event submodule uses the built-in regional intrusion detection function of the storage area camera and the material inlet camera, and then reads the alarm trigger events of the storage area camera and the material inlet camera to analyze the status of people and vehicles entering the storage area or personnel intruding the material inlet. Since the alarm event submodule has the problem of false triggering of the grab bucket, the grab bucket detection submodule is introduced to eliminate the false alarm events triggered by the grab bucket.

[0040] The grab detection submodule takes real-time stream capture from the camera in the storage area, matches it with the image template of the typical posture of the grab to detect whether the grab is in the alarm area, thereby optimizing the results of human and vehicle intrusion detection in the storage area and improving the accuracy of human and vehicle intrusion detection.

[0041] The alarm prompt module is deployed on the front-end machine of the central control room and is equipped with audio equipment, alarm channel shielding submodule and alarm information display screen. When receiving alarm messages from the human and vehicle intrusion detection module and the material inlet accumulation detection module, a language alarm and information display are performed. Through the alarm channel shielding submodule, the alarm channels irrelevant to the current operation task are shielded to avoid alarm interference outside the automatic operation area of ​​the driving.

[0042] There are 6 cameras in total, all installed on the suspension bracket under the trolley track platform on the material inlet side. Among them, 3 are used as storage area cameras to monitor the 1~3# storage areas; the other 3 are used as material inlet cameras to monitor the 1~3# material inlets. The installation location diagram of the 6 cameras is as follows: Figure 2 shown.

[0043] The LiDAR sensors in the LiDAR and 3D modeling system are installed on the sides of the platform on both sides of the traveling trolley, located at one quarter and three quarters of the two platforms respectively. Figure 3The schematic diagram of the installation position of the laser radar sensor is given. In addition to the two laser radar sensors, the laser radar and 3D modeling system also includes a vehicle-mounted edge computing module, which is used to collect the data of the two laser radars and the driving position coordinates, and synthesize the single-line three-dimensional point cloud corresponding to the x-coordinate of the large vehicle, where y and z are the values ​​​​calibrated by the laser radar projection coordinates and the trolley coordinates and the grab bucket height (X-the position of the grab bucket midpoint in the storage area in the driving direction of the large vehicle - detected by the gray bus of the large vehicle and obtained by coordinate conversion, Y-the position of the grab bucket midpoint in the storage area in the driving direction of the trolley - detected by the gray bus of the trolley and obtained by coordinate conversion, Z-"The vertical distance from the laser emission point of the laser radar on the driving vehicle to the ground of the storage area" minus "The vertical distance from the laser emission point to the material surface" is obtained), the edge computing module calculates and uploads the three-dimensional single-line point cloud to the 3D modeling module of the laser radar and 3D modeling system in real time according to the moving distance of the driving trolley. The 3D modeling module collects the single-line point cloud to synthesize the three-dimensional model of the storage area and stores it in the json file, and updates the json file every 30 seconds.

[0044] The material inlet accumulation detection module is deployed on the server, and includes a point cloud detection submodule and an image detection material submodule. The point cloud detection submodule reads the json file of the warehouse model generated by the laser radar and 3D modeling system, parses and converts it into a three-dimensional point cloud of the warehouse area, and then extracts three material inlet area point clouds from the warehouse area point cloud. By dividing the xy axis of the point cloud of each material port area into a plane grid with M×N rows and columns (for example, 11×9), the points (x, y, z) falling within the range of each xy grid are counted to form each grid point set, the z coordinate average value of each grid point set is calculated and recorded as Zavg, the grid midpoint coordinates are recorded as (Xmid, Ymid, Zavg), the grid area is recorded as Sxy, and the coordinates of the grid with the largest Zavg value in the material port area are found and recorded as (Xmid_zm, Ymid_zm, Zavg_zm). The volume of a single grid is calculated by the grid area Sxy×Zavg, and the volume of the material surface accumulated at the material port is calculated by accumulating the volumes of each grid. The material surface volume of the material port is compared with the material accumulation threshold (for example, 9.0) to determine whether there is material accumulation on the material port. When the calculated material surface volume of the material port does not change for multiple times (for example, 5 times), it is determined that the material port point cloud data has not been updated, and the material port accumulation status is marked as "no data update". If the calculated material surface volume of the material port changes and exceeds the accumulation threshold, it is marked as "large material accumulation at the material port", otherwise it is marked as "small material accumulation at the material port". Finally, the accumulation detection results and the grid coordinates with the largest Zavg value in the material port area are uploaded as the material port cleaning point (Xmid_zm, Ymid_zm, Zavg_zm) to the alarm prompt module and the intelligent driving dispatch system.

[0045] The point cloud detection method for material accumulation at the material inlet is highly reliable and can obtain the coordinates of the material inlet cleaning point, but it relies on the movement of the driving vehicle and the laser radar scanning to update the material inlet point cloud. The driving operation is in the designated warehouse area, and the point cloud of the material inlet in other warehouse areas cannot be scanned and updated in time, which will lead to untimely detection results of these material inlets. In order to overcome the shortcomings of point cloud detection, the image detection submodule is introduced as a supplement to point cloud detection.

[0046] The material inlet accumulation image detection submodule divides the empty material inlet area image into multiple (for example, 36) slices to make a template image (200×200 pixels), and slices the template image from the left side of the material inlet ( Figure 4 (a) top left) to right ( Figure 4 (a) Lower right) The templates are numbered in order of position. The weight of the template from the left side of the material opening to the center line ranges from 1 to 7, and the weight of the template from the right side of the material opening to the center line ranges from -1 to -7. The larger the absolute value of the template weight, the closer it is to the center line of the material opening, and the smaller the area of ​​the material opening occupied by the accumulated material (usually the grab bucket discharges the material at the center line of the material opening). Take a stream capture from the camera of the material inlet, extract the material inlet ROI area from the real-time image, match it with 36 template images and normalize it, then locate the best matching position (template matching, normalization and best matching position determination are achieved by calling OpenCV library functions, and six template matching algorithms are selected based on effect and time consumption, such as normalized correlation matching method). Check that the corresponding position of the template image relative to the collected image should be within the threshold range (for example, the position deviation in the length and width direction is less than 4 pixels), randomly select 3 points in the neighborhood of the best matching position to check that the matching degree is greater than the matching degree threshold (for example, 0.9), then the template can be determined to be successfully matched and marked, and the absolute value of the weight of the successfully matched template is accumulated to calculate the total matching weight, and the absolute maximum weight and number of the successfully matched templates and the position number are sorted and searched to obtain the absolute maximum weight and number of the successfully matched templates and the position number. Finally, the material accumulation status of the material in the material inlet is analyzed and judged by the total matching weight, the maximum weight and number of the successfully matched templates, and the position number. Figure 4 The detection effect diagram of typical material accumulation at the 2# feed port is given. The three feed port image detection methods are the same. Although the image collected by this image detection method is restricted by factors such as lighting and the small color distinction between the material and the feed port area, the detection result still has good robustness. The accuracy of the detection is further improved by combining the material accumulation point cloud detection method and the image detection method.

[0047] The human and vehicle intrusion detection module is deployed on the server, including an alarm event submodule and a grab detection submodule. The alarm event submodule reads human and vehicle intrusion detection events in the area from the 1~3# storage area cameras and the 1~3# material port cameras to alarm. All cameras need to be configured with detection ROI area, detection target (storage area cameras detect people and vehicles, material port cameras detect people), sensitivity, cycle and confidence. Before reading the camera alarm event, the module needs to initialize the SDK library, register and log in, set alarm callback, alarm arm, alarm event analysis, network communication processing, cancel the alarm arm, log out, and release SDK resources.

[0048] The grab detection submodule takes real-time stream captures from the 1~3# storage area cameras, extracts the ROI area for grab detection according to the human-vehicle detection area, captures the typical postures of 1~3# driving grabs in the ROI area to make templates, and 1~3# driving grabs may appear in each storage ROI area. Take 5 typical postures of grabs for a total of 15 template images. The more typical posture templates of grabs are taken, the lower the probability of missed detection, but the more computing power is consumed. The image ROI area collected by the camera in each storage area is matched with the 15 template images to detect whether the grab is in the alarm area. Before taking stream captures from the camera, this module also needs to initialize the SDK, register the device, start preview, capture, grab detection, stop preview at the end, cancel the device and release SDK resources. The principles of the grab image detection method and the material image detection method are similar, so they will not be repeated.

[0049] Finally, the human and vehicle intrusion signals of the comprehensive alarm event submodule and the grab detection results of the corresponding area of ​​the grab detection submodule are used to determine whether to output an alarm event. Then, the human and vehicle intrusion result register values ​​of the three storage areas and three material ports are packaged and uploaded to the alarm prompt module and the intelligent driving dispatching system through network communication.

[0050] The alarm prompt module is deployed on the front-end machine of the central control room and is equipped with audio equipment, alarm channel shielding submodule and alarm information display screen. When receiving alarm messages from the human and vehicle intrusion detection module and the material inlet accumulation detection module, a language alarm and information display are performed. Through the alarm channel shielding submodule, the alarm channels irrelevant to the current operation task are shielded to avoid the impact of alarms outside the automatic operation area of ​​the driving.

[0051] The alarm channels of the alarm prompt module include the 1# storage area human and vehicle intrusion alarm, 2# storage area human and vehicle intrusion alarm, 3# storage area human and vehicle intrusion alarm, 1# material port personnel intrusion alarm, 2# material port personnel intrusion alarm, 3# material port personnel intrusion alarm from the human and vehicle intrusion detection module, and the 1# material port material accumulation alarm, 2# material port material accumulation alarm, 3# material port material accumulation alarm from the material port material accumulation detection module, totaling 9 alarm channels.

[0052] The shielding alarm channel submodule is used to shield the above 9 alarm channels. After checking the shielding channel operation through the checkbox, the alarm prompt module will disable the prompt of the corresponding alarm channel. After the alarm prompt module application is closed and reopened, the alarm channel shielding operation needs to be re-checked to remind the operator to select the current automatic operation warehouse area for the driving.

[0053] Corresponding to the above 9 alarm channels, the voice broadcasts include "Alarm: Someone enters 1# storage area!", "Alarm: Someone enters 2# storage area!", "Alarm: Someone enters 3# storage area!", "Alarm: Someone enters 1# material port!", "Alarm: Someone enters 2# material port!", "Alarm: Someone enters 3# material port!", "Please note: Material accumulation at 1# material port!", "Please note: Material accumulation at 2# material port!", "Please note: Material accumulation at 3# material port!", a total of 9 voice messages.

[0054] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A system and method for detecting human and vehicle intrusion and material accumulation at the coal storage area, characterized in that: It includes material accumulation detection module at the material inlet, human and vehicle intrusion detection module, alarm prompt module, storage area camera and material inlet camera; The warehouse area camera is used to obtain images of the warehouse area and detect intrusion of personnel, trucks and forklifts in the warehouse area; The material inlet camera is used to obtain images of the material inlet and detect intrusion of personnel in the material inlet area; The material inlet accumulation detection module is composed of a point cloud detection submodule and an image detection submodule; The point cloud detection submodule is connected to the laser radar and 3D modeling system, the storage area camera, and the material inlet camera respectively. It is used to read the storage area point cloud data from the laser radar and 3D modeling system, extract the point cloud of the material inlet, calculate the volume of the material surface accumulated in the material inlet area through gridding, and compare it with the material accumulation threshold to determine the material accumulation status at the material inlet. The image detection submodule is connected to the material inlet camera and is used to capture the flow in real time from the material inlet camera and detect the material inlet logistics accumulation state through image template matching analysis; The human-vehicle intrusion detection module is composed of an alarm event submodule and a grab bucket detection submodule; The alarm event submodule is connected to the storage area camera and the material port camera respectively. Through the built-in regional intrusion detection function of the storage area camera and the material port camera, the alarm triggering events of the storage area camera and the material port camera are read to analyze the entry of people and vehicles into the storage area or the intrusion of personnel at the material port. The grab detection submodule is connected to the storage area camera and is used to obtain pictures from the storage area camera in real time and match them with the image template of the typical position of the grab to detect whether the grab is in the alarm area; The alarm prompt module is connected to the material inlet accumulation detection module and the human and vehicle intrusion detection module respectively; The alarm prompt module includes an audio device, an alarm channel shielding submodule and an alarm information display screen; When receiving alarm messages from the human and vehicle intrusion detection module and the material accumulation detection module, the alarm channel shielding submodule is used to shield the alarm channels that are not related to the current operation task from the alarm message; then the remaining alarm channels are given a language alarm through the audio equipment, and the corresponding alarm information is displayed on the alarm information display screen.

2. The system for detecting human and vehicle intrusion and material accumulation at the coal storage area as claimed in claim 1 is characterized in that: The alarm prompt module is deployed in the front-end machine of the central control room.

3. The system for detecting human and vehicle intrusion and material accumulation at the coal storage area as claimed in claim 1 is characterized in that: If the material surface volume calculated by the point cloud detection submodule remains unchanged for five consecutive times, it is judged that the point cloud data has not been updated.

4. The system for detecting human and vehicle intrusion and material accumulation at the coal storage area as claimed in claim 1 is characterized in that: If the image detection is inconsistent with the point cloud detection result, no alarm will be given for the time being; after five point cloud detections, if the five results are still inconsistent, the intelligent driving will use the point cloud detection result as the standard for the current operating material opening area and the image detection result as the standard for the non-operating material opening area.

5. The system for detecting human and vehicle intrusion into the coal storage area and material accumulation at the material opening according to claim 1 is characterized in that: The point cloud detection submodule reads the json file of the warehouse model generated by the laser radar and 3D modeling system, parses it into a three-dimensional point cloud of the warehouse area, and then extracts the point cloud of each material port area from the warehouse area point cloud; by dividing the xy axis of each material port area point cloud into a plane grid with M×N rows and columns, the points (x, y, z) falling within the range of each xy grid are counted to form each grid point set, and the z coordinate average value of each grid point set is calculated and recorded as Zavg, the grid midpoint coordinates are recorded as Xmid, Ymid, Zavg), the grid area is recorded as Sxy, and the midpoint coordinates of the grid with the largest Zavg value in the material port area are found and recorded as (Xmid_zm, Ymid_zm, Zavg_zm). The volume of a single grid is calculated by the grid area Sxy×Zavg, and the volume of the material surface accumulated at the material port is calculated by accumulating the volumes of each grid, and the material surface volume of the material port is compared with the accumulation threshold to determine whether there is material accumulation on the material port; When the material surface volume of the material port calculated for multiple times does not change, it is determined that the material port point cloud data has not been updated, and the material port accumulation status is marked as no data update; if the calculated material surface volume of the material port changes and exceeds the accumulation threshold, it is marked as too much material accumulation at the material port, otherwise it is marked as too little material accumulation at the material port; The material accumulation detection module at the material inlet and the vehicle intrusion detection module are also connected to the intelligent vehicle dispatching system; Finally, the material accumulation detection results and the grid coordinates with the largest Zavg value in the material port area are uploaded to the alarm prompt module and the intelligent driving dispatching system as the material port cleaning points (Xmid_zm, Ymid_zm, Zavg_zm).

6. The system for detecting human and vehicle intrusion and material accumulation at the coal storage area as claimed in claim 1 is characterized in that: The material opening accumulation image detection submodule divides the empty opening area image into multiple slices to make a template image, and rotates the template image to make the opening correct. The templates are numbered in order from the left to the right of the opening. The weight of the template from the left to the center line of the opening is from 1 to 7, and the weight of the template from the right to the center line of the opening is from -1 to -7. The material inlet ROI area is extracted from the images periodically collected by the material inlet camera, matched with each template image one by one and normalized, and then the best matching position is located. The corresponding position of the template image relative to the collected image is checked to be within the threshold range. Three points in the neighborhood of the best matching position are randomly selected to check that the matching degree is greater than the matching degree threshold. The template can be determined to be successfully matched and marked. At the same time, the absolute values ​​of the weights of the successfully matched templates are accumulated to calculate the total matching weight, and the absolute maximum weight, number and position number of the successfully matched templates are sorted and searched to obtain the absolute maximum weight and number of the successfully matched templates; finally, the material accumulation status of the material in the material inlet is determined by analyzing the total matching weight, maximum weight-number-position number of the successfully matched templates.

7. The system for detecting human and vehicle intrusion and material accumulation at the coal storage area as claimed in claim 1 is characterized in that: The alarm channel includes the human and vehicle intrusion alarms of all storage areas to be detected from the human and vehicle intrusion detection module, the intrusion alarms of all personnel at the material openings to be detected, and the material accumulation alarms of all material openings to be detected from the material opening accumulation detection module.

Images