A safety monitoring method and system for a warehousing workshop based on image recognition

Abstract

The present invention relates to the technical field of image processing. More specifically, the present invention relates to a safety monitoring method and system for a warehousing workshop based on image recognition. The method includes: using a neural network to obtain a first rectangle corresponding to a stone stack and a second rectangle corresponding to dust and smoke in each frame of the warehousing workshop image; determining the stone stack characteristics of the first rectangle according to the distribution of edge pixel points in the first rectangle; determining the similarity of the stone stacks between different frames according to the similarity between the stone stack characteristics; determining the first possibility of the stone stack collapsing according to the similarity of the stone stacks between consecutive multiple frames and the change of the first rectangle; determining the second possibility of the stone stack collapsing according to the second rectangle and the first possibility; and performing a safety alarm in response to the second possibility being greater than the collapse threshold. The present invention accurately monitors the storage situation of the stone stacks in the warehousing workshop, which is beneficial to ensuring the safety of the lives of on-site workers in the warehousing workshop.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing. More specifically, the present invention relates to a method and system for safety monitoring of a warehousing workshop based on image recognition. Background Art

[0002] A stone warehousing workshop is a facility specifically used for storing and managing various stones (such as crushed stones, sand and gravel, gravel, etc.). The stones in the stone warehousing workshop are usually stored in piles, and roads are left between the piles for inspectors to conduct inspections. In a large stone warehousing workshop, the stone stacks are usually very huge and are very likely to collapse. However, the human perspective is limited, and it is impossible to detect the collapse of the stone stack in the initial stage of the collapse in time, and then escape from the warehousing workshop, which endangers the lives of relevant staff.

[0003] Currently, a neural network is usually used to identify the stone stacks in the overhead view images of the warehousing workshop, and whether the stone stack has collapsed is judged according to the change in the area of the stone stack in the image. For example, the paper (Redmon, Joseph, et al. "You Only Look Once: Unified, Real-Time Object Detection." ComputerVision&Pattern Recognition IEEE, 2016.) discloses a neural network model that can be used to accurately identify the stone stacks in the images of the warehousing workshop.

[0004] In the initial stage of the collapse of the stone stack, only the top of the stone stack collapses, and the area of the stone stack in the overhead view image of the warehousing workshop does not change. If it is only judged whether the stone stack has collapsed based on the change in the area of the stone stack in the warehousing workshop image, there may be a situation of delayed collapse warning. When the stone stack collapses in a large range in a linked manner and then a warning is given, it may be too late, which may cause the on-site staff in the stone warehousing workshop to be unable to escape in time, posing a potential safety hazard to the staff in the stone warehousing workshop. Summary of the Invention

[0005] The present invention provides a method and system for safety monitoring of a warehousing workshop based on image recognition, aiming to solve the technical problem of the lag in identifying the collapse according to the change in the area of the stone stack in the related art.

[0006] In a first aspect, the present invention provides a method for safety monitoring of a warehousing workshop based on image recognition. The method includes:

[0007] Use a neural network to obtain the first rectangle corresponding to the stone stack and the second rectangle corresponding to the dust and smoke in each frame of the warehouse workshop image; determine the stone stack characteristics of the first rectangle according to the distribution of the edge pixel points in the first rectangle; determine the similarity of the stone stacks between different frames according to the similarity between the stone stack characteristics corresponding to the first rectangle in different frames of the warehouse workshop image; determine the first possibility of the stone stack collapse at the current moment according to the similarity of the stone stacks between the warehouse workshop image at the current moment and several previous frames of the warehouse workshop image and the change of the first rectangle, where the first possibility is negatively correlated with the similarity of the stone stack and positively correlated with the area ratio of all the first rectangles in the warehouse workshop image at the current moment and the difference between the areas of all the first rectangles in the warehouse workshop image at the current moment and several previous frames of the warehouse workshop image; determine the second possibility of the stone stack collapse according to the second rectangle in the warehouse workshop image at the current moment and the first possibility, where the second possibility is positively correlated with the first possibility and the area ratio of all the second rectangles in the warehouse workshop image at the current moment, and positively correlated with the difference between the area of the minimum circumscribed rectangle of all the second rectangles and the area of all the second rectangles; in response to the second possibility of the stone stack collapse being greater than the collapse threshold, issue a safety alarm.

[0008] The effects are as follows: The present invention combines the similarity between the stone stack characteristics of the first rectangle in each frame and the change in the size and shape of the first rectangle to obtain the first possibility of the stone stack collapse, which can detect the collapse situation in time at the initial stage of the stone stack collapse and issue a safety alarm, being beneficial to ensuring the safety of the staff in the stone storage workshop; the present invention combines the changes of the stone stack and the dust and smoke in the warehouse workshop to predict the possibility of large-scale linkage collapse of the stone stack, which can measure the possibility of the stack collapse from all stages of the development of the stack collapse, and issue a pre-alarm in time when a large-area stone stack collapse is about to occur, further ensuring the life safety of the on-site staff in the stone storage workshop.

[0009] The determining of the stone stack characteristics of the first rectangle includes: adjusting the image within the first rectangle to a fixed size, performing edge detection on the adjusted first rectangle to obtain an edge image; determining the stone stack characteristics of the first rectangle according to the edge image : , is a preset side length; , , respectively represent the cumulative sum of the gray values of the pixel points in the th row, the th row, and the th row of the edge image; , , respectively represent the Column, the Column, the Sum of the grayscale values of the pixel points in the column; , , Respectively represent the Row, the Row, the Row with a grayscale value of Mean of the abscissas of the pixel points; , , Respectively represent the Column, the Column, the Column with a grayscale value of Mean of the ordinates of the pixel points.

[0010] The effect is that the positions of the stones in the stone stack are random. When the stones in the first rectangle are in different positions, the edges generated will also show different forms. Therefore, the present invention reflects the stone stack characteristics according to the sum of the grayscale values of the pixel points in each row of the edge image, the sum of the grayscale values of the pixel points in each column, the mean of the abscissas of the edge pixel points in each row, and the mean of the ordinates of the edge pixel points in each column, so that the stone stack characteristics corresponding to the first rectangle in different frames can reflect the changes in the stone stack, which is beneficial to timely detecting the collapse of the stone stack at the initial stage of the collapse of the stone stack.

[0011] Preferably, determining the similarity of the stone stack between different frames includes: using each of several previous frames of the warehouse workshop image at the current moment as a reference image; determining the similarity of the stone stack between the warehouse workshop image at the current moment and each reference image: , Represents the similarity of the stone stack between the warehouse workshop image at the current moment and the th reference image; Represents the stone stack characteristics of the th first rectangle in the warehouse workshop image at the current moment, Taking all the integers in, Represents the number of first rectangles in the warehouse workshop image at the current moment; The th first rectangle in the warehouse workshop image at the current moment corresponds to the th reference image Represents the cosine similarity function; Represents the minimum value function.

[0012] Preferably, the first possibility of the stone material stack collapse at the current moment satisfies the expression: ; where represents the first possibility of the stone material stack collapse at the current moment; represents the sum of the areas of all the first rectangles in the image of the storage workshop at the current moment; represents the area size of the image of the storage workshop; represents the similarity of the stone material stack between the image of the storage workshop at the current moment and the th reference image, ranges over the integers in represents a preset first quantity; represents the th reference image, the sum of the areas of all the first rectangles in it; represents the natural exponential function.

[0013] Its effect is that: the present invention combines the change situation of the size and shape of the first rectangles corresponding to the stone material stack in different frames, as well as the similarity between the stone material stack features of the first rectangles, to obtain the first possibility of the stone material stack collapse, avoiding the situation that the size and shape of the first matrix may not change in the initial stage of the stone material stack collapse, resulting in the inability to detect the stone material stack collapse in time. The present invention can detect the collapse situation in time at the initial stage of the stone material stack collapse and give a safety alarm, which is beneficial to ensuring the safety of the staff in the stone material storage workshop.

[0014] Preferably, the second possibility of the stone material stack collapse satisfies the expression: ; where represents the second possibility of the stone material stack collapse; represents the sum of the areas of all the second rectangles in the image of the storage workshop at the current moment; represents the area size of the image of the storage workshop; represents the area of the minimum circumscribed rectangle of all the second rectangles in the image of the storage workshop at the current moment; represents the first possibility of the stone material stack collapse at the current moment.

[0015] Its effect is that: the present invention uses the change situation of the size and shape of the second rectangles corresponding to the dust and smoke to reflect the diffusion situation of the dust and smoke, and combines the diffusion situation of the dust and smoke and the first possibility of the stone material stack collapse to predict whether the stone material stack will have a linked collapse, and the result is more accurate.

[0016] Preferably, the second possibility of the stone material stack collapse satisfies the expression: ; where represents the second possibility of the stone material stack collapse, reflecting the possibility of the occurrence of the linked collapse of the stone material stack; represents the sum of the areas of all the second rectangles in the image of the storage workshop at the current moment; represents the area size of the image of the storage workshop; represents the area of the minimum bounding rectangle of all the second rectangles in the image of the storage workshop at the current moment; represents the wind speed at the current moment; represents a hyperparameter; represents the first possibility of the stone pile collapsing at the current moment; represents the natural exponential function.

[0017] Its effect is that: in the process of calculating the second possibility of the stone pile collapsing, the present invention excludes the influence of the wind speed in the storage workshop on the diffusion of dust and smoke, making the obtained result more capable of reflecting the possibility of the stone pile collapsing in a chain reaction.

[0018] Preferably, it further includes: in response to the non-existence of the second rectangle in the image of the storage workshop at the current moment, if the first possibility of the stone pile collapsing at the current moment is greater than the collapse threshold, a safety alarm is given.

[0019] Preferably, the neural network adopts the YOLOv5 model.

[0020] In a second aspect, the present invention proposes a safety monitoring system for a storage workshop based on image recognition, including a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement a safety monitoring method for a storage workshop based on image recognition as described in any one of the above-mentioned invention contents.

[0021] The present invention has the following beneficial effects: by obtaining the first possibility of the stone pile collapsing, the present invention can timely detect the collapse situation at the initial stage of the stone pile collapse. The present invention combines the change of dust and smoke to predict the possibility of large-scale chain collapse of the stone pile, can measure the possibility of the pile collapse at each stage of the development of the pile collapse and give an alarm in time, which is beneficial to ensuring the life safety of the on-site staff in the stone storage workshop. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By referring to the accompanying drawings and reading the following detailed description, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become easy to understand. In the drawings, several embodiments of the present invention are shown in an exemplary rather than restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0023] Figure 1 is a flowchart of the steps of a safety monitoring method for a storage workshop based on image recognition according to an embodiment of the present invention;

[0024] Figure 2Schematic diagram of the minimum circumscribed rectangle of all the second rectangles;

[0025] Figure 3 It is a structural block diagram of a safety monitoring system for a warehousing workshop based on image recognition according to an embodiment of the present invention. Detailed implementation manners

[0026] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0027] Next, the detailed implementation manners of the present invention will be described in conjunction with the accompanying drawings.

[0028] Figure 1 It is a step flowchart schematically showing a safety monitoring method for a warehousing workshop based on image recognition according to an embodiment of the present invention.

[0029] As Figure 1 shown, a safety monitoring method for a warehousing workshop based on image recognition includes steps S1 to S5.

[0030] Step S1: Collect multiple frames of images of the warehousing workshop, and use a neural network to obtain the first rectangles corresponding to the stone stacks and the second rectangles corresponding to the dust and smoke in each frame of the warehousing workshop image.

[0031] The warehousing workshop stores stones. Images of the warehousing workshop are collected in real time through a camera at an overhead view angle inside the warehousing workshop, and a neural network is used to identify the stone stacks and dust and smoke in the warehousing workshop images.

[0032] In one embodiment, the specific content of the neural network includes: the neural network adopts the YOLOv5 network; the input of the neural network is the image of the warehousing workshop, and the output is the bounding boxes corresponding to the stone stacks and the bounding boxes corresponding to the dust and smoke in the warehousing workshop image; the dataset used by the neural network is a dataset composed of images containing stone stacks or dust and smoke at various overhead view angles, and the labels are the bounding boxes of the stone stacks and the bounding boxes of the dust and smoke in the images containing stone stacks or dust and smoke; the loss function of the neural network is the cross-entropy loss function and the intersection over union (IOU) loss function.

[0033] Among them, YOLOv5 is a neural network algorithm in the You Only Look Once (YOLO) series of object detection models. Based on YOLO, YOLOv5 integrates architecture design. Cross-entropy loss function and IOU loss function are commonly used loss functions in YOLOv5. Implementers can also select a neural network model according to the actual implementation situation.

[0034] Input each frame of the warehousing workshop image into the trained neural network to obtain the bounding box corresponding to the stone stack and the bounding box corresponding to the dust and smoke in the warehousing workshop image. Denote the bounding box corresponding to the stone stack as the first rectangle, and the bounding box corresponding to the dust and smoke as the second rectangle. Since there may be multiple stone stacks in the warehousing workshop, there may be multiple first rectangles and multiple second rectangles in the warehousing workshop image.

[0035] So far, the first rectangle corresponding to the stone stack and the second rectangle corresponding to the dust and smoke in each frame of the warehousing workshop image have been obtained.

[0036] Step S2: Determine the stone stack characteristics of the first rectangle according to the distribution of edge pixel points in the first rectangle.

[0037] It should be noted that if the stone stacks in the warehousing workshop are stacked improperly, they may collapse. In the early stage of the collapse of the stone stack, local collapse will occur at the top of the stone stack, and the local collapse will gradually spread from the top of the stack until the whole stack collapses. During the collapse of the stone stack, the positions of the stones in the stone stack change, and the edges generated when the stones in the first rectangle are in different positions due to the collapse will also show different shapes. Therefore, the present invention determines the stone stack characteristics of the warehousing workshop image according to the distribution of edge pixel points in the first rectangle in the warehousing workshop image.

[0038] Specifically, each first rectangle in the warehousing workshop image corresponds to a stone stack. For any first rectangle in any frame of the warehousing workshop image, for the convenience of subsequent calculation, the image within the first rectangle is adjusted to a fixed size and denoted as the stone stack image. It should be noted that in the present invention is a preset side length, which is set by the implementer according to the actual implementation situation and is not specifically limited. For example . When the size of the first rectangle exceeds , the method of linear interpolation is used to adjust the image within the first rectangle. When the size of the first rectangle is less than , the method of downsampling is used to adjust the image within the first rectangle.

[0039] In one embodiment, the Canny operator is used to perform edge detection on the stone stack image to obtain the edge image of the stone stack. The edge image is a binary image. The pixel points with a gray value of 1 in the edge image are the edge pixel points in the stone stack image, and the pixel points with a gray value of 0 are the non-edge pixel points in the stone stack image. In other embodiments, the implementer can select an edge detection algorithm according to the actual implementation situation, such as the Sobel operator.

[0040] In one embodiment, the stone stack feature of the first rectangle is determined according to the gray value distribution of the pixel points in the edge image:

[0041] ;

[0042] where represents the stone stack feature of the first rectangle; is the preset side length; , , respectively represent the cumulative sum of the gray values of the pixel points in the th row, the th row, and the th row in the edge image; , , respectively represent the cumulative sum of the gray values of the pixel points in the th column, the th column, and the th column in the edge image; , , respectively represent the average value of the abscissas of the pixel points with a gray value of in the th row, the th row, and the th row in the edge image; , , respectively represent the average value of the ordinates of the pixel points with a gray value of in the th column, the th column, and the th column in the edge image.

[0043] Since the positions of the stones in the stone stack within the first rectangle are random, when the stones within the first rectangle are in different positions, the edges they generate will also show different forms. Therefore, there is a certain difference between the cumulative sum of the gray values of each row of pixel points in the edge image corresponding to the stone stack in the first rectangle before collapse and the cumulative sum of the gray values of each row of pixel points in the edge image corresponding to the collapse process. It reflects the cumulative sum of the gray values of each pixel point in each row of the edge image corresponding to the first rectangle. Similarly, there is a certain difference between the cumulative sum of the gray values of each pixel point in each column of the edge image corresponding before the collapse and the edge image corresponding during the collapse process. It reflects the cumulative sum of the gray values of each pixel point in each column of the edge image corresponding to the first rectangle. Similarly, since the edges generated when the stones in the first rectangle are in different positions will also show different forms, and the coordinates of the corresponding edge pixel points are also in different distribution situations, therefore and can also represent the uniqueness of the stones in the first rectangle. Therefore, in this embodiment, , , and are combined to obtain the stacking characteristics of the stones in the first rectangle .

[0044] In another embodiment, the stacking characteristics of the stones in the first rectangle are determined according to the gray distribution of the pixel points in the edge image:

[0045] ;

[0046] In the formula, represents the stacking characteristics of the stones in the first rectangle; is the preset side length; , , respectively represent the cumulative sum of the gray values of the pixel points in the th row, the th row, and the th row of the edge image; , , respectively represent the cumulative sum of the gray values of the pixel points in the th column, the th column, and the th column of the edge image; , , respectively represent the mean value of the abscissas of the pixel points with a gray value of in the th row, the th row, and the th row of the edge image; , , respectively represent the mean value of the abscissas of the pixel points with a gray value of in the th column, the th column, and the The mean value of the vertical coordinates of the pixel points. To further increase the feature gap of the stone stacks in different states, in this embodiment, , , and are combined to obtain the stone stack feature . In this embodiment, the stone stack feature is calculated by taking the exponent and then subtracting. Implementers can adjust , , and according to the actual situation.

[0047] Similarly, the stone stack features of each first rectangle in each frame of the storage workshop image are obtained.

[0048] Step S3: Determine the similarity of the stone stacks between different frames according to the similarity of the stone stack features of the corresponding first rectangles in different frames of the storage workshop images. According to the similarity of the stone stacks between the storage workshop image at the current moment and the previous several frames of storage workshop images and the change of the first rectangle, determine the first possibility of the stone stack collapse at the current moment.

[0049] It should be noted that when the entire stone stack collapses, the size and shape of the first rectangle corresponding to the stone stack in the storage workshop images of the front and back frames will change greatly. To accurately detect whether the stone stack collapses, the present invention obtains the first possibility of the stone stack collapse through the changes in the size and position of the first rectangle in the storage workshop image.

[0050] Specifically, obtain the center points of each first rectangle in each frame of the storage workshop image. For the target first rectangle in the storage workshop image at the current moment, use the first rectangle with the smallest distance between the center point and the center point of the target first rectangle in any frame of the storage workshop image as the first rectangle corresponding to the target first rectangle in that frame of the storage workshop image. Similarly, obtain the first rectangles corresponding to each first rectangle in the storage workshop image at the current moment in each frame of the storage workshop image.

[0051] Take the frames of storage workshop images before the storage workshop image at the current moment as a reference image respectively, where represents a preset first quantity, and implementers can set the first quantity according to the actual implementation situation. For example, . Determine the similarity of the stone stack between the storage workshop image at the current moment and the reference image according to the similarity of the stone stack features of each first rectangle in the storage workshop image at the current moment and the corresponding first rectangle in the reference image:

[0052] ;

[0053] Among them, represents the similarity of the stone stack between the image of the storage workshop at the current moment and the th reference image; represents the stone stack feature of the th first rectangle in the image of the storage workshop at the current moment, taking all integers in represents the number of the first rectangles in the image of the storage workshop at the current moment; the stone stack feature of the corresponding first rectangle of the th first rectangle in the image of the storage workshop at the current moment in the th reference image; represents the cosine similarity function; represents the minimum value function; since the value range of the cosine similarity is [-1, 1], the present invention uses to map the cosine similarity to the range of [0, 1] for subsequent calculations. When the cosine similarity between the stone stack features of the corresponding first rectangles in two frames of storage workshop images is larger, it indicates that the stone stack in the first rectangle has basically not changed, and the possibility of the collapse of the stone stack is smaller; on the contrary, when the cosine similarity between the stone stack features of the corresponding first rectangles in two frames of storage workshop images is smaller, the possibility of the collapse of the stone stack is larger. The present invention takes the minimum value among the cosine similarities between the stone stack features of all corresponding first rectangles in two frames of storage workshop images as the similarity of the stone stack between the two frames of storage workshop images.

[0054] Determine the first possibility of the collapse of the stone stack at the current moment according to the changes of the first rectangles in the image of the storage workshop at the current moment and the previous consecutive frames of storage workshop images:

[0055] ;

[0056] Among them, represents the first possibility of the collapse of the stone stack at the current moment; represents the sum of the areas of all the first rectangles in the image of the storage workshop at the current moment; represents the area size of the storage workshop image; represents the similarity of the stone stack between the image of the storage workshop at the current moment and the th reference image, taking all integers in represents a preset first quantity; represents the th sum of the areas of all the first rectangles in the reference image; represents the natural exponential function; represents the minimum value function.

[0057] The image of the storage workshop is taken from an aerial view, and there are usually multiple stone stacks in the storage workshop. When the stone stacks do not collapse, the area of the first rectangle corresponding to each stone stack is small. When the stone stacks collapse, the area of the corresponding rectangle of each stone stack gradually increases, and multiple scattered stone stacks will gradually merge into one stone stack. represents the proportion of the areas of all the first rectangles in the image of the storage workshop at the current moment, reflecting the range of the stone stacks in the image of the storage workshop at the current moment. When is larger, it is more likely that the stone stacks have collapsed at the current moment; conversely, when is smaller, it is less likely that the stone stacks have collapsed at the current moment. Since only considers the range of the stone stacks in the image of the storage workshop at the current moment and cannot measure the motion state of the stone stacks, the present invention measures the motion state of the stone stacks by the size change of the first rectangles corresponding to the stone stacks within a period of time before the current moment, and corrects through the motion state of the stone stacks. reflects the degree of size change of the first rectangles corresponding to the stone stacks. The larger it is, the faster the stone stacks change within a period of time before the current moment, and the more likely the stone stacks have collapsed at the current moment. At this time, the degree of expansion of for is greater, increasing the first possibility of the stone stacks collapsing; conversely, when is smaller, it indicates that the stone stacks are more stable within a period of time before the current moment, and the lower the possibility of the stone stacks collapsing at the current moment. At this time, the degree of expansion of for is smaller, making the first possibility of the stone stacks collapsing more dependent on .

[0058] reflects the minimum similarity of the stone stacks between the image of the storage workshop at the current moment and the images of the storage workshop within a previous period of time. In the early stage of the collapse of the stone stacks, local collapse will occur at the top of the stone stacks. Since the image of the storage workshop is taken from an aerial view, when local collapse occurs at the top of the stone stacks, the first rectangles corresponding to the stone stacks may not change. Therefore, in order to identify the local collapse of the stone stacks, the present invention uses the minimum similarity of the stone stacks between the image of the storage workshop at the current moment and the images of the storage workshop within a previous period of time When the minimum similarity is smaller, it means that the characteristics of the stone pile at the current moment have changed significantly compared with the previous stone pile. This change is more likely to be caused by the collapse of the stone pile. right The greater the expansion, the greater the first possibility of the stone pile collapse; conversely, when the minimum similarity is larger, it means that the stone pile characteristics at the current moment are basically unchanged compared with the previous stone pile, and the possibility of the stone pile collapse is smaller. right The smaller the expansion.

[0059] At this point, the first possibility of the stone pile collapse at the current moment is obtained.

[0060] Step S4: determining the second possibility of the stone pile collapse according to the second rectangle in the storage workshop image at the current moment and the first possibility of the stone pile collapse at the current moment.

[0061] It should be noted that, according to the change of the first rectangle corresponding to the stone stack, the first possibility of the collapse of the stone stack can be reflected. However, when the stone stack in the storage workshop collapses, they usually do not all collapse at the same time. Instead, one stack collapses first and then drives the other stacks to collapse in conjunction. Since the stone stack often contains a large amount of dust, when the stack collapses, a certain amount of dust and smoke will be generated. The diffusion of the dust and smoke occurs earlier than the coordinated collapse of the stacks. Therefore, in order to monitor whether the stone stack is about to collapse in conjunction before the coordinated collapse, the present invention calculates the second possibility of the collapse of the stone stack at the current moment according to the second rectangle corresponding to the dust and smoke in the storage workshop image and the first possibility of the stone stack collapse at the current moment, and uses the second possibility to reflect the possibility of the coordinated collapse of the stone stack.

[0062] In one embodiment, in response to the presence of a second rectangle in the warehouse image at the current moment, the second possibility of the collapse of the stone pile is calculated according to the second rectangle corresponding to the dust and smoke in the warehouse image at the current moment and the first possibility of the collapse of the stone pile at the current moment:

[0063] ;

[0064] in, It indicates the second possibility of the collapse of the stone pile, reflecting the possibility of the linkage collapse of the stone pile; Represents the sum of the areas of all second rectangles in the warehouse workshop image at the current moment; Indicates the area size of the warehouse workshop image; Represents the area of ​​the minimum circumscribed rectangle of all second rectangles in the warehouse image at the current moment, Figure 2Schematic diagram of the minimum circumscribed rectangle for all second rectangles Figure 2 The gray part is the second rectangle; Represents the first possibility of the stone pile collapse at the current moment.

[0065] Represents the proportion of the second rectangle corresponding to the dust and smoke in the image of the storage workshop at the current moment, The larger it is, the more dust and smoke exist at the current moment. The dust and smoke are more likely to be generated due to the local collapse of the stone pile, and subsequent linkage collapse of the stone pile is more likely; on the contrary, The smaller it is, the less dust and smoke exist at the current moment, and the less likely it is for the stone pile to have a linkage collapse. Reflects the diffusion degree of the dust and smoke at the current moment, The larger it is, the wider the diffusion range of the dust and smoke, and the more likely the dust and smoke are generated due to the collapse of the stone pile; on the contrary, The smaller it is, the smaller the diffusion range of the dust and smoke, and the less likely it is for the stone pile to have a linkage collapse. Since the first rectangle corresponding to the stone pile has a lag in detecting the linkage collapse, when the first possibility of the stone pile collapse obtained through the first rectangle corresponding to the stone pile is small, in fact, a large amount of dust and smoke may have been generated, forming a large-scale pile collapse. Therefore, in the present invention, For Make an upward correction so that The larger it is, the second possibility of the stone pile collapse The larger it is.

[0066] It should be further noted that in order to maintain ventilation in the storage workshop, a fan is usually installed in the storage workshop, so that the wind speed in the storage workshop has a certain impact on the dust and smoke. When the wind speed is relatively large, the dust and smoke with a relatively small original range may be blown to a larger range. Therefore, in another embodiment of the present invention, the second possibility of the stone pile collapse is calculated in combination with the wind speed.

[0067] In another embodiment, a wind speed sensor is deployed inside the storage workshop, and the wind speed sensor is used to collect the wind speed at each moment in real time. In response to the presence of the second rectangle in the image of the storage workshop at the current moment, the second possibility of the stone pile collapse is calculated according to the second rectangle corresponding to the dust and smoke in the image of the storage workshop at the current moment, the wind speed at the current moment, and the first possibility of the stone pile collapse at the current moment:

[0068] ;

[0069] Among them, Represents the second possibility of the stone pile collapse, reflecting the possibility of the stone pile having a linkage collapse; represents the sum of the areas of all the second rectangles in the image of the storage workshop at the current moment; represents the area size of the image of the storage workshop; represents the area of the minimum bounding rectangle of all the second rectangles in the image of the storage workshop at the current moment; represents the wind speed at the current moment; represents a hyperparameter used to obtain the relative magnitude of the wind speed with an empirical value of 4 m / s, which can be set by the implementer according to the actual implementation situation; represents the first possibility of the stone pile collapse at the current moment; represents the natural exponential function.

[0070] represents the proportion of the second rectangle corresponding to the dust and smoke in the image of the storage workshop at the current moment. reflects the diffusion degree of the dust and smoke at the current moment, represents the wind speed influence factor. Considering the influence of the wind speed in the storage workshop, when the wind speed is relatively high, it may blow the dust and smoke with a small original range to a larger range, resulting in a certain deviation in judging the possibility of the stone pile generating a linkage collapse based on the diffusion degree of the dust and smoke. Therefore, through the wind speed influence factor the diffusion degree is corrected. When is larger, through adjustment makes decrease, thereby reducing the influence of the wind speed on judging the possibility of the stone pile generating a linkage collapse; when is smaller, the diffusion degree of the dust and smoke is more likely to be caused by the pile collapse. Therefore, when is smaller, the correction degree for is smaller, making the result closer to . Since the first rectangle corresponding to the stone pile has a lag in detecting the linkage collapse, when the first possibility of the stone pile collapse obtained through the first rectangle corresponding to the stone pile is relatively small, in fact, a large amount of dust and smoke may have been generated to form a large-scale pile collapse. Therefore, in the present invention, through the is corrected upward, so that when is larger, the second possibility of the stone pile collapse is larger.

[0071] Thus, the second possibility of the stone pile collapse is obtained.

[0072] Step S5: In response to the second possibility of the stone pile collapse being greater than the collapse threshold, a safety alarm is issued.

[0073] Among them, performing safety alarm includes: emitting an acoustic and optical alarm signal and sending a distress signal.

[0074] In one embodiment, in response to the presence of a second rectangle in the image of the storage workshop at the current moment, when the second possibility of the stone stack collapsing is greater than a preset collapse threshold, a sound alarm signal is emitted through a buzzer to remind the on-site staff to evacuate, and a light alarm signal is emitted through the rapid flashing of a red LED. A distress signal can also be sent to the fire department through the wireless network.

[0075] In response to the absence of a second rectangle in the image of the storage workshop at the current moment, when the first possibility of the stone stack collapsing is greater than a preset collapse threshold, an acoustic and optical alarm signal and a distress signal are emitted.

[0076] Among them, the collapse threshold is set by the implementer according to the actual implementation situation, and specific limitations are not made. For example, it is 0.6.

[0077] Figure 3 It schematically shows a structural block diagram of a storage workshop safety monitoring system based on image recognition according to an embodiment of the present invention.

[0078] The present invention also provides a storage workshop safety monitoring system based on image recognition. As Figure 3 shown, the system includes a processor and a memory, and the memory stores computer program instructions. When the computer program instructions are executed by the processor, a storage workshop safety monitoring method according to the first aspect of the present invention is implemented.

[0079] Those skilled in the art can understand that Figure 3 the structure shown in

[0080] is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device of the present invention. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0081] In the present invention, the foregoing memory may be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, or any other medium that can be used to store the required information and can be accessed by an application program, module, or both. Any such computer storage medium may be part of the device or accessible or connectable to the device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that can be stored or otherwise held by such a computer-readable medium.

[0082] In the description of this specification, the meanings of "a plurality of" and "several" are at least two, such as two, three, or more, etc., unless otherwise specifically defined.

[0083] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0084] The above-described embodiments merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. A warehouse safety monitoring method based on image recognition, characterized in that: include: A neural network is used to obtain a first rectangle corresponding to the stone pile and a second rectangle corresponding to the dust and smoke in each frame of the storage workshop image; the stone pile feature of the first rectangle is determined according to the distribution of edge pixel points in the first rectangle; Determine the similarity of the stone stacks between different frames according to the similarity between the stone stack features corresponding to the first rectangle in the warehouse images of different frames; According to the similarity of the stone pile between the warehouse image at the current moment and the warehouse images of several previous frames and the change of the first rectangle, determine the first possibility of the collapse of the stone pile at the current moment, wherein the first possibility is negatively correlated with the similarity of the stone pile, and positively correlated with the area proportion of all the first rectangles in the warehouse image at the current moment and the difference between the areas of all the first rectangles in the warehouse image at the current moment and the warehouse images of several previous frames; Determine a second possibility of the collapse of the stone pile according to the second rectangle in the storage workshop image at the current moment and the first possibility, wherein the second possibility is positively correlated with the first possibility and the area ratio of all second rectangles in the storage workshop image at the current moment, and is positively correlated with the difference between the area of ​​the minimum circumscribed rectangle of all second rectangles and the area of ​​all second rectangles; In response to the second possibility of the rock pile collapsing being greater than the collapse threshold, a safety alarm is issued.

2. According to the method of claim 1, the method is characterized in that: The step of determining the first rectangular stone stacking feature comprises: Resize the image in the first rectangle to The fixed size of the first rectangle is used to detect the edge of the adjusted first rectangle to obtain an edge image; the stone stacking features of the first rectangle are determined according to the edge image. : , is the preset side length; , , They represent the edge images Row, No. Row, No. The cumulative sum of the grayscale values ​​of the pixels in the row; , , They represent the edge images Column, No. Column, No. The cumulative sum of the grayscale values ​​of the pixels in the column; , , They represent the edge images Row, No. Row, No. The gray value of the row is The mean of the horizontal coordinates of the pixels; , , They represent the edge images Column, No. Column, No. The gray value of the column is The mean of the vertical coordinates of the pixels.

3. The method for warehouse safety monitoring based on image recognition according to claim 1 is characterized in that: The determining of the similarity of the stone stacks between different frames comprises: The warehouse images of several frames before the current warehouse image are used as reference images respectively; the similarity of the stone stacking between the warehouse image at the current moment and each reference image is determined: , The image of the warehouse at the current moment and the Similarity of stone piles between reference images; Indicates the current moment of the warehouse image The first rectangular stone pile features, Take all Integers in Indicates the number of first rectangles in the warehouse workshop image at the current moment; The current warehouse image The first rectangle is in A first rectangular stone stack feature corresponding to a reference image; represents the cosine similarity function; Represents the minimum function.

4. A warehouse safety monitoring method based on image recognition according to claim 3, characterized in that: The first possibility of the stone pile collapse at the current moment satisfies the expression: ； in, Indicates the first possibility of the stone pile collapsing at the current moment; Represents the sum of the areas of all first rectangles in the warehouse workshop image at the current moment; Indicates the area size of the warehouse workshop image; The image of the warehouse at the current moment and the The similarity of stone stacks between reference images, Take all Integers in represents a preset first quantity; Indicates The sum of the areas of all first rectangles in the reference images; represents the natural exponential function.

5. A warehouse safety monitoring method based on image recognition according to any one of claims 1-4, characterized in that: The second possibility of the stone pile collapse satisfies the expression: ； in, Indicates the second possibility of the collapse of the stone pile; Represents the sum of the areas of all second rectangles in the warehouse workshop image at the current moment; Indicates the area size of the warehouse workshop image; Indicates the area of ​​the minimum circumscribed rectangle of all second rectangles in the warehouse workshop image at the current moment; Indicates the first possibility of the stone pile collapsing at the current moment.

6. A warehouse safety monitoring method based on image recognition according to any one of claims 1-4, characterized in that: The second possibility of the stone pile collapse satisfies the expression: ； in, It indicates the second possibility of the collapse of the stone pile, reflecting the possibility of the linkage collapse of the stone pile; Represents the sum of the areas of all second rectangles in the warehouse workshop image at the current moment; Indicates the area size of the warehouse workshop image; Indicates the area of ​​the minimum circumscribed rectangle of all second rectangles in the warehouse workshop image at the current moment; Indicates the wind speed at the current moment; represents a hyperparameter; Indicates the first possibility of the stone pile collapsing at the current moment; represents the natural exponential function.

7. A warehouse safety monitoring method based on image recognition according to any one of claims 1-4, characterized in that: Also includes: In response to the absence of the second rectangle in the storage workshop image at the current moment, if the first possibility of the stone pile collapse at the current moment is greater than the collapse threshold, a safety alarm is issued.

8. The method for warehouse safety monitoring based on image recognition according to claim 1 is characterized in that: The neural network adopts the YOLOv5 model.

9. A warehouse safety monitoring system based on image recognition, comprising a processor and a memory, wherein the memory stores a computer program, characterized in that: The processor executes the computer program to implement a warehouse workshop safety monitoring method based on image recognition as described in any one of claims 1-8.

Images