An image-based forklift violation monitoring method, medium, and device

Abstract

The application discloses a forklift illegal use monitoring method, medium and equipment based on images, and relates to the technical field of forklift monitoring. Two images captured by a camera at an interval of a first preset time length are acquired; frame difference binarization processing is performed on the two images to obtain a frame difference image; a target minimum circumscribed rectangle is determined for a connected region of the frame difference image; whether a width-length ratio of the target minimum circumscribed rectangle is less than a first preset threshold value is judged for each target minimum circumscribed rectangle; if the width-length ratio is less than the first preset threshold value, it is determined that the target minimum circumscribed rectangle corresponds to a forklift illegal use. Frame difference binarization processing is performed on two images captured by a camera at an interval of a first preset time length to obtain a frame difference image, a target minimum circumscribed rectangle is determined for a connected region in the frame difference image, and whether a forklift illegal use behavior exists is determined according to whether a width-length ratio of the target minimum circumscribed rectangle is less than a first preset threshold value, so that consumption of network resources can be reduced, and labor cost can be lowered.

Description

Technical Field

[0001] This invention relates to the field of forklift monitoring technology, and in particular to an image-based method, medium, and device for monitoring forklift violations. Background Technology

[0002] A forklift is an industrial material handling vehicle, specifically a wheeled transport vehicle used for loading, unloading, stacking, and short-distance transport of palletized goods. When sorting express deliveries, due to the weight of the goods, forklift operators typically use small electric forklifts to move and load goods within the work area. Regulations stipulate that forklift operators can only move one pallet at a time; moving multiple pallets can easily lead to goods falling or colliding, so pushing more than one pallet is a violation.

[0003] To avoid the aforementioned violations, the following two measures are typically taken:

[0004] One approach is to assign safety monitoring personnel to supervise workers on-site. However, this leads to increased labor costs. When the volume of deliveries surges and the number of forklift operators increases, safety monitoring personnel cannot guarantee that they can monitor every worker. Violations will still occur, causing losses to the factory and posing a threat to the safety of workers.

[0005] Another method involves using surveillance cameras to film and store the workers' work process. Inspectors then sample and download several segments for inspection. During off-peak hours, surveillance images within a few minutes can be almost identical. Downloading videos consumes a lot of bandwidth and network resources. Directly extracting video segments for inspection is not only a waste of manpower and resources, but also inefficient. Summary of the Invention

[0006] In order to solve at least one of the technical problems mentioned in the background art, the present invention aims to provide an image-based forklift violation detection method, medium and device, which can monitor the illegal use of forklifts, reduce labor costs and reduce the consumption of network resources.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide an image-based forklift violation monitoring method, comprising:

[0009] S101, acquire two images captured by the camera at a first preset time interval;

[0010] S102, perform frame difference binarization processing on the two images to obtain a frame difference image;

[0011] S103, for the connected regions of the frame difference image, determine the target minimum bounding rectangle, wherein the connected region is an image region composed of pixels with the same pixel value and adjacent positions in the frame difference image;

[0012] S104, for each of the target minimum bounding rectangles, determine whether the width-to-length ratio of the target minimum bounding rectangle is less than a first preset threshold.

[0013] S105, if the width-to-length ratio is less than the first preset threshold, then the forklift corresponding to the target minimum bounding rectangle is determined to be in violation.

[0014] Furthermore, the step of performing frame difference binarization on the two images to obtain a frame difference image includes:

[0015] The two images are subjected to frame difference binarization to obtain the first black and white image;

[0016] A morphological erosion operation is performed on the first black and white image to obtain a second black and white image;

[0017] A morphological dilation operation is performed on the second black-and-white image to obtain a frame difference image.

[0018] Furthermore, determining the minimum bounding rectangle of the target for the connected regions of the frame difference image includes:

[0019] For the connected regions of the frame difference image, a set of several minimum bounding rectangles is obtained;

[0020] Determine whether the area of ​​each of the minimum bounding rectangles is less than a second preset threshold;

[0021] If the area is less than the second preset threshold, then the smallest bounding rectangle smaller than the second preset threshold will be removed from the set;

[0022] Each of the minimum bounding rectangles in the set of several minimum bounding rectangles after deletion is determined as the target minimum bounding rectangle.

[0023] Furthermore, the step of determining the forklift violation corresponding to the target minimum bounding rectangle if the width-to-length ratio is less than a first preset threshold includes:

[0024] If the aspect ratio of the target minimum bounding rectangle is less than the first preset threshold, the target minimum bounding rectangle region is cropped from the two images corresponding to the frame difference image to which the target minimum bounding rectangle belongs, resulting in two sub-images.

[0025] The two sub-graphs are input into the classification model for classification, resulting in the first sub-graph containing the forklift.

[0026] The forklift in the first sub-graph is identified as an illegal forklift.

[0027] Furthermore, before the step of identifying the forklift in the first sub-graph as an illegal forklift, the method further includes:

[0028] Based on the monitoring time in the image to which the first sub-image belongs, retrieve video clips of a second preset duration before and after the monitoring time;

[0029] Get two images of any two adjacent frames in the video segment, repeat steps S101 to S104, and for the target minimum bounding rectangle whose aspect ratio is less than the first preset threshold, determine whether the distance between the center point of the target minimum bounding rectangle in the two images is greater than the third preset threshold.

[0030] If the distance is greater than the third preset threshold, the forklift in the first sub-graph is identified as an illegal forklift.

[0031] Furthermore, the classification model is the deep learning model EfficientNet.

[0032] Secondly, embodiments of the present invention also provide an image-based forklift violation monitoring method, which, after the step of identifying the forklift in the first sub-image as a violation forklift, further includes:

[0033] Obtain the image of the person violating the rules from the first sub-image;

[0034] Based on the employee database information, the images of the individuals who violated the rules are input into a facial recognition model for comparison to obtain the corresponding information of the employees who violated the rules. The employee database information is obtained in advance.

[0035] Based on the information of the employee who violated the rules, a violation tag is added to the employee database.

[0036] Thirdly, embodiments of the present invention also provide a computer storage medium storing a computer program thereon, which, when executed by a processor, implements any of the image-based forklift violation monitoring methods described above.

[0037] Fourthly, embodiments of the present invention also provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the image-based forklift violation monitoring methods described above.

[0038] Compared with the prior art, the beneficial effects of the present invention are: performing frame difference binarization processing on two images captured by the camera at a first preset time interval to obtain a frame difference image; determining the target minimum bounding rectangle for the connected regions in the frame difference image; and determining whether there is a forklift misuse behavior based on whether the width-to-length ratio of the target minimum bounding rectangle is less than a first preset threshold, which can reduce the consumption of network resources and reduce labor costs. Attached Figure Description

[0039] Figure 1 A first flowchart of an image-based forklift violation monitoring method provided in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the first image obtained according to an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a second image obtained after a 2-minute interval, provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a frame difference image provided in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of a frame difference image obtained by a pre-erosion and post-dilation operation according to an embodiment of the present invention.

[0044] Figure 6 A schematic diagram of a connected region provided in an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of the minimum bounding rectangle provided in an embodiment of the present invention;

[0046] Figure 8 This is a schematic diagram of all the smallest bounding rectangles within a frame difference image provided in an embodiment of the present invention.

[0047] Figure 9 This is a schematic diagram of a target minimum bounding rectangle provided in an embodiment of the present invention;

[0048] Figure 10 A second flowchart of an image-based forklift violation monitoring method provided in an embodiment of the present invention;

[0049] Figure 11 This is a schematic diagram of the target minimum bounding rectangle in the first image provided by an embodiment of the present invention;

[0050] Figure 12 A schematic diagram of the target minimum bounding rectangle in a second image provided by an embodiment of the present invention;

[0051] Figure 13A schematic diagram of the first sub-diagram provided in an embodiment of the present invention;

[0052] Figure 14 This is a schematic diagram excluding forklifts provided in an embodiment of the present invention;

[0053] Figure 15 This is a third flowchart of an image-based forklift violation monitoring method provided in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] To address the aforementioned problems in the prior art, this invention discloses an image-based forklift violation monitoring method. It should be noted that the executing entity in this invention can be a computer or any electronic device with data processing capabilities. For ease of description, electronic devices will be used as the executing entity in the following detailed description.

[0056] Example 1:

[0057] like Figure 1 As shown, Figure 1 A flowchart of an image-based forklift violation monitoring method provided in an embodiment of the present invention includes:

[0058] Step S101: Acquire two images taken by the camera at a first preset time interval.

[0059] In work environments such as express delivery sorting where forklifts are prone to improper operation, surveillance cameras are pre-installed to monitor and capture images of the work site in real time. Electronic devices acquire two images captured by the camera at a first preset time interval. In this embodiment of the invention, the first preset time interval is set to 2 minutes. The specific first preset time interval can be set according to actual conditions such as the volume of express deliveries and the number of employees. This invention does not limit this setting.

[0060] like Figure 2 and Figure 3 As shown, Figure 2 This is a schematic diagram of the first image obtained according to an embodiment of the present invention. Figure 3 This is a schematic diagram of a second image obtained after a 2-minute interval, as provided in an embodiment of the present invention. Figure 2 The monitoring time was 14:12, and no forklift was present in the work area. Figure 3The monitoring time is 14:14, which shows that a forklift was used in violation of regulations. The person in violation placed two pallets one in front of the other, which could easily lead to the goods falling or colliding.

[0061] Step S102: Perform frame difference binarization on the two images to obtain the frame difference image.

[0062] The frame difference algorithm is applied to the two images obtained in step S101. When a moving object appears in the monitored scene, there will be a significant difference between the frames. The absolute value of the brightness difference between the two frames is obtained by subtracting the two frames. It is then determined whether the value is greater than a threshold to analyze the motion characteristics of the image sequence and determine whether there is any moving object in the image sequence. If it is determined that there is moving object, a binarization operation is performed. The binarization threshold is set to 30. That is, if the pixel value of a pixel is greater than 30, the pixel value is set to 255, i.e., black; otherwise, it is set to 0, i.e., white, thus obtaining the frame difference image.

[0063] like Figure 4 As shown, Figure 4 This is a schematic diagram of a frame difference image provided in an embodiment of the present invention. Because... Figure 3 and Figure 2 In contrast, newly appearing or moved objects will be captured by the frame difference algorithm, and the resulting frame difference image after binarization is a black and white image.

[0064] Generally, one second of video consists of 24 images, so downloading video consumes significantly more bandwidth and network resources compared to downloading images. During off-peak hours, surveillance images within a few minutes are almost identical. Directly extracting video segments for classification is not only time-consuming but also wastes network resources. This invention utilizes two images spaced several minutes apart, combined with frame difference binarization processing, to filter out time periods where the monitored content remains unchanged, thereby saving time and network resources.

[0065] In one embodiment, step S102 above can be further subdivided into the following steps:

[0066] Step S1021: Perform frame difference binarization on the two images to obtain the first black and white image.

[0067] Similar to step S102, frame difference binarization is performed on the two images, and the resulting image is used as the first black and white image. Figure 4 .

[0068] Step S1022: Perform morphological erosion on the first black and white image to obtain the second black and white image.

[0069] The first black and white image is eroded to obtain the second black and white image. The effect of erosion is to "slim down" the image. The principle is to find the local minimum value in a small area of ​​the original image. Because it is a binary image, the pixel value is only 0 and 255. Therefore, if there is a value of 0 in a small area, the pixel value of that pixel will be 0.

[0070] Step S1023: Perform morphological dilation on the second black and white image to obtain a frame difference image.

[0071] Applying dilation and erosion to the second black-and-white image yields a frame difference image. Dilation is the opposite of erosion, targeting local maxima, effectively "fatting" the image. Erosion followed by dilation can be used to eliminate white speckles in a black background, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of a frame difference image obtained by erosion followed by dilation operation, provided in an embodiment of the present invention. Figure 5 and Figure 4 In comparison, some noise interference has been eliminated.

[0072] Step S103: For the connected regions of the frame difference image, determine the minimum bounding rectangle of the target, wherein the connected region is the image region composed of pixels with the same pixel value and adjacent positions in the frame difference image.

[0073] by Figure 6 For example, Figure 6 This is a schematic diagram of a connected region provided in an embodiment of the present invention. It is clearly visible that... Figure 6 There are 3 image regions consisting of pixels with the same pixel value and adjacent positions. The minimum bounding rectangle of the target is the minimum bounding rectangle of the connected region, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of the minimum bounding rectangle provided in an embodiment of the present invention. Figure 7 The region resembling the number "3" in the middle represents the outline of the connected region. Both the green and blue rectangles are the outer rectangles of this connected region, but the blue rectangle is the smallest outer rectangle, which is the target smallest outer rectangle.

[0074] In one embodiment, step S103 above can be further subdivided into the following steps:

[0075] Step S1031: For the connected regions of the frame difference image, obtain a set of several minimum bounding rectangles;

[0076] like Figure 8 As shown, Figure 8 This is a schematic diagram illustrating all minimum bounding rectangles within a frame difference image provided in an embodiment of the present invention. For all connected regions of the frame difference image, several minimum bounding rectangles are obtained, and these minimum bounding rectangles are treated as a set.

[0077] Step S1032: Determine whether the area of ​​each minimum bounding rectangle is less than the second preset threshold.

[0078] In step S1031, it is determined whether the area of ​​the smallest bounding rectangle is less than a second preset threshold. In this embodiment of the invention, the second preset threshold can be set to 2500, but this is not a limitation. If the area of ​​the smallest bounding rectangle is too small, it may be an interference item such as pedestrians or goods, so it needs to be excluded.

[0079] Step S1033: If the area is less than the second preset threshold, then the smallest bounding rectangle that is less than the second preset threshold is deleted from the set.

[0080] If the area of ​​the minimum bounding rectangle is less than the second preset threshold of 2500, then the minimum bounding rectangles with an area less than 2500 are deleted from the above set, and the minimum bounding rectangles with an area greater than 2500 are retained.

[0081] Step S1034: Determine each of the minimum bounding rectangles in the set of several minimum bounding rectangles after deletion as the target minimum bounding rectangle.

[0082] Define several minimum bounding rectangles with an area greater than 2500 as the target minimum bounding rectangle. For example... Figure 9 As shown, Figure 9 This is a schematic diagram of a target minimum bounding rectangle provided in an embodiment of the present invention.

[0083] Step S104: For each target minimum bounding rectangle, determine whether the width-to-length ratio of the target minimum bounding rectangle is less than a first preset threshold.

[0084] When a forklift is used to push a pallet normally, the width-to-length ratio is approximately 0.5. Setting the first preset threshold to 0.5, the more pallets pushed, the longer the target minimum bounding rectangle becomes, and the smaller the width-to-length ratio. For each target minimum bounding rectangle, determine whether its width-to-length ratio is less than 0.5.

[0085] Step S105: If the width-to-length ratio is less than the first preset threshold, then the forklift violation corresponding to the target minimum bounding rectangle is determined.

[0086] If the width-to-length ratio is less than 0.5, it means that multiple pallets are connected to the forklift, so it can be determined that the forklift corresponding to the smallest bounding rectangle of the target is operating in violation of regulations.

[0087] In one embodiment, such as Figure 10 As shown, Figure 10 The second flowchart of the image-based forklift violation monitoring method provided in this embodiment of the invention, step S105 can be further subdivided into the following steps:

[0088] Step S1051: If the aspect ratio of the target minimum bounding rectangle is less than the first preset threshold, the target minimum bounding rectangle region is cropped from the two images corresponding to the frame difference image to which the target minimum bounding rectangle belongs, resulting in two sub-images.

[0089] Figure 9 The aspect ratio of the minimum bounding rectangle of the target is less than 0.5, and its positions in the two corresponding images are as follows: Figure 11 and Figure 12 As shown, Figure 11 This is a schematic diagram of the target minimum bounding rectangle in the first image provided by an embodiment of the present invention. Figure 12 This is a schematic diagram of the target minimum bounding rectangle in a second image, provided by an embodiment of the present invention. Figure 11 and Figure 12 In the middle, the minimum bounding rectangle of the target is cropped to obtain two sub-images.

[0090] Step S1052: Input the two sub-images into the classification model for classification to obtain the first sub-image containing the forklift.

[0091] In one embodiment, two sub-images are input into the EfficientNet classification model for classification. This allows for the detection of whether a forklift is present in the sub-image, resulting in a first sub-image containing the forklift, as shown below. Figure 13 and Figure 14 As shown, Figure 13 This is a schematic diagram of a first sub-figure provided in an embodiment of the present invention, the first sub-figure containing a forklift. Figure 14 This is a schematic diagram of an embodiment of the present invention without a forklift.

[0092] Step S1053: Identify the forklift in the first sub-graph as an illegal forklift.

[0093] The electronic device identifies the forklift shown in the first sub-image as an illegal forklift and can store the first sub-image in its memory for inspection personnel to review.

[0094] In one embodiment, such as Figure 15 As shown, Figure 15 The third flowchart of the image-based forklift violation monitoring method provided in this embodiment of the invention further includes the following steps before step S1053:

[0095] Step S0001: Based on the monitoring time in the image to which the first sub-image belongs, retrieve video clips of a second preset duration before and after the monitoring time;

[0096] Step S0002: Obtain two images from any two adjacent frames in the video clip, and repeat steps S101 to S104.

[0097] Step S0003: For the target minimum bounding rectangle whose width-to-length ratio is less than the first preset threshold, determine whether the distance between the center point of the target minimum bounding rectangle in the two images is greater than the third preset threshold.

[0098] If the distance between the center point and the two images is greater than the third preset threshold, the forklift in the first sub-image will be identified as an illegal forklift.

[0099] The electronic device retrieves video clips, takes two images from any adjacent frames, and repeats steps S101 to S104 to obtain two minimum bounding rectangles of the target. If the forklift in the first sub-image is being pushed, the center points of the two minimum bounding rectangles of the target will be a certain distance apart. The third preset threshold can be set to 50. This invention does not limit this. If the distance between the center points is greater than 50, it indicates that the forklift is indeed being used illegally, further improving the accuracy of forklift violation monitoring.

[0100] Example 2:

[0101] Following step S105 in Embodiment 1, the method further includes:

[0102] Step S106: Obtain the image of the person violating the rules from the first sub-image;

[0103] Step S107: Based on the employee database information, input the image of the person who violated the rules into the face recognition model for comparison to obtain the corresponding information of the employee who violated the rules. The employee database information is obtained in advance.

[0104] Step S108: Based on the information of the employees who violated the rules, add a violation tag to the employee database information.

[0105] Example 3:

[0106] Corresponding to the embodiment of the image-based forklift violation monitoring method, this embodiment of the invention also provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described image-based forklift violation monitoring methods.

[0107] Example 4:

[0108] Corresponding to the embodiment of the image-based forklift violation monitoring method, this embodiment of the invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described image-based forklift violation monitoring methods.

[0109] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

Claims

1. An image-based method for detecting forklift violations, characterized in that, Includes the following steps: S101, acquire two images captured by the camera at a first preset time interval; S102, perform frame difference binarization processing on the two images to obtain a frame difference image; S103, for the connected regions of the frame difference image, determine the target minimum bounding rectangle, wherein the connected region is an image region composed of pixels with the same pixel value and adjacent positions in the frame difference image; S104, for each of the target minimum bounding rectangles, determine whether the width-to-length ratio of the target minimum bounding rectangle is less than a first preset threshold. S105, if the width-to-length ratio is less than the first preset threshold, then the forklift corresponding to the target minimum bounding rectangle is determined to be in violation. Determining the minimum bounding rectangle of the target for the connected regions of the frame difference image includes: For the connected regions of the frame difference image, a set of several minimum bounding rectangles is obtained; Determine whether the area of ​​each of the minimum bounding rectangles is less than a second preset threshold; If the area is less than the second preset threshold, then the smallest bounding rectangle smaller than the second preset threshold will be removed from the set; Each of the minimum bounding rectangles in the set of several minimum bounding rectangles after deletion is determined as the target minimum bounding rectangle; The step of determining the forklift violation corresponding to the target minimum bounding rectangle if the width-to-length ratio is less than a first preset threshold includes: If the aspect ratio of the target minimum bounding rectangle is less than the first preset threshold, the target minimum bounding rectangle region is cropped from the two images corresponding to the frame difference image to which the target minimum bounding rectangle belongs, resulting in two sub-images. The two sub-graphs are input into the classification model for classification, resulting in the first sub-graph containing the forklift. The forklift in the first sub-graph is identified as an illegal forklift. Before the step of identifying the forklift in the first sub-graph as an illegal forklift, the method further includes: Based on the monitoring time in the image to which the first sub-image belongs, retrieve video clips of a second preset duration before and after the monitoring time; Get two images of any two adjacent frames in the video segment, repeat steps S101 to S104, and for the target minimum bounding rectangle whose aspect ratio is less than the first preset threshold, determine whether the distance between the center point of the target minimum bounding rectangle in the two images is greater than the third preset threshold. If the distance is greater than the third preset threshold, the forklift in the first sub-graph is identified as an illegal forklift.

2. The image-based forklift violation monitoring method according to claim 1, characterized in that, The step of performing frame difference binarization on the two images to obtain a frame difference image includes: The two images are subjected to frame difference binarization to obtain the first black and white image; A morphological erosion operation is performed on the first black and white image to obtain a second black and white image; A morphological dilation operation is performed on the second black-and-white image to obtain a frame difference image.

3. The image-based forklift violation monitoring method according to claim 1, characterized in that, The classification model is the deep learning model EfficientNet.

4. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 3.

5. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 3.

Images