Forklift carrying length detection and track compliance judgment method and system, terminal and medium

Abstract

This invention relates to the field of visual inspection and forklift operation safety management technology, specifically a method, system, terminal, and medium for detecting the length of goods carried by a forklift and judging the compliance of its trajectory. The method includes: acquiring video frame images of goods transported by a forklift in real time at intervals over a time length t to form a sequence [F1, F2, ..., F, ..., F]; calculating the optical flow vector field between adjacent frames in the sequence; filtering the optical flow vector regions; extracting the region of interest (ROI) in F; calculating and outputting the polygonal point coordinates of the forklift outline, the bounding box coordinates of the forklift tires, and the forklift mask using a target detection model; and detecting whether the length of goods carried by the forklift and the forklift trajectory violate regulations based on the output of the target detection model. This invention can solve the problem of being unable to judge the compliance of forklift trajectories when the transported goods are unknown, and can avoid using positioning technology for forklift trajectory compliance judgment.

Description

Technical Field

[0001] This invention relates to the field of visual inspection and forklift operation safety management technology, specifically to a method, system, terminal, and medium for detecting the length of goods carried by a forklift and judging the compliance of its trajectory. Background Technology

[0002] In port logistics handling, forklifts play a crucial role in cargo transfer, such as the transfer of steel structures. When forklifts are transferring goods within the transfer aisles, if the goods being transferred are too long, they must travel in a straight line within the aisle. Violation of this rule (not traveling in a straight line) can easily lead to scraping against the aisle, colliding with equipment, or even causing the forklift to lose balance and tip over. Therefore, during the transfer process, the length of the goods carried by the forklift is often monitored to determine whether the forklift's trajectory is compliant if the length exceeds a certain threshold.

[0003] Currently, for forklift carrying length detection, traditional object detection algorithms are typically used, such as template matching based on manual features and general CNN object detection models, to identify the forklift carrying goods and then detect the length of the goods. However, these traditional object detection algorithms usually rely on the fixed shape and texture features of the goods carried by the forklift, meaning they are often based on the premise that the goods carried by the forklift are known. They are not applicable when the goods being transferred are unknown, making it impossible to determine whether the forklift trajectory is compliant when the goods being transferred are unknown.

[0004] In addition, when judging whether a forklift trajectory is compliant, positioning technology is often used to locate the forklift trajectory for the purpose of judging compliance. However, positioning technology is greatly affected by environmental interference, which can easily lead to misjudgment or omission. Summary of the Invention

[0005] To address the above shortcomings, this invention provides a method, system, terminal, and medium for detecting the length of goods carried by a forklift and judging the compliance of its trajectory, thereby solving at least one of the problems mentioned above.

[0006] In a first aspect, the present invention provides a method for detecting the length of goods carried by a forklift and determining the compliance of its trajectory, the method comprising: Step 101: Preset detection time intervals. Within a time length t, video frame images of forklifts transferring goods in a target scenario are acquired in real time, forming a temporal sequence of video frame images. F1,F2,…,F ,…,F ; Represents the total number of video frame images in a video frame image sequence; Step 102: Calculate the optical flow vector field between adjacent frames in the video frame image sequence; Step 103: For each calculated optical flow vector field, select optical flow vectors with displacements greater than a preset displacement threshold from the optical flow vector field to form an optical flow vector region; Step 104: Based on the formed optical flow vector region, extract the corresponding region of each video frame image in the video frame image sequence to obtain a set of regions of interest. , ,…, ,…, , Represents video frame image F Region of interest in; Step 105: Select each region of interest. Input the trained object detection model to identify forklifts and forklift tires, and segment the forklifts, outputting each... The coordinates of the polygonal points of the forklift outline, the bounding box coordinates of the forklift tires, and the output of each forklift mask in ; Step 106: Based on the output forklift mask and output The bounding box coordinates of the forklift tires are used to detect whether the load carried by the forklift exceeds the length limit; when the load exceeds the length limit, based on the output... The coordinates of polygonal points on the outline of the forklift are used to detect whether the forklift's trajectory violates regulations.

[0007] In an optional implementation, in step 104, based on the formed optical flow vector region, the corresponding region of each video frame in the video frame image sequence is extracted to obtain a set of regions of interest. , ,…, ,…, ,include: Step 1041: For each formed optical flow vector region, record the starting coordinates of each optical flow vector in the optical flow vector region to obtain the pixel coordinates in the previous video frame image corresponding to that optical flow vector region; for the video frame image sequence... F1,F2,…,F ,…,F The optical flow vector region formed by the optical flow vector location between the last two images is also calculated and recorded as the pixel coordinates to which the starting coordinates of each optical flow vector in the optical flow vector region move in the video frame image Fn; Step 1042: For each video frame image F The video frame image F recorded in step 1041 Find the minimum x-coordinate (xmin), maximum x-coordinate (xmax), minimum y-coordinate (ymin), and maximum y-coordinate (ymax) from all pixel coordinates in the video frame image F. Extract the rectangular region defined by the minimum x-coordinate (xmin), maximum x-coordinate (xmax), minimum y-coordinate (ymin), and maximum y-coordinate (ymax) to obtain the video frame image F. Region of Interest ,in .

[0008] In an optional implementation, in step 106, based on the output forklift mask... and output The bounding box coordinates of the forklift tires are used to detect whether the cargo carried by the forklift exceeds the length limit, including: Step 1061, based on output Calculate the bounding box coordinates of the forklift tires. Diameter length of forklift tires ; Step 1062: Mask the forklift By extracting the image difference, we can obtain Area of ​​goods carried by the forklift ; Step 1063: Inspect the area carrying the goods on the forklift. After performing morphological processing, the morphologically processed area of ​​the forklift's cargo is output. pixel outline ; Step 1064: Based on the pixel contour Calculate the predicted length of the goods carried by the forklift. ; Step 1065: Based on the calculated length , Calculate the actual length of the goods carried by the forklift. The calculation formula is: , In the formula, This represents the actual diameter length of the forklift tire. Step 1066: Calculate the actual length of the goods carried by the forklift. To check whether the goods carried by the forklift are too long.

[0009] In an optional implementation, in step 1061, based on the output Calculate the bounding box coordinates of the forklift tires. Diameter length of forklift tires ,include: Output-based Obtain the bounding box coordinates of the forklift tires. Height and width of forklift tires; Calculate the average of the height and width of the forklift tires to obtain the diameter. .

[0010] In an optional implementation, step 1064, based on the pixel contour... Calculate the predicted length of the goods carried by the forklift. ,include: Step 10641: Obtain pixel contours The minimum bounding rectangle; Step 10642: Obtain the pixel length of the longer side of the minimum bounding rectangle. This pixel length is the predicted length of the goods carried by the forklift. .

[0011] In an optional implementation, in step 106, based on the output The coordinates of polygonal points on the forklift outline are used to detect whether the forklift's trajectory violates regulations, including: Step S1, based on output Obtain the coordinates of the polygonal points of the forklift outline. The center point of the forklift is used as the video frame image F. Center point of the forklift , ; Step S2: Using the center point , Generate a straight line (Line); Step S3, from the center point Begin by iterating through the center points one by one. , … and ; Step S4: For the currently traversed center point, determine whether it is located on or near the line Line: If so, the forklift trajectory is determined to be within acceptable limits, and the process continues to iterate through the next center point until all center points have been traversed. ; If not, the forklift trajectory is deemed to be in violation.

[0012] In one optional implementation, the trained object detection model is obtained by means of: Constructing a training set: The method for constructing the training set includes: collecting video frame images of forklifts transporting goods in the target scenario, the collected images covering the forklift's empty working condition and the working condition carrying goods of each pre-specified shape, and each collected image including the side of the forklift and its tires; resampling each collected video frame image to 640 pixels × 640 pixels to obtain each resampled image; labeling each resampled image as follows: normalizing the polygon point coordinates and normalizing the bounding box of the forklift and forklift tires in the image, with the forklift and forklift tires corresponding to the category labels "0" and "1"; and combining the labeled images to form the training set. Load the pre-trained YOLOv11 model as the initial model; The initial model is trained using the training set to obtain a trained object detection model.

[0013] Secondly, this invention provides a forklift carrying length detection and trajectory compliance judgment system, the system comprising: The image acquisition module is used to perform preset detection time intervals. Within a time duration t, video frame images of forklifts transferring goods in the target scenario are acquired in real time, forming a temporally ordered video frame image sequence [F1, F2, ..., F...]. ,…,F ]; Represents the total number of video frame images in a video frame image sequence; The optical flow calculation module is used to calculate the optical flow vector field between adjacent frames in a video frame image sequence; The filtering module is used to filter out optical flow vectors with displacements greater than a preset displacement threshold from each calculated optical flow vector field to form an optical flow vector region; The image extraction module is used to extract the corresponding region of each video frame in the video frame image sequence based on the formed optical flow vector region, thereby obtaining a set of regions of interest. , ,…, ,…, , Represents video frame image F Region of interest in; The model calculation module is used to calculate the region of interest. Input a pre-trained object detection model to identify forklifts and forklift tires, and segment the forklifts; output... The coordinates of the polygonal points of the forklift outline, the bounding box coordinates of the forklift tires, and the output... forklift mask in ; The decision module is used based on the output forklift mask. and output The bounding box coordinates of the forklift tires are used to detect whether the load carried by the forklift exceeds the length limit; when the load exceeds the length limit, based on the output... The coordinates of polygonal points on the outline of the forklift are used to detect whether the forklift's trajectory violates regulations.

[0014] Thirdly, the present invention provides a terminal, comprising: Processor, memory, among which, This memory is used to store computer programs. The processor is used to retrieve and run the computer program from memory, enabling the terminal to perform the methods described above.

[0015] Fourthly, a computer storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: This invention uses a combined strategy of temporal video frame sequence and optical flow vector filtering to avoid the use of traditional target detection algorithms. This avoids the dependence of target detection on the fixed shape and texture features of the goods carried by the forklift, enabling target detection even when the forklift is carrying unknown goods. This helps to solve the problem of not being able to determine the compliance of the forklift trajectory when the transported goods are unknown.

[0017] This invention avoids using positioning technology to locate forklift trajectories for determining whether forklift trajectories are compliant, thereby solving the problem that positioning technology is easily subject to environmental interference, which can lead to misjudgments or omissions when used to determine the compliance of forklift trajectories.

[0018] This invention, when detecting that the forklift is carrying excessively long cargo, can, based on the output... Obtain the coordinates of the polygonal points of the forklift outline. The center point of the forklift is used as the video frame image F. Center point of the forklift , Then we can use the center point , A straight line (Line) is generated, and the forklift's trajectory is judged to be non-compliant by determining whether the forklift's subsequent center point lies on or near the Line. This method helps to integrate visual tracking with operational rules (i.e., judging whether the forklift's trajectory is non-compliant when the forklift is carrying excessively long goods), avoiding the use of positioning technology, and helping to more accurately identify trajectory violations, reducing false positives or false negatives. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of a computer storage medium provided in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail. Specific details, such as particular system structures and techniques, are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention may be implemented in other embodiments without these specific details.

[0025] It should be understood that, when used in this specification, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0026] The terms "one embodiment" or "some embodiments" used in this invention mean that one or more embodiments of the invention include the specific features, structures, or characteristics described in that embodiment. Therefore, phrases such as "in some embodiments" or "in other embodiments" appearing in different parts of the invention do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It should be noted that in the same scenario of transporting goods using a forklift, only one model of forklift is used for transporting the goods.

[0029] Figure 1 This is a schematic flowchart illustrating a method for detecting the length of goods carried by a forklift and determining the compliance of its trajectory, provided in an embodiment of the present invention. Figure 1 The executing entity can be a forklift carrying length detection and trajectory compliance judgment system. The forklift carrying length detection and trajectory compliance judgment method provided in this embodiment of the invention is executed by a computer device; correspondingly, the forklift carrying length detection and trajectory compliance judgment system runs on the computer device.

[0030] Please refer to Figure 1 The method includes steps 101 to 106.

[0031] Step 101: Preset detection time intervals. Within a time length t, video frame images of forklifts transferring goods in the target scenario are collected in real time, forming a time-series video frame image sequence.

[0032] The time-series video frame image sequence is [F1, F2, ..., F...]. ,…,F ], F represents the total number of video frame images in the video frame image sequence. The number representing the collection A video frame image.

[0033] In specific implementation, in step 101, video frame images of forklifts transferring goods in the target scenario are captured by a camera.

[0034] Understandably, the images captured by the aforementioned camera include forklift tires.

[0035] In practice, the camera can be installed at an appropriate location in the target scenario described above, so that the camera can simultaneously capture the side of the forklift and any one of its tires, as well as a complete image of the goods carried by the forklift.

[0036] For example, in the target scenario described above, which is a forklift transfer operation lane, the camera can be a regular industrial-grade network camera (1920×1080 resolution, 25fps frame rate, 4mm focal length), deployed 30 to 50 meters above the forklift transfer operation lane. When deploying, the camera should cover the forklift operation area within the forklift transfer operation lane and be able to capture the working forklift, the goods carried by the forklift, and the side of one of the forklift's tires in the same frame.

[0037] In practical implementation, t cannot be too small or too large. After obtaining the video frame image sequence, this invention requires subsequent calculations including optical flow calculation, region of interest extraction, and target detection model computation. t is too small, for example If t < 10ms, subsequent calculations such as optical flow calculation, region of interest extraction, and target detection model operation may not be completed, leading to image stacking and latency in the acquired video frames, affecting the real-time performance of forklift carrying length detection and forklift trajectory violation judgment; if If t is too large, there may be a long period of time before detection is performed after multiple steps of calculation, such as optical flow calculation, region of interest extraction, and target detection model operation, thus missing the opportunity for risk warning.

[0038] Therefore, it can be understood that the time interval The settings can be configured by those skilled in the art based on the time required to complete multiple steps such as optical flow calculation, region of interest extraction, and target detection model calculation.

[0039] For example, assuming that the maximum time required for subsequent steps 102 to 106 to complete normally is t1, then it can be set to... Or slightly more big.

[0040] For example, the aforementioned time length t can be set by those skilled in the art according to actual conditions, such as 15 to 30 seconds. Setting a lower limit of 15 seconds helps ensure the number of sampling points within t, so as to avoid capturing only 1 to 2 frames of images, which would lead to detection failure when encountering brief occlusions (such as ground debris obscuring the tires). Setting an upper limit of 30 seconds helps control the amount of sampling data, so as to avoid a decrease in algorithm efficiency and the accumulation of interference.

[0041] Step 102: Calculate the optical flow vector field between adjacent frames in the video frame image sequence.

[0042] Step 102 specifically includes: Calculate the optical flow vector field GS1 from F1 to F2; Calculate the optical flow vector field GS2 from F2 to F3; … Calculate F To F Optical flow vector field GS , .

[0043] Step 103: For each calculated optical flow vector field, select optical flow vectors with displacements greater than a preset displacement threshold from the optical flow vector field to form an optical flow vector region.

[0044] Specifically, it includes: Optical flow vectors with displacements greater than a preset displacement threshold are selected from the optical flow vector field GS1 to form the optical flow vector region QY1; Optical flow vectors with displacements greater than a preset displacement threshold are selected from the optical flow vector field GS2 to form the optical flow vector region QY2; … From the optical flow vector field GS Optical flow vector regions QY are formed by selecting optical flow vectors whose displacement is greater than a preset displacement threshold. .

[0045] In this embodiment, each optical flow vector in the optical flow vector field is represented as (x,y,u,v), where (x,y) are the pixel coordinates in the previous frame image, representing the starting coordinates of the optical flow vector; (u,v) is the vector component, and the pixel coordinates of the position moved by the pixel coordinates (x,y) in the next frame are (x+u,y+v), and the displacement of the vector component (u,v) is... .

[0046] In practice, the aforementioned preset displacement threshold can also be set by those skilled in the art based on actual conditions. For example, the preset displacement threshold can be set to 5 pixels per frame.

[0047] Forklifts typically operate at low speeds in ports, usually ≤8km / h when unloaded and ≤5km / h when fully loaded. The standard resolution for video capture is 1080P with a frame rate of 30 frames per second. The pixel displacement of a single frame for normal movement with slight jitter is usually 3 to 4 pixels, and 5 pixels per frame can completely cover the movement, avoiding missing any valid motion.

[0048] Step 104: Based on the formed optical flow vector region, extract the corresponding region of each video frame image in the video frame image sequence to obtain a set of regions of interest.

[0049] The region of interest for this group is , ,…, ,…, ,in Represents video frame image F The region of interest in the data.

[0050] Understandably, The video frame image F is extracted based on the formed optical flow vector region. The corresponding area.

[0051] Step 105: Input each region of interest into the trained target detection model to identify forklifts and forklift tires, and to segment the forklifts. Output the polygon point coordinates of the forklift outline and the bounding box coordinates of the forklift tires in each region of interest, as well as the forklift mask in each region of interest.

[0052] The forklift mask in the region of interest, i.e., the mask image of the forklift in the region of interest.

[0053] In this embodiment, step 105 specifically involves: selecting the region of interest. Input a pre-trained object detection model to identify forklifts and forklift tires, and segment the forklifts; output... The coordinates of the polygonal points of the forklift outline, the bounding box coordinates of the forklift tires, and the output... forklift mask in , .

[0054] The trained object detection model is able to detect regions of interest. Perform calculations and output. The bounding box coordinates of the forklift and forklift tires and forklift mask in .

[0055] Step 106: Based on the output of the target detection model, detect whether the goods carried by the forklift are too long, and when the goods carried by the forklift are detected to be too long, detect whether the forklift trajectory is illegal.

[0056] In this embodiment, step 106 specifically involves: based on the output forklift mask. and output The bounding box coordinates of the forklift tires are used to detect whether the load carried by the forklift exceeds the length limit; when the load exceeds the length limit, based on the output... The coordinates of polygonal points on the outline of the forklift are used to detect whether the forklift's trajectory violates regulations.

[0057] In practice, a length threshold (e.g., 8 meters) can be set by those skilled in the art according to industry requirements to detect whether the goods carried by the forklift exceed the length limit. If the length of the goods carried by the forklift exceeds the length threshold, the goods carried by the forklift are considered to be over-length; otherwise, they are not considered to be over-length.

[0058] In one embodiment, in step 104 above, based on the formed optical flow vector region, the corresponding region of each video frame image in the video frame image sequence is extracted to obtain a set of regions of interest. , ,…, ,…, The implementation methods include: Step 1041: For each formed optical flow vector region, record the starting coordinates of each optical flow vector in the optical flow vector region to obtain the pixel coordinates in the previous video frame image corresponding to that optical flow vector region; for the video frame image sequence... F1,F2,…,F ,…,F The optical flow vector region formed by the optical flow vector location between the last two images is also calculated and recorded as the pixel coordinates to which the starting coordinates of each optical flow vector in the optical flow vector region move in the video frame image Fn; Step 1042: For each video frame image F ( ), from the video frame image F recorded in step 1041 Find the minimum x-coordinate (xmin), maximum x-coordinate (xmax), minimum y-coordinate (ymin), and maximum y-coordinate (ymax) from all pixel coordinates in the video frame image F. Extract the rectangular region defined by the minimum x-coordinate (xmin), maximum x-coordinate (xmax), minimum y-coordinate (ymin), and maximum y-coordinate (ymax) to obtain the video frame image F. Region of Interest .

[0059] Taking the optical flow vector region QY1 as an example, QY1 is formed by optical flow vectors whose displacement is greater than a preset displacement threshold selected from the optical flow vector field GS1. GS1 is the calculated optical flow vector field from F1 to F2. The previous video frame image corresponding to the optical flow vector region QY1 is F1. The starting coordinates of each optical flow vector in the optical flow vector region QY1 are the pixel coordinates in the previous video frame image F1 corresponding to the optical flow vector region QY1. These pixel coordinates are recorded in step 1041, which is the video frame image F1 recorded in step 1041 mentioned in step 1042. All pixel coordinates in.

[0060] In another embodiment, in step 106 above, based on the output forklift mask and output The bounding box coordinates of the forklift tires are used to detect whether the cargo carried by the forklift exceeds the length limit, including: Step 1061, based on output Calculate the bounding box coordinates of the forklift tires. Diameter length of forklift tires ; Step 1062: Mask the forklift By extracting the image difference, we can obtain Area of ​​goods carried by the forklift ; Step 1063: Inspect the area carrying the goods on the forklift. After performing morphological processing, the morphologically processed area of ​​the forklift's cargo is output. pixel outline ; Step 1064: Based on the pixel contour Calculate the predicted length of the goods carried by the forklift. ; Step 1065: Based on the calculated length , Calculate the actual length of the goods carried by the forklift. The calculation formula is: , In the formula, This represents the actual diameter length of the forklift tire. Step 1066: Calculate the actual length of the goods carried by the forklift. To check whether the goods carried by the forklift are too long.

[0061] Understandably, this applies to the calculation of the actual length of the goods carried by the forklift. , As long as one of them If the value exceeds the length threshold, the forklift is carrying goods that are too long. ( If the values ​​of all items do not exceed the length threshold, then the goods carried by the forklift are not excessive in length.

[0062] In step 1066, if the forklift is found to be carrying goods that do not exceed the length limit, the detection result can be reported to the monitoring center for monitoring purposes only; if the forklift is found to be carrying goods that exceed the length limit, the detection result can be reported to the monitoring center, and the forklift's trajectory can be checked for violations.

[0063] Morphological treatment can enhance the continuity of cargo outlines and reduce edge breaks caused by changes in lighting or occlusion.

[0064] This invention is based on the calculation formula Calculate the actual length of the goods carried by the forklift. Calculation formula A mapping between dynamic pixels and actual length was established, enabling adaptation to different forklift operating scenarios, such as forklift fork replacement and camera angle adjustment.

[0065] Optionally, in step 1061, based on the output Calculate the bounding box coordinates of the forklift tires. Diameter length of forklift tires ,include: Output-based Obtain the bounding box coordinates of the forklift tires. Height and width of forklift tires; Calculate the average of the height and width of the forklift tires to obtain the diameter. .

[0066] Optionally, the bounding box coordinates of the forklift tire output by the object detection model are represented as (xcenter, ycenter, width, height), where (xcenter, ycenter) represents the center coordinates of the bounding box, and width and height represent the width and height of the bounding box, with coordinate values ​​in pixels.

[0067] Understandably, in the bounding box coordinates of the forklift tire output by the object detection model, the width and height of the bounding box represent the width and height of the forklift tire.

[0068] In step 1062 above, the forklift mask In the image, areas with a pixel value of 1 represent forklifts, and areas with a pixel value of 0 represent goods.

[0069] Optionally, in step 1063 above, the area of ​​the goods carried by the forklift is... Morphological processing includes: right First, a 3×3 rectangular structuring element is used to perform a closing operation to fill small holes in the cargo area and make the outline more complete. Then, a 3×3 structuring element is used to perform an opening operation to remove isolated noise points.

[0070] Understandably, the pixel contour in this embodiment refers to the set of pixel coordinates of the contour.

[0071] Optionally, in step 1064, based on the pixel contour Calculate the predicted length of the goods carried by the forklift. ,include: Step 10641: Obtain pixel contours The minimum bounding rectangle; Step 10642: Obtain the pixel length of the longer side of the minimum bounding rectangle. This pixel length is the predicted length of the goods carried by the forklift. .

[0072] In practice, those skilled in the art can implement this step 10641 using any relevant existing technology. For example, the minAreaRect function provided by OpenCV can be used to calculate the pixel contour. The minimum bounding rectangle of the contour. The input to the minAreaRect function is a two-dimensional point set, i.e., the outline. Given a list of pixel coordinates, this function returns a RotatedRect object containing the center coordinates, width (in pixels), and height (in pixels) of the smallest bounding rectangle. The length is obtained by taking the maximum of the width and height of this smallest bounding rectangle. .

[0073] In some embodiments of the present invention, in step 106, based on the output The coordinates of polygonal points on the forklift outline are used to detect whether the forklift's trajectory violates regulations, including: Step S1, based on output Obtain the coordinates of the polygonal points of the forklift outline. The center point of the forklift is used as the video frame image F. Center point of the forklift , ; Step S2: Using the center point , Generate a straight line (Line); Step S3, from the center point Begin by iterating through the center points one by one. , … and ; Step S4: For the currently traversed center point, determine whether it is located on or near the line Line: If so, the forklift trajectory is determined to be within acceptable limits, and the process continues to iterate through the next center point until all center points have been traversed. ; If not, the forklift trajectory is deemed to be in violation, and the process ends.

[0074] Optionally, based on output Obtain the coordinates of the polygonal points of the forklift outline. The center point of the forklift includes: Get the output The minimum bounding rectangle corresponding to the polygonal points of the forklift outline is the obtained coordinate system. The center point of the forklift.

[0075] In the method of this embodiment, each video frame image F is first acquired. Center point of the forklift A set of forklift center coordinates was obtained. … … Then based on the center coordinates of that group of forklifts … … Determine if the forklift is engaging in any trajectory violations.

[0076] Optionally, the method for determining whether the coordinates of each traversed center point are located on or near the line Line in step S4 includes: Calculate the distance d from the current center point coordinates to the line Line; Determine if the currently calculated distance d satisfies : If the conditions are met, then the forklift trajectory is not in violation of regulations; If the conditions are not met, the forklift trajectory is in violation; Among them, the above This is the set distance threshold.

[0077] Optionally, in specific implementations, the above Generally, 3 to 5 pixels can be used.

[0078] If Th is less than 3 pixels, it is easy to frequently trigger false violations. For example, pixel-level offsets caused by lighting fluctuations or slight lens shaking may be mistakenly identified as trajectory violations, which will greatly reduce the reliability of detection.

[0079] If Th exceeds 5 pixels, it usually corresponds to a path deviation of centimeters in physical space. This offset exceeds the tolerance range of normal operation and is not conducive to timely warning to avoid the risk of collision.

[0080] Understandably, in calculating the distance At that time, the center point lies on the straight line Line. The calculated distance d is greater than 0 and less than or equal to (…). ) At that time, the center point is located near the straight line Line.

[0081] As an illustrative embodiment of the present invention, the method for obtaining the trained target detection model includes: Construct the training set; Load the pre-trained YOLOv11 model as the initial model; The initial model is trained using the training set to obtain a trained object detection model.

[0082] In this embodiment, the target scenario for forklift cargo transfer is a port transfer scenario for steel structure cargo, and the cargo transferred by the forklift is steel structure cargo.

[0083] In this embodiment, the method for constructing the training set includes: Collect video frame images of forklifts transporting goods in the target scenario. The collected images cover the forklift's empty working condition and the working condition of carrying steel structure goods of each pre-specified shape. Each collected image includes the side of the forklift and its tires. Each collected video frame image was resampled to 640 pixels × 640 pixels, resulting in a resampled image. The following annotations were applied to each resampled image: Normalized polygon point coordinates and normalized bounding box labels are applied to the forklifts and forklift tires in the images, respectively. The forklifts and forklift tires are labeled with category labels "0" and "1". The labeled images are then collected to form a training set.

[0084] Understandably, the forklift and forklift tires in the image are sequentially annotated with normalized polygonal point coordinates and normalized bounding boxes, including: Normalize the polygonal point coordinates of the forklift in the image; Normalize the bounding box annotations for the forklift tires in the image.

[0085] When collecting video frame images of forklifts transporting goods in the target scenario, it is necessary to cover as many and comprehensive working conditions as possible, including those involving steel structure goods of different shapes.

[0086] In practical implementation, the pre-trained YOLOv11 model mentioned above can be yolov11s-seg.pt. During training, the training process can be started directly using the train() method of the YOLO class, with just one line of code. The train() method has encapsulated complete logic for data loading, forward inference, loss calculation, and backpropagation.

[0087] Before training begins, the following preparations should be made: 1. Create a dataset folder, and within the dataset folder, create two subfolders: "images" and "labels". Under "images", create two subfolders: "train" and "val". Under "labels", create two subfolders: "train" and "val".

[0088] The “images / train” folder contains images from the training set.

[0089] The "images / val" folder contains images from the verification set.

[0090] The “labels / train” folder contains the label files for the images in the training set; The “labels / val” folder contains the label files for the images in the validation set.

[0091] The validation set can be obtained using the same construction method as the training set. The training set contains more than 5000 images. The ratio of training set to validation set data size is 8:2.

[0092] 2. Each image in the training and validation sets is paired with a .txt file of the same name as a label file.

[0093] The format for recording tags in the tag file is as follows: Forklift labeling: 0 x_1 y_1 x_2 y_2 … x_j y_j … xm ym; Forklift tire markings: 1 x_center y_center w_x h_x.

[0094] In this context, "0" represents the forklift category label, (x_j, y_j) represents the j-th forklift outline point (marked clockwise), "1" represents the forklift tire category number, (x_center, y_center) represents the center point of the bounding box, w_x and h_x represent the width and height of the bounding box, and m represents the total number of marked points.

[0095] 3. Configuration file: Create a file named "dataset.yaml" in the dataset folder and specify the following: Image paths for training and validation sets; Number of categories (nc=2 in this invention); Category name (in this invention, forklift and forklift tire).

[0096] When starting to train the model: 1. Load the base model: Use a pre-trained YOLOv11 model (such as yolov11s-seg.pt, which balances speed and accuracy) as the starting point. The pre-trained model has already learned the general target features and can adapt to specific targets such as forklifts more quickly.

[0097] 2. Set training parameters: ① Number of training rounds: 500 rounds; ② Input dimensions: Use 640x640 (for medium-sized forklifts, this size can balance details and speed); ③ Batch size: Adjust according to the graphics card memory (for example, a single 16G graphics card can be set to 16-32 cards / batch).

[0098] 3. Start training: Specify the initial model to be loaded, the configuration file in the built dataset folder, and the training parameters set above through the tool, and start training using the train() method of the YOLO class.

[0099] During training, the model will automatically learn the shape characteristics of forklifts and forklift tires and their distribution patterns in the target scene.

[0100] The `train()` method automatically performs validation after each training round (or can be set to validate every Q rounds via `-val-intervalQ`) and saves the prediction results image for the validation set. The prediction results image includes a prediction comparison image: the predicted bounding boxes of the model for the validation set images (including the bounding boxes of the forklift and its tires, and the category labels) are overlaid with the ground truth label boxes to visually demonstrate whether the model can accurately identify the forklift.

[0101] The model obtained after training is named best.pt.

[0102] Evaluate the performance of the forklift model: Evaluate the model best.pt using the validation set.

[0103] Specifically, this includes calculating the recognition accuracy and false detection rate of the best.pt model.

[0104] If both the recognition accuracy and false detection rate meet the standards, then the model best.pt is a well-trained object detection model. Otherwise, more images can be added to the training set (such as forklifts operating at night or in different working postures), or the labels can be corrected and the model can be retrained.

[0105] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0106] As an illustrative embodiment of the present invention, before step 101, the method further includes: Real-time acquisition of video frame images of the target scene via camera; The system uses a target recognition algorithm in real time to identify forklifts in the captured video frame images. After a forklift is identified, the camera is triggered to focus on the identified forklift, and then step 101 is executed until the identified forklift leaves the field of view.

[0107] When the camera is triggered to focus, it can focus according to the actual size of the forklift. In practice, the forklift target can occupy about one-third or one-third of the image after focusing, ensuring that the goods carried by the forklift are complete.

[0108] Figure 2 The present invention provides a forklift carrying length detection and trajectory compliance judgment system, which includes: Image acquisition module 201 is used for preset detection time intervals. Within a time duration t, video frame images of forklifts transferring goods in the target scenario are acquired in real time, forming a temporally ordered video frame image sequence [F1, F2, ..., F...]. ,…,F ]; Represents the total number of video frame images in a video frame image sequence; Optical flow calculation module 202 is used to calculate the optical flow vector field between adjacent frames in a video frame image sequence; The filtering module 203 is used to filter out optical flow vectors with displacements greater than a preset displacement threshold from each calculated optical flow vector field to form an optical flow vector region; Image extraction module 204 is used to extract the corresponding region of each video frame image in the video frame image sequence based on the formed optical flow vector region, to obtain a set of regions of interest. , ,…, ,…, ], For video frame image F Region of interest in; Model calculation module 205 is used to calculate the region of interest. Input a pre-trained object detection model to identify forklifts and forklift tires, and segment the forklifts; output... The coordinates of the polygonal points of the forklift outline, the bounding box coordinates of the forklift tires, and the output... forklift mask in ; Module 206 is used to determine the output forklift mask. and output The bounding box coordinates of the forklift tires are used to detect whether the load carried by the forklift exceeds the length limit; when the load exceeds the length limit, based on the output... The coordinates of polygonal points on the outline of the forklift are used to detect whether the forklift's trajectory violates regulations.

[0109] This embodiment of the forklift carrying length detection and trajectory compliance judgment system belongs to the same inventive concept as the forklift carrying length detection and trajectory compliance judgment methods in the above embodiments. For details not described in detail in the embodiments of the forklift carrying length detection and trajectory compliance judgment system, please refer to the embodiments of the forklift carrying length detection and trajectory compliance judgment methods mentioned above.

[0110] Figure 3 This is a schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the method provided in the embodiment of the present invention.

[0111] The terminal 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0112] The memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile memory terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 is able to perform some or all of the steps in the above method embodiments.

[0113] The processor 310 serves as the control center of the storage terminal, connecting various parts of the electronic terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0114] The communication unit 330 is used to establish a communication channel, enabling the storage terminal to communicate with other terminals. It can receive user data sent by other terminals or send user data to other terminals.

[0115] The present invention also provides a computer storage medium 400, such as... Figure 4 As shown, the computer storage medium 400 may store a program 410, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0116] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0117] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0120] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting the carrying length of a forklift truck and judging the compliance of the trajectory, characterized in that, The methods include: Step 101, every interval preset detection time interval Real-time acquisition of video frame images of the forklift transfer goods under the forklift transfer goods target scene in the time length t, forming a video frame image sequence with time sequence F1,F2,…,F ,…,F ; represent the total number of video frame images in the video frame image sequence; Step 102: Calculate the optical flow vector field between adjacent frames in the video frame image sequence; Step 103: For each calculated optical flow vector field, select optical flow vectors with displacements greater than a preset displacement threshold from the optical flow vector field to form an optical flow vector region; Step 104, based on the formed optical flow vector region, extracting a corresponding region of each video frame image in the sequence of video frame images to obtain a set of regions of interest , ,…, ,…, , representing a region of interest in the video frame image F . Step 105, for each region of interest The trained target detection model is input to recognize the target of the forklift and the tire of the forklift, and to segment the forklift, and output the polygon point coordinates of the forklift contour in each The bounding box coordinates of the tire of the forklift, and the forklift mask in each ;​ Step 106: Based on the output forklift mask and output The bounding box coordinates of the forklift tires are used to detect whether the load carried by the forklift exceeds the length limit; when the load exceeds the length limit, based on the output... The coordinates of polygonal points on the outline of the forklift are used to detect whether the forklift's trajectory violates regulations.

2. The method according to claim 1, characterized in that, In step 104, based on the formed optical flow vector region, the corresponding region of each video frame in the video frame image sequence is extracted to obtain a set of regions of interest. , ,…, ,…, ,include: Step 1041: For each formed optical flow vector region, record the starting coordinates of each optical flow vector in the optical flow vector region to obtain the pixel coordinates in the previous video frame image corresponding to that optical flow vector region; for the video frame image sequence... F1,F2,…,F ,…,F The optical flow vector region formed by the optical flow vector location between the last two images is also calculated and recorded as the pixel coordinates to which the starting coordinates of each optical flow vector in the optical flow vector region move in the video frame image Fn; Step 1042: For each video frame image F The video frame image F recorded in step 1041 Find the minimum x-coordinate (xmin), maximum x-coordinate (xmax), minimum y-coordinate (ymin), and maximum y-coordinate (ymax) from all pixel coordinates in the video frame image F. Extract the rectangular region defined by the minimum x-coordinate (xmin), maximum x-coordinate (xmax), minimum y-coordinate (ymin), and maximum y-coordinate (ymax) to obtain the video frame image F. Region of Interest ,in .

3. The method according to claim 1, characterized in that, In step 106, based on the output forklift mask... and output The bounding box coordinates of the forklift tires are used to detect whether the cargo carried by the forklift exceeds the length limit, including: Step 1061, based on output Calculate the bounding box coordinates of the forklift tires. Diameter length of forklift tires ; Step 1062: Mask the forklift By extracting the image difference, we can obtain Area of ​​goods carried by the forklift ; Step 1063: Inspect the area carrying the goods on the forklift. After performing morphological processing, the morphologically processed area of ​​the forklift's cargo is output. pixel outline ; Step 1064: Based on the pixel contour Calculate the predicted length of the goods carried by the forklift. ; Step 1065: Based on the calculated length , Calculate the actual length of the goods carried by the forklift. The calculation formula is: ， In the formula, This represents the actual diameter length of the forklift tire. Step 1066: Calculate the actual length of the goods carried by the forklift. To check whether the goods carried by the forklift are too long.

4. The method according to claim 3, characterized in that, In step 1061, based on the output Calculate the bounding box coordinates of the forklift tires. Diameter length of forklift tires ,include: Output-based Obtain the bounding box coordinates of the forklift tires. Height and width of forklift tires; Calculate the average of the height and width of the forklift tires to obtain the diameter. .

5. The method according to claim 3, characterized in that, Step 1064: Based on the pixel contour Calculate the predicted length of the goods carried by the forklift. ,include: Step 10641: Obtain pixel contours The minimum bounding rectangle; Step 10642: Obtain the pixel length of the longer side of the minimum bounding rectangle. This pixel length is the predicted length of the goods carried by the forklift. .

6. The method according to claim 1, characterized in that, In step 106, based on the output The coordinates of polygonal points on the forklift outline are used to detect whether the forklift's trajectory violates regulations, including: Step S1, based on output Obtain the coordinates of the polygonal points of the forklift outline. The center point of the forklift is used as the video frame image F. Center point of the forklift , ; Step S2: Using the center point , Generate a straight line (Line); Step S3, from the center point Begin by iterating through the center points one by one. , … and ; Step S4: For the currently traversed center point, determine whether it is located on or near the line Line: If so, the forklift trajectory is determined to be within acceptable limits, and the process continues to iterate through the next center point until all center points have been traversed. ; If not, the forklift trajectory is deemed to be in violation.

7. The method according to claim 1, characterized in that, The methods for obtaining a trained object detection model include: Constructing a training set: The method for constructing the training set includes: collecting video frame images of forklifts transporting goods in the target scenario, the collected images covering the forklift's empty working condition and the working condition carrying goods of each pre-specified shape, and each collected image including the side of the forklift and its tires; resampling each collected video frame image to 640 pixels × 640 pixels to obtain each resampled image; labeling each resampled image as follows: normalizing the polygon point coordinates and normalizing the bounding box of the forklift and forklift tires in the image, with the forklift and forklift tires corresponding to the category labels "0" and "1"; and combining the labeled images to form the training set. Load the pre-trained YOLOv11 model as the initial model; The initial model is trained using the training set to obtain a trained object detection model.

8. A forklift carrying length detection and trajectory compliance judgment system, characterized in that, The system includes: The image acquisition module is used to perform preset detection time intervals. Within a time duration t, video frame images of forklifts transferring goods in the target scenario are acquired in real time, forming a temporally ordered video frame image sequence [F1, F2, ..., F...]. ,…,F ]; Represents the total number of video frame images in a video frame image sequence; The optical flow calculation module is used to calculate the optical flow vector field between adjacent frames in a video frame image sequence; The filtering module is used to filter out optical flow vectors with displacements greater than a preset displacement threshold from each calculated optical flow vector field to form an optical flow vector region; The image extraction module is used to extract the corresponding region of each video frame in the video frame image sequence based on the formed optical flow vector region, thereby obtaining a set of regions of interest. , ,…, ,…, , Represents video frame image F Region of interest in; The model calculation module is used to calculate the region of interest. Input a pre-trained object detection model to identify forklifts and forklift tires, and segment the forklifts; output... The coordinates of the polygonal points of the forklift outline, the bounding box coordinates of the forklift tires, and the output... forklift mask in ; The decision module is used based on the output forklift mask. and output The bounding box coordinates of the forklift tires are used to detect whether the load carried by the forklift exceeds the length limit; when the load exceeds the length limit, based on the output... The coordinates of polygonal points on the outline of the forklift are used to detect whether the forklift's trajectory violates regulations.

9. A terminal, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the forklift carrying length detection and trajectory compliance judgment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the forklift carrying length detection and trajectory compliance judgment method as described in any one of claims 1 to 7.

Images