Vision-based crossing container truck deadbolt state detection method

Abstract

The invention relates to a vision-based crossing container truck deadbolt state detection method, and belongs to the technical field of image processing. The crossing container truck deadbolt state detection method based on vision comprises the steps that an image of a lock hole area is obtained through a long-focus camera on a mechanical arm, and in order to judge the opening and closing state of a lock pin, edge features of the lock hole and the lock pin need to be extracted; based on the edge strength and in combination with keyhole priori knowledge, carrying out deadbolt state optimization judgment; lock pin state confirmation, result output and linkage control are carried out, a detection result is directly linked with a crossing control system to realize safety management and control of the container truck, and if the lock pin is in a locked state, the system forbids the container truck to pass through the crossing handrail and keeps closed; the container truck lock pin state detection method based on machine vision solves the problem that in an existing container truck lock pin state detection method based on machine vision, the detection precision of Hough transform is low under complex conditions such as uneven illumination, noise interference, lock pin contamination and slight deformation, and is used for improving the accuracy of container truck lock pin state detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method for detecting the locking and twisting state of a container truck at a crossing based on vision. Background Art

[0002] Currently, the transportation of containers mostly uses container trucks. When transporting on roads, in order to prevent the container from moving on the vehicle and causing safety accidents, a locking pin is usually used to lock between the container and the truck; when handling operations need to be carried out in a yard or warehouse, the locking pin needs to be opened to prevent potential safety hazards caused by the vehicle being moved or lifted. In the field of container terminal loading and unloading, generally, a laser scanner and a vision system are used on the loading and unloading side of the crane to realize the real-time position detection of containers, trucks, etc., and the operation control of the hoisting mechanism is realized according to the safety control strategy to prevent accidents caused by the truck and the container not being unlocked and the truck being lifted.

[0003] However, the current laser scanning technology has a strong dependence on environmental stability. Weather factors, light, obstacles and other occlusions have a great impact on laser scanning. As a result, situations of missed detection will occur. Currently, the situation where the truck is lifted and not detected by the laser scanning detection technology occurs every year. And in the crane loading and unloading link, it is already the last link of the container loading and unloading working condition. Once a missed detection situation occurs, accidents will be inevitable and the resulting losses will be irreparable.

[0004] In view of the defects of the prior art, a method for detecting the state of the truck locking pin is designed to observe the locking relationship between the container and the truck. This method can reduce the risk of lifting the truck to the lowest level in advance. A typical scenario application is to use this method at the approach crossing, and place the locking pin state detection process in front of the approach crossing to the port, that is, before the container enters the port operation area, to ensure that the container locking pin has been opened, thus avoiding the situation where the truck is lifted during loading and unloading operations due to the locking pin not being opened.

[0005] Existing methods for detecting the locking pin state based on machine vision generally adopt the following steps: First, obtain an image of the keyhole area; then use an edge detection algorithm ( operator) to extract the edges of the image; then detect the straight line or circular features in the edge image through the Hough transform ( ); finally, judge the state (open / closed) of the locking pin according to the geometric relationship of these features. This method can achieve good detection results under ideal conditions (uniform illumination, clear image, no occlusion and no contamination of the locking pin).

[0006] However, in the actual crossing application scenario, due to complex lighting conditions (shadows, reflections, day / night), dirt, rust, paint peeling, etc. on the surface of the locking pin, and differences in camera angles and container truck docking positions, the quality of the obtained keyhole images varies, and the edge detection results are not ideal. The traditional Hough transform method mainly relies on the cumulative voting of edge pixel points and is sensitive to image noise, lighting changes, and slight deformations of the locking pin. When there are problems such as noise interference, uneven lighting, dirt on the locking pin, and slight deformation in the keyhole image, the detection accuracy of the Hough transform will significantly decrease, making it difficult to accurately detect the keyhole features, thereby affecting the accuracy of the locking pin state judgment. Summary of the Invention

[0007] Based on this, it is necessary to provide a vision-based detection method for the locking and unlocking state of container trucks at crossings to address the problem that the detection accuracy of the Hough transform is affected by external factors, which in turn affects the accuracy of the locking pin state judgment.

[0008] A vision-based detection method for the locking and unlocking state of container trucks at crossings includes: S1: An image of the keyhole area is obtained through a long-focus camera on the robotic arm. To determine the opening and closing state of the locking pin, it is necessary to extract the edge features of the keyhole and the locking pin. S2: Optimize the judgment of the locking and unlocking state based on the edge intensity and combined with the prior knowledge of the keyhole. S3: Confirm the locking pin state, output the result, and perform linkage control. The detection result will directly link to the crossing control system to achieve the safety management and control of container trucks. If the locking pin is in the locked state, the system will prohibit the container truck from passing through, and the crossing railing will remain closed, and the driver will be prompted to unlock through the display screen and speaker. If the locking pin is in the open state, the system will allow the container truck to pass through, and the crossing railing will automatically lift. Among them, S1 includes the following steps: I. Camera deployment and image acquisition: Two cameras, a short-focus camera and a long-focus camera, are used in combination. In this step, the short-focus camera is used. The short-focus camera is fixedly installed on one side of the crossing and is used to continuously capture images of the container and the container truck. The images captured by the camera are transmitted to the algorithm server through the network for subsequent processing. II. Container corner detection: The following specific algorithm is used to detect the container corners in the images captured by the short-focus camera: ① Image preprocessing; The image size is uniformly adjusted to 640x640 pixels. The Padding resize method is used for scaling to maintain the aspect ratio of the image. After scaling, the short side of the image is filled with gray scale with RGB values of (128, 128, 128). ② Edge feature extraction; Two convolutional kernels are used to extract the edge features of the image in the horizontal and vertical directions respectively; Vertical direction convolutional kernel Horizontal convolution kernel (3) Box corner feature extraction and localization; Send the preprocessed and edge-extracted image into the neural network model for feature extraction; The backbone network consists of multiple convolutional layers ( ), normalization layers ( ), activation functions ( ) and modules. The specific structure is that multiple convolutional modules ( ) are connected in series. Each convolutional module consists of a convolutional layer, a normalization layer and an activation function. Each convolutional module forms a unit. Multiple units are alternately combined with convolutional modules, and finally a module is connected. The module consists of max-pooling layers ( ) and 1 connection layer ( ); Calculate the loss function and use (non-maximum suppression) to process, and obtain the final box corner detection area ; where: is the center point coordinate of the box corner area, is the width of the box corner area, is the height of the box corner area. The loss function includes classification loss, localization loss and confidence loss. The specific formula is as follows:

[0009] Since this task only detects one category of box corners, the classification loss , and the localization loss is calculated using ; The confidence loss uses binary cross-entropy loss ( ), and the formula is as follows:

[0010] where: is the true label. When there is a sample in the regression box, is , otherwise it is . is the predicted confidence III. Judging whether the container truck is parked; Judge whether the container truck is parked according to the box corner detection result. Calculate the distance difference between the center point of the currently detected box corner and the center point detected last time :

[0011] If is less than the set value ( pixels), it is considered that the container carrier has stopped. To avoid misjudgment, it is also necessary to determine whether the stop time of the container carrier has reached the set value ( seconds); IV. Manipulator guidance and keyhole image acquisition. When it is determined that the container carrier has stopped and is stable, trigger the movement of the manipulator. A long - focal camera is installed on the manipulator, which is used to capture detailed images of the keyhole area, and a distance sensor is used to measure the shooting distance. According to the detected corner position of the box, calculate the movement path of the manipulator. During the movement of the manipulator, start the long - focal camera and save the captured images as the input for subsequent lock pin state detection; at the same time, the system gives a text prompt of "Detecting" through the display unit and switches the light state to the yellow working state.

[0012] In one embodiment, the optimization factor α based on edge intensity in S2; To achieve the optimization of the Hough transform based on edge intensity, it is necessary to calculate the weight of each edge pixel point ; The final optimization factor The calculation formula is as follows:

[0013] Explanation of formula symbols: : Edge pixel point The weight of, the value range is The larger the value, the greater the contribution of the pixel point to the Hough transform; Edge intensity response ; Reflects the intensity of the pixel point as an edge. For a clear edge, is close to ; For a blurred edge or noise, is close to 0; : Edge confidence; is the gradient amplitude at pixel point , indicating the intensity of the edge. The larger the gradient amplitude, the more obvious the edge at this pixel point and the more likely it is a real edge. In this application, the operator is used to calculate the gradient amplitude; is the maximum gradient amplitude in the entire image, which is used for normalization; The function maps the normalized gradient magnitude to the interval. When the gradient magnitude is close to the maximum value, it is close to ; when the gradient magnitude is close to , it is close to , which can smoothly adjust the weights and avoid sudden weight changes caused by directly using the gradient magnitude; Edge confidence:

[0014] is the distance from the pixel point to the nearest non-zero gradient pixel point. If is an isolated noise point and there are no real edge points around it, then will be very large. If there are other pixel points with non-zero gradient values around a pixel point, it means that the pixel point is located on a real edge, which is used to measure this possibility; is the length of the image diagonal and is used for normalization. is a tuning parameter that controls the steepness of the confidence curve. According to actual tests, it is recommended to take values between ; is the exponential function.

[0015] In one embodiment, the result in S2 is reversed, so that the closer to the real edge, the larger; Generally speaking reflects the reliability of the pixel point as an edge. For isolated noise points, since the value is large, will be very small. For pixel points on real edges, will be very large. By multiplying the edge strength response and the edge confidence , we obtain the final optimization factor which comprehensively considers the strength and reliability of the pixel point as an edge. For clear and real edge points: is large, is close to ; is small, is close to ; Therefore, is close to , this pixel point has a relatively large weight in the Hough transform. For blurred edges or noise points: is small, close to ; therefore, close to 0, this pixel point contributes little to the Hough transform. For isolated noise points, is large, close to , so close to , this pixel point hardly participates in the voting of the Hough transform.

[0016] In one of the embodiments, in S2, an optimization factor β is constructed based on the prior knowledge of the keyhole structure; Formula logic: Use the prior knowledge of the keyhole and lock pin structures to construct the optimization factor , and constrain the parameter space of the Hough transform; since this step focuses on horizontal and vertical lines, therefore the construction of is divided into two parts: and ; The optimized Hough transform formula:

[0017] Where: : The cumulative voting value in the Hough space after being optimized by step (i.e., using the optimization factor ), : The cumulative voting value in the Hough space further optimized by step , : The optimization factor based on the prior knowledge of the keyhole and lock pin structures, and its value depends on the parameter whether it conforms to the prior knowledge; The calculation of is as follows:

[0018] Where represents the logical OR operation, that is, as long as meets one of the conditions of the horizontal line or the vertical line, will be ; The horizontal line detection optimization factor:

[0019] Where: : The angle tolerance (selected according to the actual screening requirements and different lock twist designs or ), which is used to control the requirement for the straightness of the line; , : The search range determined according to the pixel length range of the long side of the keyhole and the possible horizontal edge of the lock pin in the image, and the position range where they may appear; According to the actual image height H collected, the coordinate range corresponding to the long side of the keyhole is estimated as: (Considering that the long side of the keyhole may appear anywhere in the image), according to the above y coordinate range, the range of ρ can be estimated: Using the origin of the image coordinate system in the upper left corner, and the y axis downward: ; ; . Among them, is an additional margin used to compensate for various errors and uncertainties, and the value here is 5 pixels.

[0020] In one embodiment, the internal parameters of the camera (focal length f, principal point coordinates (c_x, c_y)) and the distortion coefficients are obtained; then, according to the distance sensor, the vertical distance D from the optical center of the camera to the shooting plane (the side of the container) is obtained, and the upper limit ΔD of the distance measurement error is estimated; the actual length L_long of the long side of the keyhole is determined; according to the principle of similar triangles, the pixel length range of the long side of the keyhole in the image is calculated : = f * L_long / (D + ΔD), = f * L_long / (D - ΔD).

[0021] In one embodiment, the vertical line detection optimization factor:

[0022] Among them: : The angle tolerance, which is used to control the requirement for the perpendicularity of the line; , : The search range determined according to the pixel length range of the long side of the keyhole and the possible horizontal edge of the lock pin in the image, and the position range where they may appear; of the search range; Locking state judgment logic: After the Hough transform and optimization factor screening, a set of horizontal lines and vertical lines are obtained. Next, the locking state is judged according to these lines: Finding the pair of vertical short sides: Looking for a pair of lines in the detected line set that meet the following conditions, : and are parallel to each other (or the included angle is less than the set threshold ); and The distance between ( and are respectively the minimum and maximum values of the long side length of the keyhole); and The lengths of and Satisfy: and . are respectively the minimum and maximum values of the short side length of the keyhole or the short side length of the lock pin).

[0023] In one embodiment, the pixel distance mapping is calculated as follows: According to the obtained camera internal parameters (focal length f, principal point coordinates ) and distortion coefficients; According to the distance sensor, obtain the vertical distance D from the camera optical center to the shooting plane (side of the container), and estimate the upper limit ΔD of the distance measurement error; Determine the actual length of the short side of the keyhole; Calculate the pixel length range of the short side of the keyhole in the image according to the principle of similar triangles: , . Similarly, calculate the pixel length range of the long side L_{long} in the image: , ; Judge the locking state: If two pairs of vertical short side pairs satisfying the above conditions are found( , , , , then further check whether there is a straight line cluster formed by and approximately perpendicular (that is, the included angle between and is close to 90°, allowing a certain angle tolerance); If the above conditions are met, it is considered that the lock pin is in the locked state; If no vertical short side pair satisfying the conditions is found, it is considered that the lock pin is not in the locked state.

[0024] In one embodiment, the system has initially determined the lock pin state in S3; To ensure the accuracy of the results and reduce misjudgments, this step will comprehensively confirm the final detection results of multiple consecutive frames of images; if the locking pin is determined to be in the locked state for multiple consecutive frames, it is confirmed that the locking pin is in the locked state; if the locking pin is determined to be in the open state for multiple consecutive frames, it is confirmed that the locking pin is in the open state; if the detection results of multiple consecutive frames are inconsistent, or the state of the locking pin cannot be clearly determined, an alarm will be triggered to prompt manual intervention.

[0025] In one embodiment, after the system confirms the state of the locking pin in S3, the detection results will be output in multiple ways: the state of the locking pin will be displayed on the crossing display screen in text and color (red indicates locked, green indicates open); a voice prompt will be played through the speaker to inform the driver; at the same time, the detection results will be uploaded to the crossing management software platform for management personnel to view and record.

[0026] In one embodiment, the detection results in S3 will directly link with the crossing control system to achieve the safe management and control of container trucks; if the locking pin is in the locked state, the system will prohibit the container truck from passing through and keep the crossing railing closed, and prompt the driver to unlock through the display screen and speaker; if the locking pin is in the open state, the system will allow the container truck to pass through and the crossing railing will automatically lift; through this step, it can be ensured that the locking pin of the container truck has been correctly opened before entering the port area, effectively preventing safety accidents caused by the locking pin not being opened, and improving the crossing traffic efficiency and safety.

[0027] The present invention solves the problem of low detection accuracy of the Hough transform in the existing method for detecting the state of the locking pin of a container truck based on machine vision under complex conditions, such as uneven illumination, noise interference, dirt on the locking pin, slight deformation, etc., and is used to improve the accuracy of detecting the state of the locking pin of a container truck. This method is to arrange a vision unit in the crossing area. At the same time, in order to solve the problems of various types of container trucks and different parking positions of drivers, a robotic arm is used to guide the vision unit; ensure that the image of the locking pin inside the lock hole can be accurately obtained, and an algorithm recognition module is carried out to obtain the open / closed state of the locking pin, so as to ensure that the locking relationship between the container truck and the container meets the expectations and ensure the safety of the entire transportation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0031] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or there may also be an intermediate component. When a component is considered to be "connected" to another component, it can be directly connected to the other component or there may be an intermediate component at the same time. The terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used in the specification of the present invention are only for the purpose of illustration and do not represent the only implementation manner.

[0032] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0033] In the present invention, unless otherwise clearly specified and defined, the first feature being "on" or "under" the second feature may be that the first feature is in direct contact with the second feature, or the first feature and the second feature are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may be that the first feature is directly above or obliquely above the second feature, or only indicates that the first feature is at a higher horizontal height than the second feature. The first feature being "under", "below" and "beneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or only indicates that the first feature is at a lower horizontal height than the second feature.

[0034] Unless otherwise defined, all technical and scientific terms used in the specification of the present invention have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in the specification of the present invention are only for the purpose of describing specific implementation manners and are not intended to limit the present invention. The term "and / or" used in the specification of the present invention includes any and all combinations of one or more of the related listed items.

[0035] The following Figure 1 describes the vision-based detection method for the locking and twisting state of container trucks at the crossing of the present invention.

[0036] In one embodiment, a vision-based detection method for the locking and twisting state of container trucks at the crossing includes: S1: Images of the keyhole area are obtained through a long-focus camera on the robotic arm. To determine the opening and closing state of the locking pin, it is necessary to extract the edge features of the keyhole and the locking pin; S2: Based on the edge intensity and combined with the prior knowledge of the keyhole, an optimized judgment of the locking and twisting state is made; S3: Confirm the state of the locking pin, output the result and perform linkage control. The detection result will directly link to the crossing control system to achieve the safe management and control of container trucks. If the locking pin is in the locked state, the system will prohibit the container truck from passing through and keep the crossing railing closed, and prompt the driver to unlock through the display screen and the speaker; if the locking pin is in the open state, the system will allow the container truck to pass through and the crossing railing will automatically lift; Among them, S1 includes the following steps: I. Camera deployment and image acquisition: Two cameras, a short-focus camera and a long-focus camera, are used in combination. In this step, the short-focus camera is used. The short-focus camera is fixedly installed on one side of the crossing and is used to continuously capture images containing containers and container trucks. The images captured by the camera are transmitted to the algorithm server through the network for subsequent processing; II. Corner detection of the container: Perform corner detection on the images captured by the short-focus camera. The specific algorithm is as follows: ① Image preprocessing; The image size is uniformly adjusted to 640x640 pixels. The Padding resize method is used for scaling to keep the aspect ratio of the image. After scaling, the short side of the image is filled with gray scale with RGB value (128, 128, 128); ② Edge feature extraction; Two convolution kernels are used to extract the edge features of the image in the horizontal and vertical directions respectively; Vertical direction convolution kernel Horizontal direction convolution kernel (3) Corner feature extraction and positioning; The preprocessed and edge-extracted images are sent into the neural network model for feature extraction; The backbone network consists of multiple convolutional layers ( ), normalization layers ( ), activation functions ( ) and modules, and the specific structure is a series connection of multiple convolutional modules ( ). Each convolutional module consists of a convolutional layer, a normalization layer and an activation function. Each convolutional module forms a unit. Multiple units are alternately combined with convolutional modules, and finally a module is connected. The module consists of maximum pooling layers ( ), and one connection layer ( ); Calculate the loss function and use (non-maximum suppression) to process, and obtain the final corner detection area ; where: is the coordinate of the center point of the corner area, is the width of the corner area, is the height of the corner area. The loss function includes classification loss, localization loss, and confidence loss. The specific formulas are as follows:

[0037] Since only one category of corner is detected in this task, the classification loss , and the localization loss is calculated using ; The confidence loss uses binary cross-entropy loss ( ), and the formula is as follows:

[0038] where: is the true label. When there is a sample in the regression box, is , otherwise it is . is the predicted confidence III. Judging whether the container truck has stopped; Judge whether the container truck has stopped according to the corner detection result. Calculate the distance difference between the center point of the corner detected currently and the center point detected last time:

[0039] If is less than the set value ( pixels), it is considered that the container truck has stopped. To avoid misjudgment, it is also necessary to judge whether the stop time of the container truck has reached the set value ( seconds); IV. Manipulator Guidance and Keyhole Image Acquisition: When it is determined that the truck has stopped and is stable, the manipulator is triggered to move. A long-focus camera is installed on the manipulator to capture detailed images of the keyhole area, and a distance sensor is used to measure the shooting distance. According to the detected corner position of the box, the movement path of the manipulator is calculated. During the movement of the manipulator, the long-focus camera is started and the captured images are saved as the input for subsequent lock pin state detection. At the same time, the system gives a text prompt of "Detecting..." through the display unit and switches the light state to the yellow working state.

[0040] As shown in Figure 1, the optimization factor α based on edge intensity in S2; To achieve the optimization of the Hough transform based on edge intensity, the weight of each edge pixel point needs to be calculated ; The final optimization factor The calculation formula is as follows:

[0041] Explanation of formula symbols: : Edge pixel point The weight of, with the value range of The larger the value, the greater the contribution of the pixel point to the Hough transform; Edge intensity response ; Reflects the intensity of the pixel point as an edge. For a clear edge, Is close to ; For a blurred edge or noise, Is close to 0; : Edge confidence; Is the pixel point The gradient amplitude at indicates the intensity of the edge. The larger the gradient amplitude, the more obvious the edge at the pixel point and the more likely it is a real edge. In this application, the Operator is used to calculate the gradient amplitude; Is the maximum gradient amplitude in the entire image for normalization; The function maps the normalized gradient amplitude to the Interval. When the gradient amplitude is close to the maximum value, Is close to ; When the gradient amplitude is close to At that time, Is close to , This can smoothly adjust the weight and avoid the weight mutation caused by directly using the gradient amplitude; Edge confidence:

[0042] is the distance from a pixel point to the nearest non-zero gradient pixel point. If is an isolated noise point and there are no real edge points around it, then will be large. If there are other pixel points with non-zero gradient values around a pixel point, it means that this pixel point is located on a real edge, which is used to measure this possibility; is the length of the image diagonal, used for normalization, is a tuning parameter that controls the steepness of the confidence curve. According to actual tests, it is recommended to take values in the range of between; is an exponential function.

[0043] In S2, the result is inverted so that the closer to the real edge, the larger; Generally speaking reflects the reliability of a pixel point as an edge. For an isolated noise point, since the value is large, will be very small. For a pixel point on a real edge, will be large. By multiplying the edge intensity response and the edge confidence we obtain the final optimization factor which comprehensively considers the intensity and reliability of a pixel point as an edge. For a clear and real edge point: is large, is close to ; is small, is close to ; Therefore, is close to , and this pixel point has a large weight in the Hough transform. For a blurred edge or noise point: is small, is close to ; Therefore, is close to 0, and this pixel point makes little contribution to the Hough transform. For an isolated noise point, is large, is close to , so is close to , this pixel hardly participates in the voting of Hough transform.

[0044] In this embodiment, in step The image of the keyhole area is obtained by the telephoto camera on the robot arm. In order to determine the open or closed state of the lock pin, it is necessary to extract the edge features of the keyhole and the lock pin. Using the edge detection algorithm ( In practical applications, the steps The keyhole image obtained has the following problems, which leads to poor results of traditional edge detection and Hough transform methods: Uneven lighting: The lighting environment at the crossing is complex and changeable, and there may be shadows, reflections, etc. This will cause some areas of the keyhole image to be too dark or too bright, affecting the accuracy of edge detection. The edges of the dark areas may not be detected, and the bright areas may produce false edges.

[0045] Lock pin contamination: Lock pins are exposed to outdoor environments for a long time, and there may be stains, rust, paint peeling, etc. on the surface. These contaminations will change the original texture characteristics of the lock pin, resulting in breakage, burrs or false edges in the edge detection results.

[0046] Image noise: The image sensor itself has noise, and the interference in the signal transmission process makes the collected image inevitably noisy. These noises will produce random and irregular edge points in the edge detection results.

[0047] The above problems will lead to inaccurate and incomplete edge detection results, which will affect the effect of subsequent Hough transform. The traditional Hough transform treats all edge points equally and performs indiscriminate cumulative voting. This means that noise, false edges, and broken edges in the image will interfere with the results of the Hough transform and reduce its detection accuracy. In order to solve this problem, the Hough transform needs to be optimized. By distinguishing the importance of different pixels in the edge image. Real, clear edge points should have higher weights than those caused by noise, stains, or uneven lighting.

[0048] Therefore, before performing the Hough transform, the edge image is analyzed to evaluate the "intensity" of each edge pixel. Pixels with high edge intensity are considered more likely to be real edges, and are given a greater voting weight in the Hough transform; pixels with low edge intensity are considered more likely to be noise or false edges, and are given a smaller voting weight in the Hough transform, or even do not participate in the vote.

[0049] like Figure 1 As shown, in S2, the optimization factor β is constructed a priori based on the keyhole structure; Formula logic: Use prior knowledge of the keyhole and pin structure to build optimization factors , the parameter space of the Hough transform is constrained; since this step focuses on horizontal and vertical lines, is constructed in two parts: and ; The optimized Hough transform formula:

[0050] where: : The cumulative voting value in the Hough space after being optimized by step (i.e., the optimization factor is used), : The cumulative voting value in the Hough space further optimized by step ; : The optimization factor based on the prior knowledge of the keyhole and lock pin structure, and its value depends on the parameter whether it conforms to the prior knowledge; The calculation of

[0051] where represents the logical OR operation, that is, as long as meets one of the conditions of the horizontal line or the vertical line, is ; The optimized factor for horizontal line detection:

[0052] where: : The angle tolerance (selected as or according to the actual screening requirements and different lock twist designs) is used to control the requirement for the horizontal degree of the line; , : The search range of determined according to the pixel length range of the long side of the keyhole and the possible horizontal edge of the lock pin in the image, and the possible position range where they may appear; according to the actual image height H collected, the coordinate range corresponding to the long side of the keyhole is estimated as: (considering that the long side of the keyhole may appear at any position in the image), according to the above y - coordinate range, the range of ρ can be estimated: using the image coordinate system with the origin at the upper left corner and the axis downward: ; . Among them, is an additional margin used to compensate for various errors and uncertainties, and the value here is 5 pixels.

[0053] Obtain the internal parameters of the camera (focal length f, principal point coordinates (c_x, c_y)) and distortion coefficients; then obtain the vertical distance D from the camera optical center to the shooting plane (side of the container truck) according to the distance sensor, and estimate the upper limit ΔD of the distance measurement error; determine the actual length L_long of the long side of the keyhole; calculate the pixel length range of the long side of the keyhole in the image according to the principle of similar triangles : = f * L_long / (D + ΔD), = f * L_long / (D - ΔD).

[0054] The vertical line detection optimization factor:

[0055] Wherein: : Angle tolerance, used to control the requirement for the perpendicularity of the line; , : Determined according to the pixel length range of the long side of the keyhole and the possible horizontal edge of the lock pin in the image, and the possible position range where they may appear search range; Locking state judgment logic: After the Hough transform and optimization factor screening, a set of horizontal lines and vertical lines are obtained. Next, the locking state is judged according to these lines: Find the pair of vertical short sides: Find the line pair that meets the following conditions in the detected line set, : and are parallel to each other (or the included angle is less than the set threshold ); and The distance between ( and are the minimum and maximum values of the long side length of the keyhole respectively); and The lengths of and Meet: and . are the minimum and maximum values of the short side length of the keyhole or the short side length of the lock pin respectively).

[0056] The pixel distance mapping is calculated as follows: Based on the obtained camera internal parameters (focal length f, principal point coordinates ), and distortion coefficients; obtain the vertical distance D from the camera optical center to the shooting plane (the side of the container truck) according to the distance sensor, and estimate the upper limit ΔD of the distance measurement error; determine the actual length of the short side of the keyhole ; calculate the pixel length range of the short side of the keyhole in the image according to the principle of similar triangles : , . Similarly, calculate the pixel length range of the long side L_{long} in the image : , ; Judge the locking state: If two pairs of vertical short side pairs that meet the above conditions are found ( , , , ), then further check whether there is a straight line cluster formed by and approximately perpendicular (that is, the included angle between and is close to 90°, allowing a certain angular tolerance); If the above conditions are met, it is considered that the locking pin is in the locked state; if no vertical short side pair that meets the conditions is found, it is considered that the locking pin is not in the locked state.

[0057] In this embodiment, edge enhancement has been performed on the keyhole image to obtain an edge image. This step aims to detect the straight line features in the edge image using the Hough transform, and judge whether the locking pin is in the locked state according to the unique geometric features presented by the keyhole and the locking pin in the locked state. Since there are multiple intermediate states in the unlocked state and it is difficult to judge with a unified standard, this step only focuses on and judges the locked state. In the locked state, the locking pin and the keyhole form a "cross" shape. From the image taken from the front, it can be observed that: the two long sides (horizontal) and two short sides (vertical) of the locking pin itself form a rectangle; at the same time, the two short sides (vertical) of the keyhole also appear in the image; while the two long sides of the keyhole are partially blocked by the locking pin and truncated into four shorter line segments (parallel to the short side of the locking pin, that is, in the vertical direction).

[0058] Based on the above characteristics of the locked state, the judgment process of this step is as follows: First, perform the Hough line transform on the edge image obtained in step , and apply the optimization factor to enhance the edge weight. Then use the optimization factor obtained in this step to constrain the parameter space of the Hough transform, and filter out those close to horizontal ( or , allow angular tolerance ) and close to vertical ( or , allow angular tolerance ) of the straight line. Next, focus on the short side features of the selected straight lines: find pairs of perpendicular short sides (i.e., one close to horizontal, one close to vertical, and the length is within the range of the short side lengths of the keyhole or the lock pin). If at least one pair of perpendicular short sides is found, further judgment is made: whether these two short sides belong to the keyhole and the lock pin respectively (judgment is made based on their positional, length, spacing, etc. relationships, combined with the prior dimension information of the keyhole and the lock pin); and whether there are short line segments parallel to the short side of the lock pin (i.e., in the vertical direction) near the short side of the keyhole (these short line segments may be formed by the long side of the keyhole being truncated by the lock pin). If all the above conditions are met, it is considered that the lock pin is in the locked state; if no pair of perpendicular short sides is found, or the found pair of short sides does not meet the above conditions, it is considered that the lock pin is not in the locked state.

[0059] The core of this step lies in using the unique geometric relationship between the keyhole and the lock pin in the locked state, and through detecting and analyzing the straight line features in the image, realizing the accurate judgment of the locked state of the lock pin.

[0060] As Figure 1 shown, this step is based on the preliminary judgment result of the lock pin state in step to conduct the final confirmation and output the detection result, and link with the crossing control system to realize the safety control of the container truck.

[0061] After the analysis system has preliminarily determined the lock pin state through step . To ensure the accuracy of the result and reduce misjudgment, this step will comprehensively confirm the final result based on the detection results of multiple consecutive frames of images. If the lock pin is determined to be in the locked state in multiple consecutive frames, it is confirmed that the lock pin is in the locked state; if the lock pin is determined to be in the open state in multiple consecutive frames, it is confirmed that the lock pin is in the open state; if the detection results of multiple consecutive frames are inconsistent, or the lock pin state cannot be clearly determined, an alarm is triggered to prompt manual intervention.

[0062] After confirming the lock pin state, the system will output the detection result in multiple ways: display the lock pin state on the crossing display screen in text and color (red indicates locked, green indicates open); play a voice prompt through the speaker to inform the driver; at the same time, upload the detection result to the crossing management software platform for the management personnel to view and record.

[0063] The detection results will directly interact with the crossing control system to achieve the safety control of container trucks. If the locking pin is in the locked state, the system will prohibit the container truck from passing through, keep the crossing barrier closed, and prompt the driver to unlock through the display screen and speaker. If the locking pin is in the open state, the system will allow the container truck to pass through, and the crossing barrier will automatically lift. Through this step, it can be ensured that the locking pin of the container truck is correctly opened before entering the port area, effectively preventing safety accidents caused by the locking pin not being opened, and improving the crossing passing efficiency and safety.

[0064] Thus, the present invention is completed.

[0065] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0066] The above-described embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.

Claims

1. A vision-based method for detecting the locking state of a container truck at a crossing, characterized in that: include: S1: The telephoto camera on the robot arm obtains an image of the lock hole area. In order to determine the open or closed state of the lock pin, it is necessary to extract the edge features of the lock hole and lock pin. S2: Optimize the lock twisting state judgment based on edge strength and combined with the prior knowledge of the lock hole; S3: Lock pin status confirmation, result output and linkage control. The test results will directly link the crossing control system to achieve safety management of container trucks. If the lock pin is in the locked state, the system will prohibit container trucks from passing through the crossing barrier, which will remain closed, and prompt the driver to unlock it through the display and speaker; if the lock pin is in the open state, the system will allow container trucks to pass through, and the crossing barrier will automatically lift up; Wherein, S1 comprises the following steps:

1. Camera deployment and image acquisition. A short-focus camera and a long-focus camera are used together. This step uses a short-focus camera, which is fixedly installed on one side of the crossing to continuously capture images of containers and trucks. The images captured by the camera are transmitted to the algorithm server via the network for subsequent processing.

2. Box corner detection: perform box corner detection on the image collected by the short-focus camera. The specific algorithm is as follows: ① Image preprocessing; The image size is uniformly adjusted to 640x640 pixels. Padding resize is used to maintain the image aspect ratio. After scaling, the short side of the image is filled with grayscale with RGB values ​​of (128, 128, 128); ②Edge feature extraction; Use two convolution kernels to extract the horizontal and vertical edge features of the image respectively; Vertical convolution kernel Horizontal convolution kernel (3) Box corner feature extraction and positioning; The preprocessed and edge-extracted images are fed into the neural network model for feature extraction; the backbone network consists of multiple convolutional layers ( ), Normalization layer( ), activation function( )and The specific structure is composed of multiple convolution modules ( ) are connected in series, each convolution module consists of a convolution layer, a normalization layer and an activation function. Convolution modules constitute a Unit, multiple Units are combined alternately with convolutional modules and finally connected to a Modules, Modules by Max pooling layers ( ) and 1 connection layer ( )constitute; Calculate the loss function and use (Non-maximum suppression) processing to obtain the final box corner detection area ;in: is the coordinate of the center point of the box corner area, is the box corner area width, is the height of the box corner area. The loss function includes classification loss, positioning loss and confidence loss. The specific formula is as follows:

2. Since this task only detects one category of box corners, the classification loss , positioning loss use calculate; Confidence loss Using binary cross entropy loss ( ), the formula is as follows:

3. Among them: is the true label. When there are samples in the regression box, for , otherwise . The confidence level of the prediction 3. Truck parking judgment; Determine whether the truck should stop based on the box corner detection results and calculate the center point of the currently detected box corner and the center point detected last time The distance difference between :

4. If Less than the set value ( pixels), it is considered that the truck has stopped. To avoid misjudgment, it is also necessary to determine whether the truck stop time reaches the set value ( Second); 4. Robotic arm guidance and keyhole image acquisition: When it is determined that the container truck has stopped and stabilized, the robot arm is triggered to move. A telephoto camera is installed on the robot arm to capture detailed images of the keyhole area, and a distance sensor is used to measure the shooting distance. The movement path of the robot arm is calculated based on the detected box corner position. During the movement of the robot arm, the telephoto camera is started and the captured image is saved as input for subsequent lock pin status detection. At the same time, the system gives a text prompt of "Detection in Progress" through the display unit and switches the light status to the yellow working status.

5. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 1 is characterized in that: The optimization factor α is obtained based on the edge strength in S2; In order to achieve Hough transform optimization based on edge strength, it is necessary to calculate the weight of each edge pixel ; Final optimization factor The calculation formula is as follows:

6. Explanation of formula symbols: : Edge pixels The weight range is The larger the value, the greater the contribution of the pixel to the Hough transform; Edge intensity response ; Reflects the strength of the pixel as an edge. For a clear edge, Close to ; For blurred edges or noise, Close to 0; : edge confidence; It's a pixel The gradient amplitude at the pixel indicates the strength of the edge. The larger the gradient amplitude, the more obvious the edge at the pixel point is and the more likely it is a real edge. In this application, The operator calculates the gradient magnitude; is the maximum gradient magnitude in the entire image, used for normalization; The function maps the normalized gradient magnitude to interval, when the gradient amplitude is close to the maximum value, Close to ; When the gradient amplitude is close to hour, Close to , so that the weights can be adjusted smoothly and the weight mutation caused by directly using the gradient amplitude can be avoided; Edge Confidence:

7. It's a pixel The distance to the nearest non-zero gradient pixel, if is an isolated noise point, and there is no real edge point around it, then will be very large if there are other gradient values ​​around a pixel that are not , indicating that the pixel is located on the real edge. To measure this possibility; is the length of the image diagonal, used for normalization, is an adjustment parameter that controls the steepness of the confidence curve. According to actual tests, The recommended value is between; is an exponential function.

8. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 2 is characterized in that: The result is reversed in S2 so that the closer to the real edge, The bigger; In summary It reflects the reliability of the pixel as an edge. For isolated noise points, The value is larger, will be very small, for pixels on the real edge, will be large, by taking the edge intensity response and edge confidence Multiplying together, we get the final optimization factor The strength and reliability of the pixel as an edge are comprehensively considered. For clear and real edge points: Larger, Close to ; Smaller, Close to ;therefore, Close to , this pixel has a larger weight in the Hough transform, for blurred edges or noise points: Smaller, Close to ;therefore, Close to 0, the pixel contributes little to the Hough transform. For isolated noise points, Larger, Close to , therefore Close to , this pixel hardly participates in the voting of Hough transform.

9. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 1 is characterized in that: In S2, an optimization factor β is constructed based on a priori keyhole structure; Formula logic: Use prior knowledge of the keyhole and pin structure to build optimization factors , constrains the parameter space of the Hough transform; since this step focuses on horizontal and vertical lines, The construction is divided into two parts: and ; Optimized Hough transform formula:

10. Among them: : After steps The optimized Hough space cumulative voting value (i.e., using the optimization factor ), : After steps The further optimized Hough space cumulative voting value, : An optimization factor based on the prior structure of the lock hole and the lock pin, whose value depends on the parameter Is it consistent with prior knowledge? The calculation of is as follows:

11. Among them Represents a logical OR operation, that is, as long as Meets one of the conditions of a horizontal straight line or a vertical straight line, Just for ; The horizontal line detection optimization factor:

12. Among them: : Angle tolerance (depending on the actual screening requirements and different lock twist design options or ), used to control the requirements for straight line horizontality; , : Determined based on the pixel length range of the long side of the keyhole and the possible horizontal edge of the lock pin in the image, as well as the range of their possible positions According to the actual image height H collected, the keyhole long side corresponding to The coordinate range is estimated to be: (Considering that the long side of the keyhole may appear anywhere in the image), based on the above y coordinate range, the range of ρ can be estimated: using the image coordinate system with the origin in the upper left corner, Axis down: ; .in, is an additional margin used to compensate for various errors and uncertainties, and the value here is 5 pixels.

13. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 4 is characterized in that: Obtain the camera's internal parameters (focal length f, principal point coordinates (c_{x}, c_{y})) and distortion coefficients; then obtain the vertical distance D from the camera's optical center to the shooting plane (side of the truck) based on the distance sensor, and estimate the upper limit of the distance measurement error ΔD; determine the actual length L_{long} of the long side of the keyhole; calculate the pixel length range of the long side of the keyhole in the image based on the principle of similar triangles : = f * L_{long} / (D + ΔD), = f * L_{long} / (D - ΔD).

14. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 4, characterized in that: The vertical line detection optimization factor:

15. Among them: : Angle tolerance, used to control the verticality requirement of the straight line; , : Determined based on the pixel length range of the long side of the keyhole and the possible horizontal edge of the lock pin in the image, as well as the range of their possible positions Search scope; Locking state judgment logic: After Hough transformation and After the optimization factors are screened, a set of horizontal and vertical lines are obtained. Next, the locking state is determined based on these lines: Find pairs of vertical short sides: Find pairs of lines that meet the following conditions in the detected line set. : and Parallel to each other (or the angle is less than the set threshold ); and The distance between ( and are the minimum and maximum values ​​of the long side length of the keyhole respectively); and Length and satisfy: and . are the minimum and maximum lengths of the short side of the keyhole or the short side of the lock pin, respectively).

16. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 1, characterized in that: The pixel distance mapping is calculated as follows: ) and distortion coefficient; obtain the vertical distance D from the camera optical center to the shooting plane (side of the truck) according to the distance sensor, and estimate the upper limit ΔD of the distance measurement error; determine the actual length of the short side of the keyhole ; Calculate the pixel length range of the short side of the keyhole in the image based on the principle of similar triangles : , Similarly, calculate the pixel length range of the long side L_{long} in the image : , ; Determine the locked state: If two pairs of vertical short sides that meet the above conditions are found ( , , , , then further check whether and The cluster of straight lines is approximately vertical (i.e. and The angle is close to 90°, allowing a certain angle tolerance); If the above conditions are met, the lock pin is considered to be in a locked state; if no pair of vertical short sides meeting the conditions is found, the lock pin is considered to be not in a locked state.

17. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 1, characterized in that: In S3, the system has preliminarily determined the state of the lock pin; To ensure the accuracy of the results and reduce misjudgment, this step will conduct a final confirmation based on the detection results of multiple consecutive frames of images; if multiple consecutive frames determine that the lock pin is in a locked state, then it is confirmed that the lock pin is in a locked state; if multiple consecutive frames determine that the lock pin is in an open state, then it is confirmed that the lock pin is in an open state; if the detection results of multiple consecutive frames are inconsistent, or the lock pin state cannot be clearly determined, an alarm is triggered to prompt manual intervention.

18. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 8, characterized in that: After confirming the lock pin status in S3, the system will output the test results in a variety of ways: display the lock pin status in text and color (red for locked, green for open) on the crossing display screen; play voice prompts through the speaker to inform the driver; and upload the test results to the crossing management software platform for management personnel to view and record.

19. The method for detecting the locking state of a container truck at a crossing based on vision according to claim 8, characterized in that: The detection result in S3 will directly link the crossing control system to achieve safety management of container trucks; if the lock pin is in the locked state, the system will prohibit container trucks from passing through the crossing barrier, keep it closed, and prompt the driver to unlock it through the display screen and speaker; if the lock pin is in the open state, the system will allow container trucks to pass through, and the crossing barrier will automatically lift up; This step can ensure that the locking pin of the container truck is correctly opened before entering the port area, effectively prevent safety accidents caused by the locking pin not being opened, and improve the efficiency and safety of crossing traffic.

Images