Vision-based support rod deformation detection method and system

Abstract

The invention relates to the technical field of computer vision detection, and discloses a supporting rod deformation detection method and system based on vision, and the method comprises the steps: obtaining a first supporting rod image; generating a second support rod image with a clear structure; identifying overlapped and non-overlapped areas of the support rods by adopting an instance segmentation network; judging whether a direct bending section and an indirect bending section exist or not based on the non-overlapping area and the overlapping area; and rendering the judgment result in real time. Compared with the prior art, the method only can perform bending detection on a single rod body under a non-shielding condition, and especially under a complex environment condition that supporting rods are densely arranged and shielding overlapping exists, the technical problem that deformation identification and continuous judgment of a shielding area cannot be realized is solved. Due to the fact that modes of instance segmentation, shielding contour reconstruction, skeleton line offset modeling and the like are introduced, indirect inference of shielding section deformation is achieved, and the efficiency of supporting rod structure deformation monitoring in an industrial storage scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision detection technology, and in particular to a vision-based support rod deformation detection method and system. Background Art

[0002] Currently, traditional visual methods such as manual inspections, fixed surveillance cameras, or image detection and edge extraction are primarily used to monitor support pole deformation in warehouse structures. Typically, a rough estimate of the degree of bending is achieved by extracting the edge contour or skeleton of the support pole, combined with linear fitting and offset determination. These image detection methods are somewhat effective under ideal conditions, particularly in scenarios with orthogonal viewing angles, unobstructed single poles, and a simple background, enabling basic deformation recognition and comparative analysis. However, in real-world industrial warehouse environments, support poles are often densely arranged, with their orientation at an angle to the camera's viewing angle. Furthermore, visual interference factors such as cargo obstruction and front-to-back overlap often exist, causing the outlines of the target poles to be partially or completely obscured. Furthermore, RGB images inherently lack depth information, making it difficult for traditional edge detection or single-frame image analysis methods to determine which structures are foreground poles and which are background poles. This often results in overlapping multiple pole contours, segmentation failures, or the misidentification of occluded shadows as deformed areas. Therefore, existing technologies cannot fully meet the demand for high-precision, robust, and intelligent visual image detection of support pole structural status in densely occluded environments such as industrial warehouses. In particular, when the support rods are partially visible and the deformation area is located within an obstructed section, traditional methods cannot effectively determine the structure, easily leading to missed detections or misjudgments, which affect the timeliness and accuracy of structural safety assessments. Therefore, an image detection method that can adapt to multiple rod obstructions and complex background conditions while still providing accurate segmentation, deformation judgment, and visual output is urgently needed to improve the system's adaptability in actual working conditions. Summary of the Invention

[0003] In response to the above-mentioned technical deficiencies, the purpose of the present invention is to propose a vision-based support rod deformation detection method, which aims to solve the problem that the existing method can only perform bending detection on a single rod under unobstructed conditions, especially in complex environmental conditions where support rods are densely arranged and there is occlusion overlap, and the deformation recognition and continuous judgment of the occluded area cannot be realized.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions: The present invention provides a method for detecting support rod deformation based on vision.

[0005] The vision-based support rod deformation detection method includes:

[0006] Step S10: Using a mobile RGB industrial camera to photograph shelf support rods in the storage area to obtain a first support rod image; performing lens distortion correction, geometric perspective calibration, support rod profile feature extraction, and multi-frame alignment and overlay processing on the first support rod image to obtain a second support rod image;

[0007] Step S20: Using an instance segmentation network to segment the second support rod image, obtaining a bounding box and a pixel-level mask of the support rod; detecting overlapping and non-overlapping areas of the support rod based on the bounding box and the pixel-level mask, and performing shape priori repair based on the overlapping areas of the support rod; performing optical flow registration and projection superposition on the images of consecutive frames after the shape priori repair process to obtain a third support rod image;

[0008] Step S30: determining whether there is a direct bending section of the support rod according to the non-overlapping area in the third support rod image, and if there is a direct bending section, calculating the local bending angle;

[0009] Step S40: determining whether there is an indirect bending section in the support rod in the overlapping area according to the overlapping area in the third support rod image;

[0010] Step S50: Rendering the bending determination results of the support rod in step S30 and step S40 into a video image in real time.

[0011] Preferably, in step S10, the arrangement angle between the mobile RGB industrial camera and the shelf support rod is set to a non-orthogonal viewing angle, and the shooting frame rate of the mobile RGB industrial camera is not less than 10 frames per second.

[0012] Preferably, in step S20, the shape prior restoration includes: if the upper and lower segment edges of the overlapping area of the support rods meet linear continuity, performing linear interpolation completion to generate a complete set mask; otherwise, retaining the original overlapping area of the support rods.

[0013] Preferably, in step S30, determining whether the support rod is bent according to the third support rod image, and if bent, calculating the local bending angle, specifically includes:

[0014] Extracting a support rod skeleton from the third support rod image to obtain a skeleton point set;

[0015] The least square method is used to fit the reference straight line equation of the support rod according to the skeleton point set;

[0016] Calculate the distance d from each skeleton point i in the skeleton point set to the reference line equation i , when the distance d i If the deformation distance is greater than the preset threshold, the support rod is determined to be bent;

[0017] When the support rod is determined to be bent, the coordinates of the skeleton points in the vicinity of the skeleton point i are extracted and the upward direction vector θ is fitted. up and the downward vector θ down , according to the upward vector θ up and the downward vector θ down Calculate the local bending angle of skeleton point i.

[0018] Preferably, in step S40, the step of determining whether there is an indirect bending segment in the support rod in the overlapping area according to the overlapping area in the third support rod image specifically includes: extracting the end point of the occluded upper segment and the starting point of the lower segment according to the overlapping area in the third support rod image, and constructing the slope m of the connecting line according to the end point of the upper segment and the starting point of the lower segment. c ; Fit the upper straight line fitting slope m of the overlapping area respectively up and the slope m of the lower straight line fitting down , preset slope difference threshold m th , when |m up -m c |>m th or|m down -m c |>m th , it is determined that there is an indirect bending section in the support rod in the overlapping area.

[0019] Preferably, in step S50, the step of performing real-time rendering of the video image on the bending judgment results of step S30 and step S40 of the support rod specifically includes: drawing a central skeleton line in the video image for the support rod that is judged to have a direct bending segment in the non-overlapping area according to the bending judgment result of step S30, using a red line to represent it, and marking the local bending angle as a label at the position corresponding to the central skeleton line; drawing a central skeleton line for a normal support rod that does not have a direct bending segment, using a green line to represent it; and drawing and highlighting the occluded area that is judged to have an indirect bending segment according to the bending judgment result of step S40 in the form of a dotted outline in the video image.

[0020] Preferably, in step S10, the lens distortion correction specifically includes: using a preset calibration plate to calculate the distortion parameters in the mobile RGB industrial camera to remove barrel distortion and pincushion distortion; the geometric perspective calibration specifically includes: transforming the image after removing barrel distortion and pincushion distortion into a view in which the support rod is oriented vertically; the support rod contour feature extraction specifically includes: performing edge enhancement, contrast stretching, adaptive histogram equalization and high-frequency noise suppression processing on the image after geometric perspective calibration to obtain the support rod contour feature; the multi-frame alignment and overlay processing specifically includes: performing motion compensation on adjacent image frames in the image after support rod contour feature extraction by the feature point matching method to obtain a second support rod image.

[0021] The present invention also provides a vision-based support rod deformation detection system comprising:

[0022] An image acquisition and processing module is configured to use a mobile RGB industrial camera to photograph shelf support rods in a storage area to obtain a first support rod image; and to sequentially perform lens distortion correction, geometric perspective calibration, support rod profile feature extraction, and multi-frame alignment and overlay processing on the first support rod image to obtain a second support rod image.

[0023] An image segmentation processing module is configured to segment the second support rod image using an instance segmentation network to obtain a bounding box and a pixel-level mask of the support rod; detect overlapping and non-overlapping areas of the support rod based on the bounding box and the pixel-level mask, and perform shape priori repair based on the overlapping areas of the support rod; and perform optical flow registration and projection superposition on the images of consecutive frames after the shape priori repair process to obtain a third support rod image;

[0024] a direct bending section determination module, configured to determine whether the support rod has a direct bending section based on the non-overlapping area in the third support rod image, and if so, calculate the local bending angle;

[0025] an indirect bending segment determination module, configured to determine whether a support rod in an overlapping area has an indirect bending segment according to the overlapping area in the third support rod image;

[0026] The bending detection result display module is used to render the bending determination results of step S30 and step S40 of the support rod into a video image in real time.

[0027] The present invention also provides a computer program product, comprising a vision-based support rod deformation detection program, wherein the vision-based support rod deformation detection program implements the vision-based support rod deformation detection method when executed by a processor.

[0028] The beneficial effects of the present invention are as follows: based on image detection technology, the present invention uses a mobile RGB camera to collect multiple frames of images and introduces instance segmentation and optical flow registration mechanisms. This can accurately extract the outline and skeleton structure of the target rod in image scenes where the support rods are partially occluded or overlapped, significantly improving the target separation and structure recovery capabilities of image detection in complex backgrounds.

[0029] The present invention combines direct deformation calculation of non-overlapping areas with indirect bending inference of overlapping areas to achieve a comprehensive judgment of the bending state of the support rod. Combined with the real-time rendering mechanism of video images, the image detection results have good visualization effect and on-site operability, thereby enhancing the real-time and robustness of support rod health status monitoring in industrial warehousing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 Schematic diagram of the flow of a first embodiment of a method for detecting support rod deformation based on vision according to the present invention.

[0032] Figure 2 A schematic diagram of bending deformation of a video image rendered in real time according to the first embodiment of the present invention.

[0033] Figure 3 Schematic diagram of a device for a vision-based support rod deformation detection method of the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] Example 1: Figure 1 2 is a flow chart of the first embodiment of the support rod deformation detection method based on vision of the present invention, which provides the first embodiment of the support rod deformation detection method based on vision of the present invention.

[0036] In a first embodiment, the vision-based support rod deformation detection method includes:

[0037] Step S10: Using a mobile RGB industrial camera to photograph shelf support rods in the storage area to obtain a first support rod image; performing lens distortion correction, geometric perspective calibration, support rod profile feature extraction, and multi-frame alignment and overlay processing on the first support rod image to obtain a second support rod image;

[0038] It should be noted that, in step S10, the lens distortion correction specifically includes: using a preset calibration plate to calculate the distortion parameters in the mobile RGB industrial camera, and removing barrel distortion and pincushion distortion; the geometric perspective calibration specifically includes: transforming the image after removing barrel distortion and pincushion distortion into a view in which the support rod is oriented vertically; the support rod contour feature extraction specifically includes: performing edge enhancement, contrast stretching, adaptive histogram equalization and high-frequency noise suppression processing on the image after geometric perspective calibration to obtain the support rod contour feature; the multi-frame alignment and overlay processing specifically includes: performing motion compensation on adjacent image frames in the image after the support rod contour feature is extracted by the feature point matching method to obtain a second support rod image.

[0039] It should be noted that image acquisition uses a mobile RGB industrial camera to achieve dynamic inspection, which can cover shooting angles of different angles, improve the comprehensiveness of image acquisition and the diversity of support rod exposure areas. Lens distortion correction specifically includes: using a preset calibration plate to capture images, estimating the camera's intrinsic parameter matrix and distortion coefficients (including radial distortion parameters k, k, tangential distortion parameters p, p), and using the reverse mapping method to eliminate barrel distortion and pincushion distortion in the original image, ensuring that the geometric shape of the support rod structure in the image is not disturbed by the optical distortion of the camera. Geometric perspective calibration specifically includes: based on the direction of known vertical line segments in the image (such as side walls, frames and other reference objects), calculating the tilt angle in the current image, and correcting the image coordinate system to the standard view of "support rod vertically upward" through affine or perspective transformation matrix (such as 4-point perspective transformation), so that the outline of the support rod is as close to the longitudinal direction of the image as possible, thereby improving the accuracy of subsequent feature extraction and skeleton line fitting. The support rod contour feature extraction specifically includes: image enhancement processing of the perspective-corrected image, including edge enhancement, contrast stretching (such as linear normalization mapping), histogram equalization, and noise suppression (median filtering or bilateral filtering) to highlight the edge features of the support rod in the image and enhance the grayscale difference between the rod and the background, thereby providing higher-quality visual feature input for the subsequent segmentation algorithm. The multi-frame alignment and overlay processing specifically includes: in image acquisition, there is a certain overlap between adjacent frames, inter-frame motion estimation is performed through feature point matching, and the RANSAC algorithm is combined to eliminate mismatched points, calculate the affine or perspective transformation matrix, and perform motion compensation on adjacent image frames; pixel-level overlay or local maximum fusion is performed on the compensated image sequence to finally form a fused second support rod image. This image has better structural continuity and detail retention capabilities, and is particularly suitable for subsequent target detection and deformation analysis.

[0040] It is understandable that the combination of mobile camera and multi-frame fusion can effectively increase the visible area of the occluded support pole in other frames, thereby providing visual redundancy for contour recovery and indirect deformation inference of the occluded area.

[0041] For example, during a typical inspection, the camera captures an image frame every 0.5 seconds. After acquiring five consecutive frames, feature point tracking and global alignment are performed on the image sequence to create a fused view containing information from multiple angles. In this fused view, a support pole that was originally only partially visible due to occlusion now appears fully formed in other frames, effectively completing the outline and supporting subsequent steps.

[0042] Step S20: Using an instance segmentation network to segment the second support rod image, obtaining a bounding box and a pixel-level mask of the support rod; detecting overlapping and non-overlapping areas of the support rod based on the bounding box and the pixel-level mask, and performing shape priori repair based on the overlapping areas of the support rod; performing optical flow registration and projection superposition on the images of consecutive frames after the shape priori repair process to obtain a third support rod image;

[0043] It should be noted that in step S20, the shape prior restoration includes: if the upper and lower segment edges of the overlapping area of the support rods meet linear continuity, then performing linear interpolation completion to generate a complete set mask; otherwise, retaining the original overlapping area of the support rods.

[0044] It should be noted that the instance segmentation network used in this step can be a deep learning model with pixel-level output capabilities such as Mask R-CNN, YOLACT, SOLOv2, etc. Its training data includes annotated images of support rods in warehousing scenes at multiple angles and under occlusion conditions. The model can output the bounding box and pixel-level mask of each support rod and distinguish different instances. The determination of overlapping areas is completed by analyzing the degree of pixel overlap and spatial proximity between multiple support rod masks. If the masks of two or more support rods have an intersecting pixel area in the image or the center line distance is less than the set overlap threshold, it is marked as having an occlusion relationship. Shape prior repair includes the following two cases:

[0045] If the overlapping area of the support rod has clear visible contours at its upper and lower ends, and the contour edges show a linear continuity trend in space (for example, the difference in edge direction is less than a preset threshold), then based on the upper and lower edge fitting straight lines, the contour mask of the blocked part in the middle is completed by linear interpolation to generate a complete set mask of the support rod;

[0046] If the edge directions of the upper and lower segments are inconsistent or the morphology is significantly different, and the linear continuity condition cannot be met, the segmentation results of the original overlapping area are retained and interpolation completion is not performed to avoid misleading subsequent deformation judgment.

[0047] As you can see, performing shape restoration priors can restore the complete structural outline of the support rod as much as possible under occlusion, facilitating the continuity of subsequent skeleton extraction and line fitting operations. Especially in lightly occluded scenes, fitting lines from the upper and lower edges of the mask and interpolating the mid-segment structure achieves good visual consistency and geometric plausibility.

[0048] It should be understood that if the shape prior inpainting strategy incorrectly performs linear complementation on nonlinear curved regions, it may obscure the true curvature. Therefore, this step is designed as a conditional interpolation inpainting step. If the continuity of the upper and lower segments is not satisfied, the original mask result is retained, and the subsequent deformation determination mechanism is used to further identify the presence of indirect curvature segments.

[0049] For example, in an actual warehouse inspection image, the middle section of a support rod is briefly blocked by the goods in front. There are visible edge segments of 15 pixels and 18 pixels in length above and below the blocked area, respectively. If the direction difference is within 5°, the two edges are connected by fitting a straight line interpolation to generate the middle section shape. However, for another support rod, since the direction difference between the upper and lower sections exceeds 25°, it is automatically determined that it cannot be completed, and the original mask shape is retained as the basis for subsequent analysis.

[0050] Step S30: determining whether there is a direct bending section of the support rod according to the non-overlapping area in the third support rod image, and if there is a direct bending section, calculating the local bending angle;

[0051] It should be noted that in step S30, the step of determining whether the support rod is bent based on the third support rod image and, if bent, calculating the local bending angle specifically includes:

[0052] Extracting a support rod skeleton from the third support rod image to obtain a skeleton point set;

[0053] The least square method is used to fit the reference straight line equation of the support rod according to the skeleton point set;

[0054] Calculate the distance d from each skeleton point i in the skeleton point set to the reference line equation i , when the distance d i If the deformation distance is greater than the preset threshold, the support rod is determined to be bent;

[0055] When the support rod is determined to be bent, the coordinates of the skeleton points in the vicinity of the skeleton point i are extracted and the upward direction vector θ is fitted. up and the downward vector θ down , according to the upward vector θ up and the downward vector θ down Calculate the local bending angle of skeleton point i.

[0056] It is understandable that, first, the central skeleton line of each support rod is extracted from the third support rod image, the mask area is shrunk to a single-pixel wide skeleton line through an image thinning algorithm, and the coordinates of the pixel points on the skeleton are recorded in sequence. Then, based on the entire skeleton line, a reference straight line representing the normal state of the support rod is fitted, which is used as the ideal structural form in the absence of deformation. Then, the degree of deviation between each point on the actual skeleton line and the reference straight line is compared point by point. When the degree of deviation of certain points exceeds a preset tolerance threshold (for example, a few millimeters), it is considered that the support rod has undergone observable deformation at the corresponding position. At this point, the point is marked as a suspected bending center. Furthermore, in order to determine whether there is a real "bend" at this position rather than just a slight offset, a skeleton line segment will be extracted upward and downward with this point as the center, and the approximate direction of the two direction segments will be fitted. If the extension direction between the two directions undergoes a significant mutation (i.e., there is an angle), it means that there is a local bend at this point.

[0057] It should be understood that when the support rod is deformed, its skeleton line will usually produce twists and turns. If we only rely on the degree of deviation from the overall straight line, we may not be able to accurately identify the local bend. Therefore, combining the directional changes of the upper and lower sections for angle analysis will help improve the accuracy and credibility of the judgment.

[0058] For example, in a warehouse inspection test, a set of shelf support poles approximately 2.5 meters high were selected as inspection targets. A mobile RGB camera, mounted on an inspection robot and moving at a speed of 0.5 meters per second, captured image sequences. After multi-frame alignment and fusion, skeleton extraction and deformation analysis were performed on non-overlapping regions of a particular support pole. During the actual inspection, the central region of the support pole exhibited significant deviations from the fitted straight line. This point was automatically identified as a suspected deformation location, and skeleton point sets within a 20-centimeter range above and below were extracted for directional fitting. The results showed a sudden angular shift of nearly 10 degrees between the upper and lower segments. Comparison with manually annotated ground truth data revealed that this point was actually a visual bend caused by a minor collision with a forklift. The error between the identified and manually annotated points was less than 1.5 degrees, and the offset error was less than 3 millimeters. This test demonstrated that, in unobstructed scenes, this method can automatically identify the direct bend of the support pole with high accuracy, with a bending angle error within ±2 degrees and an offset position error within ±5 millimeters, demonstrating excellent engineering adaptability and judgment stability.

[0059] Step S40: determining whether there is an indirect bending section in the support rod in the overlapping area according to the overlapping area in the third support rod image;

[0060] It should be noted that in step S40, the step of determining whether there is an indirect bending segment in the support rod in the overlapping area according to the overlapping area in the third support rod image specifically includes: extracting the end point of the occluded upper segment and the starting point of the lower segment according to the overlapping area in the third support rod image, constructing the slope m of the connecting line according to the end point of the upper segment and the starting point of the lower segment c ; Fit the upper straight line fitting slope m of the overlapping area respectively up and the slope m of the lower straight line fitting down , preset slope difference threshold m th , when |m up -m c |>m th or|m down -m c |>m th , it is determined that there is an indirect bending section in the support rod in the overlapping area.

[0061] As can be understood, first, the end point of the occluded upper segment and the starting point of the occluded lower segment of the support rod are extracted from the overlapping region of the third support rod image, namely, the last visible skeleton points that meet the mask boundary at the upper and lower ends of the occluded region. Then, a connecting line is constructed based on the end point of the upper segment and the starting point of the lower segment. This connecting line simulates the ideal extension path of the support rod within the occluded region. The directional slope of this connecting line is calculated as the "expected direction" assuming the support rod is unbendable. Next, local linear fitting is performed on the skeleton points above and below the occluded region, respectively, to obtain the actual slopes of the upper and lower segments of the support rod, which reflect the actual tilt direction of the current rod body at the upper and lower sides. A preset slope difference threshold is set to measure the deviation tolerance between the upper and lower segments and the connecting line. When the slope difference between the connecting line and the upper segment fitted line exceeds the threshold, or the slope difference between the connecting line and the lower segment fitted line exceeds the threshold, it is determined that the support rod in the overlapping region has an indirect bending segment, indicating that structural deflection or deformation may occur within the occluded region.

[0062] It should be understood that due to overlapping occlusion, some rods are invisible in the image, making direct skeleton extraction or deformation calculation impossible. Therefore, by comparing the geometric extension direction of the upper and lower segments with the consistency of their "virtual connecting lines," we can indirectly determine whether the area has abnormal deflection, which has strong adaptability. The identification of "indirect bending segments" in this step does not rely on pixel information within the occluded area. Instead, it establishes a morphological inference model based on the spatial geometric relationship of the visible part, thus achieving more reliable anomaly detection in the presence of incomplete information. This is particularly suitable for warehousing scenarios with multiple rods interlaced or with severe local occlusion.

[0063] For example, in actual detection, when processing an image of a support pole blocked by pillars on both sides, the connecting line between the last visible skeleton point above the blocked area of the pole and the first visible skeleton point below is extracted. Its direction deviates from the direction of the upper fitting straight line by about 7.2 degrees, and its direction deviates from the direction of the lower fitting straight line by 9.1 degrees, both exceeding the set slope difference threshold (set to 6 degrees). Therefore, it is determined that there is an indirect bending segment inside the blocked area, and the area is marked on the image in the form of a dotted line, and recorded as a structural risk point "pending further confirmation". After manual inspection, it was found that the support pole was indeed slightly deformed in the blocked position, verifying the effectiveness and engineering feasibility of the indirect recognition mechanism under blocked conditions.

[0064] Step S50: Rendering the bending determination results of the support rod in step S30 and step S40 into a video image in real time.

[0065] It should be noted that, in step S50, the step of performing real-time video image rendering of the bending judgment results of step S30 and step S40 of the support rod specifically includes: drawing a central skeleton line in the video image for the support rod that is judged to have a direct bending segment in the non-overlapping area according to the bending judgment result of step S30, using a red line to represent it, and marking the local bending angle as a label at the position corresponding to the central skeleton line; drawing a central skeleton line for a normal support rod that does not have a direct bending segment, using a green line to represent it; and drawing and highlighting the occluded area that is judged to have an indirect bending segment according to the bending judgment result of step S40 in the form of a dotted outline in the video image.

[0066] It is understandable that by performing real-time image rendering of the bending results, not only can the intelligent judgment results of the structural status be intuitively expressed, but it can also greatly improve the inspection efficiency and operability, so that operators can get key prompts of the structural status in the first time without manual comparison.

[0067] It should be understood that the "real-time rendering" in this step is not just static marking, but a visual rendering process that automatically updates in real time with the image frame input. This rendering module is typically embedded in the front-end monitoring interface, and continuous rendering is achieved through edge computing modules or lightweight visualization engines, thus meeting the needs of low-latency, highly readable image detection feedback in industrial scenarios.

[0068] For example, if Figure 2As shown in the example, during an actual video inspection, a direct bend was detected in the unobstructed area of support pole #A, automatically drawing a red skeleton line in the video. Adjacent support pole #B showed no abnormalities, and a green skeleton line was automatically drawn to indicate normal operation. The middle section of support pole #C, located within the obstructed area, was determined to have a potential for indirect bending. The corresponding obstructed area was highlighted with a red dashed border, and a prompt appeared stating "Structural anomaly in the obstructed section, review recommended." This rendering method allows security personnel to quickly focus on problem areas, improving inspection efficiency and the timeliness of structural anomaly resolution.

[0069] Embodiment 2: In addition, the present invention provides a vision-based support rod deformation detection system, which adopts a vision-based support rod deformation detection method in the above embodiment, and can solve the technical problem of vision-based support rod deformation detection. Compared with the prior art, the beneficial effects of the vision-based support rod deformation detection system provided by the present invention are the same as the beneficial effects of the vision-based support rod deformation detection method provided by the above embodiment, and the other technical features of the vision-based support rod deformation detection system are the same as the features disclosed in the above embodiment method, which are not repeated here.

[0070] Example 3: The present invention provides a support rod deformation detection device based on vision, please refer to Figure 3A vision-based support rod deformation detection device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a vision-based support rod deformation detection method in the above-mentioned embodiment 1. A vision-based support rod deformation detection device in an embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. A vision-based support rod deformation detection device is merely an example and should not bring any limitations to the functions and scope of use of the embodiments of the present invention. A vision-based support rod deformation detection device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of a vision-based support rod deformation detection device are also stored in RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow a vision-based support rod deformation detection device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a vision-based support rod deformation detection device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have instead.

[0071] Example 4: The present invention also provides a computer program product, comprising a computer program. When executed by a processor, the computer program implements the steps of the aforementioned vision-based support rod deformation detection method. The computer program product provided by the present invention can solve the technical problem of vision-based support rod deformation detection. Compared with the prior art, the beneficial effects of the computer program product provided by the present invention are the same as those of the vision-based support rod deformation detection method provided by the aforementioned embodiment, and are not further elaborated here.

[0072] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present invention are performed.

[0073] It should be understood that the various parts disclosed in the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0074] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for detecting support rod deformation based on vision, characterized in that: Methods include: Step S10: Using a mobile RGB industrial camera to photograph shelf support rods in the storage area to obtain a first support rod image; performing lens distortion correction, geometric perspective calibration, support rod profile feature extraction, and multi-frame alignment and overlay processing on the first support rod image to obtain a second support rod image; Step S20: Using an instance segmentation network to segment the second support rod image, obtaining a bounding box and a pixel-level mask of the support rod; detecting overlapping and non-overlapping areas of the support rod based on the bounding box and the pixel-level mask, and performing shape priori repair based on the overlapping areas of the support rod; performing optical flow registration and projection superposition on the images of consecutive frames after the shape priori repair process to obtain a third support rod image; Step S30: determining whether there is a direct bending section of the support rod according to the non-overlapping area in the third support rod image, and if there is a direct bending section, calculating the local bending angle; Step S40: determining whether there is an indirect bending section in the support rod in the overlapping area according to the overlapping area in the third support rod image; Step S50: Rendering the bending determination results of the support rod in step S30 and step S40 into a video image in real time.

2. A method for detecting support rod deformation based on vision according to claim 1, characterized in that: In step S10, the arrangement angle between the mobile RGB industrial camera and the shelf support rod is set to a non-orthogonal viewing angle, and the shooting frame rate of the mobile RGB industrial camera is not less than 10 frames per second.

3. The method for detecting support rod deformation based on vision according to claim 1, wherein: In step S20 , the shape prior restoration includes: if the upper and lower segment edges of the overlapping area of the support rods meet linear continuity, performing linear interpolation completion to generate a complete set mask; otherwise, retaining the original overlapping area of the support rods.

4. The method for detecting support rod deformation based on vision according to claim 1, wherein: In step S30, determining whether the support rod is bent based on the third support rod image, and if bent, calculating the local bending angle, specifically includes: Extracting a support rod skeleton from the third support rod image to obtain a skeleton point set; The least square method is used to fit the reference straight line equation of the support rod according to the skeleton point set; Calculate the distance d from each skeleton point i in the skeleton point set to the reference line equation i , when the distance d i If the deformation distance is greater than the preset threshold, the support rod is determined to be bent; When the support rod is determined to be bent, the coordinates of the skeleton points in the vicinity of the skeleton point i are extracted and the upward direction vector θ is fitted. up and the downward vector θ down , according to the upward vector θ up and the downward vector θ down Calculate the local bending angle of skeleton point i.

5. The method for detecting support rod deformation based on vision according to claim 1, wherein: In step S40, the step of determining whether there is an indirect bending segment in the support rod in the overlapping area according to the overlapping area in the third support rod image specifically includes: extracting the end point of the occluded upper segment and the starting point of the lower segment according to the overlapping area in the third support rod image, constructing the slope m of the connecting line according to the end point of the upper segment and the starting point of the lower segment c ; Fit the upper straight line fitting slope m of the overlapping area respectively up and the slope m of the lower straight line fitting down , preset slope difference threshold m th , when |m up -m c |>m th or|m down -m c |>m th , it is determined that there is an indirect bending section in the support rod in the overlapping area.

6. The method for detecting support rod deformation based on vision according to claim 5, wherein: In step S50, the bending judgment results of step S30 and step S40 of the support rod are subjected to real-time rendering of the video image, specifically including: according to the bending judgment result of step S30, drawing the center skeleton line of the support rod determined to have a direct bending segment in the non-overlapping area in the video image, using a red line to represent it, and marking the local bending angle as a label at the position corresponding to the center skeleton line; drawing the center skeleton line of the normal support rod without a direct bending segment, using a green line to represent it; according to the bending judgment result of step S40, the occluded area determined to have an indirect bending segment is drawn in the form of a dotted outline and highlighted in the video image.

7. The method for detecting support rod deformation based on vision according to claim 1, wherein: In step S10, the lens distortion correction specifically includes: using a preset calibration plate to calculate the distortion parameters in the mobile RGB industrial camera to remove barrel distortion and pincushion distortion; the geometric perspective calibration specifically includes: transforming the image after removing barrel distortion and pincushion distortion into a view in which the support rod is oriented vertically; the support rod contour feature extraction specifically includes: performing edge enhancement, contrast stretching, adaptive histogram equalization and high-frequency noise suppression processing on the image after geometric perspective calibration to obtain the support rod contour feature; the multi-frame alignment and overlay processing specifically includes: performing motion compensation on adjacent image frames in the image after the support rod contour feature is extracted by the feature point matching method to obtain a second support rod image.

8. A vision-based support rod deformation detection system, applied to a vision-based support rod deformation detection method according to any one of claims 1 to 7, characterized in that: The vision-based support rod deformation detection system includes: An image acquisition and processing module is configured to use a mobile RGB industrial camera to photograph shelf support rods in a storage area to obtain a first support rod image; and to sequentially perform lens distortion correction, geometric perspective calibration, support rod profile feature extraction, and multi-frame alignment and overlay processing on the first support rod image to obtain a second support rod image. An image segmentation processing module is configured to segment the second support rod image using an instance segmentation network to obtain a bounding box and a pixel-level mask of the support rod; detect overlapping and non-overlapping areas of the support rod based on the bounding box and the pixel-level mask, and perform shape priori repair based on the overlapping areas of the support rod; and perform optical flow registration and projection superposition on the images of consecutive frames after the shape priori repair process to obtain a third support rod image; a direct bending section determination module, configured to determine whether the support rod has a direct bending section based on the non-overlapping area in the third support rod image, and if so, calculate the local bending angle; an indirect bending segment determination module, configured to determine whether a support rod in an overlapping area has an indirect bending segment according to the overlapping area in the third support rod image; The bending detection result display module is used to render the bending determination results of step S30 and step S40 of the support rod into a video image in real time.

9. A vision-based support rod deformation detection device, characterized in that: The vision-based support rod deformation detection device includes: a memory, a processor, and a vision-based support rod deformation detection program stored in the memory and executable on the processor. When the vision-based support rod deformation detection program is executed by the processor, a vision-based support rod deformation detection method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The computer program product includes a vision-based support rod deformation detection program, which, when executed by a processor, implements a vision-based support rod deformation detection method according to any one of claims 1 to 7.

Images