Image-based controller terminal plugging integrity detection method and system

Abstract

The invention relates to the technical field of controller defect detection, and discloses an image-based controller terminal plugging integrity detection method and system, and the method comprises the steps: collecting a two-dimensional gray-scale map and three-dimensional point cloud data of a connector; terminal area fine positioning and residual reserved point cloud extraction are executed; constructing a three-dimensional many-to-many matching matrix in combination with the expected terminal coordinates; calculating a plugging deviation and a misplug probability; and comprehensively judging the plugging state of each terminal. In the prior art, only a two-dimensional image template or a simple distance threshold value is used for judgment, and especially under the working conditions that the terminal structures are highly similar, the plugging positions are dense, and shielding or depth change exists, reliable distinguishing of the empty plugging state, the offset plugging state and the wrong hole plugging state of the terminal cannot be achieved. According to the method, a deep dominant terminal multi-state recognition mechanism is constructed by fusing the two-dimensional image and the three-dimensional point cloud information, so that the automatic detection capability of the terminal connection integrity of the controller is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of controller defect detection, and in particular to a controller terminal plug-in integrity detection method and system based on images. BACKGROUND

[0002] At present, with the continuous improvement of the complexity of industrial control systems, the number, arrangement density and structural similarity of the terminals of the controller connectors have significantly increased, resulting in an increase in the risk of plug-in errors in the assembly process. In order to ensure the reliability and safety of electrical connections, high-precision detection of the plug-in integrity of the terminals of the connectors is required. Traditional methods rely on manual visual inspection, probe contact testing or two-dimensional image-based template matching technology. However, the above methods have the following significant shortcomings in practical application: on the one hand, manual visual inspection has strong subjectivity and is prone to misjudgment due to fatigue, and cannot stably support the automation production demand of large quantities and high rhythm; while probe contact testing can judge the electrical continuity of the terminals, it cannot identify structural defects such as "wrong hole plug-in" or "partial plug-in not fully contacting", and the probe device has a complex structure and is prone to wear, resulting in high maintenance costs. On the other hand, the two-dimensional image-based detection method has the advantages of non-contact and easy integration, but it only uses shallow visual information such as grayscale or edge, and cannot identify small pose differences in three-dimensional space. In the typical industrial environment where the terminals of the connectors are arranged densely, have similar appearances and are prone to reflection, the image differences of "empty plug-in", "partial plug-in" and "wrong hole plug-in" of the terminals are extremely weak, and it is difficult to stably distinguish them by template matching, edge contour or pixel similarity, especially when there are dust, obstruction and reflection, the false detection and missed detection rates increase significantly.

[0003] Therefore, it is urgent to propose an image-based controller terminal plug-in integrity detection method to accurately identify and classify the plug-in state, and to comprehensively improve the intelligent level and engineering applicability of the controller terminal plug-in integrity detection. SUMMARY

[0004] In view of the above technical deficiencies, the purpose of the present application is to propose an image-based controller terminal plug-in integrity detection method to solve the technical problem that the existing technology only relies on two-dimensional image templates or simple distance threshold values for judgment, especially in the working conditions where the terminal structures are highly similar, the plug-in positions are dense, and there are obstructions or depth changes, and cannot reliably distinguish the empty plug-in, partial plug-in and wrong hole plug-in states of the terminals.

[0005] To solve the above technical problems, the present application adopts the following technical solution: the present application provides an image-based controller terminal plug-in integrity detection method, The image-based controller terminal plug-in integrity detection method comprises: Step S10: Acquire a two-dimensional connector grayscale image of the controller terminal using a visible light industrial camera, and acquire three-dimensional connector point cloud data of the connector terminal using a structured light three-dimensional depth camera; perform a fusion registration task based on the two-dimensional connector grayscale image and the three-dimensional connector point cloud data using a coordinate fusion mechanism based on terminal region constraints, and output an enhanced fusion image; Step S20: Based on the enhanced fusion image, a region adaptive filtering mechanism based on terminal feature saliency enhancement is used to perform the terminal region fine localization task and the depth residual preservation point cloud extraction task, and output the point cloud image of the terminal to be tested; Step S30: Obtain the coordinates of the desired terminal. Based on the coordinates of the desired terminal and the point cloud image of the terminal to be tested, construct a three-dimensional many-to-many matching distance matrix using a depth-dominated perception-based minimum risk allocation mechanism. ; Step S40: Based on the three-dimensional many-to-many matching distance matrix Distance deviation is calculated using a misinsertion probability sensing mechanism based on uncertainty intervals. and the probability of misinsertion ; Step S50: Obtain the terminal normal angle based on the distance deviation. and the probability of misinsertion The current terminal integrity status is determined by combining the terminal normal angle. The current terminal integrity status includes uninserted state, offset state, and misaligned insertion state.

[0006] Preferably, step S10, which involves performing a coordinate fusion and registration task based on a terminal region constraint-based coordinate fusion mechanism using the two-dimensional connector grayscale image and the three-dimensional connector point cloud data, and outputting an enhanced fused image, specifically includes: For grayscale images of 2D connectors, the Harris corner detection method is used to extract the terminal region and generate a rectangular bounding box, outputting the rectangular region of the terminal. , Represents the coordinates of the i-th terminal region in the grayscale image of the two-dimensional connector. The corresponding rectangular area of ​​the terminal The x-coordinate is the pixel coordinate, and the y-coordinate is the pixel coordinate; the terminal rectangular area is projected using a bilinear interpolation method. Projected onto the 3D connector point cloud data, the output is the 3D point cloud centroid coordinates corresponding to the rectangular area of ​​the terminal. z is the axis coordinate in the 3D connector point cloud data that is consistent with the terminal insertion direction. Introducing an adjusted weight matrix W, based on the adjusted weight matrix W, the rectangular region of each terminal, and the centroid coordinates of the 3D point cloud. The axial weight fusion mapping mechanism is adopted to perform an axial weight coordinate fusion operation, and an enhanced fusion image is output; wherein, the weight matrix W is adjusted to enhance the registration dominance of the axial direction coordinate consistent with the terminal insertion direction, and to improve the sensitivity of the image after fusion to the terminal insertion depth.

[0007] Preferably, in step S20, the terminal region fine positioning task and the depth residual reservation point cloud extraction task are performed according to the enhanced fusion image using a region adaptive filtering mechanism based on terminal feature saliency enhancement, and the step of outputting the to-be-measured terminal point cloud image, specifically comprising: Step S201: first, the enhanced fusion image is subjected to Gaussian smoothing filtering to extract a fusion grayscale image , and then the enhanced fusion image is subjected to Sobel operator convolution to extract a fusion depth image , and based on the fusion grayscale image and the fusion depth image , a saliency response function is defined using the linear superposition principle ; according to the saliency response function , a local maximum value detection method is used to extract a terminal candidate window region; Step S202: calculate the depth variance of the terminal candidate window region , according to the depth variance , an adaptive filtering kernel size is set using a linear weighting method , according to the adaptive filtering kernel size , bilateral filtering is performed on the terminal candidate window region to output a fine positioning terminal window; Step S203: performing depth residual analysis on the fine positioning terminal window to obtain fine positioning terminal residual and regional average insertion depth; based on the fine positioning terminal residual and the regional average insertion depth, a soft interval filtering mechanism is used to perform an insertion depth screening task, and finally a to-be-measured terminal point cloud image is output.

[0008] Preferably, in step S20, the formula of the saliency response function is:

[0009] wherein, is the saliency weight coefficient of the grayscale gradient in the fusion image, used to adjust the influence intensity of the image texture on the saliency response; is the saliency weight coefficient of the depth gradient in the fusion image, used to adjust the contribution degree of the terminal depth mutation region to the saliency response; is the grayscale gradient vector of the fusion grayscale image at the terminal region coordinate ; is the grayscale gradient vector of the fusion grayscale image at the terminal region coordinate a depth map gradient vector of the current pixel; denotes the L2 norm of a vector.

[0010] Preferably, in step S20, the step of performing the plug-in depth screening task based on the fine positioning terminal residual error and the region average plug-in depth adopts a soft interval filtering mechanism, and specifically includes: a first residual error threshold and a second residual error threshold , satisfying ; When the fine positioning terminal residual error in the fine positioning terminal window is greater than the first residual error threshold , the image data corresponding to the fine positioning terminal residual error is retained; When the fine positioning terminal residual error in the fine positioning terminal window is greater than the first residual error threshold and less than the second residual error threshold , the image data corresponding to the fine positioning terminal residual error is retained by using an exponential decay weighting method; When the fine positioning terminal residual error in the fine positioning terminal window is greater than the second residual error threshold , the image data corresponding to the fine positioning terminal residual error is removed.

[0011] Preferably, in step S30, the step of obtaining the expected terminal coordinates and constructing a three-dimensional many-to-many matching distance matrix based on the expected terminal coordinates and the to-be-measured terminal point cloud image by using a depth-dominant perception-based minimum risk allocation mechanism specifically includes: Step S301: obtaining the to-be-measured terminal coordinates from the to-be-measured terminal point cloud image, obtaining the expected terminal coordinates , and introducing an insertion depth direction weighting factor ; calculating a three-dimensional Euclidean distance based on the to-be-measured terminal coordinates , the expected terminal coordinates , and the insertion depth direction weighting factor ; Step S302: defining a matching pair confidence function based on the three-dimensional Euclidean distance , and the formula expression of the matching pair confidence function is as follows:

[0012] wherein, is a matching distance strong confidence threshold, used to determine whether the three-dimensional terminal coordinate points are highly matched; ​For matching distance weak confidence threshold, used to determine whether there is a plug-in offset between three-dimensional terminal coordinate points; Based on the matching confidence function Adopting the confidence index suppression principle of residual reverse amplification to construct the matching risk matrix ; Step S303: Fusion of three-dimensional Euclidean distance And matching risk matrix Adopting piecewise confidence weighting to generate the final three-dimensional many-to-many matching distance matrix .

[0013] Preferably, in step S40, based on the three-dimensional many-to-many matching distance matrix Adopting the uncertainty interval-based misplug probability perception mechanism to calculate the distance deviation And misplug probability The steps specifically include: Step S401: Obtain the matching pair three-dimensional distance Of the matching pair In the three-dimensional many-to-many matching distance matrix And record the three-dimensional distance As the distance deviation ; Wherein, Indicates the index in the expected terminal coordinates, Indicates the index in the point cloud terminal to be tested; The preset strong confidence threshold And weak confidence threshold ; If , the matching pair Is marked as "normal plug-in"; If , the matching pair Is marked as "plug-in exception"; If , the matching pair Is marked as "uncertain state"; Step S402: For all matching pairs in "uncertain state" , based on the matching pair three-dimensional distance Construct misplug probability perception function , wherein, The steepness parameter of the misplug probability curve, The center inflection point position of the misplug risk probability curve; And the misplug probability perception function corresponding to the matching pair marked as "plug-in exception" is assigned a value of zero; Step S403: Fusion of misplug probability perception function estimation results of matching pairs marked as "uncertain state" and "plug-in exception", output the final misplug probability and distance deviation .

[0014] The present invention also provides an image-based controller terminal connection integrity detection system, comprising: The fusion registration module is used to acquire two-dimensional connector grayscale images of the controller terminals using a visible light industrial camera, and to acquire three-dimensional connector point cloud data of the connector terminals using a structured light three-dimensional depth camera. Based on the two-dimensional connector grayscale images and three-dimensional connector point cloud data, a coordinate fusion mechanism based on terminal region constraints is used to perform the fusion registration task and output an enhanced fusion image. The terminal region fine localization and point cloud extraction module is used to perform terminal region fine localization and depth residual preservation point cloud extraction tasks based on the enhanced fusion image using a region adaptive filtering mechanism based on terminal feature saliency enhancement, and outputs the point cloud image of the terminal to be tested. A 3D many-to-many matching matrix construction module is used to obtain the coordinates of the desired terminal. Based on the coordinates of the desired terminal and the point cloud image of the terminal to be tested, a minimum risk allocation mechanism based on depth-dominated perception is used to construct a 3D many-to-many matching distance matrix. ; The module for estimating misinsertion probability and distance deviation is used for estimating the distance matrix of three-dimensional many-to-many matching. Distance deviation is calculated using a misinsertion probability sensing mechanism based on uncertainty intervals. and the probability of misinsertion ; The terminal integrity determination module is used to obtain the terminal normal angle based on the distance deviation. and the probability of misinsertion The current terminal integrity status is determined by combining the terminal normal angle. The current terminal integrity status includes uninserted state, offset state, and misaligned insertion state.

[0015] The present invention also provides an image-based controller terminal connection integrity detection device, comprising: a memory, a processor, and an image-based controller terminal connection integrity detection program stored in the memory and executable on the processor. When the image-based controller terminal connection integrity detection program is executed by the processor, it implements an image-based controller terminal connection integrity detection method.

[0016] The present invention also provides a computer program product, including an image-based controller terminal connection integrity detection program, wherein the image-based controller terminal connection integrity detection program implements the image-based controller terminal connection integrity detection method when executed by a processor.

[0017] The beneficial effects of this invention are as follows: This invention enables fine differentiation of the plugging status of controller terminals, supports multi-category identification of empty plugging, off-center plugging and misaligned plugging, and by fusing two-dimensional grayscale images and three-dimensional point cloud data, it enhances the structural differences of terminals and introduces a many-to-many matching distance matrix based on expected coordinates and an uncertainty perception mechanism, which effectively improves the accuracy and anti-interference ability of plugging anomaly detection.

[0018] This invention improves the actual insertion state of terminals by introducing residual point cloud extraction and normal angle-assisted discrimination mechanism, thereby enhancing its practicality and adaptability in industrial settings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of an image-based controller terminal connection integrity detection method according to the present invention.

[0021] Figure 2 This is an enhanced grayscale schematic diagram of the first embodiment of an image-based controller terminal connection integrity detection method of the present invention.

[0022] Figure 3 This is an enhanced grayscale schematic diagram of a first embodiment of an image-based controller terminal connection integrity detection method of the present invention.

[0023] Figure 4 This is a schematic diagram of the original point cloud density of a first embodiment of an image-based controller terminal connection integrity detection method of the present invention.

[0024] Figure 5 This is a schematic diagram of the residual point cloud after retention, representing a first embodiment of an image-based controller terminal connection integrity detection method of the present invention.

[0025] Figure 6 This is a schematic diagram of a device for detecting the integrity of controller terminal connections based on an image, according to the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of the image-based controller terminal connection integrity detection method of the present invention, which presents the first embodiment of the image-based controller terminal connection integrity detection method of the present invention.

[0028] In the first embodiment, the image-based controller terminal connection integrity detection method includes: Step S10: Acquire a two-dimensional connector grayscale image of the controller terminal using a visible light industrial camera, and acquire three-dimensional connector point cloud data of the connector terminal using a structured light three-dimensional depth camera; perform a fusion registration task based on the two-dimensional connector grayscale image and the three-dimensional connector point cloud data using a coordinate fusion mechanism based on terminal region constraints, and output an enhanced fusion image; It should be noted that the "coordinate fusion mechanism based on terminal region constraints" refers to extracting the region of interest where the terminal is located in a two-dimensional grayscale image through terminal template matching, edge detection, or semantic segmentation algorithms. This region is used to constrain the cropping and registration range of the three-dimensional point cloud, including key structural features such as the terminal center point, edge contour, and corner points. On this basis, pseudo-color registration, projection remapping, or local ICP algorithms are used to match and transform the salient features of the terminal in the two-dimensional image coordinates with the physical structure points in the three-dimensional point cloud coordinates, thereby generating a fused image under a unified spatial reference. The fused image retains the image texture details and depth structure changes, and has higher edge sharpness and spatial consistency.

[0029] Understandably, by constructing fused images, a clear correspondence is established between the two-dimensional and three-dimensional information of the connector terminals, significantly enhancing the geometric separability of the terminal edges and the ability to perceive depth changes. The terminal areas in the fused images not only possess clear grayscale contrasts but also contain realistic features of insertion depth changes, thereby effectively improving the accuracy of subsequent terminal recognition and insertion status analysis, and avoiding visual misjudgments caused by light reflection, terminal occlusion, or terminal similarity.

[0030] It should be understood that, compared to traditional terminal detection methods that rely solely on two-dimensional images or three-dimensional point clouds, the former is limited by the lack of depth information and cannot determine the depth of insertion; the latter is easily affected by point cloud noise and local missing data, especially on reflective metal surfaces where there are serious data gaps. This invention, through the fusion of images and point clouds, not only bridges the perceptual differences between the two but also reduces registration errors through semantic constraints on the terminal region. It effectively avoids problems such as structural mismatch and region drift in the multimodal data decoupling process of traditional methods, improving the consistency and contextual information integrity of the fused image.

[0031] For example, in a typical industrial controller connector assembly, there are 24 terminal holes with similar structures, and some terminals may be slightly misaligned or not inserted at all during actual assembly. Under low light and complex background interference, relying solely on two-dimensional images makes it difficult to accurately identify misaligned or partially inserted states. However, the enhanced fused image generated by the aforementioned fusion mechanism in this invention can accurately extract the geometric shape and insertion depth differences corresponding to each terminal. In actual controller inspection, the terminal structure recognition accuracy is improved by approximately 42%, and the false negative rate is reduced by more than 30%, significantly improving adaptability and recognition accuracy in complex environments.

[0032] Step S20: Based on the enhanced fusion image, a region adaptive filtering mechanism based on terminal feature saliency enhancement is used to perform the terminal region fine localization task and the depth residual preservation point cloud extraction task, and output the point cloud image of the terminal to be tested; It should be noted that the "region-adaptive filtering mechanism based on terminal feature saliency enhancement" refers to the following: Addressing the subtle differences in structural texture, grayscale boundaries, and depth distribution between terminal and non-terminal regions in the fused image, a saliency weight map is constructed to assign different processing priorities to different regions, thereby enhancing the edge of the terminal region and preserving depth details. Terminal feature saliency includes several aspects, such as abrupt gradient changes at the terminal socket boundary, grayscale transitions caused by the terminal's metallic reflectivity, and local depressions or protrusions in the depth map when the terminal head is inserted. The region-adaptive filtering mechanism adjusts the shape and size of the filter kernel according to the saliency intensity, focusing image enhancement processing on the terminal region. This effectively suppresses background interference such as plastic shell texture, reflective highlights, or blurred shadows, thus retaining only data points highly correlated with the terminal's geometry in the 3D point cloud, removing redundant or false points without recognition significance. The final output point cloud image of the terminal under test has clear spatial structure, good boundary continuity, and reasonable depth distribution.

[0033] Understandably, the technical effect of this step lies in enhancing the separability of the terminal region in the fused image by significantly improving the terminal's features, thereby increasing the accuracy of terminal localization. Simultaneously, adaptive filtering preserves details of the terminal region while strongly suppressing non-target areas, significantly reducing recognition errors caused by background interference. In the 3D point cloud processing stage, the introduction of a depth residual preservation strategy accurately depicts the actual insertion depth and deformation boundaries of the plugged-in terminal, ensuring sufficient and reliable 3D structural data support for subsequent matching analysis. This step not only strengthens structural recognition capabilities in image space but also enhances depth resolution capabilities in the point cloud dimension, ensuring high-precision closed-loop control from image acquisition to structural recognition.

[0034] It should be understood that, compared to common denoising methods in existing technologies such as fixed template matching, edge extraction, or mean filtering, this step provides a novel mechanism that achieves efficient image enhancement and point cloud optimization while preserving key geometric features. Traditional methods often exhibit problems such as feature blurring, boundary offset, and damage to the 3D point cloud structure under complex backgrounds or lighting interference, especially prone to misjudgment or missed detection under non-ideal conditions such as incomplete terminal insertion or tilted insertion. This invention, by guiding the filtering process in the image saliency space, adaptively focuses on highly saliency regions, fundamentally suppressing the influence of unstructured noise. Simultaneously, by setting structural residual preservation constraints within the region during point cloud data processing, it effectively avoids the risk of erroneously deleting genuine terminal features.

[0035] For example, such as Figure 2 As shown, the original acquired 2D grayscale image reveals significant confusion between the terminal area boundary and unstructured areas such as the plastic shell and shadows under complex background and lighting conditions. This results in poor separability of terminal features, hindering subsequent precise localization and recognition tasks. Figure 3 As shown, after processing with a region adaptive filtering mechanism based on terminal feature saliency enhancement, the edge structure, grayscale gradient, and regional texture of the terminal region in the image are significantly improved, significantly suppressing interference from non-target background regions and improving the recognition accuracy and positioning stability of the terminal region. Figure 4 As shown, although the terminal area exhibits a certain degree of structural prominence, it also suffers from a large amount of non-target point cloud noise overlapping with the structure, resulting in unclear point cloud structure and blurred boundaries, thus affecting the accuracy of insertion status identification. Figure 5 As shown, after adopting the depth residual preservation strategy, only the effective point cloud that is highly correlated with the terminal geometry is retained in the figure. Non-target point clouds and redundant backgrounds are effectively filtered out. The terminal boundaries are clear and the spatial structure is complete, providing high-quality 3D input data for subsequent misinsertion discrimination and state modeling.

[0036] Step S30: Obtain the coordinates of the desired terminal. Based on the coordinates of the desired terminal and the point cloud image of the terminal to be tested, construct a three-dimensional many-to-many matching distance matrix using a depth-dominated perception-based minimum risk allocation mechanism. ; It should be noted that the "depth-dominant perception-based minimum risk allocation mechanism" means that when performing terminal pairing and identification tasks, it not only considers the Euclidean distance error of spatial location but also introduces the local structural consistency of the point cloud depth dimension as a weighting index for pairing priority. First, a set of desired terminal coordinates is generated based on a preset terminal wiring diagram or historical connection template, with each coordinate point corresponding to the location of a terminal that should exist. Then, this set is matched with the point cloud image of the terminal to be tested extracted from the image using three-dimensional features. To address uncertainties such as terminal occlusion, missing parts, and deformation in real-world scenarios, this mechanism constructs a three-dimensional many-to-many matching candidate set and calculates a set of matching risk function values ​​on this set.

[0037] Understandably, by introducing a depth-dominated perception mechanism, erroneous pairings caused by angle changes, brightness interference, or terminal occlusion in traditional two-dimensional matching can be effectively eliminated. Furthermore, by establishing a pairing optimization strategy based on a multi-objective risk function, high matching stability can be maintained even in complex mating states such as misaligned holes, empty insertions, or offset insertions. The constructed three-dimensional matching distance matrix not only reflects the matching reliability between each desired terminal position and all detected terminals but also provides structured data support for subsequent uncertainty identification, state classification, and misinsertion probability estimation.

[0038] For example, in a pin header connection scenario for an industrial controller, there is a column of 12 terminal points. The 4th position is a reserved empty space, the 7th terminal is not fully inserted, and the 9th position has an incorrect insertion. Traditional methods, when there is uneven image brightness or terminal obstruction, may incorrectly identify the 4th position as a failed insertion, while considering the 7th and 9th positions as normal insertions. This invention, however, introduces a depth-driven matching mechanism into the 3D point cloud, accurately distinguishing between the low depth density corresponding to "no terminal insertion," the depth offset residual corresponding to "offset terminal insertion," and the high-risk cross-region pairing corresponding to "incorrect insertion." This constructs a matching matrix that truly reflects the connection status, providing a reliable basis for subsequent status identification.

[0039] Step S40: Based on the three-dimensional many-to-many matching distance matrix Distance deviation is calculated using a misinsertion probability sensing mechanism based on uncertainty intervals. and the probability of misinsertion ; It should be noted that the "misinsertion probability sensing mechanism based on uncertainty intervals" refers to the following: after obtaining the 3D matching distance between each pair of "expected terminal coordinates" and "actually detected terminal point cloud centroids," different matching confidence intervals are divided based on a preset distance threshold model. Then, a probability mapping mechanism is introduced to convert the spatial deviation magnitude of each matching result into an estimated misinsertion probability value. Specifically, this mechanism typically sets two key thresholds: the first threshold is a "strong confidence threshold," used to determine terminals with very small insertion deviations and extremely high insertion integrity; the second threshold is a "weak confidence threshold," used to identify terminals with critical deviations and certain insertion risks. Matching results below the strong confidence threshold are considered normal insertions, matching pairs between the two thresholds are assigned a moderate misinsertion probability, while matching pairs above the weak confidence threshold are marked as high-risk matches, representing a high probability of misinsertion, off-center insertion, or incorrect hole insertion problems. The aforementioned confidence intervals can be set based on historical data statistics, sample training regression models, or engineering experience, and combined with methods such as Gaussian distribution and kernel density estimation to generate continuous misinsertion probability values.

[0040] It should be understood that traditional technologies often use a single threshold to determine whether a terminal has been successfully inserted, lacking an effective distinction between the spatial relationship between "empty insertion" and "incorrect insertion." This is especially problematic in real-world scenarios where a terminal may not be fully inserted but has already triggered an electrical connection, or it may be inserted into an incorrect hole but point cloud features still exist. In these typical, easily confused scenarios, the single threshold mechanism is prone to false alarms and missed detections, severely impacting the practicality and scalability of the detection. The uncertainty interval mechanism employed in this invention effectively covers these "gray areas." By setting a probability transition region, boundary samples can be processed with moderate confidence, providing more robust technical support for the operational interpretability, fault tolerance flexibility, and adaptability of the detection results.

[0041] Step S50: Obtain the terminal normal angle based on the distance deviation. and the probability of misinsertion The current terminal integrity status is determined by combining the terminal normal angle. The current terminal integrity status includes uninserted state, offset state, and misaligned insertion state.

[0042] It should be noted that the "terminal normal angle" in this step refers to the angle between the local surface normal vector extracted for each terminal under test in the 3D point cloud and the ideal insertion direction. This normal angle can be calculated by the vector angle between the normal obtained by fitting the surface within the terminal point cloud region and the standard insertion axis, reflecting the degree of deviation between the terminal insertion direction and the expected insertion path. By introducing the terminal normal angle as a supplementary judgment criterion and jointly analyzing it with the "matching distance deviation" and "misinsertion probability" output from the previous stage, a ternary judgment model can be constructed to classify and identify the insertion status of each terminal more precisely. Specifically: when the matching distance is large, the misinsertion probability is high, and the normal angle deviates significantly, it is judged as a mis-insertion state; when the matching distance is small, the misinsertion probability is high, but the normal angle is basically normal, it is judged as a gap insertion state (i.e., the insertion position is correct but not fully inserted); when the matching distance is moderate, the misinsertion probability is moderate, and the normal angle deviates from a certain threshold but is not completely reversed, it is judged as an off-center insertion state.

[0043] Understandably, this judgment logic constructs a multi-factor judgment mechanism that integrates spatial location information (distance deviation), insertion uncertainty expression (misinsertion probability), and local directional structural information (normal angle). This enables it to effectively distinguish three types of insertion defect states in more complex and similar scenarios. In particular, in scenarios where the terminal is inserted into the wrong position but is still inserted into a certain socket, the angular feature of the normal angle can effectively capture the deviation of "insertion path and hole axis not being consistent", making up for the shortcomings of traditional methods in identifying misinsertion.

[0044] For example, in a plug-in test of an industrial controller, the following situations occurred on a 24-pin terminal block: pin 8 was "incorrectly inserted," pin 15 was "empty," and pin 21 was "off-center." First, steps S30-S40 calculated the matching distance deviations to be 1.6mm, 0.9mm, and 0.7mm, with mis-insertion probabilities of 0.93, 0.75, and 0.68, respectively. At this point, relying solely on these two indicators, pins 15 and 21 might be difficult to distinguish. However, after introducing the normal angle, the normal angle of pin 8 was 22° (significantly deviating from the standard axis), the normal angle of pin 15 was 3° (almost perpendicular insertion), and the normal angle of pin 21 was 11° (slight tilt). The final judgment was: pin 8 was incorrectly inserted, pin 15 was empty, and pin 21 was off-center. The results, verified by actual manual disassembly, were consistent with the actual situation, indicating that the mechanism has good engineering application value in distinguishing abnormal connections.

[0045] Example 2: Furthermore, the present invention provides an image-based controller terminal connection integrity detection system, which employs an image-based controller terminal connection integrity detection method from the above embodiments, and can solve the technical problem of image-based controller terminal connection integrity detection. Compared with the prior art, the beneficial effects of the image-based controller terminal connection integrity detection system provided by the present invention are the same as those of the image-based controller terminal connection integrity detection method provided in the above embodiments, and other technical features of the image-based controller terminal connection integrity detection system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0046] Example 3: This invention provides an image-based controller terminal connection integrity detection device. Please refer to... Figure 6An image-based controller terminal connection integrity detection device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform an image-based controller terminal connection integrity detection method as described in Embodiment 1 above. The image-based controller terminal connection integrity detection device in this embodiment of the invention may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This image-based controller terminal connection integrity detection device is merely an example and should not impose any limitations on the functionality or scope of use of the embodiments of the invention. An image-based controller terminal connection integrity detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the image-based controller terminal connection integrity detection device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An I / O interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows an image-based controller terminal connection integrity detection device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows an image-based controller terminal connection integrity detection device with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0047] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image-based controller terminal connection integrity detection method described above. The computer program product provided by this invention can solve the technical problem of image-based controller terminal connection integrity detection. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the image-based controller terminal connection integrity detection method provided in the above embodiments, and will not be repeated here.

[0048] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0049] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0050] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An image-based controller terminal plug-in integrity detection method, characterized by, The method comprises: Step S10: acquiring a two-dimensional connector grayscale image of the terminal of the controller through a visible light industrial camera, and collecting three-dimensional connector point cloud data of the terminal of the connector through a structured light three-dimensional depth camera; performing a fusion registration task based on the two-dimensional connector grayscale image and the three-dimensional connector point cloud data by using a coordinate fusion mechanism based on terminal region constraints, and outputting an enhanced fusion image; Step S20: performing a terminal region fine positioning task and a depth residual reservation type point cloud extraction task based on the enhanced fusion image by using a region adaptive filtering mechanism based on terminal feature saliency enhancement, and outputting a to-be-measured terminal point cloud image; Step S30: Obtain the expected terminal coordinates, and construct a three-dimensional many-to-many matching distance matrix based on the expected terminal coordinates and the terminal point cloud image to be measured using a minimum risk allocation mechanism based on depth dominant perception ; Step S40: Based on the three-dimensional many-to-many matching distance matrix Calculating distance deviation using an uncertainty interval-based false insertion probability awareness mechanism and false insertion probability ; Step S50: Obtain terminal normal angle, based on distance deviation And misplug probability Determine the current terminal integrity state in combination with the terminal normal angle, the current terminal integrity state including empty insertion state, offset insertion state and misplug insertion state.

2. The image-based controller terminal plug-in integrity detection method of claim 1, wherein, In step S10, the step of performing the fusion registration task based on the two-dimensional connector grayscale image and the three-dimensional connector point cloud data by using the coordinate fusion mechanism based on the terminal region constraints, and outputting the enhanced fusion image, specifically comprises: For grayscale images of 2D connectors, the Harris corner detection method is used to extract the terminal region and generate a rectangular bounding box, outputting the rectangular region of the terminal. , Represents the coordinates of the i-th terminal region in the grayscale image of the two-dimensional connector. The corresponding rectangular area of ​​the terminal. The x-coordinate is the pixel coordinate, and the y-coordinate is the pixel coordinate; the terminal rectangular area is projected using a bilinear interpolation method. Projected onto the 3D connector point cloud data, the output is the 3D point cloud centroid coordinates corresponding to the rectangular area of ​​the terminal. z is the axis coordinate in the 3D connector point cloud data that is consistent with the terminal insertion direction. An adjustment weight matrix W is introduced, and the adjustment weight matrix W, each terminal rectangular region, and three-dimensional point cloud barycentric coordinates are used as a basis An axial weight fusion mapping mechanism is adopted to perform an axial weighted coordinate fusion operation, and an enhanced fusion image is output; wherein the adjustment weight matrix W is used to enhance the registration dominance of the axial direction coordinate consistent with the terminal insertion direction, and to improve the sensitivity of the image after fusion to the terminal insertion depth.

3. The image-based controller terminal plug-in integrity detection method of claim 1, wherein, In step S20, the step of performing the terminal region fine positioning task and the depth residual reservation type point cloud extraction task based on the enhanced fusion image by using the region adaptive filtering mechanism based on the terminal feature saliency enhancement, and outputting the to-be-measured terminal point cloud image, specifically comprises: Step S201: first, the enhanced fusion image is subjected to Gaussian smoothing filtering method to extract a fusion gray image , then the enhanced fusion image is subjected to Sobel operator convolution method to extract a fusion depth image , and based on the fusion gray image and the fusion depth image , a saliency response function is defined by using linear superposition principle ; according to the saliency response function , a local maximum value detection method is used to extract a terminal candidate window region; Step S202: calculating the depth variance of the terminal candidate window region , according to the depth variance Setting the adaptive filter kernel size in a linear weighting manner , according to the adaptive filter kernel size Performing bilateral filtering on the terminal candidate window region to output a fine positioning terminal window; Step S203: performing depth residual analysis on the fine positioning terminal window to obtain fine positioning terminal residual and region average insertion depth; performing an insertion depth screening task based on the fine positioning terminal residual and the region average insertion depth by using a soft interval filtering mechanism, and finally outputting the to-be-measured terminal point cloud image.

4. The image-based controller terminal plug-in integrity detection method of claim 3, wherein, In step S20, the saliency response function is expressed by the formula: wherein, is a saliency weight coefficient of the gradient of the grayscale image in the fusion image, used to adjust the influence intensity of the image texture on the saliency response; is a saliency weight coefficient of the gradient of the depth image in the fusion image, used to adjust the contribution degree of the terminal depth mutation region on the saliency response; is a gradient vector of the grayscale image at the terminal region coordinate in the fusion grayscale image; is a gradient vector of the depth image at the terminal region coordinate in the fusion depth image; represents the L2 norm of the vector.

5. The image-based controller terminal plug-in integrity detection method of claim 3, wherein, In step S20, the step of performing the insertion depth screening task based on the fine positioning terminal residual and the region average insertion depth by using the soft interval filtering mechanism, specifically comprises: a first residual threshold value is preset and a second residual threshold value , satisfying ;​ When the fine positioning terminal residual error in the fine positioning terminal window is greater than a first residual error threshold , the image data corresponding to the part of the fine positioning terminal residual error is reserved; When the fine positioning terminal residual error in the fine positioning terminal window is greater than a first residual error threshold and less than a second residual error threshold , an exponential decay weighting method is adopted to retain the image data corresponding to the part of the fine positioning terminal residual error . When the fine positioning terminal residual error in the fine positioning terminal window is greater than a second residual error threshold , the image data corresponding to the part of the fine positioning terminal residual error is rejected.

6. The image-based controller terminal plug-in integrity detection method of claim 1, wherein, In step S30, the expected terminal coordinates are obtained, and a three-dimensional many-to-many matching distance matrix is constructed based on the expected terminal coordinates and the terminal point cloud image to be measured using a minimum risk allocation mechanism based on depth dominant perception , specifically comprising: Step S301: Obtain the to-be-tested terminal coordinates from the to-be-tested terminal point cloud image , obtain the expected terminal coordinates , introduce the insertion depth direction weighting factor , ; calculate the three-dimensional Euclidean distance based on the to-be-tested terminal coordinates , the expected terminal coordinates and the insertion depth direction weighting factor ;​ Step S302: determining the three-dimensional Euclidean distance Defining the matching pair confidence function The formula of the matching pair confidence function is wherein, a match distance strong confidence threshold value for determining whether three-dimensional terminal coordinate points are highly matched or not; a match distance weak confidence threshold value for determining whether three-dimensional terminal coordinate points exist with a plug-in offset or not. Matching based on a confidence function Constructing a matching risk matrix using a confidence index suppression principle with residual reverse amplification ; Step S303: fusion of three-dimensional Euclidean distances and matching risk matrix The final three-dimensional many-to-many matching distance matrix is generated using piecewise confidence weighting .

7. The image-based controller terminal plug-in integrity detection method of claim 1, wherein, In step S40, based on the three-dimensional many-to-many matching distance matrix A misinsertion probability-aware mechanism based on uncertainty intervals is used to calculate the distance deviation And the step of misinsertion probability Specifically includes: Step S401: Obtain the 3D many-to-many matching distance matrix Matching pairs in Matching pairs of three-dimensional distances and three-dimensional distance Recorded as distance deviation ;in, Indicates the index in the desired terminal coordinates. This represents the index of the point to be tested in the cloud sub-database; a preset strong confidence threshold is used. With weak confidence threshold ; If then the pair of matches is labeled "plug normal"; If then the pair of matches is labeled as "plug-in anomaly"; If then the pair of matches is labeled as "Uncertain State"; Step S402: For all matching pairs in "uncertain state" , based on the three-dimensional distance of the matching pair Constructing the misplug probability perception function , wherein, is the steepness parameter of the misplug probability curve, is the central inflection point position of the misplug risk probability curve; and the misplug probability perception function corresponding to the matching pair marked as "abnormal plugging" is assigned a value of zero; Step S403: fuse the misplug probability perception function estimation results of the matching pairs labeled as "uncertain state" and "plug-in abnormality", and output the final misplug probability and distance deviation .

8. An image-based controller terminal plug-in integrity detection system applied to the image-based controller terminal plug-in integrity detection method of any one of claims 1 to 7, characterized in that, The image-based controller terminal insertion integrity detection system comprises: The fusion registration module is configured to acquire a two-dimensional connector grayscale image of the terminal of the controller through a visible light industrial camera, and collect three-dimensional connector point cloud data of the terminal of the connector through a structured light three-dimensional depth camera; perform a fusion registration task based on the two-dimensional connector grayscale image and the three-dimensional connector point cloud data by using a coordinate fusion mechanism based on terminal region constraints, and output an enhanced fusion image; The terminal region fine positioning and point cloud extraction module is configured to perform a terminal region fine positioning task and a depth residual reservation type point cloud extraction task based on the enhanced fusion image by using a region adaptive filtering mechanism based on terminal feature saliency enhancement, and output a to-be-measured terminal point cloud image. The three-dimensional many-to-many matching matrix construction module is configured to acquire expected terminal coordinates, and construct a three-dimensional many-to-many matching distance matrix based on the expected terminal coordinates and the terminal point cloud image to be measured by using a minimum risk allocation mechanism based on depth dominant perception ; a misplug probability and distance deviation estimation module configured to estimate a misplug probability based on the three-dimensional many-to-many matching distance matrix calculating the distance deviation using a misplug probability-aware mechanism based on uncertainty intervals and a misplug probability ; a terminal integrity determination module configured to obtain a terminal normal angle, determine a distance deviation based on the terminal normal angle and a misplug probability determine a current terminal integrity state based on the terminal normal angle, the current terminal integrity state including an empty insertion state, a partial insertion state, and a misalignment insertion state.

9. An image-based controller terminal plug-in integrity detection apparatus, characterized by, The image-based controller terminal insertion integrity detection device comprises a memory, a processor, and an image-based controller terminal insertion integrity detection program stored in the memory and executable on the processor, and the image-based controller terminal insertion integrity detection program, when executed by the processor, implements the image-based controller terminal insertion integrity detection method in any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program product comprises an image-based controller terminal insertion integrity detection program, and the image-based controller terminal insertion integrity detection program, when executed by a processor, implements the image-based controller terminal insertion integrity detection method in any one of claims 1 to 7.

Images