Flexible wire harness detection method and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of vision, and discloses a flexible wire harness detection method and electronic equipment, and the method comprises the steps: obtaining a wire harness image, extracting a binary mask of the wire harness image, converting the binary mask into a skeleton mask, and extracting end points and cross points of a wire harness through the skeleton mask. By calculating the minimum bending energy in the plurality of paths, the path corresponding to the minimum bending energy is determined as the path of the flexible wire harness, a large number of hyper-parameters do not need to be set, and the end points of the flexible wire harness do not need to be manually determined, so that the detection efficiency of the flexible wire harness can be improved.

Description

Technical Field

[0001] This application relates to the field of vision technology, and in particular to a method and electronic device for detecting flexible wire harnesses. Background Technology

[0002] In robotics, the automated arrangement, connection, and inspection of flexible wire harnesses are crucial for achieving flexible production in industrial robots. Flexible wire harnesses require precise inspection and segmentation.

[0003] Currently, edge detection and contour tracking methods are highly dependent on the working scenario, making them difficult to adapt to the complex environment of factories. Furthermore, they can only process single, non-intersecting wire harnesses. Once multiple wire harnesses overlap, edge information overlaps, or the topology becomes complex, the contour extracted by the edge detection algorithm will break, and the contour tracking algorithm will get stuck in a local optimum, failing to accurately distinguish the boundary and centerline of a single wire harness. The final segmentation result will contain a large number of misjudged areas.

[0004] Model-based estimation methods require setting a large number of hyperparameters and users need to manually select the endpoints of the flexible harness, resulting in cumbersome operation and low detection efficiency. Summary of the Invention

[0005] This application provides a method and electronic device for detecting flexible wire harnesses, so as to improve the detection efficiency of flexible wire harnesses.

[0006] The embodiments of this application provide the following technical solutions: In a first aspect, embodiments of this application provide a method for detecting flexible wire harnesses, the method comprising: Obtain the wire harness image of the flexible wire harness; Extract the binary mask of the wire harness image based on the wire harness image; Convert the binary mask into a skeleton mask; Based on the skeleton mask, extract the endpoints and intersections of the flexible wire harness; Calculate the bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness, where each path corresponds to a bending energy. Obtain the minimum bending energy among multiple bending energies, and determine the path corresponding to the minimum bending energy as the path of the flexible wire harness.

[0007] In some embodiments, Based on the skeleton mask, extract the endpoints and intersections of the flexible wire harness, including: Obtain the neighborhood detection template; The neighborhood detection template is convolved with the skeleton mask to extract the endpoints and intersections of the flexible wire bundle. The skeleton mask includes multiple skeleton pixels. The skeleton pixel corresponding to the endpoint of the flexible wire bundle is adjacent to only one skeleton pixel, and the skeleton pixel corresponding to the intersection of the flexible wire bundle is adjacent to at least three skeleton pixels.

[0008] In some embodiments, The flexible wire harness consists of multiple wires, each made of a different material; Calculate the bending energies of multiple paths formed by the endpoints and intersections of the flexible harness, including: Obtain the bending stiffness of flexible wire harnesses of each material; Based on the bending stiffness of the flexible wire harness of each material, and combined with the degree of bending of the flexible wire harness of each material in different segments, the bending energy of the path formed by the endpoints and intersections of the flexible wire harness of each material is calculated, so as to obtain multiple bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness.

[0009] In some embodiments, Based on the bending stiffness of the flexible wire harness of each material, and combined with the degree of bending of the flexible wire harness of each material in different segments, the bending energy of the path formed by the endpoints and intersections of the flexible wire harness of each material is calculated, including:

[0010] in, Let m be the bending energy of the flexible wire harness made of the m-th material. Let be the bending stiffness of the flexible wire harness made of the m-th material. Where is the Voronoi length, and N is the number of pixels in the skeleton mask of the flexible wire bundle. Let m be the curvature of the flexible wire harness made of the m-th material at the k-th pixel, where m ≥ 2.

[0011] In some embodiments, The method also includes: Calculate the path smoothness for each path; Based on the bending energy and path smoothness of each path, the total energy corresponding to each path is calculated, resulting in multiple total energies; The path corresponding to the minimum total energy among multiple total energies is determined as the path of the flexible wire harness.

[0012] In some embodiments, Calculate the path smoothness for each path, including:

[0013] in, For path smoothness, For smoothness weighting coefficients, Let be the material weight of the flexible wire harness of material m, and N be the number of pixels in the skeleton mask of the flexible wire harness. It is the square of the curvature difference between adjacent pixels.

[0014] In some embodiments, Based on the bending energy and path smoothness of each path, calculate the total energy corresponding to each path, including:

[0015] in, For total energy, For bending energy, For path smoothness.

[0016] In some embodiments, The method also includes: Perform a distance transformation on the binary mask to obtain a distance matrix, where each element in the distance matrix represents the distance from a pixel to the nearest background pixel in the binary mask; The maximum diameter of the flexible wire harness is determined based on the distance matrix, where the maximum diameter of the flexible wire harness is the maximum value of the distance in the distance matrix; Starting from an endpoint, traverse along the connected path of the skeleton mask to determine the traversal length of the endpoint, where the traversal length is the connected length of the endpoint and several other endpoints. If the traversal length of a certain endpoint is greater than the maximum diameter of the flexible wire bundle, and no other endpoint or intersection is encountered, then the endpoint is determined to be a real endpoint. If the traversal length of a certain endpoint is less than the maximum diameter of the flexible wire bundle, and another endpoint or intersection is encountered, then the endpoint is determined to be a false endpoint. Iterate through all endpoints to remove all dummy endpoints from the skeleton mask.

[0017] In some embodiments, The method also includes: Construct a training set, which includes multiple wire bundle images, each of which contains labeled flexible wire bundles. The YOLO model is trained based on the training set to obtain the trained YOLO model. Based on the wire harness image, extract the binary mask of the wire harness image, including: Based on the trained YOLO model, extract the binary mask of the wire harness image; Converting a binary mask to a skeleton mask includes: The binary mask is converted into a skeleton mask using a skeleton extraction algorithm.

[0018] Secondly, embodiments of this application provide an electronic device, including: At least one processor; At least one memory for storing at least one program; When at least one program is executed by at least one processor, the at least one processor implements the method for detecting flexible wire harnesses in the first aspect.

[0019] Thirdly, embodiments of this application provide a non-volatile computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the aforementioned flexible wire harness detection method.

[0020] Fourthly, embodiments of this application provide a computer program product comprising one or more lines of program code stored in a non-volatile computer-readable storage medium. A processor of an electronic device reads the program code from the non-volatile computer-readable storage medium and executes the program code to complete the steps of the above-described flexible wire harness detection method.

[0021] The beneficial effects of this application embodiment are as follows: Unlike the prior art, this application embodiment provides a method for detecting flexible wire harnesses, including: acquiring a wire harness image of the flexible wire harness; extracting a binary mask from the wire harness image; converting the binary mask into a skeleton mask; extracting the endpoints and intersections of the flexible wire harness according to the skeleton mask; calculating multiple bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness, wherein each path corresponds to a bending energy; obtaining the minimum bending energy among the multiple bending energies, and determining the path corresponding to the minimum bending energy as the path of the flexible wire harness.

[0022] By acquiring the wire harness image, extracting the binary mask of the wire harness image, and then converting the binary mask into a skeleton mask, the endpoints and intersections of the wire harness are extracted using the skeleton mask. By calculating the minimum bending energy among multiple paths, the path corresponding to the minimum bending energy is determined as the path of the flexible wire harness. This eliminates the need to set a large number of hyperparameters and eliminates the need to manually determine the endpoints of the flexible wire harness, thereby improving the detection efficiency of the flexible wire harness. Attached Figure Description

[0023] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0024] Figure 1 This is a schematic diagram of a flexible wire harness provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a flexible wire harness detection method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a training process for a YOLO model provided in an embodiment of this application; Figure 4A This is a schematic diagram of a wire harness image provided in an embodiment of this application; Figure 4B This is a schematic diagram of a binary mask for a wire harness image provided in an embodiment of this application; Figure 5 yes Figure 2 A detailed flowchart of step S204 in the process; Figure 6A This is a schematic diagram of the intersection point of a wire harness image provided in an embodiment of this application; Figure 6B This is a schematic diagram of the endpoints of a wire harness image provided in an embodiment of this application; Figure 7 This is a schematic diagram of a process for deleting fake endpoints provided in an embodiment of this application; Figure 8 This is a schematic diagram illustrating a process for deleting spurious endpoints provided in an embodiment of this application; Figure 9 yes Figure 2 A detailed flowchart of step S205 in the process; Figure 10 This is a schematic flowchart illustrating the process of determining the path of a flexible wire harness according to an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0026] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0027] Flexible wire harnesses, as components formed by the orderly or intertwined arrangement of multiple deformable linear objects such as wires, cables, and ropes, have been widely used in key fields such as industrial manufacturing, medical surgery, smart homes, and robot control due to their flexibility, bendability, and high integration. In industrial settings, flexible wire harnesses are the core carriers of signal and power transmission within automotive electronics and consumer electronics devices (such as smartphones and laptops), and their assembly quality directly determines the performance stability of the end product. In the medical field, the precise positioning and manipulation of flexible wire harnesses (such as surgical sutures and endoscopic catheters) are crucial for the success of minimally invasive surgery. In robotics, the automated arrangement, connection, and testing of flexible wire harnesses are important steps in realizing flexible production for industrial robots.

[0028] To ensure the functional reliability of flexible wire harnesses in the aforementioned applications, precise detection and segmentation are required. This involves separating the wire harness region from a complex background, extracting the centerline coordinates of individual wires, and obtaining the overall segmentation mask. This provides visual feedback for subsequent quality inspections (such as wire harness breakage and short-circuit detection), morphological analysis (such as entanglement assessment), and automated manipulation (such as robot grasping and sorting). However, due to their inherent flexibility, flexible wire harnesses are prone to complex entanglement and crossing during use or assembly, forming a topological structure with high curvature changes and near-parallel crossings. Furthermore, their operating environments are often accompanied by background interference (such as mechanical parts obscuring the view in industrial environments or tissue fluid reflection in medical settings). This makes it difficult for traditional visual inspection methods to achieve efficient and robust segmentation and detection, becoming a key technical bottleneck restricting the automation development of flexible wire harness-related fields.

[0029] Traditional wire harness detection methods, such as edge detection and contour tracking, are highly dependent on the working environment and are difficult to adapt to the complex environment of a factory. Furthermore, these methods can only process single, non-intersecting wire harnesses. When multiple wire harnesses overlap, edge information overlaps, or the topology is complex, the contours extracted by edge detection algorithms become broken, and contour tracking algorithms get stuck in local optima, failing to accurately distinguish the boundaries and centerlines of individual wire harnesses. The final segmentation result contains a large number of misclassified regions.

[0030] Model-based estimation methods require setting a large number of hyperparameters, rely excessively on the uniformity of color in deformable linear objects, and require users to manually select the endpoints of flexible wire bundles, resulting in cumbersome operation and low detection efficiency.

[0031] Based on this, embodiments of this application provide a method for detecting flexible wire harnesses to improve the detection efficiency of flexible wire harnesses.

[0032] Please see Figure 1 , Figure 1 This is a schematic diagram of a flexible wire harness provided in an embodiment of this application.

[0033] like Figure 1 As shown, each flexible wire harness includes two endpoints, and there is an intersection point between two intersecting flexible wire harnesses.

[0034] If multiple flexible wire harnesses are intertwined, the flexible wire harnesses need to be inspected to identify each flexible wire harness.

[0035] For details, please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for detecting flexible wire harnesses provided in an embodiment of this application.

[0036] The detection method for the flexible wire harness is executed by one or at least two processors of an electronic device.

[0037] like Figure 2 As shown, the flow chart of the flexible wire harness detection method includes the following steps S201-S206: Step S201: Obtain the wire harness image of the flexible wire harness; Specifically, the wire harness image is an image containing a flexible wire harness, captured by a camera or obtained from the memory of an electronic device, wherein the wire harness image is an RGB image.

[0038] Step S202: Extract the binary mask of the wire harness image based on the wire harness image; Specifically, the binary mask of the wire harness image is extracted using the YOLO model.

[0039] Before extracting the binary mask from the harness image, the YOLO model needs to be trained.

[0040] Please see Figure 3 , Figure 3 This is a schematic diagram of a process for training a YOLO model provided in an embodiment of this application.

[0041] like Figure 3 As shown, the process of training a YOLO model includes the following steps S301-S302: Step S301: Construct a training set, wherein the training set includes multiple wire harness images, each of which includes an labeled flexible wire harness.

[0042] Specifically, a training set is constructed by manually labeling the flexible wire bundles in each wire bundle image. The training set includes multiple wire bundle images, each of which contains labeled flexible wire bundles; that is, each wire bundle image is a labeled wire bundle image.

[0043] Furthermore, the labeled wire harness images are processed, for example, by rotating and simulating lighting, so that the wire harness images contain different shapes of flexible wire harnesses (straight, bent, crossed), different lighting (strong light, weak light, shadow), different backgrounds (industrial production line background, solid color background), and different wire harness specifications (diameter, color), in order to increase data diversity and enable the trained YOLO model to adapt to more complex real-world scenarios.

[0044] In this embodiment of the application, the YOLO model includes the YOLO v8 model.

[0045] Step S302: Train the YOLO model based on the training set to obtain the trained YOLO model.

[0046] Specifically, training parameters are configured through code or command line. These parameters include the number of training epochs and batch size. Based on the wire harness images in the training set, the YOLO model is trained using these parameters to obtain the trained YOLO model, which can then be used to extract the binary mask of the wire harness image.

[0047] Please refer to the following: Figure 4A and Figure 4B , Figure 4A This is a schematic diagram of a wire harness image provided in an embodiment of this application; Figure 4B This is a schematic diagram of a binary mask for a wire harness image provided in an embodiment of this application.

[0048] like Figure 4A As shown, the wire harness image is an RGB image, which contains a flexible wire harness.

[0049] like Figure 4B As shown, after extraction by the trained YOLO model, a binary mask of the wire harness image is obtained. The pixel values ​​in the binary mask are only 0 or 255, where a pixel value of 0 represents the background area and a pixel value of 255 represents the wire harness area.

[0050] In this embodiment of the application, a binary mask is extracted from the wire harness image by using a trained YOLO model. The YOLO model, for example, the YOLO v8 model, uses a segmentation network to learn the foreground region of the target, which can effectively ignore the background.

[0051] Step S203: Convert the binary mask into a skeleton mask; Specifically, the binary mask of the wire harness image is converted into a skeleton mask using a skeleton extraction algorithm.

[0052] It is understandable that after the binary mask is extracted by the skeleton, the resulting skeleton mask is a single-pixel wide mask. The pixel value of the skeleton mask is 0 or 255. In the skeleton mask, a pixel value of 0 represents the background area, i.e., the background pixel; a pixel value of 255 represents the skeleton, i.e., the skeleton pixel. This skeleton pixel is precisely aligned with the center line of the flexible wire harness (including endpoints and intersections).

[0053] In the embodiments of this application, the skeleton extraction algorithm includes: Zhang-Suen thinning algorithm, Guo-Hall algorithm, and Hit-or-Miss transform algorithm.

[0054] Step S204: Extract the endpoints and intersections of the flexible wire harness based on the skeleton mask; The skeleton mask contains the endpoints and intersections of the flexible wire harness.

[0055] For details, please refer to [link / reference]. Figure 5 , Figure 5 yes Figure 2 A detailed flowchart of step S204 is shown.

[0056] like Figure 5 As shown, step S204: Extracting the endpoints and intersections of the flexible wire harness based on the skeleton mask, includes the following steps S241-S242: Step S241: Obtain the neighborhood detection template.

[0057] Specifically, the neighborhood detection template is a 3x3 convolution kernel K = [ [1, 1, 1], [1,10, 1], [1, 1, 1] ] The center position of the convolution kernel corresponds to the target skeleton pixel to be detected, and its weight is 10; the surrounding 8 positions correspond to the 8 neighboring pixels of the target pixel (up, down, left, right + four diagonal), and their weights are 1.

[0058] It is understandable that the design principle of the convolution kernel is to amplify the features of the target pixel by using "high weight in the center + low weight in the neighborhood", while quantizing the number of adjacent skeleton pixels in the neighborhood (the more adjacent pixels, the larger the convolution result).

[0059] Step S242: Convolve the neighborhood detection template with the skeleton mask to extract the endpoints and intersections of the flexible wire bundle.

[0060] Specifically, to facilitate convolution calculations, the skeleton mask is first normalized to a 0-1 mask (denoted as M), where: M(x,y) = 1: Skeleton pixels (originally 255); M(x,y) = 0: Background pixels (originally 0).

[0061] Specifically, the neighborhood detection template and the skeleton mask are convolved. For example, for each skeleton pixel P (x,y) in the skeleton mask M, a 3×3 convolution kernel is slid around P, and the sum of "neighboring pixel value × corresponding weight" is calculated to obtain the convolution result S.

[0062] Since the skeleton mask consists of multiple skeleton pixels, each of which is a single pixel wide, the convolution result is 12 if it is convolved with the endpoints or dummy endpoints, 12 if it is convolved with the midpoints, and 14 if it is convolved with the intersections. Therefore, the endpoints and intersections of the flexible wire bundle can be determined by the convolution result. If the convolution result is 14, then the skeleton pixel is determined to be an intersection.

[0063] Furthermore, since the convolution results of endpoints and intermediate points are similar, the number of neighboring nodes N is used to distinguish between endpoints and intermediate points. The skeleton mask includes multiple skeleton pixels. The skeleton pixel corresponding to the endpoint of the flexible bundle is adjacent to only one skeleton pixel, the skeleton pixel corresponding to the intermediate point of the flexible bundle is adjacent to two skeleton pixels, and the skeleton pixel corresponding to the intersection point of the flexible bundle is adjacent to at least three skeleton pixels. Therefore, the number of neighboring nodes N for the endpoint is 1, the number of neighboring nodes N for the intermediate point is 2, and the number of neighboring nodes N for the intersection point is ≥3.

[0064] In summary, the endpoints and intersections of the flexible wire bundle can be determined by convolving the neighborhood detection template with the skeleton mask and combining the convolution result with the number of neighboring pixels of the skeleton pixels in the skeleton mask. Specifically: If the convolution result corresponding to a certain skeleton pixel is 12, and the number of neighboring nodes of the skeleton pixel is 1, then the skeleton pixel is determined to be the endpoint of the flexible wire bundle. If the convolution result corresponding to a certain skeleton pixel is 14, and the number of its neighbors is greater than or equal to 3, then the skeleton pixel is determined to be the intersection point of the flexible wire bundle.

[0065] Please see Figure 6A , Figure 6A This is a schematic diagram of the intersection point of a wire harness image provided in an embodiment of this application.

[0066] like Figure 6A As shown, the red dot is located at the intersection of the flexible wire bundle. The number of neighbors of this intersection is 3, that is, this intersection is adjacent to 3 skeleton pixels.

[0067] Please refer to the following: Figure 6B , Figure 6B This is a schematic diagram of the endpoints of a wire harness image provided in an embodiment of this application.

[0068] like Figure 6B As shown, the two solid dots are located at the endpoints of the flexible wire bundle. The number of neighbors of this endpoint is 1, that is, this endpoint is adjacent to 1 skeleton pixel.

[0069] Understandably, the jagged edges of the binary mask of the wire bundle can cause split ends in the generated skeleton mask, resulting in spurious endpoints. Therefore, this application further removes spurious endpoints from the skeleton mask to improve the accuracy of the subsequently generated path.

[0070] Please see Figure 7 , Figure 7 This is a schematic diagram of a process for deleting fake endpoints provided in an embodiment of this application.

[0071] like Figure 7 As shown, the process for deleting spoof endpoints includes the following steps S701-S709: Step S701: Perform a distance transformation on the binary mask to obtain the distance matrix.

[0072] Specifically, the binary mask of the line harness image is subjected to Euclidean distance transformation to obtain the distance matrix D, that is, the distance transformation matrix. The size of the distance matrix is ​​exactly the same as that of the binary mask, and each element in the distance matrix represents the distance from a pixel to the nearest background pixel in the binary mask. That is, each pixel value = the straight-line distance from the pixel to the nearest background pixel (the area with a value of 0) in the binary mask.

[0073] Step S702: Determine the maximum diameter of the flexible wire harness based on the distance matrix.

[0074] Specifically, the maximum diameter of the flexible wire harness is the maximum value of the distance in the distance matrix. All pixel values ​​in the distance matrix D are less than or equal to the maximum diameter d of the wire harness. By traversing all pixels in the distance matrix D, the maximum value is taken as the maximum diameter d of the wire harness. The maximum diameter d is the number of pixels corresponding to the actual physical thickness of the wire harness. For example, a wire harness with a diameter of 5mm corresponds to 8 pixels, so d=8.

[0075] Step S703: Starting from an endpoint, traverse along the connected path of the skeleton mask to determine the traversal length of the endpoint.

[0076] Specifically, for each endpoint, the traversal is performed along the connected path of the skeleton mask to determine the traversal length of the endpoint. The traversal length is the connected length of the endpoint and several other endpoints. This traversal length is used in conjunction with the maximum diameter d to determine whether the endpoint is noise, that is, whether the endpoint is a false endpoint.

[0077] Step S704: Whether the traversal length of the endpoint is greater than the maximum diameter of the flexible wire harness.

[0078] Specifically, determine whether the traversal length of the endpoint is greater than the maximum diameter of the flexible wire harness. If the traversal length is greater than the maximum diameter of the flexible wire harness, proceed to step S706: determine that the endpoint is a real endpoint. If the traversal length of the endpoint is less than or equal to the maximum diameter of the flexible wire harness, proceed to step S705: determine whether another endpoint or intersection has been encountered.

[0079] Step S705: Has another endpoint or intersection been encountered?

[0080] Understandably, if the traversal length is greater than the maximum diameter d and no endpoints or intersections are encountered, it means that the traversal is proceeding along the true branch of the skeleton mask, and the endpoint is a true endpoint; if the traversal length is less than the maximum diameter d and another endpoint or intersection is encountered, it means that the traversal is proceeding along the noisy branch of the skeleton mask, and the endpoint is a false endpoint.

[0081] If another endpoint or intersection is encountered, proceed to step S709: determine that the endpoint is a false endpoint; If no other endpoint or intersection is encountered, proceed to step S706: determine that the endpoint is a real endpoint.

[0082] Step S706: Determine that the endpoint is a real endpoint.

[0083] Step S707: Whether to traverse all endpoints.

[0084] Specifically, determine whether all endpoints in the skeleton mask have been traversed. If so, proceed to step S708: identify all spurious endpoints and remove all spurious endpoints from the skeleton mask.

[0085] If not, proceed to the next endpoint and return to step S703: Starting from an endpoint, traverse along the connected path of the skeleton mask to determine the traversal length of the endpoint.

[0086] Step S708: Identify all spurious endpoints and remove them from the skeleton mask.

[0087] Specifically, after traversing all endpoints in the skeleton mask, all spurious endpoints are identified and removed from the skeleton mask.

[0088] Step S709: Determine that the endpoint is a spurious endpoint.

[0089] Specifically, if the traversal length of an endpoint is less than or equal to the maximum diameter of the flexible wire harness, and another endpoint or intersection is encountered, then the endpoint is determined to be a false endpoint.

[0090] Please refer to the following: Figure 8 , Figure 8 This is a schematic diagram illustrating a process for deleting fake endpoints provided in an embodiment of this application.

[0091] like Figure 8 As shown, a1 and b1 are binary masks for the flexible wire harness. Due to the presence of protruding parts, false endpoints may occur.

[0092] a2 and b2 are skeletonized skeleton masks, which contain spurious endpoints; a3 and b3 are the skeleton masks obtained after removing spurious endpoints.

[0093] By removing spurious endpoints from the skeleton mask, a denoised skeleton mask is obtained.

[0094] In this embodiment of the application, by traversing all endpoints in the skeleton mask and removing spurious endpoints in the skeleton mask, noise in the skeleton mask can be eliminated, and the path of the flexible wire harness can be better determined.

[0095] Step S205: Calculate the bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness, where each path corresponds to a bending energy.

[0096] Understandably, multiple endpoints and intersections constitute multiple paths, but each flexible wire bundle corresponds to only one correct path. As a flexible object, the flexible wire bundle has the minimum bending energy in its natural state. Therefore, the optimal path at the intersection is the combination of line segments that minimizes the cumulative bending energy.

[0097] Please refer to the following: Figure 9 , Figure 9 yes Figure 2A detailed flowchart of step S205 in the process.

[0098] like Figure 9 As shown, step S205 involves calculating the bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness, including the following steps S251-S252: Step S251: Obtain the bending stiffness of the flexible wire harness for each material; It is understandable that the bending stiffness of flexible wire harnesses varies depending on the material. Assuming... Let m be the bending stiffness of the flexible wire harness made of the m-th material, for example: the bending stiffness of the flexible wire harness made of copper. ≈10 5 N m 2 The bending stiffness of a flexible nylon wire harness ≈10 3 N m 2 .

[0099] In the embodiments of this application, the bending stiffness of the flexible wire harness of each material is obtained through material manuals, engineering manuals or experimental measurements. For standard materials (such as nylon), the bending stiffness can be obtained directly through material manuals or engineering manuals; for non-standard materials (such as hybrid materials), the bending stiffness needs to be calculated through bending experiments.

[0100] Step S252: Based on the bending stiffness of the flexible wire harness of each material, and combined with the bending degree of the flexible wire harness of each material in different segments, calculate the bending energy of the path formed by the endpoints and intersections of the flexible wire harness of each material, so as to obtain multiple bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness.

[0101] Specifically, based on the bending stiffness of the flexible wire harness of each material, and combined with the degree of bending of the flexible wire harness of each material in different segments, the bending energy of the path formed by the endpoints and intersections of the flexible wire harness of each material is calculated, including:

[0102] in, Let m be the bending energy of the flexible wire harness made of the m-th material. Let be the bending stiffness of the flexible wire harness made of the m-th material. Where is the Voronoi length, and N is the number of pixels in the skeleton mask of the flexible wire bundle. Let m be the curvature of the flexible wire harness made of the m-th material at the k-th pixel, where m ≥ 2.

[0103] Specifically, assuming the flexible harness is composed of two or more materials (m≥2, such as copper core + nylon sheath, or different materials in different sections), each path corresponds to one material, that is, the path of the m-th material. Therefore, it is necessary to determine the material of each path in advance, for example, by material labeling.

[0104] Voronoi length refers to the average distance from the skeleton pixel to the edge of the wire bundle, which is equivalent to the physical half-width of the wire bundle. Its function is to normalize the pixel-level curvature calculation result into the physical level bending energy, eliminate the interference of wire bundle thickness and pixel resolution on energy calculation, and make the bending energy calculation result conform to the laws of elasticity and have engineering applicability.

[0105] Step S206: Obtain the minimum bending energy among multiple bending energies, and determine the path corresponding to the minimum bending energy as the path of the flexible wire harness.

[0106] It is understandable that, as a flexible object, the flexible wire harness has the minimum bending energy under natural conditions. Therefore, the optimal path at the intersection is the combination of line segments that minimizes the cumulative bending energy.

[0107] Specifically, after obtaining multiple bending energies, the minimum bending energy among the multiple bending energies is determined, and the path corresponding to the minimum bending energy is determined as the path of the flexible wire harness.

[0108] Furthermore, in this embodiment, path smoothness is introduced as an energy supplement to calculate the total energy corresponding to each path, and the path with the minimum total energy is determined as the path of the flexible wire harness.

[0109] Please see Figure 10 , Figure 10 This is a flowchart illustrating the process of determining the path of a flexible wire harness, as provided in an embodiment of this application.

[0110] like Figure 10 As shown, the process for determining the path of the flexible wire harness includes the following steps S1001-S1003: Step S1001: Calculate the path smoothness of each path.

[0111] Specifically, the path smoothness of each path is calculated, including:

[0112] in, For path smoothness, For smoothness weighting coefficients, Let be the material weight of the flexible wire harness of material m, and N be the number of pixels in the skeleton mask of the flexible wire harness. It is the square of the curvature difference between adjacent pixels.

[0113] It should be noted that, This is the smoothness weighting coefficient, and its value can be adjusted according to the specific scenario. For example, setting the smoothness weighting coefficient... =0.3, to ensure that smoothness does not dominate the bending energy term.

[0114] It is understandable that the material weight is related to the rigidity of the m-th material. The harder the material (such as copper), the greater the cost of a sudden curvature change (hard wire harnesses are difficult to bend suddenly). In other words, the stronger the resistance to bending, the higher the material weight. The larger the value, for example, the material weight for copper is set to 1.0; while the softer the material (such as nylon), the smaller the cost of abrupt curvature changes, that is, the weaker the resistance to bending, the higher the material weight. The smaller the value, for example, the material weight for nylon is set to 0.5. By setting the material weight, the path smoothness calculation can be adapted to the physical properties of different materials.

[0115] in, Let be the curvature of the flexible wire harness made of the m-th material at the (k+1)-th pixel. Let be the curvature of the flexible wire harness made of the m-th material at the k-th pixel, where the (k+1)-th and k-th pixels are adjacent pixels. By calculating the square of the curvature difference between adjacent pixels, the abrupt change in curvature between them can be quantified. It can be understood that the smaller the square of the curvature difference between adjacent pixels, the higher the smoothness between the two points, and the smoother the path. The smaller the value, the better.

[0116] Step S1002: Calculate the total energy corresponding to each path based on the bending energy and path smoothness of each path, and obtain multiple total energies.

[0117] Specifically, based on the bending energy and path smoothness of each path, the total energy corresponding to each path is calculated, including:

[0118] in, For total energy, For bending energy, For path smoothness.

[0119] Step S1003: Determine the path corresponding to the minimum total energy among multiple total energies as the path of the flexible wire harness.

[0120] Specifically, after obtaining the total energy corresponding to each path, the multiple total energies are aggregated, the minimum value among the multiple total energies is searched to obtain the minimum total energy, and the path corresponding to the minimum total energy is determined as the path of the flexible wire harness.

[0121] In this embodiment, different curvature weights are assigned to flexible wire harnesses of different materials, and path smoothness is introduced as a supplementary factor to select the real path that best matches the physical shape of the flexible wire harness, thereby improving the detection and segmentation success rate of different wire harnesses.

[0122] It should be noted that when two flexible wire bundles intersect, the flexible wire bundles are divided into 4 segments by the intersection point. By randomly selecting the skeleton mask of one segment of the flexible wire bundle, the bending energy corresponding to the skeleton mask is calculated, and the minimum bending energy path is generated for the other segments. The combination of the segments corresponding to the minimum total energy is determined as the correct path of the flexible wire bundle. After determining the correct path of the flexible wire bundle, the skeleton masks of the two segments are merged to generate the center pixel sequence of the flexible wire bundle, wherein the center pixel sequence includes every pixel in the skeleton mask.

[0123] In this embodiment of the application, a method for detecting a flexible wire harness is provided, comprising: acquiring a wire harness image of the flexible wire harness; extracting a binary mask of the wire harness image based on the wire harness image; converting the binary mask into a skeleton mask; extracting the endpoints and intersections of the flexible wire harness according to the skeleton mask; calculating multiple bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness, wherein each path corresponds to a bending energy; obtaining the minimum bending energy among the multiple bending energies, and determining the path corresponding to the minimum bending energy as the path of the flexible wire harness.

[0124] By acquiring the wire harness image, extracting the binary mask of the wire harness image, and then converting the binary mask into a skeleton mask, the endpoints and intersections of the wire harness are extracted using the skeleton mask. By calculating the minimum bending energy among multiple paths, the path corresponding to the minimum bending energy is determined as the path of the flexible wire harness. This eliminates the need to set a large number of hyperparameters and eliminates the need to manually determine the endpoints of the flexible wire harness, thereby improving the detection efficiency of the flexible wire harness.

[0125] Please refer to the following: Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0126] like Figure 11 As shown, the electronic device 110 includes one or more processors 111 and a memory 112. Wherein, Figure 11 Take a processor 111 as an example.

[0127] Processor 111 and memory 112 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0128] The processor 111 is configured to provide computing and control capabilities to control the electronic device 110 to perform corresponding tasks, such as controlling the electronic device 110 to perform the flexible wire harness detection method in any of the above method embodiments. The method includes: acquiring a wire harness image of the flexible wire harness; extracting a binary mask of the wire harness image based on the wire harness image; converting the binary mask into a skeleton mask; extracting the endpoints and intersections of the flexible wire harness according to the skeleton mask; calculating multiple bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness, wherein each path corresponds to a bending energy; obtaining the minimum bending energy among the multiple bending energies, and determining the path corresponding to the minimum bending energy as the path of the flexible wire harness.

[0129] By acquiring the wire harness image, extracting the binary mask of the wire harness image, and then converting the binary mask into a skeleton mask, the endpoints and intersections of the wire harness are extracted using the skeleton mask. By calculating the minimum bending energy among multiple paths, the path corresponding to the minimum bending energy is determined as the path of the flexible wire harness. This eliminates the need to set a large number of hyperparameters and eliminates the need to manually determine the endpoints of the flexible wire harness, thereby improving the detection efficiency of the flexible wire harness.

[0130] Processor 111 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0131] Memory 112, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the flexible wire harness detection method in the embodiments of this application. Processor 111 can implement the flexible wire harness detection method in any of the above method embodiments by running the non-transitory software programs, instructions, and modules stored in memory 112. Specifically, memory 112 may include volatile memory (VM), such as random access memory (RAM); memory 112 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or other non-transitory solid-state storage devices; memory 112 may also include combinations of the above types of memory.

[0132] Memory 112 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 112 may optionally include memory remotely located relative to processor 111, and these remote memories may be connected to processor 111 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0133] One or more modules are stored in memory 112. When executed by one or more processors 111, they perform the flexible wire harness detection method in any of the above method embodiments, for example, the method described above. Figure 2 The steps shown.

[0134] This application also provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors. For example, the one or more processors can execute the flexible wire harness detection method in any of the above method embodiments, such as performing the steps described above.

[0135] This application also provides a computer program product, which includes one or more lines of program code stored in a non-volatile computer-readable storage medium. The processor of the electronic device reads the program code from the non-volatile computer-readable storage medium and executes the program code to complete the steps of the above-described flexible wire harness detection method.

[0136] The apparatus or device embodiments described above are merely illustrative. The unit modules described as separate components may or may not be physically separate, and the components shown as module units may or may not be physical units; that is, they may be located in one place or distributed across multiple network module units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for a computer device (which may be a personal computer, server, or network device, etc.) to execute the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above. For the sake of brevity, they are not provided in detail; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting flexible wire harnesses, characterized in that, The method includes: Obtain an image of the flexible wire harness; Based on the wire harness image, extract the binary mask of the wire harness image; Convert the binary mask into a skeleton mask; Based on the skeleton mask, extract the endpoints and intersections of the flexible wire bundle; Calculate the bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness, wherein each path corresponds to a bending energy. The minimum bending energy among the multiple bending energies is obtained, and the path corresponding to the minimum bending energy is determined as the path of the flexible wire harness.

2. The method according to claim 1, characterized in that, The step of extracting the endpoints and intersections of the flexible wire harness based on the skeleton mask includes: Obtain the neighborhood detection template; The neighborhood detection template is convolved with the skeleton mask to extract the endpoints and intersections of the flexible wire bundle. The skeleton mask includes multiple skeleton pixels. The skeleton pixel corresponding to the endpoint of the flexible wire bundle is adjacent to only one skeleton pixel. The skeleton pixel corresponding to the intersection of the flexible wire bundle is adjacent to at least three skeleton pixels.

3. The method according to claim 1, characterized in that, The flexible wire harness includes multiple wires, each of which is made of a different material; The calculation of multiple bending energies of the multiple paths formed by the endpoints and intersections of the flexible wire harness includes: Obtain the bending stiffness of the flexible wire harness for each material; Based on the bending stiffness of the flexible wire harness of each material, and combined with the degree of bending of the flexible wire harness of each material in different segments, the bending energy of the path formed by the endpoints and intersections of the flexible wire harness of each material is calculated to obtain multiple bending energies of multiple paths formed by the endpoints and intersections of the flexible wire harness.

4. The method according to claim 3, characterized in that, The calculation of the bending energy of the path formed by the endpoints and intersections of the flexible wire bundle of each material, based on the bending stiffness of the flexible wire bundle of each material and the degree of bending of the flexible wire bundle of each material in different segments, includes: in, Let m be the bending energy of the flexible wire harness made of the m-th material. Let be the bending stiffness of the flexible wire harness made of the m-th material. Where is the Voronoi length, and N is the number of pixels in the skeleton mask of the flexible wire bundle. Let m be the curvature of the flexible wire harness made of the m-th material at the k-th pixel, where m ≥ 2.

5. The method according to claim 4, characterized in that, The method further includes: Calculate the path smoothness for each of the aforementioned paths; Based on the bending energy and path smoothness of each path, the total energy corresponding to each path is calculated to obtain multiple total energies; The path corresponding to the minimum total energy among the multiple total energies is determined as the path of the flexible wire harness.

6. The method according to claim 5, characterized in that, The calculation of the path smoothness for each of the paths includes: in, For path smoothness, For smoothness weighting coefficients, Let represent the material weight of the flexible wire harness of material m, and N represent the number of pixels in the skeleton mask of the flexible wire harness. It is the square of the curvature difference between adjacent pixels.

7. The method according to claim 5, characterized in that, The step of calculating the total energy corresponding to each path based on the bending energy and path smoothness of each path includes: in, For total energy, For bending energy, For path smoothness.

8. The method according to claim 1, characterized in that, The method further includes: A distance transformation is performed on the binary mask to obtain a distance matrix, wherein each element in the distance matrix represents the distance from a pixel to the nearest background pixel in the binary mask; The maximum diameter of the flexible wire harness is determined based on the distance matrix, wherein the maximum diameter of the flexible wire harness is the maximum value of the distance in the distance matrix; Starting from an endpoint, traverse along the connected path of the skeleton mask to determine the traversal length of the endpoint, wherein the traversal length is the connected length of the endpoint and several endpoints. If the traversal length of a certain endpoint is greater than the maximum diameter of the flexible wire bundle, and no other endpoint or intersection is encountered, then the endpoint is determined to be a real endpoint. If the traversal length of a certain endpoint is less than the maximum diameter of the flexible wire bundle, and another endpoint or intersection is encountered, then the endpoint is determined to be a false endpoint. Iterate through all endpoints to remove all dummy endpoints from the skeleton mask.

9. The method according to claim 1, characterized in that, The method further includes: Construct a training set, wherein the training set includes multiple wire harness images, each of which includes labeled flexible wire harnesses; Based on the training set, the YOLO model is trained to obtain the trained YOLO model; The step of extracting a binary mask from the wire harness image includes: Based on the trained YOLO model, extract the binary mask of the wire harness image; The step of converting the binary mask into a skeleton mask includes: The binary mask is converted into a skeleton mask using a skeleton extraction algorithm.

10. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, such that at least one of the processors implements claim 1 9. Any one of the methods described.

Images