Qualification detection method for bundled wire harness and AR equipment
By using AR devices to identify cable tie areas in bundled wire harness images and generate bounding boxes, the problem of low accuracy and efficiency of manual inspection is solved, and automated and efficient inspection is achieved.
Patent Information
- Application Number
- CN202410571701.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-09
- Publication Date
- 2025-11-18
AI Technical Summary
In existing technologies, the qualification inspection of bundled wire harnesses mainly relies on manual judgment, resulting in low inspection accuracy and efficiency.
By using AR devices to automatically identify the main body, groove, and tail area of the cable tie through the color information of the cable tie in the target image, bounding boxes are generated, and the detection results of the bundled wire harness are obtained by combining multi-frame image analysis.
It enables automatic detection of bundled wire harnesses, improving detection accuracy and efficiency while reducing manual intervention.
Smart Images
Figure CN120976086A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of augmented reality, and particularly relates to a qualified detection method of bundled wire harness and an AR device. BACKGROUND
[0002] In the industrial production process, it is inevitable to bundle the wire harness, and the quality of the bundled wire harness needs to be guaranteed. Therefore, the qualified detection of the bundled wire harness is needed. In the prior art, the qualified detection of the bundled wire harness is mainly realized by manual detection. Since the manual detection standard is not uniform and the time consumption is relatively long, the detection accuracy and efficiency are relatively low. SUMMARY
[0003] The present application provides a qualified detection method of bundled wire harness and an AR device, which is used for realizing automatic detection of whether the bundled wire harness is qualified, and does not need manual detection, thereby improving the detection accuracy and efficiency.
[0004] In a first aspect, the present application provides a qualified detection method of bundled wire harness, which is applied to an AR device, and the method comprises the following steps.
[0005] For any one target to-be-detected image containing a bundled wire harness, a first color of a main body region of a cable tie in the target to-be-detected image is used to obtain a plurality of first bounding boxes of the main body region of the cable tie in the target to-be-detected image; wherein the target to-be-detected image includes a wire harness and a cable tie used for bundling the wire harness, and the used cable tie includes a main body region, a slot region and a tail region, the main body region and the tail region are connected through the slot region, the main body region is a cable tie region for bundling the wire harness, the tail region is a cable tie region for not bundling the wire harness, the color of the main body region is the same as that of the tail region, and the color of the main body region is different from that of the slot region; and
[0006] For any one first bounding box, a target bounding box of the cable tie in the target to-be-detected image is obtained based on the first bounding box, and a bundled wire harness image is obtained through the target bounding box; and
[0007] According to the first color and a second color of the slot region, a second bounding box of the tail region in the bundled wire harness image and a third bounding box of the slot region in the bundled wire harness image are obtained.
[0008] According to the second bounding box of the strap tail area in each of the plurality of target images to be detected and the third bounding box of the strap notch area in each of the corresponding plurality of bundled wire harness images, a detection result of the bundled wire harness is obtained.
[0009] The second aspect of the application provides an AR device, comprising a processor and a memory, the processor and the memory are connected through a bus;
[0010] The memory stores a computer program, and the processor is configured to execute the following operations based on the computer program:
[0011] For any one target image to be detected containing a bundled wire harness, a plurality of first bounding boxes of the strap body area in the target image to be detected are obtained using the first color of the strap body area in the target image to be detected; wherein the target image to be detected includes a wire harness and a strap used to bundle the wire harness, and the used strap includes a strap body area, a strap notch area and a strap tail area, the strap body area and the strap tail area are connected through the strap notch area, the strap body area is the strap area for bundling the wire harness, the strap tail area is the strap area for not bundling the wire harness, the color of the strap body area is the same as the color of the strap tail area, and the color of the strap body area is different from the color of the strap notch area; and
[0012] For any one first bounding box, a target bounding box of the strap in the target image to be detected is obtained based on the first bounding box, and a bundled wire harness image is obtained through the target bounding box; and
[0013] According to the first color and the second color of the strap notch area, a second bounding box of the strap tail area in the bundled wire harness image and a third bounding box of the strap notch area in the bundled wire harness image are obtained.
[0014] According to the second bounding box of the strap tail area in each of the plurality of target images to be detected and the third bounding box of the strap notch area in each of the corresponding plurality of bundled wire harness images, a detection result of the bundled wire harness is obtained.
[0015] According to the third aspect of the embodiment of the application, a computer storage medium is provided, the computer storage medium stores a computer program, and the computer program is used to execute the method of the first aspect.
[0016] The target to-be-detected image obtained by using the AR device in the above embodiments of the present application, the first color of the cable tie body region in the target to-be-detected image is used to obtain a plurality of first bounding boxes of the cable tie body region in the target to-be-detected image, for any one first bounding box, a target bounding box of the cable tie in the target to-be-detected image is obtained based on the first bounding box, and a bundled wire harness image is obtained through the target bounding box; and a second color of the cable tie notch region is used to obtain a second bounding box of the cable tie tail region in the bundled wire harness image and a third bounding box of the cable tie notch region in the bundled wire harness image. Finally, according to the second bounding box of the cable tie tail region in each of the plurality of target to-be-detected images and the third bounding box of the cable tie notch region in the corresponding bundled wire harness image of the plurality of target to-be-detected images, a detection result of the bundled wire harness is obtained. Thus, in the embodiments of the present application, it is realized to automatically detect whether the bundled wire harness is qualified, without the need for manual judgment, thereby improving the detection accuracy and detection efficiency. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0018] Figure 1 An exemplary flowchart of a qualified detection method of a bundled wire harness provided by an embodiment of the present application is shown;
[0019] Figure 2 An exemplary schematic diagram of a used cable tie in a target to-be-detected image provided by an embodiment of the present application is shown;
[0020] Figure 3 An exemplary flowchart for determining a first bounding box provided by an embodiment of the present application is shown;
[0021] Figure 4 An exemplary schematic diagram of a target to-be-detected image provided by an embodiment of the present application is shown;
[0022] Figure 5 An exemplary flowchart for determining a second bounding box and a third bounding box provided by an embodiment of the present application is shown;
[0023] Figure 5A An exemplary schematic diagram of a cable tie provided by an embodiment of the present application is shown;
[0024] Figure 6 An exemplary flowchart for determining whether to determine a detection result provided by an embodiment of the present application is shown;
[0025] Figure 7 An exemplary flow diagram for determining a plurality of target detection images is shown.
[0026] Figure 8 An exemplary flow diagram for determining a detection result of a bundled wire harness is shown.
[0027] Figure 9 An exemplary schematic diagram of a qualified detection device for a bundled wire harness is shown.
[0028] Figure 10 An exemplary hardware structure diagram of an AR device is shown. DETAILED DESCRIPTION
[0029] In order to make the objects, implementations and advantages of the present application clearer, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application.
[0030] Based on the exemplary embodiments described in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the claims of the present application. In addition, although the disclosure in the present application is introduced according to one or more examples, it should be understood that each aspect of the disclosure can also constitute a complete embodiment.
[0031] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.
[0032] The terms "first", "second", and the like in the specification of the present application and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to the clearly listed components, but can include other components not clearly listed or inherent to the product or device.
[0033] The term "module" as used herein refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that can perform the functionality associated with that element.
[0034] The idea of the embodiments of the present application is summarized as follows.
[0035] At present, the eligibility of the bundled wire harness is mainly detected by manual detection. Since the manual judgment standards are different and the time is long, the detection accuracy and efficiency are low.
[0036] Based on the problem of low detection accuracy and efficiency caused by manual detection of the eligibility of the bundled wire harness in the prior art, the embodiments of the present application provide a bundled wire harness eligibility detection method. The target to-be-detected image obtained by using an AR device is used to obtain a plurality of first bounding boxes of the strap main body region in the target to-be-detected image by using the first color of the strap main body region in the target to-be-detected image. For any one first bounding box, the target bounding box of the strap in the target to-be-detected image is obtained based on the first bounding box, and the bundled wire harness image is obtained through the target bounding box. And the second color of the strap notch region is obtained according to the first color and the second color of the strap notch region to obtain the second bounding box of the strap tail region in the bundled wire harness image and the third bounding box of the strap notch region in the bundled wire harness image. Finally, the detection result of the bundled wire harness is obtained according to the second bounding box of the strap tail region in each of the plurality of target to-be-detected images and the third bounding box of the strap notch region in the corresponding bundled wire harness image of the plurality of target to-be-detected images. Thus, in the embodiments of the present application, automatic detection of whether the bundled wire harness is eligible is realized, and manual judgment is not required, thereby improving the detection accuracy and efficiency.
[0037] Next, the bundled wire harness eligibility detection method in the embodiments of the present application will be described in detail with reference to the accompanying drawings. As shown in Figure 1 The bundled wire harness eligibility detection method can include the following steps:
[0038] Step 101: For any target image to be detected that contains a bundled wire harness, use the first color of the main body region of the cable tie in the target image to obtain multiple first bounding boxes of the main body region of the cable tie in the target image; wherein, the target image to be detected includes a wire harness and a cable tie used to bundle the wire harness, and the used cable tie includes a main body region, a cable tie slot region, and a cable tie tail region, the main body region and the tail region are connected by the cable tie slot region, the main body region is the cable tie region for bundling the wire harness, the tail region is the cable tie region for not bundling the wire harness, the color of the main body region is the same as the color of the tail region, and different from the color of the cable tie slot region;
[0039] like Figure 2 The image shown is a frontal view of the target image using a back zip tie. Figure 2 As can be seen, the cable tie includes a main body area 201, a slot area 202, and a tail area 203. The color of the main body area is the same as the color of the tail area, but different from the color of the slot area.
[0040] In this embodiment, the main body area and the tail area of the cable tie are set to blue, and the slot area of the cable tie is set to red. However, this example is only for illustration and does not limit the colors of the main body area, slot area, and tail area of the cable tie. The colors of the main body area, slot area, and tail area of the cable tie in this embodiment can be set according to the actual situation.
[0041] The following describes the specific method for determining multiple first bounding boxes of the main cable tie region in the target image to be detected in step 101, such as... Figure 3 The diagram shown illustrates the process of determining the first bounding box, which may include the following steps:
[0042] Step 301: Convert the target image to be detected into the HSV space to obtain the HSV value of each pixel in the target image to be detected;
[0043] The HSV values in this application embodiment include hue, saturation, and lightness. The method for determining HSV values is explained below:
[0044] 1. Hue (H): For any pixel in the target image to be detected, the R-channel value, G-channel value, and B-channel value of the pixel are divided by a first specified value to obtain the normalized R-channel value, normalized G-channel value, and normalized B-channel value of the pixel. Based on the normalized R-channel value, normalized G-channel value, and normalized B-channel value, the intermediate hue of the pixel is obtained. The intermediate hue is multiplied by a second specified value to obtain the hue of the pixel. The intermediate hue can be determined in the following four ways:
[0045] Method 1: If the largest value among the normalized R channel value, normalized G channel value, and normalized B channel value of the pixel is equal to the smallest value, then the intermediate hue of the pixel is determined to be 0.
[0046] Method 2: If the R channel value is the largest among the normalized R channel value, normalized G channel value, and normalized B channel value of the pixel, then the intermediate hue of the pixel can be obtained based on the normalized G channel value and the normalized B channel value. The intermediate hue can be obtained using formula (1):
[0047] H z =((C G -C B ) / (C max -C min ))mod 6......(1);
[0048] Among them, H z C represents the intermediate hue of a pixel. G For the normalized G channel value, C B The normalized B channel value, C max C is the channel value with the largest value among the normalized R channel value, normalized G channel value, and normalized B channel value of the pixel. min It is the smallest value among the normalized R channel value, normalized G channel value, and normalized B channel value of the pixel.
[0049] Method 3: If the value of the normalized R channel value, normalized G channel value, and normalized B channel value of the pixel is the largest, then the intermediate hue of the pixel can be obtained based on the normalized R channel value and the normalized B channel value. The intermediate hue can be obtained through formula (2):
[0050] H z =((C B -C R ) / (Cmax -C min ))+2......(2);
[0051] wherein H z is the intermediate hue of the pixel point, C B is the normalized B channel value, and C R is the normalized R channel value.
[0052] Mode four: if the channel value with the largest value among the normalized R channel value, the normalized G channel value, and the normalized B channel value of the pixel point is the B channel value, then the intermediate hue of the pixel point is obtained based on the normalized R channel value and the normalized G channel value, wherein the intermediate hue can be obtained through formula (3):
[0053] H z = ((C R -C G ) / (C max -C min ))+4......(3);
[0054] wherein H z is the intermediate hue of the pixel point, C G is the normalized G channel value, and C R is the normalized R channel value.
[0055] 2. Saturation (S): for any one pixel point in the target image to be detected, the R channel value, the G channel value, and the B channel value of the pixel point are divided by a first specified value respectively to obtain the normalized R channel value, the normalized G channel value, and the normalized B channel value of the pixel point. If the channel value with the largest value among the normalized R channel value, the normalized G channel value, and the normalized B channel value of the pixel point is 0, then it is determined that the saturation of the pixel point is 0, otherwise, it is determined that the saturation of the pixel point is -1.
[0056] 3. Lightness (V): for any one pixel point in the target image to be detected, the R channel value, the G channel value, and the B channel value of the pixel point are divided by a first specified value respectively to obtain the normalized R channel value, the normalized G channel value, and the normalized B channel value of the pixel point. The channel value with the largest value among the normalized R channel value, the normalized G channel value, and the normalized B channel value of the pixel point is divided by the first specified value to obtain the lightness of the pixel point.
[0057] It should be noted that the first specified value in the embodiment of the present application is 255, and the second specified value is 60. However, the embodiment of the present application does not limit the first specified value and the second specified value, and the specific values of the first specified value and the second specified value in the embodiment of the present application can be set according to actual conditions.
[0058] Step 302: obtaining a plurality of first bounding boxes of the ribbon body region in the target image to be detected by using the HSV value of each pixel point and the HSV range corresponding to the first color.
[0059] In one embodiment, step 302 can be specifically implemented as follows: for any one pixel point, if the HSV value of the pixel point is in the HSV range corresponding to the first color, the gray value of the pixel point is set to 255; if the HSV value of the pixel point is not in the HSV range corresponding to the first color, the gray value of the pixel point is set to 0; based on the gray value of each pixel point, an intermediate target image to be detected is obtained, and the intermediate target image to be detected is input into a preset algorithm to obtain a plurality of first bounding boxes in the target image to be detected.
[0060] The preset algorithm in the embodiment of the present application is the findContours function of opencv, but the embodiment of the present application does not limit the preset algorithm, and the preset algorithm in the embodiment of the present application can be set according to specific actual conditions.
[0061] As shown in FIG. 1, it is a schematic diagram of the target image to be detected, and from the figure, the approximate shape of the ribbon body region and the ribbon tail region can be seen. Figure 4
[0062] The HSV range corresponding to the first color in the embodiment of the present application is pre-set, as described above, the ribbon body region in the embodiment of the present application is blue, so the first color in the embodiment of the present application is blue, and the HSV range of the first color in the embodiment of the present application is [110, 50, 50]-[130, 255, 255]. That is, the range of hue H is [110, 130], the range of saturation S is [50, 255], and the range of lightness V is [50, 255]. However, the embodiment of the present application does not limit the first color and the HSV range thereof, and the specific values can be set according to actual conditions.
[0063] In order to further ensure the accuracy of detection and reduce the amount of calculation, in one embodiment, after step 302 is performed, for any one first bounding box, if the aspect ratio of the first bounding box is not in a first specified range, the first bounding box is deleted, wherein the aspect ratio of the first bounding box is obtained based on the length and width of the first bounding box.
[0064] The aspect ratio of the first bounding box in the embodiment of the present application is obtained by dividing the length of the first bounding box by the width of the first bounding box.
[0065] Secondly, the first specified range in the embodiment of the present application is (3, 15), but the first specified range is not limited, and the first specified range in the embodiment of the present application can be set according to the actual situation.
[0066] Step 102: for any one first bounding box, based on the first bounding box, obtaining a target bounding box of the cable tie in the target image to be detected, and obtaining a bundled wire harness image through the target bounding box;
[0067] In one embodiment, step 102 can be specifically implemented as: magnifying the length and width of the first bounding box by a specified multiple to obtain the target bounding box, and cropping the target image to be detected based on the target bounding box to obtain the bundled wire harness image.
[0068] The specified multiple in the embodiment of the present application is three, but the specified multiple is not limited in the embodiment of the present application, and the specified multiple in the embodiment of the present application can be set according to the actual situation.
[0069] Step 103: obtaining a second bounding box of the cable tie tail area in the bundled wire harness image and a third bounding box of the cable tie notch area in the bundled wire harness image according to the first color and the second color of the cable tie notch area;
[0070] Next, the way of determining the second bounding box and the third bounding box in step 103 is introduced, as shown in the flowchart for determining the second bounding box and the third bounding box. Figure 5 The specific steps can include the following steps:
[0071] Step 501: converting the target image to be detected into HSV space to obtain the HSV value of each pixel point in the target image to be detected;
[0072] The determination of the HSV value of each pixel point in the embodiment of the present application is the same as that in step 301 described above, and the embodiment of the present application will not be repeated here.
[0073] Step 502: using the HSV value of each pixel point and the HSV range corresponding to the first color to obtain a plurality of intermediate bounding boxes of the cable tie specified area in the target image to be detected, wherein the cable tie specified area includes the cable tie main body area and / or the cable tie tail area;
[0074] The specific manner of determining the intermediate surrounding frame in the embodiment of the present application is the same as the manner of determining the first surrounding frame in step 302 described above, and the embodiment of the present application will not be described here again.
[0075] Step 503: obtaining a plurality of third surrounding frames of the cable slot area in the target image to be detected by using the HSV value of each pixel point and the HSV range corresponding to the second color.
[0076] The second color in the embodiment of the present application is red, and the corresponding HSV range is [155, 43, 35]-[180, 255, 255] and [0, 43, 35]-[11, 255, 255], that is, the range of hue is [155, 180] and [0, 11], the range of saturation is [43, 255], and the range of lightness is [35, 255]. However, the second color and the corresponding HSV range in the embodiment of the present application are not limited here, and the second color and the corresponding HSV range in the embodiment of the present application can be set according to actual conditions.
[0077] It should be noted that the execution sequence of step 502 and step 503 in the embodiment of the present application is not limited here, and step 502 can be executed first, and then step 503 can be executed. Step 503 can be executed first, and then step 502 can be executed. Step 502 and step 503 can be executed simultaneously.
[0078] In order to reduce the amount of calculation, in an embodiment, for any one third surrounding frame, if the aspect ratio of the third surrounding frame is not in the second specified range, the third surrounding frame is deleted, wherein the aspect ratio of the third surrounding frame is obtained based on the length and the width of the third surrounding frame.
[0079] The aspect ratio of the third surrounding frame in the embodiment of the present application is obtained by dividing the length of the third surrounding frame by the width of the surrounding frame. The second specified range in the embodiment of the present application can be (1, 2), but the second specified range in the embodiment of the present application is not limited, and the second specified range in the embodiment of the present application can be set according to actual conditions.
[0080] Step 504: obtaining the second surrounding frame according to the plurality of intermediate surrounding frames and the plurality of third surrounding frames.
[0081] In one embodiment, step 504 can be embodied as: for any two of the plurality of intermediate bounding boxes, determining whether there is a target third bounding box intersecting a line connecting center points of the any two intermediate bounding boxes based on the center point position coordinates of the any two intermediate bounding boxes and each third bounding box; if there is, determining the intermediate bounding box with a larger length value of the any two intermediate bounding boxes as the second bounding box.
[0082] The determination of whether there is a target third bounding box intersecting a line connecting center points of the any two intermediate bounding boxes based on the center point position coordinates of the any two intermediate bounding boxes and each third bounding box in the embodiment of the application belongs to a manner in the prior art, and the embodiment of the application will not be described here.
[0083] As shown in FIG. 6, the line connecting the center point O1 of the intermediate bounding box of the main body region of the cable tie and the center point O2 of the intermediate bounding box of the tail region of the cable tie intersects the third bounding box of the cable tie notch region. Figure 5A
[0084] It should be noted that the number of the second bounding box in the embodiment of the application can only be one, and if it is greater than one, step 101 is returned to recalculate.
[0085] In order to avoid malicious detection and ensure the accuracy of the detection result, before step 104 is executed, it is necessary to detect whether the user wearing the AR device exists the head look-around cable tie operation to avoid being maliciously detected. As shown in FIG. 7, it is a specific flowchart, which can specifically include the following steps: Figure 6
[0086] Step 601: traversing each to-be-detected image containing a bundled wire harness, for any to-be-detected image traversed, converting a target point in a device coordinate system to a world coordinate system by using pose data of an AR device corresponding to the to-be-detected image to obtain a world position coordinate of the target point;
[0087] The pose data in the embodiment of the application includes a position and a direction of the AR device.
[0088] In one embodiment, step 601 can be embodied as: obtaining the world position coordinate of the target point according to a position coordinate of the target point in the device coordinate system, the position and the direction of the AR device corresponding to the to-be-detected image. Wherein, the world position coordinate of the target point can be obtained by formula (4):
[0089]
[0090] Wherein, is a matrix corresponding to the world position coordinate of the target point z, is a matrix corresponding to a position of a target point z in a device coordinate system, R is a matrix corresponding to a position of an AR device corresponding to the to-be-detected image, and T is a matrix corresponding to a direction of the AR device corresponding to the to-be-detected image.
[0091] Step 602: Projecting the target point in a specified plane in the world coordinate system based on the world position coordinates of the target point and an origin of the world coordinate system, to obtain a projection vector corresponding to the target point.
[0092] In one embodiment, step 602 can be specifically implemented as follows: obtaining a vector between the target point and the origin in the world coordinate system according to the world position coordinates of the target point and the origin of the world coordinate system; and obtaining the projection vector corresponding to the target point based on the vector between the target point and the origin in the world coordinate system and a normal vector of the specified plane. The projection vector corresponding to the target point can be obtained through formula (5):
[0093]
[0094] wherein, is the projection vector corresponding to the target point, is the vector between the target point and the origin in the world coordinate system, is the normal vector of the specified plane.
[0095] The specified plane in the embodiment of the present application is the xoz plane. However, the specified plane in the embodiment of the present application is not limited, and the specified plane in the embodiment of the present application can be set according to actual conditions.
[0096] Step 603: Obtaining a vector angle of the target point based on the projection vector corresponding to the target point and a projection vector of a target point corresponding to pose data of the first frame of to-be-detected images. The vector angle of the target point can be obtained through formula (6):
[0097]
[0098] wherein, a is the vector angle of the target point, is the projection vector of the target point corresponding to the pose data of the first frame of to-be-detected images.
[0099] Step 604: Determining whether the to-be-detected image and the multiple frames of continuous to-be-detected images after the to-be-detected image are the multiple target to-be-detected images based on the vector angle of the target point corresponding to the to-be-detected image and the vector angles of the target points corresponding to the multiple frames of continuous to-be-detected images after the to-be-detected image. If yes, step 605 is executed, and if no, step 601 is returned.
[0100] As Figure 7 shown, a flowchart for determining a plurality of target detection images is shown, which can specifically include the following steps:
[0101] Step 701: Based on the vector angle of the target point corresponding to the to-be-detected image and the vector angles of the target points corresponding to the plurality of continuous to-be-detected images after the to-be-detected image, a to-be-detected vector angle sequence is obtained.
[0102] In an embodiment, step 701 can be specifically implemented as: adding the vector angle of the target point corresponding to the to-be-detected image and the vector angles of the target points corresponding to the plurality of continuous to-be-detected images after the to-be-detected image in sequence into the same sequence to obtain the to-be-detected vector angle sequence.
[0103] For example, the to-be-detected vector angle sequence {a1, a2, a3,... an} is obtained, where a1 is the vector angle of the target point corresponding to the to-be-detected image, and an is the vector angle of the target point corresponding to the n-1 frame to-be-detected image after the to-be-detected image. n n
[0104] Step 702: According to the to-be-detected vector angle sequence and a preset template vector angle sequence, a similarity between the to-be-detected vector angle sequence and the template vector angle sequence is obtained; wherein the similarity between the to-be-detected vector angle sequence and the template vector angle sequence can be obtained by formula (7):
[0105]
[0106] Wherein Y is the similarity between the to-be-detected vector angle sequence and the template vector angle sequence, S n is the to-be-detected vector angle sequence, and S T is the preset template vector angle sequence.
[0107] Step 703: If the similarity is greater than a specified similarity, the to-be-detected image and the plurality of continuous to-be-detected images after the to-be-detected image are determined as the plurality of target to-be-detected images.
[0108] The specified similarity in the embodiments of the present application can be set according to specific actual situations, and the present application does not limit the specified similarity.
[0109] Step 104 (step 605): According to the respective second bounding boxes of the cable tie tail region in the plurality of target to-be-detected images and the respective third bounding boxes of the cable tie notch region in the plurality of target to-be-detected images, a detection result of the bundled wire harness is obtained.
[0110] As Figure 8 described, the flowchart for determining the detection result of the bundled harness can specifically include the following steps:
[0111] Step 801: For any one of the target detection images in the plurality of target detection images, the actual length of the cable tail region in the any one of the target detection images is obtained according to the length of the second bounding box of the cable tail region corresponding to the target detection image, the length of the third bounding box of the cable notch region in the bundled harness image, and the actual length of the cable notch region preset; wherein the actual length of the cable tail region in the any one of the target detection images is obtained by formula (8):
[0112]
[0113] Wherein, L w is the actual length of the cable tail region in the any one of the target detection images, d w is the length of the second bounding box of the cable tail region corresponding to the any one of the target detection images, d c is the length of the third bounding box of the cable notch region in the any one of the bundled harness images, and L c is the actual length of the cable notch region.
[0114] Step 802: The maximum actual length in each cable tail region in the plurality of target detection images is determined as the target actual length of the cable tail region;
[0115] Step 803: It is judged whether the target actual length is within the third specified range, if yes, step 804 is executed, if not, step 805 is executed;
[0116] The third specified range in the embodiment of the application can be set according to actual conditions, and the third specified range is not limited herein.
[0117] Step 804: The detection result of the bundled harness is determined as qualified;
[0118] Step 805: The detection result of the bundled harness is determined as unqualified.
[0119] Based on the same inventive concept, the bundled harness qualified detection method as described above can also be realized by a bundled harness qualified detection device. The effect of the bundled harness qualified detection device is similar to that of the foregoing method, and will not be repeated here.
[0120] Figure 9A structural schematic diagram of a qualified detection device for a bundled harness according to one embodiment of the present disclosure.
[0121] As shown in Figure 9 The qualified detection device for a bundled harness 900 of the present disclosure can include a first positioning module 910, an image processing module 920, a second positioning module 930, and a detection module 940.
[0122] The first positioning module 910 is configured to, for any one target to-be-detected image containing a bundled harness, obtain a plurality of first bounding boxes of a strap main body region in the target to-be-detected image by using a first color of the strap main body region in the target to-be-detected image; wherein the target to-be-detected image includes a harness and a strap used to bundle the harness, and the used strap includes a strap main body region, a strap notch region, and a strap tail region, the strap main body region and the strap tail region are connected through the strap notch region, the strap main body region is a strap region for bundling the harness, the strap tail region is a strap region not for bundling the harness, the color of the strap main body region is the same as that of the strap tail region, and different from that of the strap notch region; and
[0123] The image processing module 920 is configured to, for any one first bounding box, obtain a target bounding box of the strap in the target to-be-detected image based on the first bounding box, and obtain a bundled harness image through the target bounding box; and
[0124] The second positioning module 930 is configured to obtain a second bounding box of the strap tail region in the bundled harness image and a third bounding box of the strap notch region in the bundled harness image according to the first color and a second color of the strap notch region.
[0125] The detection module 940 is configured to obtain a detection result of the bundled harness according to the second bounding box of the strap tail region in each of a plurality of target to-be-detected images and the third bounding box of the strap notch region in the corresponding bundled harness image of each of the plurality of target to-be-detected images.
[0126] In one embodiment, the first positioning module 910 is specifically configured to:
[0127] convert the target to-be-detected image into an HSV space to obtain HSV values of each pixel point in the target to-be-detected image; and obtain the plurality of first bounding boxes of the strap main body region in the target to-be-detected image by using the HSV values of the each pixel point and an HSV range corresponding to the first color;
[0128] The second positioning module 930 is specifically configured to:
[0129] convert the target image to be detected into an HSV space to obtain HSV values of each pixel point in the target image to be detected;
[0130] obtain a plurality of intermediate bounding boxes of a specified area of the cable tie in the target image to be detected by using the HSV values of each pixel point and an HSV range corresponding to the first color, wherein the specified area of the cable tie includes the main body area of the cable tie and / or the tail area of the cable tie; and
[0131] obtain a plurality of third bounding boxes of the cable tie notch area in the target image to be detected by using the HSV values of each pixel point and an HSV range corresponding to the second color;
[0132] obtain the second bounding box according to the plurality of intermediate bounding boxes and the plurality of third bounding boxes.
[0133] In an embodiment, the device further comprises:
[0134] the first filtering module 950 is configured to, after obtaining the plurality of first bounding boxes of the main body area of the cable tie in the target image to be detected by using the HSV values of each pixel point and the HSV range corresponding to the first color, delete any first bounding box if an aspect ratio of the first bounding box is not within a first specified range, wherein the aspect ratio of the first bounding box is obtained based on a length and a width of the first bounding box;
[0135] the second filtering module 960 is configured to, after obtaining the plurality of third bounding boxes of the cable tie notch area in the target image to be detected by using the HSV values of each pixel point and the HSV range corresponding to the second color, delete any third bounding box if an aspect ratio of the third bounding box is not within a second specified range, wherein the aspect ratio of the third bounding box is obtained based on a length and a width of the third bounding box.
[0136] In an embodiment, the second positioning module 930 is further configured to:
[0137] for any two intermediate bounding boxes in the plurality of intermediate bounding boxes, determine whether there is a target third bounding box intersecting a line connecting the center points of the any two intermediate bounding boxes based on the center point position coordinates of the any two intermediate bounding boxes and each third bounding box;
[0138] if there is, determine the intermediate bounding box with a larger length value in the any two intermediate bounding boxes as the second bounding box.
[0139] In an embodiment, the device further comprises:
[0140] a coordinate conversion module 970, configured to, before the detection result of the bundled wire harness is obtained according to the second bounding box of the cable tail area in each of the plurality of target images to be detected and the third bounding box of the cable notch area in each of the plurality of corresponding images of the bundled wire harness, traverse each image to be detected containing the bundled wire harness, and for any one of the images to be detected traversed, convert a target point in a device coordinate system to a world coordinate system by using pose data of an AR device corresponding to the image to be detected, to obtain a world position coordinate of the target point;
[0141] a projection module 980, configured to project the target point in a specified plane in the world coordinate system based on the world position coordinate of the target point and an origin of the world coordinate system, to obtain a projection vector corresponding to the target point; and
[0142] a vector angle determination module 990, configured to obtain a vector angle of the target point according to the projection vector corresponding to the target point and a projection vector of a target point corresponding to the pose data of the first image to be detected;
[0143] a judgment module 991, configured to determine whether the image to be detected and a plurality of continuous images to be detected after the image to be detected are the plurality of target images to be detected based on the vector angle of the target point corresponding to the image to be detected and vector angles of target points corresponding to a plurality of continuous images to be detected after the image to be detected;
[0144] if yes, performing the step of obtaining the detection result of the bundled wire harness according to the second bounding box of the cable tail area in each of the plurality of target images to be detected and the third bounding box of the cable notch area in each of the plurality of corresponding images of the bundled wire harness;
[0145] if no, returning to the step of traversing each image to be detected containing the bundled wire harness, until the plurality of target images to be detected are determined, and then ending.
[0146] In an embodiment, the pose data includes a position and a direction of the AR device; and the coordinate conversion module 970 is specifically configured to:
[0147] obtain the world position coordinate of the target point according to a position coordinate of the target point in the device coordinate system, the position and the direction of the AR device corresponding to the image to be detected;
[0148] projecting the target point in a specified plane in the world coordinate system based on the world position coordinates of the target point and the origin of the world coordinate system, to obtain a projection vector corresponding to the target point, comprising:
[0149] obtaining a vector between the target point and the origin in the world coordinate system according to the world position coordinates of the target point and the origin of the world coordinate system; and obtaining a projection vector corresponding to the target point based on the vector between the target point and the origin in the world coordinate system and a normal vector of the specified plane.
[0150] In an embodiment, the judgment module 991 is specifically configured to:
[0151] obtaining a sequence of detection vectors based on the vector angle of the target point corresponding to the to-be-detected image and the vector angles of the target points corresponding to a plurality of continuous to-be-detected images located after the to-be-detected image;
[0152] obtaining a similarity between the sequence of detection vectors and a preset sequence of template vector angles according to the sequence of detection vectors and the preset sequence of template vector angles;
[0153] if the similarity is greater than a specified similarity, determining the to-be-detected image and the plurality of continuous to-be-detected images located after the to-be-detected image as the plurality of target to-be-detected images.
[0154] In an embodiment, the detection module 940 is specifically configured to:
[0155] obtaining an actual length of the cable tail region in the arbitrary target to-be-detected image according to the length of the second bounding box of the cable tail region corresponding to the target to-be-detected image, the length of the third bounding box of the cable notch region in the bundled wire harness image, and a preset actual length of the cable notch region;
[0156] determining a maximum actual length of the cable tail region in the plurality of target to-be-detected images as a target actual length of the cable tail region;
[0157] if the target actual length is within a third specified range, determining the detection result of the bundled wire harness as qualified;
[0158] if the target actual length is not within the third specified range, determining the detection result of the bundled wire harness as unqualified.
[0159] In an embodiment, the detection module 940 is further configured to:
[0160] The actual length of the cable tie tail region in the arbitrary frame of the target image to be detected is obtained by the following formula:
[0161]
[0162] Ltail= Lbox2- Lbox1 w Ltailis the actual length of the cable tie tail region in the arbitrary frame of the target image to be detected, d w Lbox2is the length of the second bounding box of the cable tie tail region corresponding to the arbitrary frame of the target image to be detected, d c Lbox3is the length of the third bounding box of the cable tie notch region in the arbitrary frame of the cable harness image, and L c Lnotch is the actual length of the cable tie notch region.
[0163] After introducing the method and device for qualified detection of a cable harness according to an exemplary embodiment of the present application, next, the AR device according to another exemplary embodiment of the present application is introduced.
[0164] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.
[0165] In some possible embodiments, the AR device according to the present application can at least include at least one processor and at least one computer storage medium. The computer storage medium stores program codes, which, when executed by the processor, cause the processor to perform the steps in the method for qualified detection of a cable harness according to various exemplary embodiments of the present application described above in the specification. For example, the processor can perform steps 101-104 as shown in Figure 1
[0166] The AR device 1000 according to this embodiment of the present application is described below with reference to Figure 10 Figure 10 The AR device 1000 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.
[0167] As shown in Figure 10 , the AR device 1000 is in the form of a general AR device. The components of the AR device 1000 can include but are not limited to the above-mentioned at least one processor 1001, the above-mentioned at least one computer storage medium 1002, and a bus 1003 connecting different system components including the computer storage medium 1002 and the processor 1001.
[0168] Bus 1003 represents one or more of several possible bus structures, including a computer storage bus or computer storage bus controller, a peripheral bus, a processor bus, or a local bus using any of a variety of bus architectures.
[0169] Computer storage media 1002 can include read-only computer storage media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1021 and / or cache memory computer storage media 1022, and further can include read-only computer storage media (ROM) 1023.
[0170] Computer storage media 1002 can also include program / utility 1025 having a set of programs / modules 1024, including but not limited to, operating systems, one or more application programs, other program modules, and program data, each of which can implement aspects of a network environment, and each or a combination thereof, for example.
[0171] AR device 1000 can also communicate with one or more external devices 1004 such as a keyboard, a pointing device, etc.; and a device or devices that enable a user to interact with AR device 1000; and / or any devices (e.g., routers, modems, etc.) that enable AR device 1000 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 1005. Still yet, AR device 1000 can communicate with one or more networks, such as a local area network (LAN), a general area network (WAN), and / or the Internet, through network adapter 1006. As depicted, network adapter 1006 communicates with the other components of AR device 1000 via bus 1003. It should be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with AR device 1000. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.
[0172] In some possible embodiments, each of the aspects of the method for qualifying a bundled wire harness provided by the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the method for qualifying a bundled wire harness according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device.
[0173] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.
Claims
1. A method for testing the quality of bundled wire harnesses, characterized in that, When applied to AR devices, the method includes: For any target image to be detected that contains a bundled wire harness, multiple first bounding boxes of the main body region of the cable tie in the target image are obtained using a first color of the main body region of the cable tie in the target image. The target image includes a wire harness and cable ties used to bundle the wire harness. The used cable tie includes a main body region, a slot region, and a tail region. The main body region and the tail region are connected by the slot region. The main body region is the area where the wire harness is bundled, and the tail region is the area where the wire harness is not bundled. The color of the main body region is the same as the color of the tail region, but different from the color of the slot region. For any given first bounding box, based on the first bounding box, a target bounding box of the cable tie in the target image to be detected is obtained, and an image of the cable bundle is obtained through the target bounding box; and, Based on the first color and the second color of the cable tie slot area, a second bounding box of the cable tie tail area in the bundled wire harness image and a third bounding box of the cable tie slot area in the bundled wire harness image are obtained. The detection result of the bundled wire harness is obtained based on the second bounding boxes of the cable tie tail region in each of the multi-frame target images to be detected and the third bounding boxes of the cable tie slot region in each of the bundled wire harness images corresponding to the multi-frame target images to be detected.
2. The method according to claim 1, characterized in that, For any target image containing a cable tie, the method utilizes the first color of the main region of the cable tie in the target image to obtain multiple first bounding boxes of the main region of the cable tie in the target image, including: The target image to be detected is converted to HSV space to obtain the HSV value of each pixel in the target image to be detected; using the HSV value of each pixel and the HSV range corresponding to the first color, multiple first bounding boxes of the cable tie main area in the target image to be detected are obtained. The step of obtaining a second bounding box for the cable tie tail region and a third bounding box for the cable tie slot region in the bundled wire harness image based on the first color and the second color of the cable tie slot region includes: The target image to be detected is converted to the HSV space to obtain the HSV value of each pixel in the target image to be detected; Using the HSV values of each pixel and the HSV range corresponding to the first color, multiple intermediate bounding boxes are obtained for the designated area of the cable tie in the target image to be detected, wherein the designated area of the cable tie includes the main body area of the cable tie and / or the tail area of the cable tie; and, Using the HSV values of each pixel and the HSV range corresponding to the second color, multiple third bounding boxes of the cable tie slot area in the target image to be detected are obtained; The second bounding box is obtained based on the plurality of intermediate bounding boxes and the plurality of third bounding boxes.
3. The method according to claim 2, characterized in that, The method further includes obtaining the main area of the cable tie after multiple first bounding boxes in the target image by utilizing the HSV values of each pixel and the HSV range corresponding to the first color. For any given first bounding box, if the aspect ratio of the first bounding box is not within a first specified range, then the first bounding box is deleted, wherein the aspect ratio of the first bounding box is obtained based on the length and width of the first bounding box; The method further includes obtaining the cable tie slot region after multiple third bounding boxes in the target image by utilizing the HSV values of each pixel and the HSV range corresponding to the second color. For any third bounding box, if the aspect ratio of the third bounding box is not within the second specified range, then the third bounding box is deleted, wherein the aspect ratio of the third bounding box is obtained based on the length and width of the third bounding box.
4. The method according to claim 2, characterized in that, The step of obtaining the second bounding box based on the plurality of intermediate bounding boxes and the plurality of third bounding boxes includes: For any two intermediate bounding boxes among the plurality of intermediate bounding boxes, based on the center point coordinates of the two intermediate bounding boxes and each third bounding box, determine whether there is a target third bounding box that intersects with the line connecting the center points of the two intermediate bounding boxes. If it exists, the middle bounding box with the larger length value among any two middle bounding boxes is determined as the second bounding box.
5. The method according to claim 1, characterized in that, Before obtaining the detection result of the bundled wire harness based on the second bounding boxes of the cable tie tail region in each of the multiple frames of target images to be detected and the third bounding boxes of the cable tie slot region in each of the corresponding bundled wire harness images in the multiple frames of target images to be detected, the method further includes: The process iterates through each image to be detected, including the bundled wires. For any image encountered during the iteration, using the pose data of the AR device corresponding to that image, the target point in the device coordinate system is transformed to the world coordinate system to obtain the world position coordinates of the target point; and... Based on the world position coordinates of the target point and the origin of the world coordinate system, the target point is projected onto a specified plane in the world coordinate system to obtain the projection vector corresponding to the target point; and, The vector angle of the target point is obtained based on the projection vector corresponding to the target point and the projection vector of the target point corresponding to the pose data of the first frame of the image to be detected. Based on the vector angle of the target point corresponding to the image to be detected and the vector angle of the target point corresponding to multiple consecutive images to be detected after the image to be detected, it is determined whether the image to be detected and the multiple consecutive images to be detected after the image to be detected are the multiple target images to be detected. If so, then execute the step of obtaining the detection result of the bundled wires based on the second bounding boxes of the cable tie tail region in each of the multi-frame target images to be detected and the third bounding boxes of the cable tie slot region in each of the bundled wire images corresponding to the multi-frame target images to be detected. If not, return to the step of traversing each image to be detected containing the bundled wires until the target image to be detected in the multiple frames is determined, then end.
6. The method according to claim 5, characterized in that, The pose data includes the position and orientation of the AR device; The step of using the pose data of the AR device corresponding to the image to be detected to transform the target point in the device coordinate system to the world coordinate system, and obtaining the world position coordinates of the target point, includes: The world position coordinates of the target point are obtained based on the position coordinates of the target point in the device coordinate system and the position and orientation of the AR device corresponding to the image to be detected. The projection of the target point onto a specified plane in the world coordinate system, based on the world position coordinates of the target point and the origin of the world coordinate system, to obtain the projection vector corresponding to the target point, includes: Based on the world position coordinates of the target point and the origin of the world coordinate system, the vector between the target point and the origin in the world coordinate system is obtained; based on the vector between the target point and the origin in the world coordinate system and the normal vector of the specified plane, the projection vector corresponding to the target point is obtained.
7. The method according to claim 5, characterized in that, The step of determining whether the image to be detected and the multiple consecutive frames of images to be detected are the target images to be detected based on the vector angle of the target point corresponding to the image to be detected and the vector angle of the target point corresponding to the multiple consecutive frames of images to be detected after the image to be detected includes: Based on the vector angles of the target points corresponding to the image to be detected and the vector angles of the target points corresponding to multiple consecutive frames of images to be detected after the image to be detected, a sequence of vector angles to be detected is obtained. Based on the angle sequence of the vector to be detected and the preset angle sequence of the template vector, the similarity between the angle sequence of the vector to be detected and the angle sequence of the template vector is obtained; If the similarity is greater than a specified similarity, then the image to be detected and multiple consecutive images to be detected following the image to be detected are determined as the multiple target images to be detected.
8. The method according to claim 1, characterized in that, The step of obtaining the detection result of the bundled wire harness based on the second bounding boxes of the cable tie tail region in each of the multiple frames of target images to be detected and the third bounding boxes of the cable tie slot region in each of the corresponding bundled wire harness images in the multiple frames of target images to be detected includes: For any one of the multiple target images to be detected, the actual length of the cable tie tail region in the target image to be detected is obtained based on the length of the second bounding box of the cable tie tail region corresponding to the target image to be detected, the length of the third bounding box of the cable tie slot region in the bundled wire harness image, and the preset actual length of the cable tie slot region. The actual length of the largest value in each cable tie tail region in the multi-frame target detection image is determined as the target actual length of the cable tie tail region; If the actual length of the target is within the third specified range, then the test result of the bundled wire is determined to be qualified; If the actual length of the target is not within the third specified range, then the test result of the bundled wire is determined to be unqualified.
9. The method according to claim 8, characterized in that, The step of obtaining the actual length of the cable tie tail region in any one frame of the multi-frame target detection image, based on the length of the second bounding box of the cable tie tail region corresponding to the target detection image, the length of the third bounding box of the cable tie slot region in the bundled wire harness image, and the preset actual length of the cable tie slot region, includes: The actual length of the cable tie tail region in any frame of the target image to be detected can be obtained using the following formula: Among them, L w Let d be the actual length of the cable tie tail region in any frame of the target image to be detected. w d is the length of the second bounding box of the cable tie tail region corresponding to any frame of the target image to be detected. c L is the length of the third bounding box of the cable tie slot area in any frame of the bundled wire image. c This refers to the actual length of the cable tie slot area.
10. An AR device, characterized in that, It includes a processor and a memory, which are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: For any target image to be detected that contains a bundled wire harness, multiple first bounding boxes of the main body region of the cable tie in the target image are obtained using a first color of the main body region of the cable tie in the target image. The target image includes a wire harness and cable ties used to bundle the wire harness. The used cable tie includes a main body region, a slot region, and a tail region. The main body region and the tail region are connected by the slot region. The main body region is the area where the wire harness is bundled, and the tail region is the area where the wire harness is not bundled. The color of the main body region is the same as the color of the tail region, but different from the color of the slot region. For any given first bounding box, based on the first bounding box, a target bounding box of the cable tie in the target image to be detected is obtained, and an image of the cable bundle is obtained through the target bounding box; and, Based on the first color and the second color of the cable tie slot area, a second bounding box of the cable tie tail area in the bundled wire harness image and a third bounding box of the cable tie slot area in the bundled wire harness image are obtained. The detection result of the bundled wire harness is obtained based on the second bounding boxes of the cable tie tail region in each of the multi-frame target images to be detected and the third bounding boxes of the cable tie slot region in each of the bundled wire harness images corresponding to the multi-frame target images to be detected.