Method and apparatus for confirming defect by using image data

Abstract

The present invention relates to a method and a device for confirming a defect by using image data, the method comprising: a step in which an electronic device displays a plurality of image data for a product in a first area; a step in which the electronic device enlarges one of the image data displayed in the first area and displays same as target image data in a second area; and a step in which the electronic device displays a bounding box at a location of at least one defect included in the target image data. Other embodiments are possible.

Description

Method and device for identifying defects using image data

[0001] The present invention relates to a method and device for identifying defects using image data.

[0002] Recently, with the advancement of AI (artificial intelligence) technology, interest in smart factories that operate digital automation systems by combining digital data and ICT (information and communications technology) in the production process, including product design, development, and manufacturing, is increasing.

[0003] When a smart factory like this is applied to the product manufacturing process, the deployment of workers can be minimized, which reduces labor costs. In addition, multiple processes for product manufacturing can be performed continuously, which reduces product production speed and production costs.

[0004] To detect external defects in products manufactured in smart factories, vision inspection systems utilizing artificial intelligence are recently being adopted. These vision inspection systems assess the normality and abnormality of image data of a product, prioritizing the detection of defects.

[0005] When a vision inspection system determines a product's defects based on image data, an inspector directly identifies the location and type of defect in the image data identified as defective, generating labeling data. The vision inspection system then stores this labeling data in a database, continuously training the system and enhancing it.

[0006] However, even with these vision inspection systems, inspectors must manually generate labeling data by using image data to identify the location and type of defects. This wastes inspectors' manpower and time, requiring them to manually identify defects and generate labeling data.

[0007] Therefore, there is a need to develop a technology that can identify defects in the appearance of a product and label them more quickly and easily.

[0008] Embodiments of the present invention for solving these conventional problems provide a method and device for confirming defects using image data that can provide a user interface for more easily labeling defects confirmed in a plurality of image data of the appearance of a product.

[0009] In addition, embodiments of the present invention for solving conventional problems provide a method and device for confirming defects using image data, which automatically displays a bounding box at a location where a defect is confirmed in a plurality of image data, displays the defect type of the corresponding defect in a list box, and performs labeling by selecting the defect type of the corresponding defect from the list box.

[0010] In addition, embodiments of the present invention for solving conventional problems provide a method and device for confirming defects using image data, which enable an inspector to modify the location, size, type of defect, etc. of a bounding box automatically displayed at a location where a defect is confirmed in a plurality of image data.

[0011] A method for confirming a defect using image data according to an embodiment of the present invention is characterized by including a step in which an electronic device displays a plurality of image data for a product in a first area, a step in which the electronic device enlarges one of the image data displayed in the first area and displays it as target image data in a second area, and a step in which the electronic device displays a bounding box at the location of at least one defect included in the target image data.

[0012] In addition, the step of displaying as target image data is characterized by being a step of displaying as target image data image data selected from among the image data displayed in the first area or image data confirmed to be defective through an artificial intelligence algorithm.

[0013] In addition, the step of displaying the bounding box is characterized by being a step of displaying the bounding box by adding it to the target image data according to an additional signal of the bounding box.

[0014] In addition, the step of displaying the bounding box is characterized by being a step of displaying the bounding box at a location related to the defect identified in the target image data through the artificial intelligence algorithm.

[0015] In addition, after the step of displaying the bounding box, the electronic device is characterized in that it further includes a step of deleting or changing the bounding box displayed in the target image data.

[0016] In addition, after the step of displaying the bounding box, the electronic device is characterized by further including a step of applying the target image data and the addition, deletion, and change history of the bounding box to the artificial intelligence algorithm.

[0017] In addition, after the step of displaying the bounding box, the electronic device is characterized by further including a step of setting at least one of the properties and the position of the bounding box according to a setting signal for setting at least one of the properties including the size, shape and color of the bounding box and the position of the bounding box.

[0018] In addition, the step of setting at least one of the properties and the location of the bounding box is characterized by including the step of the electronic device checking at least one type of defect that may occur at the location of the defect based on previously stored labeling data related to the defect indicated by the bounding box, and the step of displaying a list box sorted by the at least one type of defect by overlaying it on the target image data.

[0019] In addition, the step of applying to the artificial intelligence algorithm is characterized in that the electronic device applies the updated labeling data by mapping it with the defective type selected from the list box to the artificial intelligence algorithm.

[0020] In addition, a device for confirming defects using image data according to an embodiment of the present invention is characterized by including a display unit that displays a main screen for confirming defects in a product, and a control unit that divides the main screen into a first area and a second area, displays a plurality of image data in the first area, enlarges one of the plurality of image data and displays it as target image data in the second area, and controls the display unit to display a bounding box at the location of at least one defect included in the target image data.

[0021] In addition, the control unit is characterized in that it displays the image data selected from among the image data displayed in the first area or the image data identified as defective through an artificial intelligence algorithm as the target image data.

[0022] In addition, the control unit is characterized in that it adds the bounding box to the target image data and displays it according to an additional signal of the bounding box.

[0023] In addition, the control unit is characterized in that it displays the bounding box at a location related to the defect identified in the target image data through the artificial intelligence algorithm.

[0024] In addition, the control unit is characterized by deleting or changing the bounding box displayed in the target image data.

[0025] In addition, the control unit is characterized in that it applies the addition, deletion, and change history of the target image data and the bounding box to the artificial intelligence algorithm.

[0026] In addition, the control unit is characterized in that it sets at least one of the properties and the position of the bounding box according to a setting signal for setting at least one of the properties including the size, shape and color of the bounding box and the position of the bounding box.

[0027] In addition, the control unit is characterized in that it overlays the target image data and displays a list box that sorts at least one type of defect that can occur at the location of the defect based on pre-stored labeling data related to the defect indicated by the bounding box.

[0028] In addition, the control unit is characterized in that it applies the updated labeling data by mapping it with the defective type selected from the list box to the artificial intelligence algorithm.

[0029] As described above, the method and device for confirming defects using image data according to the present invention have the effect of enabling faster and easier labeling by providing a user interface that can display and label defects confirmed in a plurality of image data regarding the appearance of a product.

[0030] In addition, the method and device for confirming defects using image data according to the present invention automatically displays a bounding box at a location where a defect is confirmed in a plurality of image data, displays the defect type of the corresponding defect in a list box, and performs labeling by selecting the defect type of the corresponding defect from the list box, thereby having the effect of minimizing waste of manpower and time for labeling.

[0031] In addition, the method and device for confirming defects using image data according to the present invention have the effect of enabling the inspector to more easily create labeling data by allowing the inspector to modify the location, size, type of defect, etc. of the bounding box automatically displayed at the location where a defect is confirmed in a plurality of image data.

[0032] FIG. 1 is a drawing showing the main configuration of an electronic device for checking defects using image data according to an embodiment of the present invention.

[0033] FIG. 2 is a flowchart illustrating a method for identifying defects using image data according to an embodiment of the present invention.

[0034] FIG. 3 is a detailed flowchart illustrating a method for updating labeling data based on a defect type identified in image data according to an embodiment of the present invention.

[0035] FIG. 4 is a screen example diagram for explaining the direction in which image data for a product according to an embodiment of the present invention is acquired.

[0036] FIG. 5 is a screen example of an interface for checking for defects in image data according to an embodiment of the present invention.

[0037] FIG. 6 is a screen example showing the shape of a bounding box according to a defect that can be confirmed in image data for one direction of a product according to an embodiment of the present invention.

[0038] FIG. 7 and FIG. 8 are screen examples showing defects identified in the appearance of a product according to one embodiment of the present invention.

[0039] FIGS. 9 to 14 are screen examples for explaining a method for checking for defects in the appearance of a product according to another embodiment of the present invention.

[0040] FIG. 15 is a screen example diagram illustrating a method for checking for defects in the appearance of a product according to another embodiment of the present invention.

[0041] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. In the drawings, portions irrelevant to the description may be omitted for clarity in describing the present invention, and the same reference numerals may be used throughout the specification for identical or similar components.

[0042]

[0043] FIG. 1 is a drawing showing the main configuration of an electronic device for checking defects using image data according to an embodiment of the present invention.

[0044] Referring to FIG. 1, an electronic device (100) according to the present invention may include a communication unit (110), an input unit (120), a camera (130), a display unit (130), a memory (140), and a control unit (150). In addition, the electronic device (100) according to the present invention may refer to a vision inspection system capable of controlling operations for a process of generating labeling data by detecting defects through analysis of image data among processes included in a smart factory.

[0045] The communication unit (110) can perform communication with a device placed outside the electronic device (100), for example, an electronic device (not shown) that monitors and controls the entire smart factory. In addition, the communication unit (110) can perform communication with a robot arm (not shown) equipped with a camera (not shown), receive image data acquired from the camera, and provide the same to the control unit (150). To this end, the communication unit (110) can be configured to receive image data acquired from the camera using 5G (5 th It can perform wireless communication such as LTE (long term evolution), Wi-Fi (wireless fidelity), etc., or serial modbus communication such as RS232, RS422, RS485, etc.

[0046] The input unit (120) generates input data in response to an administrator input of the electronic device (100). To this end, the input unit (120) may include at least one input means among a keyboard, a mouse, a keypad, a dome switch, a touch panel, a touch key, and a button.

[0047] The display unit (130) outputs output data according to the operation of the electronic device (100). To this end, the display unit (130) may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, and an electronic paper display. The display unit (130) may be implemented as a touch screen by being combined with the input unit (120).

[0048] The memory (140) stores the operating programs of the electronic device (100). The memory (140) can store an artificial intelligence algorithm that is trained through machine learning and can detect defects in image data. The memory (140) can create and store labeling data related to the defects detected through the artificial intelligence algorithm in a database. In this case, the labeling data may be data related to the types of defects that can be identified on the surface of a product being produced in a smart factory, such as a circuit breaker.

[0049] The control unit (150) receives a plurality of image data acquired from the camera through the communication unit (110). When a confirmation signal for confirming a defect occurring in a product is received from the input unit (120), the control unit (150) executes a program capable of confirming a defect in the image data and generating labeling data. The control unit (150) can arrange the plurality of image data provided from the communication unit (110) in a first area of ​​the main screen of the program. The control unit (150) can enlarge and display the target image data in which a defect has been confirmed based on the analysis results of the plurality of image data in a second area. In addition, the control unit (150) can confirm the image data corresponding to the selection signal received from the input unit (120) as the target image data and enlarge and display it in the second area.

[0050] The control unit (150) displays defects contained in the target image data displayed in the second region. More specifically, the control unit (150) uses an artificial intelligence algorithm to overlay a bounding box on the location where a defect is identified in the target image data displayed in an enlarged manner in the second region. At this time, the bounding box may be displayed by reflecting attributes including size, shape, and color set in response to the defect identified in the target image data. The bounding box may be displayed in various shapes, such as a square, circle, or triangle.

[0051] The control unit (150) can divide the target image data into a plurality of virtual areas and sequentially analyze the plurality of areas according to a preset order to check for defects. For example, the control unit (150) can divide the target image data into a total of 12 areas each of 4 horizontal x 3 vertical areas, and analyze the target image data in the order of (1,1) to (1,4), (2,1) to (2,4), and (3,1) to (3,4) to check for defects. At this time, the control unit (150) can analyze only the areas where defects are likely to be detected among the 12 areas, thereby having the effect of minimizing the time consumed when analyzing the target image data.

[0052] In addition, when the control unit (150) receives an additional signal for a bounding box for the target image data from the input unit (120), it can display a bounding box at the location where the additional signal is generated. The additional signal is a signal for adding a bounding box for defects not identified by the artificial intelligence algorithm as a result of an inspector visually inspecting the target image data.

[0053] When the control unit (150) receives a change signal for a bounding box displayed as an overlay on the target image data from the input unit (120), it changes at least one of the position and properties of the bounding box according to the change signal. The control unit (150) receives a signal regarding whether there is an actual defect from the input unit (120). At this time, the signal regarding whether there is an actual defect is a signal generated by an input from an inspector who is a user of the electronic device (100), and may be a distinguishing signal regarding whether the defect indicated by the bounding box is an actual defect or whether it is normal but has been confirmed as a defect.

[0054] If an actual defective signal is not received, the control unit (150) can confirm that a bounding box identified as a defective is displayed even though the defective signal is not an actual defective signal, and can delete the bounding box.

[0055] Conversely, when an actual defect signal is received, the control unit (150) calls labeling data related to the defect from the memory (140), creates the called labeling data as a list box, and displays it at a location close to the location where the bounding box is displayed. At this time, the labeling data refers to data that labels the types of defects that may occur in relation to the product, and in particular, may be a defect type for a defect that may occur at the location where the bounding box is displayed.

[0056] In addition, the control unit (150) can display the defect types in a list box by sorting them based on the occurrence frequency of the defect types that may occur in relation to the defects at the location where the bounding box is displayed. For example, if defect types of Type A, Type B, Type C, and Type D may occur in relation to the defects that may occur at the location where the bounding box is displayed and have occurrence frequencies in alphabetical order, the control unit (150) can display the defect types in a list box by sorting them in alphabetical order. At this time, the control unit (150) can display Type A, Type B, Type C, and Type D in different colors for each defect type. In addition, the list box additionally displays a number for each defect type so that the user can easily select the defect type by entering a number.

[0057] The control unit (150) updates the labeling data when at least one defect type is selected from among the defect types included in the list box from the input unit (120). The control unit (150) uses the updated labeling data to enable the artificial intelligence algorithm to perform additional learning on defects, thereby enabling the advancement of the artificial intelligence algorithm.

[0058]

[0059] FIG. 2 is a flowchart illustrating a method for identifying defects using image data according to an embodiment of the present invention.

[0060] Referring to FIG. 2, in step 201, the control unit (150) checks whether a confirmation signal for confirming a defect occurring in the product is received from the input unit (120). If the confirmation signal is received as a result of the confirmation in step 201, the control unit (150) may perform step 203, and if the confirmation signal is not received, the control unit (150) may wait for the reception of the confirmation signal. At this time, the confirmation signal may mean an execution signal for a program that can confirm a defect in image data and generate labeling data. In addition, the main screen of the program may be configured to be divided into a first area and a second area. This will be described using FIG. 5 below. In the embodiment of the present invention, the main screen displayed on the display unit (130) is described as being divided into a first area and a second area as an example, but this is for the convenience of explanation and is not necessarily limited thereto.

[0061] In step 203, the control unit (150) can arrange the plurality of image data provided from the communication unit (110) in the first area of ​​the main screen. In step 205, the control unit (150) can enlarge and display the target image data that has been confirmed to be defective based on the analysis results of the plurality of image data in the second area. In addition, the control unit (150) can confirm the image data corresponding to the selection signal received from the input unit (120) as the target image data and enlarge and display it in the second area.

[0062] In step 207, the control unit (150) updates the labeling data based on the defect type included in the target image data displayed in the second region. This will be described in more detail with reference to FIG. 3 below. FIG. 3 is a detailed flowchart illustrating a method for updating labeling data based on the defect type identified in image data according to an embodiment of the present invention.

[0063] Referring to FIG. 3, in step 301, the control unit (150) checks whether there is a defect in the target image data enlarged and displayed in the second area. As a result of the check in step 301, if there is a defect in the target image data, the control unit (150) performs step 303, and if there is no defect, the control unit performs step 305. At this time, in step 303, the control unit (150) can check for a defect included in the target image data using an artificial intelligence algorithm. In step 305, if an additional signal of a bounding box is received from the input unit (120), the control unit (150) performs step 303, and if no additional signal is received, the control unit performs step 321. The additional signal is a signal for adding a bounding box for a defect that has not been confirmed by the artificial intelligence algorithm based on the result of an inspector visually inspecting the target image data.

[0064] In step 303, the control unit (150) displays a bounding box by overlaying it on the location where the defect is identified, or displays a bounding box at a location corresponding to the additional signal received in step 305. At this time, the bounding box may be displayed by reflecting attributes including size, shape, and color set corresponding to the location of the defect identified in the target image data. The bounding box may be displayed in various shapes such as a square, circle, or triangle.

[0065] The control unit (150) can divide the target image data into a plurality of virtual areas and sequentially analyze the plurality of areas according to a preset order to check for defects. For example, the control unit (150) can divide the target image data into a total of 12 areas each of 4 horizontal x 3 vertical areas, and analyze the target image data in the order of (1,1) to (1,4), (2,1) to (2,4), and (3,1) to (3,4) to check for defects. At this time, the control unit (150) can analyze only the areas where defects are likely to be detected among the 12 areas, thereby having the effect of minimizing the time consumed when analyzing the target image data.

[0066] In step 307, the control unit (150) checks whether a change signal for the bounding box displayed as an overlay on the target image data is received from the input unit (120). If the change signal is received as a result of the check in step 307, the control unit (150) can perform step 309 to change at least one of the position and properties of the bounding box according to the change signal received from the input unit (120), and then perform step 311. If the change signal is not received as a result of the check in step 307, the control unit (150) can perform step 311.

[0067] In step 311, the control unit (150) receives a signal from the input unit (120) regarding whether there is an actual defect. At this time, the signal regarding whether there is an actual defect is a signal generated by an input from an inspector who is a user of the electronic device (100), and may be a distinguishing signal regarding whether a defect indicated by a bounding box is an actual defect or a defect that was confirmed as normal. In step 311, if the distinguishing signal received from the input unit (120) is an actual defect signal, the control unit (150) performs step 313, and if the distinguishing signal is a normal signal, the control unit (150) performs step 319. In step 319, the control unit (150) deletes a bounding box that is indicated as a defect even though it is not a defect, and then performs step 321.

[0068] In step 313, the control unit (150) calls labeling data related to defects corresponding to the bounding box displayed in step 303 from the memory (140), creates the called labeling data as a list box, and displays it at a location close to the location where the bounding box is displayed. At this time, the labeling data refers to data that labels the types of defects that may occur in relation to the product, and in particular, may be a type of defect for a defect that may occur at the location where the bounding box is displayed.

[0069] In addition, the control unit (150) can display the defect types in a list box by sorting them based on the occurrence frequency of the defect types that may occur in relation to the defects at the location where the bounding box is displayed. For example, if defect types of Type A, Type B, Type C, and Type D may occur in relation to the defects that may occur at the location where the bounding box is displayed and have occurrence frequencies in alphabetical order, the control unit (150) can display the defect types in a list box by sorting them in alphabetical order. At this time, the control unit (150) can display Type A, Type B, Type C, and Type D in different colors for each defect type. In addition, the list box additionally displays a number for each defect type so that the user can easily select the defect type by entering a number.

[0070] In step 315, the control unit (150) performs step 317 when at least one defect type is selected from among the defect types included in the list box from the input unit (120). In step 317, the control unit (150) updates the labeling data and performs step 321. At this time, the control unit (150) updates the labeling data, thereby enabling the artificial intelligence algorithm to additionally learn about defects, thereby enabling the advancement of the artificial intelligence algorithm.

[0071] In step 321, if the control unit (150) receives a signal from the input unit (120) to check the next defect in the target image data, the control unit (150) may return to step 301 and re-perform steps 301 to 319. Conversely, if the control unit (150) does not receive a signal from step 321 to check the next defect in the target image data, the control unit (150) may perform step 323. In step 323, the control unit (150) may store the target image data in the memory (140) so that the artificial intelligence algorithm may perform learning on the target image data as well.

[0072] And the control unit (150) returns to step 209 of Fig. 2. In step 209, if defects are confirmed in all image data from the first image data to the seventh image data, the control unit (150) terminates the process, and if defects are not confirmed in all image data, the control unit returns to step 205 and steps 205 to 209 can be re-performed.

[0073]

[0074] FIG. 4 is a screen example diagram for explaining the direction in which image data for a product according to an embodiment of the present invention is acquired.

[0075] Referring to FIG. 4, the control unit (150) receives first image data (401) to seventh image data (407) for the product (P) acquired from the camera (130). The first image data (401) may be image data for the Top direction of the product (P), the second image data (402) may be image data for the Load_Tap direction of the product (P), the third image data (403) may be image data for the Load direction of the product (P), the fourth image data (404) may be image data for the Right direction, the fifth image data (405) may be image data for the Line_Tap direction, the sixth image data (406) may be image data for the Line direction, and the seventh image data (407) may be image data for the Left direction.

[0076]

[0077] FIG. 5 is a screen example of an interface for checking for defects in image data according to an embodiment of the present invention.

[0078] Referring to FIG. 5, the control unit (150) can divide the display unit (130) into a first region (510) and a second region (520), and divide the first region (510) into a plurality of regions. The control unit (150) can display image data for the product (P) acquired by the camera (130) in each region into which the first region (510) is divided. In addition, the control unit (150) can enlarge and display target image data that is confirmed to have a defect among the image data displayed in the first region (510) in the second region (520).

[0079] At this time, the first area (510) is explained as being divided into four as an example, but this is for the convenience of explanation, and is not necessarily limited to four, and may be divided into seven so that a total of seven image data, which are all image data for the product (P) acquired from the camera (130), can be displayed. In addition, the first area (510) can be implemented so that all image data displayed in the first area (510) can be scrolled left and right so that the scroll can be used to check all image data.

[0080]

[0081] FIG. 6 is a screen example showing the shape of a bounding box according to a defect that can be confirmed in image data for one direction of a product according to an embodiment of the present invention.

[0082] Referring to FIG. 6, among the types of defects that may occur in the image data for the TOP, LOAD (LD), LINE (LN), LEFT (LFT), RIGHT (RGT), LOAD_TAP (LDT), and LINE_TAP (LNT) directions for the circuit breaker, the types of defects that may be confirmed in the image data for the TOP direction are as shown in Table 1 below. In addition, the types of defects that may be confirmed in each of the image data for the LOAD (LD), LINE (LN), LEFT (LFT), RIGHT (RGT), LOAD_TAP (LDT), and LINE_TAP (LNT) directions are as shown in Tables 2 to 7 below.

[0083] Bad shooting direction Occurrence Bad position TypeTOPAllScrewingDamage, Contamination, ContaminationCriticalAux CoverAuxCoverNone, AuxCoverUnformed, AuxCoverDamaged, TripButtonNone, SlideButtonLocation, YellowButtonMixed, SlideButtonMixedCase, Main coverCaseMainCoverDamage, CaseMainCoverUnformedMechanismMechanismDamage, Rubber, SwitchOffLocation, SemiMixedScrewScrewNoneTerminalTerminalScrewNone

[0084] The control unit (150) can display bounding boxes with different properties, such as size, shape, and color, depending on the type of defect that can be identified in the image data for the TOP direction, as shown in FIG. 6.

[0085] Type of shooting direction error: LOADAllCrack, Contamination, ContaminationCriticalBarrierBarrierDamage, BarrierCableMisassembleCase, Main coverAssemblyGap, CaseMainCoverDamage, CaseMainCoverUnformed

[0086] 촬영 방향불량발생 위치불량 유형LINEAllCrack, ScrewingDamage, Contamination, ContaminationCriticalArc BarrierAcrBarrierNone, AcrBarrierRippedAux CoverAuxCoverUnformed, AuxCoverDamageBaseBaseDamageCase, Main CoverCaseMainCoverDamage, CaseTerminalDamage, CaseMainCoverUnformedScrewAuxScrewNone, AuxScrewPoorConnection, MainCoverScrewNone, MainCoverScrewPoorConnectionTerminalTapNone, TerminalNone

[0087] 촬영 방향불량발생 위치불량 유형LEFTAllCrack, Contamination, ContaminationCriticalAux CoverAuxCoverUnformed, AuxCoverDamageCase, Main CoverAssemblyGap, Misassembley, CaseDamage, CaseMainCoverDamage, TriangularCaseMainCoverUnformed, CaseMainCoverUnformedScrewScrewNone, ScrewPoorConnection

[0088] 촬영 방향불량발생 위치불량 유형RIGHTAllCrack, Contamination, ContaminationCriticalAux CoverAuxCoverUnformed, AuxCoverDamageCase, Main coverAssemblyGap, Misassembly, CaseDamage, CaseMainCoverDamage, TriangularCaseMainCoverUnformed, CaseMainCoverUnformedScrewScrewNone, ScrewPoorConnectionStickerNameTagDirt, NameTagRipped, SealedLabelNone, SealedLabelMislocation, SealedLabelMultiple, SealedLabelFolding, SealedLabelRipped, StickerFolding, NameTagNone, NameTagMislocation

[0089] 촬영 방향불량발생 위치불량 유형LOAD_TAPAllCrack, ScrewingDamage, Contamination, ContaminationCriticalAux CoverAuxCoverUnformed, AuxCoverDamage, LSEStickerMixedBarrierBarrierDamage, BarrierCableMisassembleCase, Main CoverCaseMainCoverDamage, CaseMainCoverUnformedScrewAuxScrewNone, AuxScrewPoorConnection, MainCoverScrewNone, MainCoverScrewPoorConnectionTerminalTapNone, TerminalScrewNone, TerminalNone

[0090] Shooting direction error Occurrence position error typeLINE_TAPAllCrack, ScrewingDamage, Contamination, ContaminationCriticalArc BarrierArcBarrierNone, ArcBarrierRippedAux CoverAuxCoverUnformed, AuxCoverDamageBaseBaseDamageCase, Main coverCaseMainCoverDamage, CaseTerminalDamate, CaseMainCoverUnforemdScrewAuxScrewNone, AuxScrewPoorConnection, MainCoverScrewNone, MainCoverScrewPoorConnectionTerminalTapNone, TerminalNone

[0091] FIG. 7 and FIG. 8 are screen examples showing defects identified in the appearance of a product according to one embodiment of the present invention.

[0092] Referring to FIGS. 7 and 8, the control unit (150) can display first image data (401), second image data (402), fourth image data (404), and fifth image data (405) in the first area (510) of the display unit (130). The control unit (150) can display a display window (711a) that can indicate the number of defects in the upper left corner of each image data displayed in the first area (510), and can change the color of the area where the target image data being checked for defects is displayed, as shown by reference numeral 711, to display it.

[0093] For example, v displayed in the display window (711a) may mean that confirmation of defects identified in the corresponding image data has been completed, and the numbers 4 and 2 may mean that there are 4 and 2 defects in the first image data (401) and the second image data (402), respectively. If nothing is displayed in the display window (711a), this may mean that there are no defects in the fourth image data (404).

[0094] When the first image data (401) is selected from the input unit (120), the control unit (150) can enlarge the first image data (401) in the second area (520) and display it as target image data. In addition, since four defects are identified as indicated in the display window (711a) displayed on the first image data (401), the control unit (150) can enlarge the first image data (401) in the second area (520) and display it as target image data. In addition, the control unit (150) can display bounding boxes at positions related to the defects (731, 732, 733, 734) identified in the target image data.

[0095] In addition, the control unit (150) may add a new bounding box to a location for a new defect (735) selected by the input unit (120) as in FIG. 8, even though the location is not identified as a defect in the target image data.

[0096] In this case, the control unit (150) can confirm the location of the defect occurrence based on the location selected as the new defect (735) and display the defect types that may occur at the confirmed defect occurrence location in a labeling list (745). At this time, since the defect occurrence location selected as the new defect (735) is the location corresponding to Screw as shown in FIG. 6, the control unit (150) can display a labeling list (745) including ScrewNone, which is the defect type that may occur at the Screw location as shown in Table 1. In addition, since ScrewNone is the only defect type that may occur at the Screw location, the control unit (150) can automatically select ScrewNone as the defect type and can receive the defect type in text form from the input unit (120).

[0097] When a defect type is selected or input, the control unit (150) can update labeling data using the location of the corresponding bounding box and the defect type. The control unit (150) can retrain the artificial intelligence algorithm using the updated labeling data.

[0098]

[0099] FIGS. 9 to 14 are screen examples for explaining a method for checking for defects in the appearance of a product according to another embodiment of the present invention.

[0100] Referring to FIGS. 9 to 14, if four defects are confirmed in the first image data (401) as in FIG. 7, the control unit (150) can divide the second area (520) in which the first image data (401) is indicated as target image data into a plurality of virtual areas using a plurality of virtual lines (721). At this time, the control unit (150) can perform virtual numbering such as (1,1), (1,2), etc. according to the number of the plurality of virtual areas, and can sequentially check the defects confirmed in the target image data included in the divided areas.

[0101] At this time, the control unit (150) can confirm that there are no defects in the target image data at (1,1) to (1,4) and that the first defect (731) exists at (2,1) and (2,2). The control unit (150) can display a bounding box at the location where the first defect (731) is confirmed. At this time, since the confirmed first defect (731) is a defect related to a screw, the control unit (150) can display a bounding box having a green circular attribute related to a screw.

[0102] If the first defect (731) detected through the artificial intelligence algorithm as shown in Fig. 9 is an actual defect, the inspector can select the right arrow (742) displayed in the second area (520) through the input unit (120). When the control unit (150) receives an input signal for the right arrow (742) from the input unit (120), it can confirm the input signal as a signal for confirming the second defect (732) and switch the screen to Fig. 10.

[0103] Conversely, if the first defect (731) detected by the artificial intelligence algorithm as shown in FIG. 9 is not an actual defect, the inspector can delete the bounding box displayed at the location detected as the first defect (731) through the input unit (120). In this way, when the inspector selects the bounding box through the input unit (120) to delete the bounding box, the control unit (150) can display a menu list providing a function to delete the bounding box on the display unit (130). When the inspector selects deletion of the bounding box from the menu list, the control unit (150) can cause the corresponding image data to be relearned.

[0104] In addition, when an input signal for the right arrow (742) in FIG. 9 is received, the control unit (150) can confirm that a second defect (732) exists at (2,2) and (2,3) and display a screen as in FIG. 10 on the display unit (130). The control unit (150) can display a bounding box at a location where the second defect (732) is confirmed in the target image data as in FIG. 10. At this time, since the second defect (732) is a defect existing at a location related to the mechanism, the control unit (150) can display a bounding box having a purple square attribute related to the mechanism.

[0105] As shown in FIG. 10, when a signal for changing the position of the bounding box is received while the bounding box for the second defect (732) is displayed, the inspector can move the position of the bounding box. For example, when the cursor (743) for changing the position of the bounding box is activated through the input unit (120), the control unit (150) can select and move the bounding box according to the movement of the cursor (743). More specifically, the inspector can select and move the bounding box using the cursor (743) or change the position of the bounding box using the arrow keys (↑, ↓, ←, →) of the keyboard. At this time, when the input unit (120) is combined with the display unit (130) and implemented in the form of a touch screen, the inspector can touch the display unit (130) to activate the cursor (743) and then select and move the bounding box to change the position of the bounding box.

[0106] The control unit (150) can confirm the movement of the cursor (743) input by the inspector as a change signal for changing the position of the bounding box and change the position of the bounding box according to the change signal.

[0107] In addition, if the inspector determines that the size of the bounding box needs to be changed as in Fig. 11, the inspector can activate the cursor (743) for changing the size of the bounding box using the input unit (120). The inspector can change the size of the bounding box using the cursor (743). The inspector can select the bounding box using the cursor (743) and then drag it to change the size of the bounding box. In addition, if the input unit (120) is implemented in the form of a touch screen by being combined with the display unit (130), the inspector can change the size of the bounding box using multi-touch, etc.

[0108] At this time, if there is no need to change the position or size of the bounding box, the control unit (150) can switch from the screen of FIG. 9 to the screen of FIG. 12. As in FIG. 9, when an input signal for the right arrow (742) is received, the control unit (150) can confirm the input signal as a signal for checking the second defect (732) and switch the screen to FIG. 12. In addition, as in FIGS. 10 and 11, when an input signal for the right arrow (742) is received after the position or size change of the bounding box is completed, the control unit (150) can switch the screen to FIG. 12.

[0109] When an inspector selects a bounding box using a cursor (743) while a bounding box is displayed at the location corresponding to the second defect (732) as in Fig. 12, the control unit (150) can display a menu list (744) related to the bounding box according to a selection signal generated from the input unit (120). At this time, the menu list (744) can include confirmation / change and deletion of the defect type.

[0110] When the inspector selects deletion from the menu list (744), the control unit (150) determines that the second defect (732) detected through the artificial intelligence algorithm is not an actual defect and can delete the bounding box displayed at the location detected as the second defect (732).

[0111] Conversely, when the inspector selects the defect type confirmation / change from the menu list (744), the control unit (150) can determine that the second defect (732) detected through the artificial intelligence algorithm is an actual defect and confirm the selection signal as a signal for labeling the second defect (732). The control unit (150) can display a labeling list (745) that displays labeling data as a list for labeling the second defect (732).

[0112] At this time, since the location where the second defect (732) occurred is a location related to the mechanism, as shown in Table 1, mechanism damage, rubber, switchofflocation, and semimixed, which are defect types related to the mechanism, can be displayed as items in the labeling list (745). At this time, among the items constituting the labeling list (745), the item identified as the defect type of the second defect (732) can be displayed in a different color from the other items, as shown in FIG. 12. The inspector can check the items displayed in different colors in the labeling list (745) to confirm the defect type of the second defect (732), and can change the defect type of the second defect (732) by selecting any one of the items included in the labeling list (745). In addition, the number written in the defect type of each item constituting the labeling list (745) may indicate the number of times the corresponding defect type occurred.

[0113] Next, when an input occurs in the right arrow (742) after an item is selected from the labeling list (745), the control unit (150) can switch the screen as shown in Fig. 13. The control unit (150) can display a pop-up window (751) asking whether to update the labeling data using the position and size of the bounding box changed according to the input of the input unit (120) as shown in Figs. 10 and 11, and the type of defect selected in Fig. 12. When YES is selected in the pop-up window (751), the control unit (150) can update the labeling data using the position of the bounding box changed as shown in Fig. 10, the size of the bounding box changed as shown in Fig. 11, and the type of defect selected in Fig. 12. The control unit (150) can retrain the artificial intelligence algorithm using the updated labeling data and target image data.

[0114] In addition, the control unit (150) can switch to the screen of Fig. 14 after updating the labeling data. The control unit (150) can confirm that a third defect (733) exists at (2,3) and can display a bounding box at the location where the third defect (733) is confirmed. At this time, the control unit (150) can display a bounding box with a blue square attribute because the third defect (733) is a defect type related to switchofflocation among mechanism-related defects.

[0115] As shown in FIGS. 9, 12, and 14, the inspector can sequentially perform actions such as checking for the presence or absence of actual defects and selecting the type of defects existing in the target image data, and can easily change the position and size of the bounding box, as shown in FIGS. 10 and 11, thereby creating labeling data more easily.

[0116] In addition, changing the position of the bounding box as in Fig. 10, changing the size of the bounding box as in Fig. 11, and checking / changing and deleting the type of defect as in Fig. 12 can be performed in the same manner by selecting each bounding box while all bounding boxes are displayed in the target image data as in Fig. 7.

[0117] FIG. 15 is a screen example diagram illustrating a method for checking for defects in the appearance of a product according to another embodiment of the present invention.

[0118] Referring to FIG. 15, the control unit (150) can divide the first area (510) of the display unit (130) into seven to display all image data. At this time, although the seventh image data (407) is not displayed in the first area (510), the control unit (150) can scroll the first area (510) according to the left and right scroll signal for the first area (510) received from the input unit (120) to display the seventh image data (407) in the first area (510).

[0119] In addition, other configurations except for the first region (510) as described above are described in Fig. 7, so a detailed description thereof will be omitted.

[0120]

[0121] The embodiments of the present invention disclosed in this specification and drawings are merely specific examples intended to facilitate understanding and easily explain the technical content of the present invention, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be interpreted to include all modifications or variations derived based on the technical concept of the present invention, in addition to the embodiments disclosed herein.

Claims

1. A step in which an electronic device displays multiple image data for a product in a first area; A step in which the electronic device enlarges one of the image data displayed in the first area and displays it as target image data in the second area; and A step in which the electronic device displays a bounding box at the location of at least one defect included in the target image data; A defect identification method characterized by including a .

2. In paragraph 1, The step of displaying the above target image data is as follows: A method for confirming a defect, characterized by comprising a step of indicating selected image data from among the image data displayed in the first area above or image data confirmed to be defective through an artificial intelligence algorithm as the target image data.

3. In paragraph 2, The steps for displaying the above bounding box are: A defect confirmation method characterized by comprising a step of adding and displaying the bounding box to the target image data according to an additional signal of the bounding box.

4. In paragraph 3, The steps for displaying the above bounding box are: A defect confirmation method characterized by a step of displaying the bounding box at a location related to the defect confirmed in the target image data through the artificial intelligence algorithm.

5. In paragraph 4, After the step of displaying the above bounding box, A step of the electronic device deleting or changing the bounding box displayed on the target image data; A defect identification method characterized by further including:

6. In paragraph 5, After the step of displaying the above bounding box, A step in which the electronic device applies the addition, deletion and change history of the target image data and the bounding box to the artificial intelligence algorithm; A defect identification method characterized by further including:

7. In paragraph 6, After the step of displaying the above bounding box, A step of setting at least one of the properties and the position of the bounding box according to a setting signal for setting at least one of the properties including the size, shape and color of the bounding box and the position of the bounding box by the electronic device; A defect identification method characterized by further including:

8. In paragraph 7, The step of setting at least one of the properties and position of the above bounding box is: A step in which the electronic device identifies at least one type of defect that may occur at the location of the defect based on the labeling data stored in advance in relation to the defect indicated by the bounding box; and A step of displaying a list box sorting at least one of the above defective types by overlaying it on the target image data; A defect identification method characterized by including a .

9. In paragraph 8, The steps to apply the above artificial intelligence algorithm are: A defect confirmation method characterized in that the electronic device applies the updated labeling data by mapping it with the defect type selected from the list box to the artificial intelligence algorithm.

10. A display unit that displays the main screen for checking for defects in the product; and A control unit that divides the main screen into a first area and a second area, displays a plurality of image data in the first area, enlarges one of the plurality of image data and displays it as target image data in the second area, and controls the display unit to display a bounding box at the location of at least one defect included in the target image data; A defect checking device characterized by including a .

11. In paragraph 10, The above control unit, A defect confirmation device characterized in that it displays image data selected from among the image data displayed in the first area above or image data confirmed to be defective through an artificial intelligence algorithm as the target image data.

12. In paragraph 11, The above control unit, A defect confirmation device characterized by adding and displaying the bounding box to the target image data according to an additional signal of the bounding box.

13. In paragraph 12, The above control unit, A defect confirmation device characterized by displaying the bounding box at a location related to the defect confirmed in the target image data through the artificial intelligence algorithm.

14. In paragraph 13, The above control unit, A defect confirmation device characterized by deleting or changing the bounding box displayed in the target image data.

15. In paragraph 14, The above control unit, A defect confirmation device characterized by applying the addition, deletion and change history of the target image data and the bounding box to the artificial intelligence algorithm.

16. In paragraph 15, The above control unit, A defect verification device characterized in that at least one of the properties and the position of the bounding box is set according to a setting signal for setting at least one of the properties including the size, shape and color of the bounding box and the position of the bounding box.

17. In paragraph 16, The above control unit, A defect confirmation device characterized in that it overlays and displays a list box that sorts at least one defect type that can occur at the location of the defect based on pre-stored labeling data related to the defect indicated by the above bounding box on the target image data.

18. In paragraph 17, The above control unit, A defect identification device characterized in that it applies the updated labeling data by mapping it with the defect type selected from the above list box to the above artificial intelligence algorithm.

Images