Electronic device providing user interface for process inspection, and process inspection method using same

Abstract

The present invention relates to an electronic device providing a user interface for process inspection, and a process inspection method using same. The electronic device according to one embodiment of the present invention may comprise a processor providing a user interface for process inspection, including: a first area in which a process inspection list and a process inspection result of a product introduced into a process are displayed; and a second area in which an image acquired by capturing a process progress status in real time is displayed.

Description

Electronic device providing a user interface for process inspection and a process inspection method using the same

[0001] The present invention relates to an electronic device providing a user interface for process inspection and a process inspection method using the same.

[0002] With the recent advancements in related technologies like the Internet of Things (IoT) and Artificial Intelligence (AI), smart factories are becoming a hot topic. A smart factory is a manufacturing system that operates by applying Information and Communications Technology (ICT) combined with digital automation solutions throughout the production process, from design and development to manufacturing.

[0003] In sites where smart factory technology is applied, labor costs can be reduced because worker deployment is eliminated. Furthermore, multiple processes for product manufacturing can be performed continuously, reducing both production speed and unit costs. Consequently, active efforts are being made to apply smart factories in various field situations.

[0004] Meanwhile, manufacturing a single product may require multiple processes, and within each process, the product may be manufactured through the sequential operation of multiple equipment modules. During process operation, process inspections are also conducted to detect missing components or defective products.

[0005] Efforts are also ongoing to automate these process inspections by incorporating smart factory technology to reduce labor costs and maintain consistent quality.

[0006] However, the above-described content merely provides background information on the present invention and does not correspond to previously disclosed technology.

[0007] The purpose of the present invention is to provide an electronic device that provides a user interface for process inspection to prevent defects that may occur during a process, and a process inspection method using the same.

[0008] However, the problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0009] In an electronic device according to one embodiment of the present invention, a processor may be included that provides a user interface for process inspection, including a first area displaying a process inspection list of products input into a process and a process inspection result, and a second area displaying a real-time image of the process progress status.

[0010] The above processor can use an object recognition algorithm to check whether an object corresponding to the process inspection list is recognized in the image, and display a bounding box for the recognized object in the image.

[0011] The above processor can display object information and object recognition rate together in the above bounding box.

[0012] The above processor can display the progress status of each task of the process inspection list in the first area according to the object recognition result.

[0013] The above processor can display a judgment reference line on the image, and identify the process inspection result according to the overall work progress status of the process inspection list when the product passes the judgment reference line.

[0014] The above processor may display notification information indicating an error on the user interface when the process inspection result is determined to be an error.

[0015] The above processor may transmit a request signal to a facility control device that controls the facility module to prevent shipment of the product if the process inspection result is determined to be erroneous.

[0016] The above processor can display specific details about the progress status of the process inspection list in the third area of ​​the user interface.

[0017] The above processor can display a function implemented to set display conditions of the image in a fourth area of ​​the user interface.

[0018] In one embodiment of the present invention, an electronic device may include a display for displaying a user interface for process inspection consisting of a plurality of areas; a processor for displaying a process inspection list identified based on a product code of a product input into a process in a first area of ​​the user interface, displaying an image captured in real time showing the progress of the process in a second area of ​​the user interface, checking whether an object corresponding to the process inspection list is recognized in the image using an object recognition algorithm corresponding to the product, and displaying the progress status of the process inspection list and the process inspection result in the first area according to the object recognition result.

[0019] The above processor can identify an object recognition algorithm corresponding to the product among object recognition algorithms provided for each product code.

[0020] The above processor can display a judgment reference line on the image, and identify the process inspection result according to the overall work progress status of the process inspection list when the product passes the judgment reference line.

[0021] The above processor may transmit a request signal to a facility control device that controls the facility module to prevent shipment of the product if the process inspection result is determined to be erroneous.

[0022] A process inspection method performed by an electronic device according to one embodiment of the present invention may include: a step of displaying a process inspection list identified based on a product code of a product input into a process in a first area of ​​a user interface for process inspection; a step of displaying an image captured in real time of the process progress status in a second area of ​​the user interface; a step of checking whether an object corresponding to the process inspection list is recognized in the image using an object recognition algorithm corresponding to the product; and a step of displaying the progress status of the process inspection list and the process inspection result in the first area according to the object recognition result.

[0023] The above-mentioned checking step may include a step of identifying an object recognition algorithm corresponding to the product among object recognition algorithms provided for each product code.

[0024] The above-described inspection step may include a step of displaying a judgment reference line on the image; and a step of identifying the process inspection result according to the overall work progress status of the process inspection list when the product passes the judgment reference line.

[0025] The above-described inspection step may include a step of transmitting a request signal to a facility control device that controls the corresponding facility module to prevent shipment of the product if the process inspection result is determined to be erroneous.

[0026] In an electronic device according to one embodiment of the present invention, a processor may be included that displays an image captured in real time of the process progress status of a product input into a process, checks whether an object corresponding to a process inspection list in the image is recognized using an object recognition algorithm corresponding to the product, and identifies a process inspection result according to the overall work progress status of the process inspection list when the product passes a judgment reference line displayed in the image.

[0027] According to one embodiment of the present invention, an auxiliary system can be constructed to prevent human error of a worker in a manual process or error occurrence in an automated process during repetitive work.

[0028] According to one embodiment of the present invention, a smart quality maintenance system can be secured through artificial intelligence-based image analysis.

[0029] According to one embodiment of the present invention, an artificial intelligence-based image analysis solution can be provided to prevent defects that may occur in manual and automatic packaging processes.

[0030] According to one embodiment of the present invention, a product can be maintained at a consistent quality by detecting defects before shipping the product and taking measures therefor before shipping.

[0031] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

[0032] FIG. 1 is a schematic diagram illustrating a process inspection system according to one embodiment of the present invention.

[0033] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0034] FIG. 3 is a diagram illustrating an operation flow chart of an electronic device according to one embodiment of the present invention.

[0035] FIG. 4 is a drawing illustrating the operation of a process inspection system according to one embodiment of the present invention.

[0036] FIG. 5 is a drawing illustrating a user interface according to a first embodiment of the present invention.

[0037] FIG. 6 is a drawing illustrating a user interface according to a second embodiment of the present invention.

[0038] FIG. 7 is a drawing illustrating a user interface according to a third embodiment of the present invention.

[0039] FIG. 8 is a drawing illustrating a user interface according to a fourth embodiment of the present invention.

[0040] FIG. 9 is a drawing illustrating a user interface according to a fifth embodiment of the present invention.

[0041] Hereinafter, specific embodiments according to the present disclosure will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the present invention is not limited thereto.

[0042] In describing embodiments of the present disclosure, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the embodiments, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and these may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing one embodiment and should not be limited in any way. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts, or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts, or combinations thereof other than those described. Additionally, terms such as “part,” “unit,” “module,” and “block” described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software.

[0043] FIG. 1 is a schematic diagram illustrating a process inspection system according to one embodiment of the present invention.

[0044] Referring to FIG. 1, the process inspection system (1) (hereinafter also referred to as system (1)) may include an electronic device (100), a server (200), a power management device (310) for each process line (300), an equipment control device (320), and an equipment module (330).

[0045] An electronic device (100) according to one embodiment of the present invention provides a user interface for process inspection and is a device that performs process inspection. For example, the electronic device (100) may be implemented as a computer, a programmable logic controller (PLC), a server, a smart phone, a tablet PC, a smart pad, a laptop, etc., but is not limited thereto. The electronic device (100) may have a built-in manufacturing execution system (MES) to manage the process of the process line (300).

[0046] A server (200) according to one embodiment of the present invention is a device that communicates with an electronic device (100), a device on a process line (300), etc., and collects or provides data, and may have an employee assistance program (EAP), a recipe parameter management system (RPMS), etc. built in.

[0047] The server (200) according to one embodiment of the present invention may be implemented as a separate device from the electronic device (100), but is not limited thereto and may of course be implemented as a single device. Furthermore, the electronic device (100) and the server (200) may be implemented in various ways, such as not being physically separated but operating separately in terms of control.

[0048] A process line (300) (process line #1, process line #2, etc.) according to one embodiment of the present invention may include a plurality of equipment modules (330) configured to operate sequentially according to the process order. At least one process line (300) may be installed depending on the on-site situation.

[0049] A plurality of equipment modules (330) (equipment module #1, equipment module #2, equipment module #3, ...) may be connected to an equipment control device (320) (equipment control device #1, equipment control device #2, equipment control device #3, ...) that controls the operation of each equipment module. In addition, the process line (300) may separately be equipped with a power management device (310) that manages power, etc. of the plurality of equipment control devices (320).

[0050] At this time, the power management device (310) and the equipment control device (320) may be implemented as a PLC (Programmable Logic Controller). However, the present invention is not limited thereto, and the power management device (310) and the equipment control device (320) may also be implemented as a computer, tablet PC, smart pad, laptop, etc.

[0051] As previously described, manufacturing a single product may require multiple processes, and within each process, the product may be manufactured through the sequential operation of multiple equipment modules. In this case, process inspection is essential to ensure consistent quality control.

[0052] The present invention proposes an electronic device that automatically performs process inspection of a process line (300) and a process inspection method using the same.

[0053] Hereinafter, the configuration and operation of a remote management system (1) according to one embodiment of the present invention will be specifically described with reference to the drawings.

[0054] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0055] An electronic device (100) according to one embodiment of the present invention includes a storage unit (140) and a processor (150). Of course, the electronic device (100) may further include an input unit (110), a communication unit (120), or a display unit (130).

[0056] The input unit (110) generates input data in response to user input of the electronic device (100). For example, the user input may be a user input for manipulating a user interface, for example, a user input for setting display conditions of an image, a user input for entering a product code, etc. In addition, the user input may be an input required to provide a user interface for process inspection and perform process inspection.

[0057] The input unit (110) includes at least one input means. For example, the input unit (110) may include, but is not limited to, a keyboard, a keypad, a dome switch, a touch panel, a touch key, a mouse, or a menu button.

[0058] The communication unit (120) is a component that performs communication with other devices. That is, the communication unit (120) can perform communication with external devices such as a server (200), a power management device (310), an equipment control device (320), and an equipment module (330) to transmit and receive product codes, process inspection results, and request signals based on the results.

[0059] For example, the communication unit (120) can perform wireless communication such as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), Wi-Fi (wireless fidelity), Bluetooth, or wired communication such as LAN (local area network), WAN (Wide Area Network), and power line communication, but is not limited thereto.

[0060] The display unit (130) displays display data according to the operation of the electronic device (100). The display unit (130) may display a screen that displays a user interface for process inspection, a screen that receives user input, etc.

[0061] For example, the display unit (130) may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, or an electronic paper display. The display unit (130) may be implemented as a touch screen in combination with the input unit (110).

[0062] The storage unit (140) stores information necessary for the operation of the electronic device (100). That is, the storage unit (140) can store a program for the operation of the electronic device (100). The storage unit (140) includes a non-volatile storage that can preserve data (information) regardless of whether power is supplied, and a volatile memory that loads data to be processed by the processor (150) and cannot preserve data if power is not supplied. The storage includes a flash memory, a hard-disc drive (HDD), a solid-state drive (SSD), a read-only memory (ROM), etc., and the memory includes a buffer, a random access memory (RAM), etc.

[0063] The storage unit (140) can store a Manufacturing Execution System (MES), a process inspection list for each product, an object recognition algorithm for each product, a video showing the process progress, product information, etc. The storage unit (140) can store operation programs required in the process of receiving user input, displaying a user interface, recognizing objects, etc.

[0064] The processor (150) can control at least one other component (e.g., hardware or software component) of the electronic device (100) by executing software such as a program, and can perform various data processing or operations.

[0065] A processor (150) according to one embodiment of the present invention may provide a user interface for process inspection including a first area displaying a process inspection list of products input into a process and process inspection results, and a second area displaying a real-time image of the process progress status.

[0066] A processor (150) according to one embodiment of the present invention may display a process inspection list identified based on a product code of a product input into a process in a first area of ​​the user interface, display an image captured in real time of the process progress status in a second area of ​​the user interface, and check whether an object corresponding to the process inspection list is recognized in the image using an object recognition algorithm corresponding to the product, and display the progress status of the process inspection list and the process inspection result in the first area according to the object recognition result.

[0067] Meanwhile, the processor (150) may perform at least a portion of the data analysis, processing, and result information generation for performing the above operations using at least one of a machine learning, neural network, or deep learning algorithm as a rule-based or artificial intelligence (AI) algorithm. Examples of the neural network may include models such as a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), and an RNN (Recurrent Neural Network).

[0068] FIG. 3 is a diagram illustrating an operation flow chart of an electronic device according to one embodiment of the present invention.

[0069] According to one embodiment of the present invention, the processor (150) can display a process inspection list identified based on the product code of a product input into the process in a first area of ​​the user interface (S10).

[0070] The user interface may be visualized for performing process inspection and may consist of multiple areas. Implementation examples of the user interface are illustrated in Figures 5 to 9.

[0071] The process line (300) can accommodate multiple products, and each product may require different inspection items. To address this, a process is required to verify product information, specifically the product code. The product code is engraved on the product and can be implemented as a barcode, QR (Quick Response) code, or other format. It can also be written as a serial number, and the format of the product code is not limited to a single format.

[0072] Additionally, each equipment module (330) installed in the process line (300) may be equipped with a scanner for scanning a product code. When a product is introduced into the equipment module (330), the equipment control device (320) may read the product code through the installed scanner and transmit it to the electronic device (100). In addition, the process of verifying the product code may be applied in various ways and is not limited to any one.

[0073] The process inspection list includes at least one item to be inspected for each product, and may be prepared for each product. The process inspection list may be stored in the storage unit (140) or stored in a separate product database, but is not limited to either one.

[0074] According to one embodiment of the present invention, the processor (150) can display a real-time image of the process progress status in a second area of ​​the user interface (S20).

[0075] In the present invention, process inspection is performed through object recognition using at least one camera installed on a process line (300). The camera installation locations and number of cameras are not limited to any one. If multiple cameras are installed along the process line (300), the images displayed in the second area may be images acquired from cameras installed corresponding to each process. The processor (150) may display images corresponding to the current process in the second area based on the process progress status.

[0076] Meanwhile, to perform process inspection, an object recognition algorithm corresponding to the product-specific process inspection list must be developed. A single object recognition algorithm can be applied to multiple products, and there is no requirement that the object recognition algorithm and product correspond one-to-one.

[0077] The object recognition algorithm may likewise be stored in the storage unit (140) or stored by constructing a separate product database, and is not limited to either. The processor (150) may identify an object recognition algorithm corresponding to the recognized product code among the object recognition algorithms provided for each product code.

[0078] According to one embodiment of the present invention, the processor (150) can check whether an object corresponding to a process inspection list in an image is recognized using an object recognition algorithm corresponding to a product (S30).

[0079] Object recognition algorithms can be designed based on the shape of the object, the size of the object, the location of the object, the number of objects, etc. for each item included in the process inspection list.

[0080] According to one embodiment of the present invention, the processor (150) can display the progress status of the process inspection list and the process inspection results in the first area according to the object recognition results (S40).

[0081] Process inspection results indicate whether a problem occurred during the manufacturing process of a product. These can be expressed in various ways. For example, a normal product may be indicated as OK, while a defective product may be indicated as NG. Defects can refer to missing parts or accessories within the product, or incorrectly positioned parts. The type of defect can vary depending on the product.

[0082] The processor (150) considers an object as normal if it is recognized for all items in the process inspection list, and considers it as defective if it is not recognized for any item.

[0083] The processor (150) can display the progress status according to whether or not the object is recognized for each item as the process proceeds sequentially.

[0084] According to one embodiment of the present invention, an auxiliary system can be constructed to prevent human error of a worker in a manual process or error occurrence in an automated process during repetitive work.

[0085] According to one embodiment of the present invention, a smart quality maintenance system can be secured through artificial intelligence-based image analysis.

[0086] FIG. 4 is a drawing illustrating the operation of a process inspection system according to one embodiment of the present invention.

[0087] Specifically, Figure 4 illustrates the product packaging process and the corresponding process inspection. It goes without saying that the process depicted in Figure 4 applies equally to both manual and automated packaging processes.

[0088] The packaging process for this product consists of a shield attachment step (S410), a box forming step (S420), an inner material insertion step (S430), a product and spare parts insertion step (S440), and a product shipment step (S450). The process inspection list can be set up in the same way as the packaging process.

[0089] In the shield attachment step (S410), the electronic device (100) can use an object recognition algorithm to recognize the shield in the image (410) and thereby determine whether a shield is attached to the product. Specifically, the electronic device (100) can recognize the product and determine whether a shield is attached to a specific location on the product. If multiple shields are to be attached, it can be determined whether a predetermined number are attached.

[0090] In the box forming step (S420), the electronic device (100) can confirm whether the box has been formed properly by recognizing the box shape in the image (420) using an object recognition algorithm.

[0091] In the internal material insertion step (S430), the electronic device (100) can confirm whether the internal material has been inserted into the box by recognizing the internal material in the image (430) using an object recognition algorithm.

[0092] In the product and spare parts insertion step (S440), the electronic device (100) can confirm whether the product and spare parts are placed in the box by recognizing the product and spare parts in the image (440) using an object recognition algorithm.

[0093] In the product shipment step (S450), the electronic device (100) can ship the product as shown in the image (450) if the process is properly performed in steps S410 to S440.

[0094] If an error occurs in any one of the processes in steps S410 to S440, the electronic device (100) can transmit a request signal to the equipment control device (310) that controls the corresponding equipment module (330) to prevent shipment of the product.

[0095] There may be various methods for preventing product shipment. For example, a stopper that physically prevents product shipment may be installed in the equipment module (330), and when the equipment control device (310) receives a request signal from the electronic device (100) to prevent product shipment, the stopper may be controlled to operate.

[0096] According to one embodiment of the present invention, an artificial intelligence-based image analysis solution can be provided to prevent defects that may occur in manual and automatic packaging processes.

[0097] According to one embodiment of the present invention, a product can be maintained at a consistent quality by detecting defects before shipping the product and taking measures therefor before shipping.

[0098] According to one embodiment of the present invention, the possibility of expansion to similar processes in the future can be considered.

[0099] Below, FIGS. 5 to 9 illustrate user interfaces according to various embodiments of the present invention. The design, including the size and layout of each area of ​​the user interface illustrated in FIGS. 5 to 9, is merely exemplary and does not limit the scope of the present invention.

[0100] FIG. 5 is a diagram illustrating a user interface according to a first embodiment of the present invention. FIG. 5, like FIG. 4, describes a product packaging process. It goes without saying that the process illustrated in FIG. 5 is equally applicable to both manual and automatic packaging processes.

[0101] The user interface (500) of FIG. 5 consists of a first area (510) that displays a process inspection list of products introduced into the process and the results of the process inspection, and a second area (520) that displays a video taken in real time of the process progress status.

[0102] The electronic device (100) can display a process inspection list (work progress status - box forming, cushion insertion, SEALED attachment 1, SEALED attachment 2, product insertion, spare part insertion) and a process inspection result (OK / NG reading - OK / NG) in the first area (510).

[0103] The electronic device (100) can use an object recognition algorithm to check whether an object corresponding to a process inspection list is recognized within an image of the second area (520).

[0104] When an object corresponding to each task is recognized, the electronic device (100) reports that the task is completed and can indicate that the task is completed in the process inspection list of the first area (510). Various methods for indicating this can be employed, such as changing the circled portion to the left of each task in the first area (510) to a different color, placing a check mark in the circled portion, or erasing text describing the task.

[0105] The electronic device (100) can display a bounding box (521) on an object recognized in an image.

[0106] When recognizing an object, the electronic device (100) may display a bounding box around the object if the recognition rate is higher than a threshold value (e.g., 60%) set during the learning of the object recognition algorithm. If the recognition rate is lower than the threshold value, the object is not recognized and the bounding box is not displayed. However, the threshold value may be adjusted depending on the performance of the object recognition algorithm, and a user interface for adjusting the threshold value is illustrated in FIG. 8.

[0107] The electronic device (100) can display object information and an object recognition rate together in a bounding box (521). For example, the electronic device (100) can display object information "model" and an object recognition rate (not shown) of 85% at the top of a bounding box (521) displayed around a recognized object (product) (it goes without saying that there is no limitation on the location).

[0108] In addition, the electronic device (100) can display a judgment reference line (522) on the image of the second area (520), and identify the process inspection result according to the overall work progress status of the process inspection list when the product passes the judgment reference line (522). For example, the electronic device (100) determines whether the bounding box (521) of the product passes the judgment reference line (522), and when determining that all processes according to the process inspection list of the first area (510) are completed, causes the OK of the first area (510) to light up.

[0109] Meanwhile, in the case of an automatic packaging process, the product may appear in the image of the second area (520) (image acquired by a camera installed in the final process area) after all packaging processes are completed. When the product appears in the image, the electronic device (100) may identify the progress of the work performed in the process prior to the image and display it in the first area (510). For example, if the overall progress of the work is appropriately completed, the completion of all tasks in the process inspection list of the first area (510) may be displayed based on one of the display methods described above.

[0110] The electronic device (100) can display the process inspection results based on the overall work progress status of the first area (510) when the product passes the judgment reference line (522) in the image of the second area (520).

[0111] FIG. 6 is a drawing illustrating a user interface according to a second embodiment of the present invention.

[0112] The same content described with reference to Figure 5 above is adopted, and any overlapping content is omitted for explanation.

[0113] The user interface (600) of FIG. 6 is composed of a first area (610) that displays a process inspection list and process inspection results of products input into the process, a second area (620) that displays a real-time video of the process progress, a third area (630) that displays specific details about the progress of the process inspection list, a fourth area (640) that displays a function implemented to set the display conditions of the video, a fifth area (650) that displays product information, and a sixth area (660) that displays other basic settings.

[0114] The electronic device (100) can display specific details regarding the progress status of the process inspection list in the third area (630). The specific details are contents displayed in real time regarding the progress status of the process. For example, after the electronic device (100) confirms that four seals are attached to the product using an object recognition algorithm, "Confirmation of attachment of four seals" can be displayed in the third area (630). Then, after confirming box forming, "Confirmation of box forming" can be displayed in the third area (630). Although the third area (630) illustrated displays specific details in text format, it is obvious that the display method and form are not limited to any one.

[0115] The electronic device (100) may display a function implemented to set the display conditions of an image in the fourth area (640). The display conditions of the image may include a function for reversing the image up, down, left, and right, a function for adjusting the position of a judgment reference line (process reference line), and a function for adjusting the range of a judgment area (an area including the process reference line). In addition, a function for initializing the display conditions of the image, a recording function, and a function for displaying information related to program execution (## Program Start ##, etc.) may be included.

[0116] The electronic device (100) can display final OK, NG in the OK / NG reading section of the first area (610) when the final product passes the judgment reference line (622).

[0117] If the electronic device (100) determines that the process inspection result is an error (NG), it may display notification information (not shown) indicating the error on the user interface. The notification information may be displayed in the form of a pop-up window, but may also be displayed in various other ways.

[0118] Additionally, if the electronic device (100) determines that the process inspection result is an error (NG), it can record the situation before / after the error occurs based on the time of occurrence. At this time, the electronic device (100) can provide a function to set the video recording range in seconds.

[0119] The electronic device (100) can be returned to its initial state when it receives a user input requesting initialization after the defect-related rework is completed.

[0120] The electronic device (100) can display product information in the fifth area (650). The electronic device (100) can identify product information corresponding to the product code of the product.

[0121] Product information may include the name of the process line that produces the product, communication information with the equipment control device (320), information read from the product database or storage unit (140), and product code (ITEMCODE), product description (DESCRIPTION), color of spare parts (SPAREPART), etc.

[0122] Communication information may include the serial port number (e.g., COM 1) and connection status (e.g., Connect) of the equipment control device (320) that is connected to the electronic device (100) via serial communication (e.g., RS232). Through this, the current connection status between the electronic device (100) and the equipment control device (320) may be displayed.

[0123] The electronic device (100) can receive a product code from the equipment control device (320) and transmit the process inspection results to the equipment control device (320). The equipment control device (320) can control a stopper for preventing shipment based on the process inspection results. In addition, the electronic device (100) can transmit a request signal to the equipment control device that controls the corresponding equipment module to prevent shipment of the product.

[0124] The electronic device (100) can display other basic settings in the sixth area (660). Specifically, when the "DB" button is pressed, the storage (140) of the electronic device (100) or a separately constructed product database can be accessed to access the DB settings window. At this time, a separate login may be required to access the storage (140) or the product database. A specific example of the DB settings window is illustrated in FIG. 7.

[0125] The electronic device (100) can store information about an error that has occurred (date and time of occurrence, product name, type of error (shield not attached, spare part not attached), etc.), and can open a folder of the corresponding path when the "NG LOG" button is entered.

[0126] When the “Start Process” and “End Process” buttons are pressed, the electronic device (100) can start and end the object recognition algorithm for process inspection.

[0127] When the "SETTING" button is pressed, the electronic device (100) can open the setting window of the process inspection program.

[0128] When the “SERIAL” button is pressed, the electronic device (100) can open the serial communication setting window of the process inspection program.

[0129] When the “CLOSE” button is pressed, the electronic device (100) can terminate the process inspection program.

[0130] FIG. 7 is a drawing illustrating a user interface according to a third embodiment of the present invention.

[0131] The user interface (700) of FIG. 7 illustrates a user interface that appears when the “DB” button is entered in the sixth area (660) of FIG. 6.

[0132] The user interface (700) includes an area (710) for displaying product information, an area (720) for displaying a product list, and an area (730) for displaying a DB setting function.

[0133] The area (710) displaying product information is similar to the information displayed in the fifth area (650) of FIG. 6, and may include product code (ITEMCODE), product description (DESCRIPTION), product (PRODUCT), color of spare parts (SPAREPART), etc.

[0134] The electronic device (100) can output product information read from the storage unit (140) or the product database (hereinafter, both cases are collectively referred to as DB) in a table format.

[0135] The area (730) displaying the DB setting function may be provided with buttons for performing functions such as inquiry, save, search, delete, item selection, PW change, and close.

[0136] The electronic device (100) can search for product information in the DB (search), store new product information in the DB (save), search for product information in the DB (search), delete product information in the DB (delete), manually select a product code to be used in the process among product information in the DB (item selection), or change the password of a user who can access the DB (password change) based on a user input that presses the button.

[0137] If the inserted parts are different for each product code, the electronic device (100) can modify and store the DB for the corresponding contents based on user input.

[0138] FIG. 8 is a drawing illustrating a user interface according to a fourth embodiment of the present invention.

[0139] The user interface (800) of FIG. 8 illustrates a user interface that appears when the “SETTING” button is input in the sixth area (660) of FIG. 6.

[0140] The user interface (800) includes a storage path and image display condition setting area (810), an area for adjusting the reference value of the object recognition algorithm (820), and an other button area (830). In each area, the electronic device may operate according to user input, but is not limited thereto.

[0141] Referring to area (810), the electronic device (100) can set a database storage path and an error log storage path. In addition, the electronic device (100) can set the display conditions of the image in the same manner as the fourth area (640) of FIG. 6.

[0142] Referring to the area (820), the electronic device (100) can set the number of frames of the image captured by the camera (process frame count) and the number of frames for determining that the object has been recognized (process OK count). For example, in the box recognition process, if the number of frames recognized as boxes is 10 or more out of 20 frames captured by the camera in the process, the box is considered to have been recognized (OK). In other words, this means that the standard value for box recognition is set to 50%. The electronic device (100) can adjust the standard value by adjusting the process frame count and the process OK count.

[0143] At this time, the area (820) can have setting items determined based on the process inspection list corresponding to the product.

[0144] Referring to the area (830), the electronic device (100) can initialize the settings in the corresponding settings window, save the modified settings, or close the settings window.

[0145] FIG. 9 is a drawing illustrating a user interface according to a fifth embodiment of the present invention.

[0146] The user interface (900) of FIG. 9 can be implemented to set matters related to SERIAL communication.

[0147] The user interface (900) displays serial port number, serial communication speed, data bits, parity bits, stop bits, etc., and can perform a serial communication test to establish a communication connection between the equipment control device (320) and the electronic device (100).

[0148] The electronic device (100) can change the port of the serial cable connected to the equipment control device (320) or change the port number, etc. through the user interface (900), and can test the results of message reception / transmission with the equipment control device (320).

[0149] The electronic device (100) can establish serial communication with the equipment control device (320) based on a user input of pressing the Connect button, and if connected, the button changes to Disconnect. Conversely, the electronic device (100) can release serial communication with the equipment control device (320) based on a user input of pressing the Disconnect button, and if released, the button changes to Connect.

[0150] While the present invention has been described in detail above through representative examples, those skilled in the art will appreciate that various modifications and equivalent embodiments are possible. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the appended claims.

[0151] The present invention relates to an electronic device providing a user interface for process inspection and a process inspection method using the same. Since the present invention can provide an electronic device providing a user interface for process inspection to prevent defects that may occur during a process and a process inspection method using the same, it has industrial applicability.

Claims

1. In electronic devices, An electronic device including a processor that provides a user interface for process inspection, including a first area displaying a process inspection list and process inspection results of products input into the process, and a second area displaying a real-time video of the process progress.

2. In paragraph 1, The above processor, Using an object recognition algorithm, check whether an object corresponding to the process inspection list is recognized in the image, An electronic device that displays a bounding box, object information, and object recognition rate for the recognized object in the image.

3. In paragraph 2, The above processor, An electronic device that displays the progress status of each task in the process inspection list based on the object recognition results in the first area.

4. In paragraph 3, The above processor, Mark the judgment criteria on the above video, An electronic device that identifies the process inspection results according to the overall work progress status of the process inspection list when the product passes the judgment criteria.

5. In paragraph 1, The above processor, If the above process inspection result is determined to be an error, notification information indicating the error is displayed on the user interface, An electronic device that transmits a request signal to a facility control device that controls the facility module to prevent shipment of the above product.

6. In paragraph 1, The above processor, An electronic device that displays specific details about the progress status of the above process inspection list in the third area of ​​the above user interface.

7. In paragraph 1, The above processor, An electronic device that displays a function implemented to set display conditions of the above image in a fourth area of ​​the user interface.

8. In electronic devices, A display showing a user interface for process inspection consisting of multiple areas; A process inspection list identified based on the product code of the product input into the process is displayed in the first area of ​​the user interface, A video of the process progress in real time is displayed in the second area of ​​the user interface. Using an object recognition algorithm corresponding to the above product, check whether an object corresponding to the process inspection list in the above image is recognized, An electronic device including a processor that displays the progress status of the process inspection list and the process inspection results in the first area according to the object recognition results.

9. In paragraph 8, The above processor, An electronic device that identifies an object recognition algorithm corresponding to the product among object recognition algorithms provided for each product code.

10. In paragraph 8, The above processor, Mark the judgment criteria on the above video, An electronic device that identifies the process inspection results according to the overall work progress status of the process inspection list when the product passes the judgment criteria.

11. In paragraph 8, The above processor, An electronic device that transmits a request signal to a facility control device that controls the facility module to prevent shipment of the product if the above process inspection result is determined to be erroneous.

12. In a process inspection method performed by an electronic device, A step of displaying a process inspection list identified based on the product code of a product input into the process in a first area of ​​a user interface for process inspection; A step of displaying a video of the process progress in real time in the second area of ​​the user interface; A step of checking whether an object corresponding to the process inspection list in the image is recognized using an object recognition algorithm corresponding to the product; A process inspection method comprising a step of displaying the progress status and process inspection results of the process inspection list in the first area according to the object recognition results.

13. In paragraph 12, The above inspection steps are: A step of displaying a judgment reference line on the above image; A process inspection method comprising a step of identifying the process inspection results according to the overall work progress status of the process inspection list when the product passes the judgment criterion.

14. In paragraph 12, The above inspection steps are: A process inspection method comprising the step of transmitting a request signal to a facility control device that controls the facility module to prevent shipment of the product if the process inspection result is determined to be erroneous.

15. In electronic devices, Displays real-time video footage of the progress of products introduced into the process. Using an object recognition algorithm corresponding to the above product, check whether an object corresponding to the process inspection list in the image is recognized, An electronic device including a processor that identifies the process inspection results according to the overall work progress status of the process inspection list when the product passes the judgment criteria indicated in the image.

Images