Product maintenance method and system, control device, storage medium and program product

Abstract

The invention discloses a product maintenance method and system, a control device, a storage medium and a program product. The method comprises the steps that the control device detects whether a to-be-maintained product is in place or not; when the to-be-maintained product is in place, the control device obtains a defect list, and the defect list comprises one or more defect positions; for each defect position, the control device detects a defect type corresponding to the defect position, obtains a predefined maintenance process corresponding to the defect type according to the detected defect type, executes the maintenance process, and detects whether the defect position still has a defect after maintenance; when the defect position still has the defect after maintenance, stopping the maintenance process and giving an alarm; wherein the maintenance process comprises one or more operations for maintenance software, and the maintenance software is used for controlling a maintenance device to maintain the to-be-maintained product.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to, but are not limited to, the field of computer technology, and in particular to a product maintenance method and system, a control device, a storage medium, and a program product. Background Art

[0002] In the production process of liquid crystal display (LCD), many processes involve manufacturing defect repair. For example, in the color filter (CF) process section, the traditional working method is that the defective glass plate carries the defect list identified by the upstream automatic optical inspection (AOI) equipment into the maintenance workshop, and the maintenance software matched with the maintenance robot arm enters the glass plate into the (GlassIn) state. The maintenance personnel click on each defect in the maintenance software in turn, identify the defect type based on experience, click the button or shortcut key of the maintenance software, and control the maintenance robot arm to perform the maintenance operation.

[0003] Some maintenance robot arm suppliers provide built-in automated maintenance solutions in maintenance software. The implementation method is: modify the existing maintenance software code, use image recognition algorithms in the code to identify defect types, and call interfaces to drive maintenance robot arms for maintenance. However, this implementation method of providing automated maintenance solutions is only applicable to maintenance robot arms of a certain supplier and is not universal. Factories often have maintenance robot arms from multiple suppliers. If each supplier implements automated maintenance by modifying the code, the cost and risk are high. In addition, when adding new maintenance processes, this method requires modifying the maintenance software code and performing system testing, which has a long cycle, high risks, and is difficult to expand. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The present disclosure provides a product maintenance method, including:

[0006] The control device detects whether the product to be repaired is in place;

[0007] When the product to be repaired is in place, the control device obtains a defect list, the defect list including one or more defect locations;

[0008] For each defect location, the control device detects the defect type corresponding to the defect location, obtains a predefined maintenance process corresponding to the defect type according to the detected defect type, executes the maintenance process, and detects whether the defect location is still defective after the maintenance. When the defect location is still defective after the maintenance, the maintenance process is terminated and an alarm is issued; wherein, the maintenance process includes one or more operations for maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired.

[0009] An embodiment of the present disclosure also provides a control device, including a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the product repair method described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0010] The embodiments of the present disclosure further provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the product repair method described in any embodiment of the present disclosure is implemented.

[0011] An embodiment of the present disclosure further provides a program product, comprising instructions. When the computer program product is executed by a computer, the instructions execute the product maintenance method as described in any embodiment of the present disclosure.

[0012] The embodiment of the present disclosure also provides a control device, including a maintenance engine module, a maintenance processing module and a maintenance process module, wherein:

[0013] The maintenance process module is configured to define corresponding maintenance processes for one or more defect types;

[0014] The maintenance engine module is configured to detect whether the product to be repaired is in place; when the product to be repaired is in place, obtain a defect list, the defect list includes one or more defect locations; for each defect location, detect the defect type corresponding to the defect location, obtain a predefined maintenance process corresponding to the defect type according to the detected defect type, call the maintenance processing module to execute the maintenance process, and detect whether the defect location still has defects after the maintenance, and when the defect location still has defects after the maintenance, terminate the maintenance process and issue an alarm; wherein the maintenance process includes one or more operations for maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired;

[0015] The maintenance processing module is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module.

[0016] The embodiment of the present disclosure also provides a product maintenance system, including an automatic maintenance server, an automatic maintenance client, and a maintenance device, wherein:

[0017] The automatic maintenance server includes a maintenance engine module, a maintenance processing module and a maintenance process module, wherein the maintenance process module is configured to define corresponding maintenance processes for one or more defect types; the maintenance engine module is configured to call the execution function encapsulated by the maintenance processing module to drive maintenance according to the maintenance process defined by the maintenance process module; the maintenance processing module is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module;

[0018] The automatic maintenance client includes an automated UI maintenance module and an RPC maintenance processing module, wherein the automated UI maintenance module is configured to encapsulate operations on a maintenance software interface, the maintenance software being provided on a host computer connected to the maintenance device, and being used to control the maintenance device to repair the product to be repaired; the RPC maintenance processing module is configured to encapsulate one or more maintenance business-related execution functions, each execution function including one or more operations on the maintenance software interface.

[0019] The product maintenance method and system, control device, storage medium and program product of the embodiments of the present disclosure detect whether the product to be repaired is in place through the control device; when the product to be repaired is in place, the control device obtains a defect list, which includes one or more defect locations; for each defect location, the control device detects the defect type corresponding to the defect location, obtains a predefined maintenance process corresponding to the defect type according to the detected defect type, executes the maintenance process, and detects whether the defect location is still defective after the maintenance, and when the defect location is still defective after the maintenance, the maintenance process is terminated and an alarm is issued; wherein the maintenance process includes one or more operations for maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired, so that the effect of realizing automated maintenance without modifying the original maintenance software code is achieved. When a new defect is found, it is only necessary to define a new corresponding maintenance process without modifying the original maintenance software code. Therefore, the cycle is short, the cost is low, the risk is low, and the scalability is strong.

[0020] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. Other advantages of the present disclosure can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide an understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0022] Figure 1 A flowchart of a product maintenance method provided for an exemplary embodiment of the present disclosure;

[0023] Figure 2 A structural schematic diagram of a product maintenance system provided for an exemplary embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of the structure of another product maintenance system provided by an exemplary embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a communication method between an MQTT server and an MQTT client;

[0026] Figure 5 A schematic diagram of the structure of an automatic maintenance server provided by an exemplary embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of a defect position in a video stream window provided by an exemplary embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of a defect position in a defect image provided by an exemplary embodiment of the present disclosure;

[0029] Figure 8 A schematic diagram of the structure of an automatic maintenance client provided by an exemplary embodiment of the present disclosure;

[0030] Fig. 9 A schematic diagram of an automatic maintenance server remotely calling a measurement and grinding operation of an automatic maintenance client provided by an exemplary embodiment of the present disclosure;

[0031] Fig.10 for Fig. 9 The MQTT data flow diagram corresponding to the remote call shown;

[0032] Fig.11 A structural schematic diagram of another product maintenance system provided for an exemplary embodiment of the present disclosure;

[0033] Fig.12 A schematic diagram of an interface of an automatic maintenance client provided by an exemplary embodiment of the present disclosure;

[0034] Fig.13 A schematic diagram of a system startup process provided for an exemplary embodiment of the present disclosure;

[0035] Fig.14 A schematic diagram of an automatic maintenance process provided for an exemplary embodiment of the present disclosure;

[0036] Fig.15 A schematic diagram of the structure of a control device provided for an exemplary embodiment of the present disclosure;

[0037] Fig.16 A schematic structural diagram of another control device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is apparent to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0039] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features, and elements disclosed in the present disclosure may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present disclosure may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other limitations. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0040] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be appreciated by those of ordinary skill in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.

[0041] like Figure 1 As shown, the embodiment of the present disclosure provides a product maintenance method, including:

[0042] Step 101, the control device detects whether the product to be repaired is in place;

[0043] Step 102: When the product to be repaired is in place, the control device obtains a defect list, which includes one or more defect locations;

[0044] Step 103, for each defect location, the control device detects the defect type corresponding to the defect location, obtains a pre-defined maintenance process corresponding to the defect type according to the detected defect type, executes the maintenance process, and detects whether the defect location is still defective after the maintenance. When the defect location is still defective after the maintenance, the maintenance process is terminated and an alarm is issued; wherein each maintenance process corresponding to the defect type includes one or more operations for the maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired.

[0045] The product repair method provided by the embodiment of the present disclosure detects the defect type corresponding to each defect location, obtains a pre-defined repair process corresponding to the defect type according to the detected defect type, executes the repair process, and detects whether the defect location is still defective after the repair. When the defect location is still defective after the repair, the repair process is terminated and an alarm is issued, thereby achieving the effect of realizing automated repair without modifying the original repair software code. When a new defect is found, only a new corresponding repair process needs to be defined, and there is no need to modify the original repair software code. Therefore, the cycle is short, the cost is low, the risk is low, and the scalability is strong.

[0046] The use scenario of the product repair method of the embodiment of the present disclosure is not limited to the LCD manufacturing process or the CF process, and can be applied to the repair of products of many different types or different process stages.

[0047] The CF process is a very important part of LCD production. This process is responsible for adding red, green and blue filters to the pixel array to achieve color display. In the CF process, a special color resin (usually red, green and blue) is used to accurately deposit the color resin on each pixel through photolithography, development and other steps to form a structure we call a color filter. Each pixel has three sub-pixels, red, green and blue, and blocks or passes light of a specific color by vibrating the liquid crystal elements to achieve color display. The CF process usually includes three process segments: Black Matrix (BM), Color Layer (COLOR), Photo Spacer (PS).

[0048] In some exemplary embodiments, Figure 2As shown, the maintenance software is set on a host computer (not shown) connected to the maintenance device, and the control device can be directly connected to the host computer where the maintenance software is located.

[0049] In the embodiment of the present disclosure, the maintenance device may be a maintenance robot arm, however, the embodiment of the present disclosure is not limited thereto. The maintenance device is usually provided with a camera, which can be used to take a picture of the product to be repaired and detect the status of the product to be repaired accordingly. Therefore, in step 101, the control device can detect whether the product to be repaired is in place through the camera on the maintenance device.

[0050] In the embodiment of the present disclosure, in step 102, the control device may obtain a defect list of the product to be repaired through the repair software. The defect list may be obtained by detection by an upstream automated optical inspection (AOI) device.

[0051] AOI equipment is widely used in various electronic manufacturing industries to detect the soldering quality and component placement on printed circuit boards (PCBs) to ensure that there are no defects such as missing solder joints, incorrect polarity placement, and component misalignment. Its operating principle is to capture the image of the board through a high-definition camera and use image processing and analysis software to compare it with the pre-defined correct parameters to automatically detect any non-compliant areas. AOI equipment can greatly improve the efficiency and accuracy of inspection and reduce the pressure of manual inspection. In LCD production, AOI equipment is often used to check various defects on the panel to ensure product quality. After the AOI equipment detects and obtains the defect list, it sends the defect list to the host computer where the repair software is located through the network.

[0052] Normally, the defect list provided by the AOI device only includes the approximate location of one or more defects, but not the specific defect type. The traditional repair method is to manually click on the defects in the defect list, manually determine the defect type for each defect, and decide how to repair it. The disclosed embodiment detects the defect type corresponding to the defect location through a control device, and then uses a pre-defined repair process corresponding to the defect type to perform automated repair.

[0053] In the embodiments of the present disclosure, each maintenance process corresponding to a defect type includes one or more operations on the user interface of the maintenance software. For example, the user interface of the maintenance software may include interface elements such as windows, buttons, keys, scroll bars, and input boxes. Operations on the user interface of the maintenance software may include obtaining window content, pressing buttons, clicking keys, dragging scroll bars, and entering parameters in input boxes. The "window" in the embodiments of the present disclosure is a broad concept. A rectangular or rectangular-shaped area on the user interface of the maintenance software may be called a window. For example, a window may include: a pop-up window, a title bar, a menu bar, a toolbar, a status bar, and the like.

[0054] In other exemplary embodiments, Figure 3 As shown, the maintenance software is set on a host computer (not shown) connected to the maintenance device, and the control device includes an automatic maintenance server (AutoRepairServer) and an automatic maintenance client (AutoRepairClient). The automatic maintenance server and the host computer are respectively connected to the automatic maintenance client. The automatic maintenance server notifies the automatic maintenance client to execute one or more operations for the maintenance software through a remote procedure call (Remote Procedure Call, RPC) to control the maintenance device to repair the product to be repaired.

[0055] In the disclosed embodiment, each automatic maintenance client is connected to a maintenance device (and the maintenance software corresponding to the maintenance device), and an automatic maintenance server can be connected to multiple automatic maintenance clients. By setting up an automatic maintenance server and an automatic maintenance client, the disclosed embodiment improves the flexibility and scalability of the system and reduces the computing power requirements of the automatic maintenance client.

[0056] In some exemplary embodiments, a message channel may be established between the automatic maintenance server and the automatic maintenance client via a message server, and both the automatic maintenance server and the automatic maintenance client may be message subscribers.

[0057] For example, the message server may be a message queuing telemetry transport protocol (MQTT) server, which is also called an MQTT broker. It may be a server or a server cluster running MQTT server software. Figure 4As shown in the figure, the MQTT server is responsible for receiving network connections from MQTT clients (message subscribers), processing the subscription / unsubscription and message publishing requests of MQTT clients, and forwarding messages published by MQTT clients to other subscribers. MQTT servers are widely used in industries such as electricity, new energy, smart cities, smart homes, smart meter reading, Internet of Vehicles, finance and payment, and operators.

[0058] Publish-subscribe is a messaging paradigm where the sender of a message (called a publisher) does not send the message directly to a specific receiver (called a subscriber). Instead, the published messages are divided into different categories, so the publisher does not need to know which subscribers (if any) may exist. Similarly, subscribers can express interest in one or more categories and only receive messages of interest, without having to know which publishers exist.

[0059] In the embodiment of the present disclosure, the automatic maintenance server and the MQTT server may be set on one computer. However, the present disclosure is not limited to this. The automatic maintenance server and the MQTT server may also be set on different computers.

[0060] In the embodiment of the present disclosure, the automatic maintenance server and the automatic maintenance client may also be connected via a computer network and communicate with each other via the Transmission Control / Internet Protocol (TCP / IP) protocol, which is not limited in the present disclosure.

[0061] In some exemplary embodiments, the automatic maintenance server may include a maintenance engine module, a maintenance processing module, and a maintenance process module, wherein:

[0062] A maintenance process module is configured to define corresponding maintenance processes for one or more defect types;

[0063] The maintenance engine module is configured to call the execution function encapsulated by the maintenance processing module to drive maintenance according to the maintenance process defined by the maintenance process module;

[0064] The maintenance processing module is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module.

[0065] In the disclosed embodiment, before using the product maintenance method, the operator needs to define a corresponding maintenance process according to the defect type on the automatic maintenance server.

[0066] In some exemplary embodiments, the automatic repair server further includes an AI defect recognition service, the AI ​​defect recognition service includes a pre-trained deep learning model, and the deep learning model is used to recognize the input image to determine whether the input image has defects and the corresponding defect type. The control device detects the defect type corresponding to the defect position through the AI ​​defect recognition service.

[0067] In the embodiments of the present disclosure, the deep learning model may adopt a convolutional neural network (CNN) model or a recurrent neural network (RNN) model, however, the present disclosure is not limited thereto. In an exemplary embodiment, the deep learning model may be a YOLO (You Only Look Once) target detection model.

[0068] In the disclosed embodiment, before using the AI ​​defect recognition service, it is necessary to first collect normal pictures and defect pictures, use a deep learning algorithm (such as CNN, RNN) to train a corresponding deep learning model, and use the deep learning model to identify the specific defect type and location.

[0069] In other exemplary embodiments, the AI ​​defect recognition service may also be provided on the automatic maintenance client, which is not limited by the present disclosure.

[0070] In the disclosed embodiment, the automatic maintenance server calls the remote operation capability provided by the automatic maintenance client according to the defined maintenance process to take screenshots, and calls the AI ​​defect recognition service to identify the defect type and location. After the recognition is completed, the RPC process call operation provided by the automatic maintenance client is called to complete the remote automatic maintenance.

[0071] The disclosed embodiment introduces a deep learning mechanism to identify defect types, allowing customers to collect defect images and train the algorithm themselves to identify new defect types. A new repair process is defined for the new defect type in combination with the repair process module, allowing the system to automatically repair the new defect type.

[0072] In the present disclosure, Figure 5 As shown, the automatic maintenance server may include the following components or modules:

[0073] (1)MQTT client: responsible for encapsulating the connection with the remote MQTT server, disconnection and reconnection, topic subscription, message reception, message reporting, etc.

[0074] (2) Parameter configuration module: responsible for the configuration of different product maintenance equipment and used to provide parameters to the maintenance engine module.

[0075] (3) Repair Process module: responsible for configuring the repair process. Provides repair process definitions for different defects. Uses configuration files to define repair steps, conditional judgments, branch judgments, etc.

[0076] (4) Repair Engine module: According to the definition of the maintenance process module, the runtime executes specific functional steps (functions), and the function of each step is encapsulated in the Repair Handler module.

[0077] (5) Repair Handler module: encapsulates the execution functions required by the maintenance engine module, which are functions related to maintenance business.

[0078] (6) AI defect recognition service: This is the image defect type and defect location recognition and reasoning service interface.

[0079] The input of the AI ​​defect recognition service is defect image data, and the output is the defect type and specific defect location identified by the deep learning model. An exemplary deep learning model output is:

[0080] {

[0081] "imageSize":[1024,1024],

[0082] "defectCode":"PS-DEFECT-NORMAL",

[0083] "defectDescription":"General foreign matter",

[0084] "defectCodeL2":"",

[0085] "defectDescriptionL2":"",

[0086] "defectCenter":[124,345],

[0087] "defectPositions":[[50,50,5,5],[100,100,6,6]]

[0088] }

[0089] Among them, imageSize indicates the size of the uploaded image, [1024,1024] is the width and height information of the image; defectCode indicates the string of the defect code; defectDescription is used to describe the defect type; defectCodeL2 indicates the code of the specific defect subtype, such as fiber type, scratch, etc. Some defect types cannot be repaired, in which case the defect subtype needs to be set to NG (Not Good) on the repair software. defectDescriptionL2: used to describe the defect subtype; defectCenter indicates the defect center point (relative to the image size), which is used for double-clicking to locate before repair; defectPositions is used to describe the defect position, which may be one or more, and each defect position is a relative position relative to the image size (imageSize), [upper left corner X, upper left corner Y, width, height].

[0090] In some exemplary embodiments, the automatic maintenance server and / or the automatic maintenance client further includes a coordinate conversion module, wherein the coordinate conversion module is configured to convert the coordinates of the defect position between a screen coordinate system and a camera coordinate system.

[0091] Since some maintenance operations such as inkjet (INK) and manual grinding need to drag an area position on the interface and select the parameter point "execute", the automatic maintenance server and / or automatic maintenance client need to convert the image coordinates into screen coordinates for operation.

[0092] The aspect ratio of the image received by the AI ​​defect recognition service must be consistent with the video stream window, so that the position coordinates returned by the AI ​​defect recognition service are relative coordinates based on the image (i.e., the defect image), and the automatic maintenance server and / or automatic maintenance client performs screen coordinate conversion.

[0093] like Figure 6 and Figure 7 As shown, assuming that the window width and window height are windowWidth and windowHeight respectively, the image width (pixel width of the image) and image height (pixel height of the image) are imageWidth and imageHeight respectively, in the disclosed embodiment, the window refers to the window that displays the image (defective image). It is known that the aspect ratio of the window and the image is consistent. Let the coordinates of the coordinate point on the image be (x, y), we need to calculate the corresponding coordinates of the point in the window (x1, y1) and its corresponding coordinates on the screen (x2, y2), the calculation method is as follows:

[0094] (1) Calculate the aspect ratio of the window and image:

[0095] (2) Calculate window coordinates: x1 = x*widthRatio; y1 = y*HeightRatio;

[0096] (3) Use the coordinate conversion function to convert the window coordinates (x1, y1) into screen coordinates (x2, y2) and simulate mouse operations on (x2, y2).

[0097] In the present disclosure, Figure 7 As shown, the window coordinates (x1, y1) are the coordinates with the upper left corner of the window as the coordinate origin (0, 0), the screen coordinates (x2, y2) are the coordinates with the upper left corner of the screen as the coordinate origin (0, 0), and the coordinates of the mouse operation are screen coordinates. Therefore, the coordinates of the coordinate points on the window need to be converted to screen coordinates. This coordinate conversion can be achieved through the coordinate conversion function.

[0098] In some exemplary embodiments, the automatic maintenance client includes an automated UI maintenance module and an RPC maintenance processing module, wherein the automated UI maintenance module is configured to encapsulate operations on a maintenance software interface; the RPC maintenance processing module is configured to encapsulate one or more maintenance business-related execution functions, each execution function including one or more operations on the maintenance software interface.

[0099] In the present disclosure, Figure 8 As shown, the automatic maintenance client may include the following components or modules:

[0100] (1)MQTT Client: This module is not related to the business and is mainly used to realize connection with the remote MQTTBroker, reconnection after disconnection, TOPIC subscription, message reception, message distribution to the RPC repair processing (RepairRPCHandler) module, message reporting, etc.

[0101] (2) RPC Repair Processing (RepairRPCHandler) module: This module encapsulates functions related to the repair business, combines basic calls to the automated UI repair (RepairUIAutomation) module, and returns the results through the MQTT client reporting interface after the operation is completed. The RPC Repair Processing module is responsible for processing business-related information.

[0102] (3) RepairUIAutomation module: This module is independent of the business and encapsulates operations on window elements, button operations, mouse operations, and possibly file operations on the maintenance software interface. The parameters of this module include element ID, text, file path, etc. This module does not need to know specific business information.

[0103] In the embodiment of the present disclosure, the RPC maintenance processing module includes a combined call to the operations of one or more maintenance software interfaces encapsulated by the automated UI maintenance module, and may also include other business logic, such as defect judgment, etc. In the embodiment of the present disclosure, the automated UI maintenance module is irrelevant to the business, and its encapsulated operations include clicking a button, moving the mouse, obtaining window content, etc.; the RPC maintenance processing module encapsulates functions related to the maintenance business, for example, maintenance defect 1 includes operations of one or more maintenance software interfaces, maintenance defect 2 includes operations of one or more maintenance software interfaces, etc.

[0104] In the embodiment of the present disclosure, the automatic maintenance server and the automatic maintenance client communicate by calling the MQTT protocol. The topic setting rules of the MQTT server are as follows:

[0105] Uplink topic (automatic maintenance client to automatic maintenance server, hereinafter referred to as client to server): up / $SN / #;

[0106] Downlink topic (from automatic maintenance server to automatic maintenance client, hereinafter referred to as server to client): down / $SN / #;

[0107] Among them, SN is the device serial number, and the MQTT client ID and the device serial number are unique identifiers of the device. The MQTT client ID and the device serial number may be the same or different, and the present disclosure does not impose any restrictions on this.

[0108] The MQTT topic description is shown in Table 1.

[0109]

[0110] Table 1

[0111] In the disclosed embodiment, no matter the automatic maintenance client loses power, is closed, or is powered off, the MQTT client will send a last wish message to the MQTT server.

[0112] The definitions of the topic (TOPIC) and payload (Payload) related to the remote operation (Method) are shown in Table 2. The payload includes down payload (down payload) and up payload (up payload). The terminal receives the corresponding remote call TOPIC: down / $SN / rpc / request / $id / $Method payload data, and uses up / $SN / rpc / response / $id / $Method TOPIC to reply data after execution.

[0113] Table 2 takes the grinding operation corresponding to the conventional foreign body defect as an example to illustrate the topics and loads related to the remote operation. In actual application, the defect can also be any other type of defect, and the present disclosure does not limit this.

[0114]

[0115]

[0116]

[0117]

[0118] Table 2

[0119] Robotic arms from different manufacturers have corresponding host computer controls, and the corresponding host computer operation buttons may be different. In the disclosed embodiment, the automated UI maintenance module obtains the interface structure of the maintenance software, captures the elements on the interface, obtains the ID of the interface element to be operated, and performs maintenance by operating the corresponding interface elements.

[0120] When there are too many defects on the glass plate (such as exceeding the preset defect threshold) or when the number of repairs reaches the preset repair number threshold and defects still exist, the glass plate is judged to be Not Good (NG). The grinding operation is a maintenance operation performed by the maintenance robot arm for conventional foreign body defects. Grinding refers to grinding the foreign body smaller by the robot arm. When the defect is ground small enough or flat enough (that is, the defect height is lower than the preset height threshold), the repair is considered to have passed. Measure height first and then grind (MeasureLapping) means measuring the height of the foreign body first and then performing the grinding operation; grind first and then measure height (LappingMeasure) means performing the grinding operation first and then measuring the height of the foreign body. When the defect area is relatively large, you can use Scan Area Grinding (Scan LappingMeasure) to set a rectangular area and perform grinding within the rectangular area. At this time, the grinding time will be relatively long.

[0121] The automatic maintenance client synchronizes the status of the operation on the terminal (ie, the automatic maintenance client) to the automatic maintenance server through TOPIC:up / $SN / status / report / $Status. As shown in Table 3, during the automatic operation, the operator can start / pause / resume the maintenance operation on the terminal.

[0122]

[0123] Table 3

[0124] Fig. 9The interaction process between the server and the client is illustrated by taking the automatic repair server (server) calling the automatic grinding MeasureLapping operation of the automatic repair client (client) CF-REPAIR-01 in a stage of automatic repair as an example, where CF-REPAIR-01 is the device number of the automatic repair client. When the terminal is started, it has subscribed to TOPIC:down / CF-REPAIR-01 / rpc / request / #; when the server is started, it has subscribed to TOPIC:up / CF-REPAIR-01 / rpc / response / #.

[0125] like Fig.10 As shown, first, the server assigns a call ID, and the call ID can use the auto-increment logic, for example, 100; the server's MeasureLapping function uses the MQTT client to send a maintenance message and waits for a return message; the client receives the maintenance message, operates the maintenance software, and publishes a return message after completion; the server's MeasureLapping function receives the return message, parses the message and returns.

[0126] Fig.11 A structural diagram of a product maintenance system provided by an embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the product maintenance system includes an automatic maintenance server and one or more automatic maintenance clients. Each automatic maintenance client is connected to a maintenance software. The automatic maintenance server establishes one or more maintenance objects according to the number of automatic maintenance clients connected to itself, and communicates with the automatic maintenance client through the MQTT service.

[0127] In some examples, the product maintenance system performs an automatic maintenance process including the following steps:

[0128] (1) The maintenance software is started at the same time as the automatic maintenance client is started. In other examples, the maintenance software may not be started from the automatic maintenance client, and the automatic maintenance client establishes a connection with the maintenance software by searching for the main window title of the maintenance software. An exemplary interface diagram of the automatic maintenance client is shown in FIG. Fig.12 shown.

[0129] (2) After the automatic maintenance client is successfully started, it connects to the MQTT server according to the preset MQTT server address, user name, password, and device number as the MQTT client ID.

[0130] (3) The automatic maintenance server receives the online message and creates the corresponding maintenance object.

[0131] (4) The maintenance object loads the defined automatic maintenance process to start automatic maintenance.

[0132] For example, Fig.13 As shown, the client CF-REPAIR-01 comes online, the server creates a maintenance object, and loads the user-defined process.

[0133] like Fig.14 As shown, the maintenance object performs the following automatic maintenance process:

[0134] (I) The server detects whether the glass plate is ready to be loaded by remotely calling the client's interface. If the client returns that it is not ready, the server checks in a loop until it is ready. (In actual implementation, the loop check may wait for a short period of time, such as 0.1 seconds);

[0135] (II) The server obtains the number of defects by remotely calling the client's interface. (The client checks the number of defects in the repair software list control);

[0136] (III) If the number of defects is greater than the preset value, the server remotely calls the client to determine that the glass plate is NG, and remotely calls the client to prepare for the glass removal operation. (The client clicks the GlassOut button and waits for the maintenance robot to move the glass plate away), and then returns to step (I);

[0137] (IV) If the number of defects is less than a preset value, the server remotely calls the client to obtain defect images, and the client intercepts the defect images of the video area and returns them;

[0138] (V) The server calls the AI ​​defect identification service and returns the defect type and location;

[0139] (VI) The server determines the defect type. If there is no defect (indicating that the upstream equipment has passed the inspection), the server calls the next operation of the remote call client (if the next operation returns that there is no next defect, the remote call client prepares to produce the glass);

[0140] (VII) The server performs corresponding operations according to the defect type. The defect type and repair process can be freely defined. Here, a conventional foreign body defect is used as an example to introduce the subsequent branches);

[0141] (VIII) If it is a conventional foreign body, the server remotely calls the client's measurement / lapping (a repair method) operation, and after the repair operation is completed, the defect height after the repair is returned;

[0142] (IX) If the defect height is greater than the predetermined height (repair is not completed), the server continues to remotely call the client's measurement and grinding operation;

[0143] (X) If the defect height is less than the preset height (preliminary repair is completed), the server calls the AI ​​defect recognition service to determine whether the repair is completed. If the repair is completed, the client's Next operation is remotely called. Otherwise, the automatic process is paused, and an alarm dialog box pops up, and a sound is called to notify the operator to intervene manually. After the manual processing is completed, the automatic repair process is resumed;

[0144] (XI) If the defect height is greater than the preset height (repair is not completed) and the number of repairs is greater than the preset number, the automatic process is suspended, and an alarm dialog box pops up, and a sound is called to notify the operator to intervene manually. After the manual processing is completed, the automatic repair process is resumed.

[0145] In the disclosed embodiment, the robot arm software preliminarily determines whether the current defect has been repaired by the height of the foreign object. When it is initially determined that the current defect has been repaired, it then uses the AII defect recognition service to determine whether the repair is complete, which can enhance the accuracy of the repair result.

[0146] In other examples, the automatic maintenance client can only provide basic maintenance software encapsulation operations that are not related to the business, and the automatic maintenance server implements the encapsulation of business functions and remotely calls the basic operations of the automatic maintenance client such as keyboard, mouse, and screenshot operations. This is equivalent to putting more operations on the server side, and the automatic maintenance client has better reusability.

[0147] In some other examples, the process defined on the automatic maintenance server can be sent to the automatic maintenance client for execution. That is, the maintenance process is defined on the automatic maintenance server, and then the maintenance process is sent to the automatic maintenance client, and the automatic maintenance client implements the maintenance engine to automatically execute the process. In this way, during the automatic maintenance process, the automatic maintenance client completes the process-driven operation, the automatic maintenance client and the automatic maintenance server have less interaction, and the timeliness is higher.

[0148] The disclosed embodiment achieves the effect of automated operation by deploying an automated maintenance client on a maintenance operating system and using an automated UI maintenance module to interact with the original maintenance software, thereby enabling automated maintenance to be implemented without modifying the original maintenance software.

[0149] The disclosed embodiment defines a maintenance process according to the defect type in the automatic maintenance server, and realizes the effect of automated maintenance by remotely calling the automatic maintenance client at runtime, thereby solving the problem of users adding new defect processes.

[0150] like Fig.15 As shown, the embodiment of the present disclosure further provides a control device, including a maintenance process module 1510, a maintenance engine module 1520 and a maintenance processing module 1530, wherein:

[0151] A maintenance process module 1510 is configured to define corresponding maintenance processes for one or more defect types;

[0152] The maintenance engine module 1520 is configured to detect whether the product to be repaired is in place; when the product to be repaired is in place, obtain a defect list, which includes one or more defect locations; for each defect location, detect the defect type corresponding to the defect location, obtain a predefined maintenance process corresponding to the defect type according to the detected defect type, call the maintenance processing module 1530 to execute the maintenance process, and detect whether the defect location still has defects after the maintenance. When the defect location still has defects after the maintenance, the maintenance process is terminated and an alarm is issued; wherein the maintenance process includes one or more operations for the maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired;

[0153] The maintenance processing module 1530 is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module 1520.

[0154] In some exemplary embodiments, the maintenance software is provided on a host computer connected to the maintenance device, and the control device is directly connected to the host computer.

[0155] In some exemplary embodiments, the maintenance software is set on a host computer connected to the maintenance device, the control device includes an automatic maintenance server and an automatic maintenance client, the automatic maintenance server and the host computer are respectively connected to the automatic maintenance client, and the automatic maintenance server notifies the automatic maintenance client to execute one or more operations for the maintenance software through remote call to control the maintenance device to repair the product to be repaired.

[0156] In some exemplary embodiments, a message channel is established between the automatic maintenance server and the automatic maintenance client via a message server, and both the automatic maintenance server and the automatic maintenance client are message subscribers.

[0157] In some exemplary embodiments, the automatic maintenance server includes a maintenance engine module 1520, a maintenance processing module 1530, a maintenance process module 1510 and an AI defect recognition service, wherein the AI ​​defect recognition service includes a pre-trained deep learning model, and the deep learning model is used to recognize the input image to determine whether the input image has defects and the corresponding defect type.

[0158] In some exemplary embodiments, the automatic maintenance server and / or the automatic maintenance client comprises a coordinate conversion module, wherein the coordinate conversion module is configured to convert the coordinates of the defect position between a screen coordinate system and a camera coordinate system.

[0159] In some exemplary embodiments, the automatic maintenance client includes an automated UI maintenance module and an RPC maintenance processing module, wherein the automated UI maintenance module is configured to encapsulate operations on the maintenance software interface; and the RPC maintenance processing module is configured to encapsulate one or more maintenance business-related execution functions, each execution function including one or more operations on the maintenance software interface.

[0160] like Figure 3 As shown, the embodiment of the present disclosure also provides a product maintenance system, including an automatic maintenance server, an automatic maintenance client and a maintenance device, wherein:

[0161] The automatic maintenance server includes a maintenance engine module, a maintenance processing module and a maintenance process module, wherein the maintenance process module is configured to define corresponding maintenance processes for one or more defect types; the maintenance engine module is configured to call the execution function encapsulated by the maintenance processing module to drive maintenance according to the maintenance process defined by the maintenance process module; the maintenance processing module is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module;

[0162] The automatic maintenance client includes an automated UI maintenance module and an RPC maintenance processing module, wherein the automated UI maintenance module is configured to encapsulate operations on a maintenance software interface, the maintenance software being provided on a host computer connected to the maintenance device, and being used to control the maintenance device to repair the product to be repaired; the RPC maintenance processing module is configured to encapsulate one or more maintenance business-related execution functions, each execution function including one or more operations on the maintenance software interface.

[0163] An embodiment of the present disclosure also provides a control device, including a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the product repair method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0164] like Fig.16As shown, in one example, the control device may include: a processor 1610, a memory 1620, a bus system 1630 and a transceiver 1640, wherein the processor 1610, the memory 1620 and the transceiver 1640 are connected via the bus system 1630, the memory 1620 is used to store instructions, and the processor 1610 is used to execute the instructions stored in the memory 1620 to control the transceiver 1640 to send and receive signals. Specifically, the transceiver 1640 can obtain a defect list and a defect picture under the control of the processor 1610, and the processor 1610 detects whether the product to be repaired is in place; when the product to be repaired is in place, the defect type corresponding to each defect position in the defect list is detected according to the defect picture, and a pre-defined maintenance process corresponding to the defect type is obtained according to the detected defect type, and a maintenance processing module is called to execute the maintenance process, and it is detected whether the defect position is still defective after the maintenance. When the defect position is still defective after the maintenance, the maintenance process is terminated and an alarm is issued; wherein, the maintenance process includes one or more operations for maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired.

[0165] It should be understood that the processor 1610 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0166] The memory 1620 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1610. A portion of the memory 1620 may also include a nonvolatile random access memory. For example, the memory 1620 may also store information on the device type.

[0167] In addition to the data bus, the bus system 1630 may also include a power bus, a control bus, and a status signal bus. Fig.16 Various buses are labeled as bus system 1630 .

[0168] In the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the processor 1610 or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1620, and the processor 1610 reads the information in the memory 1620 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0169] The embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the product maintenance method described in any embodiment of the present disclosure is implemented. The product maintenance method driven by executing executable instructions is basically the same as the product maintenance method provided in the above embodiment of the present disclosure, and will not be repeated here.

[0170] In some possible implementations, various aspects of the product repair method provided by the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the product repair method according to the various exemplary embodiments of the present disclosure described above in this specification. For example, the computer device may execute the product repair method recorded in the embodiments of the present disclosure.

[0171] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0172] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0173] It should be noted that the above embodiments or implementations are merely illustrative and not restrictive. Therefore, the present disclosure is not limited to what is specifically shown and described herein. Various modifications, substitutions or omissions may be made to the form and details of the implementation without departing from the scope of the present disclosure.

Claims

1. A product maintenance method, characterized in that: include: The control device detects whether the product to be repaired is in place; When the product to be repaired is in place, the control device obtains a defect list, the defect list including one or more defect locations; For each defect location, the control device detects the defect type corresponding to the defect location, obtains a predefined maintenance process corresponding to the defect type according to the detected defect type, executes the maintenance process, and detects whether the defect location is still defective after the maintenance. When the defect location is still defective after the maintenance, the maintenance process is terminated and an alarm is issued; wherein, the maintenance process includes one or more operations for maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired.

2. The method according to claim 1, characterized in that The maintenance software is arranged on a host computer connected to the maintenance device, and the control device is directly connected to the host computer.

3. The method according to claim 1, characterized in that The maintenance software is set on a host computer connected to the maintenance device, and the control device includes an automatic maintenance server and an automatic maintenance client. The automatic maintenance server and the host computer are respectively connected to the automatic maintenance client. The automatic maintenance server notifies the automatic maintenance client to execute one or more operations for the maintenance software through remote calling, so as to control the maintenance device to repair the product to be repaired.

4. The method according to claim 3, characterized in that A message channel is established between the automatic maintenance server and the automatic maintenance client via a message server, and both the automatic maintenance server and the automatic maintenance client are message subscribers.

5. The method according to claim 3, characterized in that: The automatic maintenance server includes a maintenance engine module, a maintenance processing module and a maintenance flow module, wherein: The maintenance process module is configured to define corresponding maintenance processes for one or more defect types; The maintenance engine module is configured to call the execution function encapsulated by the maintenance processing module to drive maintenance according to the maintenance process defined by the maintenance process module; The maintenance processing module is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module.

6. The method according to claim 5, characterized in that The automatic repair server also includes an AI defect recognition service, which includes a pre-trained deep learning model. The deep learning model is used to recognize the input image to determine whether the input image has defects and the corresponding defect type.

7. The method according to claim 6, characterized in that The automatic maintenance server and / or the automatic maintenance client comprises a coordinate conversion module, wherein the coordinate conversion module is configured to convert the coordinates of the defect position between a screen coordinate system and a camera coordinate system.

8. The method according to claim 3, characterized in that The automatic maintenance client includes an automated UI maintenance module and an RPC maintenance processing module, wherein the automated UI maintenance module is configured to encapsulate operations on the maintenance software interface; the RPC maintenance processing module is configured to encapsulate one or more maintenance business-related execution functions, each execution function including one or more operations on the maintenance software interface.

9. A control device, characterized in that: comprising a memory; and a processor connected to the memory, the memory being used to store instructions, the processor being configured to execute the steps of the product maintenance method according to any one of claims 1 to 8 based on the instructions stored in the memory.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the product maintenance method according to any one of claims 1 to 8 is implemented.

11. A computer program product, characterized in that The computer program product comprises instructions, and when the computer program product is executed by a computer, the instructions execute the product maintenance method according to any one of claims 1 to 8.

12. A control device, characterized in that: It includes maintenance engine module, maintenance processing module and maintenance process module, among which: The maintenance process module is configured to define corresponding maintenance processes for one or more defect types; The maintenance engine module is configured to detect whether the product to be repaired is in place; when the product to be repaired is in place, obtain a defect list, the defect list includes one or more defect locations; for each defect location, detect the defect type corresponding to the defect location, obtain a predefined maintenance process corresponding to the defect type according to the detected defect type, call the maintenance processing module to execute the maintenance process, and detect whether the defect location still has defects after the maintenance, and when the defect location still has defects after the maintenance, terminate the maintenance process and issue an alarm; wherein the maintenance process includes one or more operations for maintenance software, and the maintenance software is used to control the maintenance device to repair the product to be repaired; The maintenance processing module is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module.

13. A product maintenance system, characterized in that: It includes an automatic maintenance server, an automatic maintenance client, and a maintenance device, wherein: The automatic maintenance server includes a maintenance engine module, a maintenance processing module and a maintenance process module, wherein the maintenance process module is configured to define corresponding maintenance processes for one or more defect types; the maintenance engine module is configured to call the execution function encapsulated by the maintenance processing module to drive maintenance according to the maintenance process defined by the maintenance process module; the maintenance processing module is configured to encapsulate the execution function corresponding to each execution step of the maintenance engine module; The automatic maintenance client includes an automated UI maintenance module and an RPC maintenance processing module, wherein the automated UI maintenance module is configured to encapsulate operations on a maintenance software interface, the maintenance software being provided on a host computer connected to the maintenance device, and being used to control the maintenance device to repair the product to be repaired; the RPC maintenance processing module is configured to encapsulate one or more maintenance business-related execution functions, each execution function including one or more operations on the maintenance software interface.

Images