Module assembly unit detection method, processing method, and detection processing system

Abstract

The application relates to a detection method, a processing method and a detection processing system of a module assembly unit. The method comprises the following steps: receiving an image of a module assembly unit uploaded by a target camera and identity information of the module assembly unit uploaded by a first target device; uploading the image of the module assembly unit to a target cloud server, receiving a detection result of the module assembly unit issued by the target cloud server; generating a processing strategy of the module assembly unit based on the detection result of the module assembly unit; and storing the identity information of the module assembly unit and the processing strategy of the module assembly unit in a target database table after binding, so that a second target device obtains the processing strategy of the module assembly unit from the target database table according to the identity information of the module assembly unit. The method solves the problem of high cost of the module assembly unit defect detection method in the prior art.

Description

Technical Field

[0001] This application relates to the field of module assembly technology, and in particular to a method for detecting, processing, and a detection and processing system for module component units. Background Technology

[0002] Production lines for modules such as camera modules and communication modules involve multiple processing steps, including cleaning, soldering, mounting, and dispensing. Currently, some processing steps have incorporated AOI (Automated Optical Inspection) equipment. AOI equipment detects defects in module components on the production line, and some AOI equipment has a defect detection function, allowing for the removal of defective module components. However, AOI equipment is expensive, and the number of AOI equipment with defect detection capabilities will continue to increase. Summary of the Invention

[0003] To address the high cost of existing module component unit defect detection methods, this application provides a module component unit detection method, a module component unit processing method, and a module component unit detection and processing system.

[0004] Firstly, this application provides a method for detecting module component units, applied to a supporting MES system. The method includes: receiving an image of the module component unit uploaded by a target camera and identification information of the module component unit uploaded by a first target device, wherein the target camera is a camera on the unloading side of the first target device, and the first target device is a processing and assembly device performing the Nth processing step, where N is a positive integer; uploading the image of the module component unit to a target cloud server; and receiving the detection result of the module component unit issued by the target cloud server, wherein the target cloud server is a cloud server corresponding to the first target device, and the target cloud server is used to receive and store the image of the module component unit and perform... A target detection model is used to verify and detect the stored images. The target detection model is configured to detect the type of appearance defects and / or the size information of the components introduced by the processing steps performed by the first target device. Based on the detection results of the module component unit, a processing strategy for the module component unit is generated. The identification information of the module component unit and the processing strategy of the module component unit are bound and stored in the target database table. The second target device obtains the processing strategy of the module component unit from the target database table according to the identification information of the module component unit. The second target device is a processing and assembly device that performs the N+1th processing step. The target database table is the database table corresponding to the second target device.

[0005] Optionally, the second target device is configured to process only one module component unit at a time. The target detection model is configured to detect the size information of the components introduced by the processing steps performed by the first target device, and bind the identification information of the module component unit and the processing strategy of the module component unit to the target database table. This allows the second target device to obtain the processing strategy of the module component unit from the target database table based on the identification information of the module component unit. This includes binding the identification information of the module component unit, the processing strategy of the module component unit, and the size information of the components in the detection results of the module component unit to the target database table, and then storing these components in the target database table. This allows the second target device to obtain the processing strategy of the module component unit and the size information of the components in the detection results of the module component unit from the target database table based on the identification information of the module component unit.

[0006] Optionally, the second target device is configured to process only one module component unit at a time. Based on the detection result of the module component unit, a processing strategy for the module component unit is generated, including: based on the detection result of the module component unit and the operating condition of the second target device, generating a processing strategy for the module component unit, wherein the operating condition is one of the following: supports the removal of the module component unit, does not support the removal of the module component unit, and the processing strategy for the module component unit is one of the following: process, skip, rework, intercept and scrap, wherein processing means processing the module component unit, skipping means not processing the module component unit, rework means sending the module component unit back to the first target device for reprocessing, and intercept and scrap means removing the module component unit and placing it in the throwing tray.

[0007] Optionally, the second target device is configured to process one module component unit on a vehicle at a time. The identification information of the module component unit is the acupoint coordinate identifier of the module component unit. Based on the detection results of the module component unit, a processing strategy for the module component unit is generated. The identification information of the module component unit and the processing strategy of the module component unit are bound and stored in a target database table. The second target device then retrieves the processing strategy of the module component unit from the target database table based on the identification information of the module component unit, including: based on the detection results of each module component unit in the vehicle... Based on the test results, a processing strategy for the module component unit is generated. The processing strategy for the module component unit is one of the following: processing or skipping. Processing means processing the module component unit, and skipping means not processing the module component unit. The vehicle's identification information, the acupoint identifier of each module component unit in the vehicle, and the processing strategy of the module component unit are bound and stored in the target database table. The second target device then retrieves the processing strategy of the module component unit from the target database table based on the vehicle's identification and the acupoint identifier of each module component unit in the vehicle.

[0008] Optionally, the second target device is configured to process only one module component unit at a time, and the target detection model is further configured to detect the level of appearance defect type introduced by the processing procedure performed by the first target device. After generating a processing strategy for the module component unit based on the detection results of the module component unit, and binding the identity information of the module component unit and the processing strategy of the module component unit to the target database table, the method further includes: binding the level of appearance defect type in the detection results of the module component unit to the identity information of the module component unit.

[0009] Secondly, this application provides a method for processing a module component unit. The method is applied to a second target device, which is the processing and assembly equipment performing the (N+1)th processing step, where N is a positive integer. The method includes: obtaining identification information of the module component unit; obtaining a processing strategy for the module component unit from a target database table based at least on the identification information of the module component unit; and executing the processing strategy of the module component unit. The target database table is the database table corresponding to the second target device. The processing strategy of the module component unit is generated by a supporting MES system based on the detection results of the module component unit and is implemented by the supporting MES system. After being bound to the identity information of the module component unit, it is stored in the target database table. The detection result of the module component unit is sent to the supporting MES system by the target cloud server. The target cloud server is the cloud server corresponding to the first target device. The target cloud server is used to receive and store the image of the module component unit uploaded by the supporting MES system and run the target detection model to verify and detect the stored image. The target detection model is configured to detect the appearance defect type and / or the size information of the components introduced by the processing steps performed by the first target device. The first target device is the processing and assembly equipment that performs the Nth processing step.

[0010] Optionally, the second target device is configured to process only one module component unit at a time, and the target detection model is configured to detect the size information of the components introduced by the processing steps performed by the first target device, and at least obtain the processing strategy of the module component unit from the target database table based on the identity information of the module component unit, and execute the processing strategy of the module component unit, including: obtaining the processing strategy of the module component unit and the size information in the detection result of the module component unit from the target database table based on the identity information of the module component unit; and executing the processing strategy of the module component unit based on the size information in the detection result of the module component unit.

[0011] Optionally, the second target device is configured to process one module component unit on a vehicle at a time. The identification information of the module component unit is the acupoint identifier of the module component unit. At least based on the identification information of the module component unit, the processing strategy of the module component unit is obtained from the target database table, and the processing strategy of the module component unit is executed. This includes: obtaining the processing strategy of the module component unit from the target database table based on the identification information of the vehicle and the acupoint identifier of each module component unit in the vehicle, and executing the processing strategy of the module component unit. The processing strategy of the module component unit is one of the following: processing or skipping. Processing means processing the module component unit, and skipping means not processing the module component unit.

[0012] Thirdly, this application provides a detection and processing system for module component units. The system includes: multiple processing and assembly devices for implementing the processing method of module component units as described in any of the above applications; multiple cameras, each corresponding to a processing and assembly device, and the cameras are disposed on the discharge side of the corresponding processing and assembly device; a supporting MES system for implementing the detection method of module component units as described in any of the above applications; a cloud server, each corresponding to a processing and assembly device, for storing and running a detection model, the detection model being configured to detect the appearance defect type and / or size information introduced by the processing steps performed by the processing and assembly devices corresponding to the cloud server; and a database including multiple database tables, each corresponding to a processing and assembly device.

[0013] Optionally, the detection model includes at least one defect sub-detection model. One of the defect sub-detection models is configured to detect a type of appearance defect introduced by the processing procedure performed by the cloud server corresponding to the processing and assembly equipment, as well as the level of the appearance defect type. The defect sub-detection model is obtained by training a neural network using multiple sets of first training data. Each set of first training data includes images of module component units acquired in historical time periods, and the appearance defect type corresponding to the images of the module component units.

[0014] Optionally, the detection model includes at least one size sub-detection model. One of the size sub-detection models is configured to detect the size information of components introduced by the processing steps performed by the cloud server corresponding to the processing and assembly equipment. The size sub-detection model is obtained by training a neural network using multiple sets of second training data. Each set of second training data includes historical data: images of module component units, parameters of the camera corresponding to the processing and assembly equipment, and size information of components.

[0015] In this embodiment, a camera is installed on the discharge side of the processing and assembly equipment to capture images of the processed module component units. The images are uploaded to a cloud server with a detection model deployed via a MES server. The cloud server sends the detection results to the MES server. The MES server generates a processing strategy based on the detection results and binds the identification information of the module component units and the processing strategy to the database table of the next processing and assembly equipment. The next processing and assembly equipment can obtain and execute the processing strategy from the database table. Compared with the prior art using AOI inspection equipment to inspect module component units, this application uses a cloud server with a detection model to handle defect detection. Compared with expensive AOI inspection equipment, the cost of the cloud server is very low, which greatly reduces the cost of defect detection of module component units and solves the problem of high cost of defect detection methods for module component units in the prior art. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 A schematic flowchart illustrating a detection and execution method in the production and processing stage provided in this application embodiment; Figure 2 A flowchart illustrating a method for processing a module component unit, as provided in an embodiment of this application; Figure 3 A structural block diagram of a module component unit inspection and processing system provided in this application embodiment; Figure 4 A top view of a heat map of a module component unit provided in this application embodiment; Figure 5 A side view of a heat map of a module component unit provided in this application embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0022] To address the high cost of existing module component unit defect detection methods, this application provides a module component unit detection method, a module component unit processing method, and a module component unit detection and processing system, which can reduce the cost of module component unit defect detection.

[0023] Figure 1 This application provides a method for detecting module component units, which is applied to a supporting MES (Manufacturing Execution System). The method includes: Step S101: Receive the image of the module component unit uploaded by the target camera and the identity information of the module component unit uploaded by the first target device; Wherein, the aforementioned target camera is the output camera of the aforementioned first target device, and the aforementioned first target device is the processing and assembly equipment that performs the Nth processing step, where N is a positive integer; For example, the camera can be a 20-megapixel CMOS (Complementary Metal-Oxide-Semiconductor) camera, coupled with a ring LED (Light Emitting Diode Light Source) light source, which can achieve defect recognition at the 0.02mm² level. Alternatively, the camera can be a 365nm ultraviolet 3D camera (such as the Keyence LJ-X8000A) to obtain a thermal map of the module assembly, which contains the height values ​​of various positions on the surface of the module assembly unit.

[0024] Specifically, the aforementioned module component unit refers to a semi-finished product that has not yet been fully processed and assembled on the electronic product module production line. For example, an electronic product module can be a camera module, a communication module, a battery module, etc.

[0025] Specifically, for each processing step, attention is paid to whether the processing step introduces appearance defects and / or introduces the size information of components. Therefore, the camera is set on the discharge side of the corresponding processing and assembly equipment to capture images of the module component unit after the processing step is completed.

[0026] Specifically, each processing step corresponds to a processing and assembly device, which is used to perform that processing step.

[0027] For example, an electronic product module production line includes multiple processing steps such as welding, mounting, and dispensing.

[0028] For example, given that the processing and assembly requirements of different electronic product modules vary, each processing and assembly equipment on the electronic product module production line is equipped with a camera to identify the identity information of the module component unit. For example, the identity information can be a QR code, thereby ensuring that the processing and assembly equipment accurately matches the processing and assembly requirements of the electronic product module. Therefore, the camera built into the processing and assembly equipment is used to upload the identity information of the module component unit to the supporting MES system.

[0029] Step S102: Upload the image of the module component unit to the target cloud server and receive the detection result of the module component unit sent by the target cloud server. The target cloud server is the cloud server corresponding to the first target device. The target cloud server is used to receive and store the images of the module component unit and run the target detection model to verify and detect the stored images. The target detection model is configured to detect the appearance defect type and / or the size information of the components introduced by the processing steps performed by the first target device. Specifically, some processing steps focus on the types of appearance defects introduced by the process, some focus on the dimensional information of the components introduced by the process, and some focus on both the types of appearance defects introduced by the process and the dimensional information of the components introduced by the process. Since the types of appearance defects introduced by different processing steps are different, the inspection requirements for different processing steps are also different. Therefore, each processing and assembly equipment is configured with a corresponding cloud server, and the corresponding inspection model for the processing and assembly equipment is deployed on the cloud server. This inspection model meets the inspection requirements of the processing step.

[0030] For example, the size information can be: length, width, and height.

[0031] Step S103: Based on the detection results of the module component unit, generate the processing strategy of the module component unit, bind the identity information of the module component unit and the processing strategy of the module component unit and store them in the target database table, so that the second target device can obtain the processing strategy of the module component unit from the target database table according to the identity information of the module component unit. The aforementioned second target equipment is the processing and assembly equipment that performs the N+1th processing step, and the aforementioned target database table is the database table corresponding to the aforementioned second target equipment.

[0032] Specifically, the identification information of the module component unit and the processing strategy of the module component unit are bound and stored in the table column of the target database table.

[0033] Specifically, each processing and assembly device is configured with a corresponding database table. The processing and assembly device retrieves the processing strategy from the database table and executes the processing strategy.

[0034] Through the above embodiments, a camera is set on the discharge side of the processing and assembly equipment to capture images of the processed module component units. The images are uploaded to a cloud server with a detection model deployed via the MES server. The cloud server sends the detection results to the MES server. The MES server generates a processing strategy based on the detection results and binds the identification information of the module component unit and the processing strategy to the database table of the next processing and assembly equipment. The next processing and assembly equipment can obtain and execute the processing strategy from the database table. Compared with the prior art of using AOI inspection equipment to inspect module component units, this application uses a cloud server with a detection model to be responsible for defect detection. Compared with the expensive AOI inspection equipment, the cost of the cloud server is very low, which greatly reduces the cost of defect detection of module component units, thereby solving the problem of high cost of defect detection methods for module component units in the prior art.

[0035] In an optional embodiment, the second target device is configured to process only one module component unit at a time, and the target detection model is configured to detect the size information of the components introduced by the processing steps performed by the first target device. The step S103, which binds the identity information of the module component unit and its processing strategy to a target database table, and then stores them, allows the second target device to obtain the processing strategy of the module component unit from the target database table based on its identity information. This can be achieved as follows: The identification information of the module component unit, the processing strategy of the module component unit, and the component size information in the test results of the module component unit are bound and stored in the target database table. This enables the second target device to obtain the processing strategy of the module component unit and the component size information in the test results of the module component unit from the target database table based on the identification information of the module component unit.

[0036] Specifically, based on the detection results of the above module component units, a processing strategy for generating the above module component units is generated. For example, the Nth processing step can be to package and cut a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and the N+1th processing step is to mount the CMOS image sensor onto the substrate. In the Nth processing step, the size information of the CMOS image sensor is detected. If the supporting MES system finds that the size information of the CMOS image sensor has a certain deviation from the standard, the generated processing strategy is to automatically correct the mounting coordinates of the CMOS image sensor according to the deviation in the N+1th processing step. The second target device executes the processing strategy based on the size information of the CMOS image sensor to ensure the optical axis consistency of the camera module and ensure the final imaging quality of the camera module.

[0037] In this embodiment, if there is a need to detect the size information of components, it is necessary not only to bind the identity information of the module component unit with the processing strategy of the module component unit and store it in the database table of the next processing and assembly equipment, but also to bind the size information of the components with the identity information of the module component unit and the processing strategy of the module component unit and store it in the next processing and assembly equipment, so that the next processing and assembly equipment executes the processing strategy based on the size information of the components.

[0038] In an optional embodiment, the second target device is configured to process only one module component unit at a time, and the processing strategy for generating the module component unit based on the detection result of the module component unit in step S103 can be implemented as follows: Based on the detection results of the module component unit and the operating conditions of the second target device, a processing strategy for the module component unit is generated. The operating conditions are one of the following: support the removal of the module component unit, or do not support the removal of the module component unit. The processing strategy for the module component unit is one of the following: process, skip, rework, or intercept and scrap. Processing means processing the module component unit; skipping means not processing the module component unit; rework means sending the module component unit back to the first target device for reprocessing; and intercept and scrap means removing the module component unit and placing it in the throwing tray.

[0039] In this embodiment, on the electronic product module production line, some processing steps only process and assemble a single module component unit at a time. For the processing and assembly equipment corresponding to these processing steps, some processing and assembly equipment supports the removal of module component units, while others do not. The supporting MES system can generate a processing strategy based on the detection results of the module component unit and whether the second target equipment supports the removal of the module component unit. For example, if the detection result of the module component unit shows that the flexible circuit board and substrate of the module component unit have poor soldering and the second target equipment supports the removal of the module component unit, then the processing strategy is rework, and the second target equipment sends the module component unit back to the first target equipment for reprocessing. For example, if the detection result of the module component unit is that the flexible circuit board and substrate of the module component unit are poorly soldered and the second target device does not support the removal of the module component unit, the processing strategy is to skip it. The second target device does not perform the N+1 processing step on the module component unit and directly transfers it to the processing and assembly equipment of the next processing step. When the processing and assembly equipment of a certain processing step supports the removal of the module component unit, the module component unit is removed and scrapped. If the detection result of the module component unit is that the surface of the CMOS image sensor of the module component unit is scratched and the second target device supports the removal of the module component unit, the processing strategy is to intercept and scrap it. The second target device removes the module component unit and puts it into the throwing tray.

[0040] In an optional embodiment, the second target device is configured to process one module component unit on a carrier at a time, and the identification information of the module component unit is the acupoint coordinate identifier of the module component unit. Step S103 can be implemented as follows: Based on the detection results of each module component unit in the above-mentioned vehicle, a processing strategy for the above-mentioned module component unit is generated. The processing strategy for the above-mentioned module component unit is one of the following: processing or skipping. Processing means processing the above-mentioned module component unit, and skipping means not processing the above-mentioned module component unit. For example, on an electronic product module production line, some processing steps process and assemble all module component units on a single carrier each time. For the processing and assembly equipment corresponding to these processing steps, module component units cannot be removed. In this case, the generated processing strategy can only process or skip. For example, if the detection result of a module component unit on the carrier is that the component is dirty, the corresponding processing strategy is to skip. If the detection result of a module component unit on the carrier is that the component is not dirty, the corresponding processing strategy is to process.

[0041] The identification information of the aforementioned vehicle, the acupoint identifier of each module component unit in the aforementioned vehicle, and the processing strategy of the aforementioned module component unit are bound and stored in the aforementioned target database table, so that the aforementioned second target device can obtain the processing strategy of the aforementioned module component unit from the aforementioned target database table based on the identification of the aforementioned vehicle and the acupoint identifier of each module component unit in the aforementioned vehicle.

[0042] For example, the vehicle's identification information can be in the form of a QR code. The acupoint identifier of each module component unit in the vehicle and the processing strategy of the module component unit are stored in the database table in the form of a Map table. For example, the Map table includes: (A1, OK), (A2, NG), (B1, NG), (B2, NG), where A1, A2, B1, and B2 are the acupoint identifiers of the module component units in the first column of row A, the second column of row A, the first column of row B, and the second column of row B, respectively. OK indicates processing, and NG indicates skipping. The processing and assembly equipment has a built-in camera. Therefore, when the vehicle arrives at the second target device, the camera of the second target device identifies the vehicle's identification information, retrieves the Map table of the vehicle from the database table corresponding to the second target device based on the vehicle's identification information, and executes the processing strategy of each module component unit on the vehicle according to the Map table.

[0043] In an optional embodiment, the second target device is configured to process only one module component unit at a time, and the target detection model is further configured to detect the level of the appearance defect type introduced by the processing steps performed by the first target device. After step S103, the method further includes: The level of the appearance defect type in the detection results of the above module component unit is bound to the identification information of the above module component unit.

[0044] In this embodiment, it is acceptable for module component units to have a certain degree of appearance defects. In this case, it is necessary to bind the level of appearance defect type of the module component unit with the identity information of the module component unit. When the finished electronic product module is obtained later, the level of appearance defect type of the module component unit can be obtained according to the identity information of the module component unit, thereby determining the quality of the module component unit.

[0045] like Figure 2 As shown, this application embodiment provides a method for processing a module component unit. The method is applied to a second target device, which is the processing and assembly equipment that performs the (N+1)th processing step, where N is a positive integer. Figure 2 As shown, the above method includes: Step S201: Obtain the identification information of the module component unit; Specifically, given that the processing and assembly requirements of different electronic product modules vary, each processing and assembly equipment on the electronic product module production line is equipped with a camera to identify the identity information of the module component unit.

[0046] Step S202: At least based on the identity information of the module component unit, obtain the processing strategy of the module component unit from the target database table and execute the processing strategy of the module component unit. The aforementioned target database table is the database table corresponding to the aforementioned second target device. The processing strategy of the aforementioned module component unit is generated by the supporting MES system based on the detection results of the aforementioned module component unit, and is stored in the aforementioned target database table after being bound by the identity information of the aforementioned module component unit to the aforementioned supporting MES system. The detection results of the aforementioned module component unit are sent to the aforementioned supporting MES system by the target cloud server. The aforementioned target cloud server is the cloud server corresponding to the first target device. The aforementioned target cloud server is used to receive and store the images of the aforementioned module component unit uploaded by the aforementioned supporting MES system and run the target detection model to verify and detect the stored images. The aforementioned target detection model is configured to detect the appearance defect type and / or the size information of the components introduced by the processing steps performed by the aforementioned first target device. The aforementioned first target device is the aforementioned processing and assembly equipment that performs the Nth processing step.

[0047] The aforementioned supporting MES system is used to receive images of the module component units uploaded by the target camera and the identity information of the module component units uploaded by the first target device.

[0048] Through the above embodiments, a camera is set on the discharge side of the processing and assembly equipment to capture images of the processed module component units. The images are uploaded to a cloud server with a detection model deployed via the MES server. The cloud server sends the detection results to the MES server. The MES server generates a processing strategy based on the detection results and binds the identification information of the module component unit and the processing strategy to the database table of the next processing and assembly equipment. The next processing and assembly equipment can obtain and execute the processing strategy from the database table. Compared with the prior art of using AOI inspection equipment to inspect module component units, this application uses a cloud server with a detection model to be responsible for defect detection. Compared with the expensive AOI inspection equipment, the cost of the cloud server is very low, which greatly reduces the cost of defect detection of module component units, thereby solving the problem of high cost of defect detection methods for module component units in the prior art.

[0049] In an optional embodiment, the second target device is configured to process only one module component unit at a time, and the target detection model is configured to detect the size information of the components introduced by the processing steps performed by the first target device. Step S202 can be implemented as follows: Based on the identification information of the module component unit, the processing strategy of the module component unit and the size information in the detection results of the module component unit are obtained from the target database table. Based on the size and position information in the detection results of the above module component units, the processing strategy of the above module component units is executed.

[0050] In this embodiment, if there is a need to detect the size information of components, it is necessary not only to bind the identity information of the module component unit with the processing strategy of the module component unit and store it in the database table of the next processing and assembly equipment, but also to bind the size information of the components with the identity information of the module component unit and the processing strategy of the module component unit and store it in the next processing and assembly equipment, so that the next processing and assembly equipment executes the processing strategy based on the size information of the components.

[0051] In an optional embodiment, the second target device is configured to process one module component unit on a carrier at a time, and the identification information of the module component unit is the acupoint identifier of the module component unit. Step S202 can be implemented as follows: Based on the vehicle's identification information and the acupoint identifier of each module component unit in the vehicle, the processing strategy of the module component unit is obtained from the target database table, and the processing strategy of the module component unit is executed. The processing strategy of the module component unit is one of the following: processing or skipping. Processing means processing the module component unit, and skipping means not processing the module component unit.

[0052] For example, on an electronic product module production line, some processing steps process and assemble all module component units on a single carrier each time. For the processing and assembly equipment corresponding to these processing steps, module component units cannot be removed. In this case, the generated processing strategy can only process or skip. For example, if the detection result of a module component unit on the carrier is that the component is dirty, the corresponding processing strategy is to skip. If the detection result of a module component unit on the carrier is that the component is not dirty, the corresponding processing strategy is to process.

[0053] For example, the vehicle's identification information can be in the form of a QR code. The acupoint identifier of each module component unit in the vehicle and the processing strategy of the module component unit are stored in the database table in the form of a Map table. For example, the Map table includes: (A1, OK), (A2, NG), (B1, NG), (B2, NG), where A1, A2, B1, and B2 are the acupoint identifiers of the module component units in the first column of row A, the second column of row A, the first column of row B, and the second column of row B, respectively. OK indicates processing, and NG indicates skipping. The processing and assembly equipment has a built-in camera. Therefore, when the vehicle arrives at the second target device, the camera of the second target device identifies the vehicle's identification information, retrieves the Map table of the vehicle from the database table corresponding to the second target device based on the vehicle's identification information, and executes the processing strategy of each module component unit on the vehicle according to the Map table.

[0054] Figure 3 An inspection and processing system for module component units is provided for embodiments of this application, such as... Figure 3 As shown, the above system includes: Multiple processing and assembly equipment, which are used to implement the processing method of the module component unit as described above; Multiple cameras, each corresponding to a processing and assembly device, are installed on the discharge side of the corresponding processing and assembly device. A matching MES system is provided, which is used to implement the detection method of the module component unit as described above. The cloud server corresponds one-to-one with the processing and assembly equipment. The cloud server is used to store and run the detection model. The detection model is configured to detect the appearance defect type and / or size information introduced by the processing steps performed by the processing and assembly equipment corresponding to the cloud server. The database includes multiple database tables, each corresponding one-to-one with the aforementioned processing and assembly equipment.

[0055] Specifically, Figure 3 The example only provides three processing and assembly devices, three cameras, three cloud servers, and three databases. This application does not limit the number of processing and assembly devices, cameras, cloud servers, and databases in the module component unit detection and processing system.

[0056] Through the above embodiments, a camera is set on the discharge side of the processing and assembly equipment to capture images of the processed module component units. The images are uploaded to a cloud server with a detection model deployed via the MES server. The cloud server sends the detection results to the MES server. The MES server generates a processing strategy based on the detection results and binds the identification information of the module component unit and the processing strategy to the database table of the next processing and assembly equipment. The next processing and assembly equipment can obtain and execute the processing strategy from the database table. Compared with the prior art of using AOI inspection equipment to inspect module component units, this application uses a cloud server with a detection model to be responsible for defect detection. Compared with the expensive AOI inspection equipment, the cost of the cloud server is very low, which greatly reduces the cost of defect detection of module component units, thereby solving the problem of high cost of defect detection methods for module component units in the prior art.

[0057] In an optional embodiment, the detection model includes at least one defect sub-detection model. The defect sub-detection model is configured to detect a type of appearance defect introduced by the processing procedure performed by the cloud server corresponding to the processing and assembly equipment, as well as the level of the appearance defect type. The defect sub-detection model is obtained by training a neural network using multiple sets of first training data. Each set of first training data includes images of module component units acquired in historical time periods, and the appearance defect type corresponding to the images of the module component units.

[0058] For example, if the appearance defect type detected by the defect sub-detection model is whether there is dirt, the process of acquiring the first training data is as follows: collect image samples: dirty image samples (the area to be detected in the module component unit image has dirt) and no dirty image samples (the area to be detected in the module component unit image has no dirt). Labeling process: use a label box to label the location of dirt in the dirty image samples and label the classification label as dirty. No dirty image samples do not need to use a label box for location labeling, only label the classification label as no dirt. Before the labeling process, the area of ​​dirt to be detected can also be cropped from the module component unit image to focus on detecting the dirt target and improve the detection accuracy of the model.

[0059] For example, if the defect detection model is used to detect whether the glue coverage is qualified, where qualified coverage means there are no abnormal glue amounts such as insufficient glue or overflow, otherwise the coverage is unqualified, the acquisition process of the first training data is as follows: collect image samples: qualified coverage image samples (the glue coverage of the area to be glued in the module component unit image is qualified) and unqualified coverage image samples (the glue coverage of the area to be glued in the module component unit image is unqualified), annotation processing: use annotation boxes to mark the position of the area to be glued and the actual glue coverage area in the qualified coverage image samples, and label them as qualified coverage. use annotation boxes to mark the position of the area to be glued and the actual glue coverage area in the unqualified coverage image samples, and label them as unqualified coverage. Before the annotation processing, the area to be detected as qualified glue coverage can also be cropped from the module component unit image to focus on detecting whether the glue coverage of this area is qualified, thereby improving the detection accuracy of the model.

[0060] For example, if the appearance defect type detected by the defect sub-detection model is whether there is a missing component, the process of acquiring the first training data is as follows: collect image samples: image samples with missing components (the area to be detected in the module component unit image has missing components) and image samples without missing components (the area to be detected in the module component unit image has no missing components). Labeling process: for image samples with missing components, there is no need to use a label box to label the position, and label the classification as having missing components. For image samples without missing components, the position of the component is labeled using a label box, and the classification label is as having no missing components. Before the labeling process, the area of ​​missing components to be detected can also be cropped from the module component unit image to focus on detecting missing components and improve the detection accuracy of the model.

[0061] For example, for processing steps that require the detection of the level of appearance defect type, the level of appearance defect type also needs to be marked in the above-mentioned marking process.

[0062] For example, Figure 4 This is a top view of the heat map of the module component unit. Figure 5 This is a side view of the heat map of a module component unit. Figure 4 and Figure 5In the diagram, X represents the horizontal axis of the heatmap, Y represents the vertical axis, and Z represents the vertical axis. The coordinate units for all axes are (μm). S represents the color scale axis, where each color represents a height value. Different colors at different locations within the module component unit indicate different height values ​​at those locations. If the defect detection model is used to detect whether the flatness of the module component unit is up to standard, the acquisition process for the first training data is as follows: Image samples (heatmaps of the module component unit) are acquired using the aforementioned 365nm ultraviolet 3D camera: Flatness-compliant image samples (the flatness of the area to be detected in the thermal image of the module component unit is up to standard) and flatness-incompatible image samples (the flatness of the area to be detected in the thermal image of the module component unit is down to standard). The thermal image of the module component unit contains all locations of the module component unit. The height value of the point; the flatness of the area to be tested is the vertical distance between the highest point and the lowest point of the surface of the area to be tested. The smaller the difference, the better the flatness. Labeling process: Use a labeling box to label the location of the area to be tested in the image sample with qualified flatness, label the vertical coordinate of the highest point of the surface of the area to be tested, label the vertical coordinate of the lowest point of the surface of the area to be tested, label the flatness of the area to be tested (flatness is the difference between the vertical coordinate of the highest point and the lowest point of the surface of the area to be tested), and label it as qualified flatness. Use a labeling box to label the location of the area to be tested in the image sample with unqualified flatness, label the vertical coordinate of the highest point of the surface of the area to be tested, label the vertical coordinate of the lowest point of the surface of the area to be tested, label the flatness of the area to be tested, and label it as unqualified flatness.

[0063] In an optional embodiment, the detection model includes at least one size sub-detection model. The size sub-detection model is configured to detect the size information of components introduced by the processing steps performed by the cloud server corresponding to the processing and assembly equipment. The size sub-detection model is obtained by training a neural network using multiple sets of second training data. Each set of second training data includes historical data: images of module component units, parameters of the camera corresponding to the processing and assembly equipment, and size information of components.

[0064] For example, for detecting the length of a component, the process of acquiring the second training data is as follows: Image samples (planar images of the module component unit) are acquired using the aforementioned CMOS camera, and annotation processing is performed: In the image samples, the x-coordinate and y-coordinate of a key point are marked on each wide side of the component, and the line connecting these two key points is perpendicular to the two wide sides of the component (i.e., the y-coordinates of these two key points are the same). The pixel distance between these two key points (the absolute value of the difference between the x-coordinates of these two key points), the actual distance (the distance between these two key points in real physical space), and the camera parameters of the CMOS camera are also marked. The actual distance between these two key points is the length of the component. Through the above annotation... Image samples are used to train the size detection model, enabling it to learn to locate key points on the two wide sides of a component in an image of a module unit. The model then learns the relationship between the pixel distance, the actual distance, and camera parameters of these two key points. This allows the size detection model to convert the pixel distance between these two key points into the actual distance (the length of the component) based on the camera parameters. The camera parameters include at least: resolution, focal length, aperture, intrinsic parameters (pixel focal length, etc.), and extrinsic parameters (distance between the camera and the component). These camera parameters are the key basis for the size detection model to convert the image pixel size into the actual physical size of the component. The process of obtaining the second training data for detecting the width of the component is similar.

[0065] For example, to detect the height of a component, the second training data acquisition process is as follows: Image samples (heatmaps of module component units) are acquired using the aforementioned 365nm ultraviolet 3D camera, and annotation processing is performed: For example, in the image samples, the x, y, and y coordinates of a key point on the top surface of the component to be detected are annotated, as are the x, y, and y coordinates of a key point on the bottom surface of the component. The x-coordinates of the key points on the top and bottom surfaces of the component are the same, and the y-coordinates of the key points on the top and bottom surfaces of the component are the same. The pixel distance between these two key points (the key point on the top surface) is also annotated. The camera parameters of the 365nm ultraviolet 3D camera are marked by the difference between the vertical coordinates of the vertical coordinates and the vertical coordinates of the key points on the bottom surface, as well as the actual distance (the vertical distance between the key points on the top surface and the key points on the bottom surface in real physical space). The actual distance between these two key points is the height of the component. The size detection model is trained by the above-mentioned marked image samples, so that the size detection model can learn to locate the above two key points of the component in the heat map of the module component unit, and learn the relationship between the pixel distance, actual distance and camera parameters of these two key points. This allows the size detection model to learn to convert the pixel distance of these two key points into the actual distance (the height of the component) based on the camera parameters.

[0066] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the module component unit detection method or the module component unit processing method provided in any of the foregoing method embodiments.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0069] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0070] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for detecting module component units, characterized in that, The method is applied to a supporting MES system, and the method includes: The system receives images of the module component unit uploaded by the target camera and the identification information of the module component unit uploaded by the first target device. The target camera is the camera on the discharge side of the first target device, and the first target device is a processing and assembly device that performs the Nth processing step, where N is a positive integer. The image of the module component unit is uploaded to the target cloud server, and the detection result of the module component unit is received from the target cloud server. The target cloud server is the cloud server corresponding to the first target device. The target cloud server is used to receive and store the image of the module component unit and run the target detection model to verify and detect the stored image. The target detection model is configured to detect the appearance defect type and / or the size information of the components introduced by the processing steps performed by the first target device. Based on the detection results of the module component unit, a processing strategy for the module component unit is generated. The identity information of the module component unit and the processing strategy of the module component unit are bound and stored in the target database table. The second target device obtains the processing strategy of the module component unit from the target database table according to the identity information of the module component unit. The second target device is a processing and assembly device that performs the N+1th processing step, and the target database table is the database table corresponding to the second target device.

2. The method according to claim 1, characterized in that, The second target device is configured to process only one module component unit at a time. The target detection model is configured to detect the size information of components introduced by the processing steps performed by the first target device, bind the identification information of the module component unit and the processing strategy of the module component unit, and store them in the target database table. This allows the second target device to retrieve the processing strategy of the module component unit from the target database table based on the identification information of the module component unit, including: The identification information of the module component unit, the processing strategy of the module component unit, and the component size information in the detection result of the module component unit are bound and stored in the target database table, so that the second target device can obtain the processing strategy of the module component unit and the component size information in the detection result of the module component unit from the target database table according to the identification information of the module component unit.

3. The method according to claim 1 or 2, characterized in that, The second target device is configured to process only one module component unit at a time. Based on the detection results of the module component unit, a processing strategy for the module component unit is generated, including: Based on the detection results of the module component unit and the operating conditions of the second target device, a processing strategy for the module component unit is generated. The operating conditions are one of the following: support the removal of the module component unit, or do not support the removal of the module component unit. The processing strategy for the module component unit is one of the following: process, skip, rework, or intercept and scrap. Processing means processing the module component unit. Skipping means not processing the module component unit. Rework means sending the module component unit back to the first target device for reprocessing. Intercept and scrap means removing the module component unit and placing it in the throwing tray.

4. The method according to claim 1 or 2, characterized in that, The second target device is configured to process one module component unit on a vehicle at a time. The identification information of the module component unit is the acupoint coordinate identifier of the module component unit. Based on the detection results of the module component unit, a processing strategy for the module component unit is generated. The identification information of the module component unit and the processing strategy of the module component unit are bound and stored in a target database table. The second target device then retrieves the processing strategy of the module component unit from the target database table according to the identification information of the module component unit, including: Based on the detection results of each module component unit in the vehicle, a processing strategy for the module component unit is generated. The processing strategy for the module component unit is one of the following: process or skip. Processing means processing the module component unit, and skipping means not processing the module component unit. The vehicle's identification information, the acupoint identifier of each module component unit in the vehicle, and the processing strategy of the module component unit are bound and stored in the target database table, so that the second target device can obtain the processing strategy of the module component unit from the target database table based on the vehicle's identification and the acupoint identifier of each module component unit in the vehicle.

5. The method according to claim 1 or 2, characterized in that, The second target device is configured to process only one module component unit at a time. The target detection model is also configured to detect the level of appearance defect type introduced by the processing steps performed by the first target device. After generating a processing strategy for the module component unit based on the detection results of the module component unit, and binding the identification information of the module component unit and the processing strategy of the module component unit to the target database table, the method further includes: The level of the appearance defect type in the detection results of the module component unit is bound to the identity information of the module component unit.

6. A method for processing a module component unit, characterized in that, The method is applied to a second target device, which is a processing and assembly device that performs the (N+1)th processing step, where N is a positive integer. The method includes: Obtain the identity information of the module component unit; At least based on the identity information of the module component unit, the processing strategy of the module component unit is obtained from the target database table, and the processing strategy of the module component unit is executed. The target database table is the database table corresponding to the second target device. The processing strategy of the module component unit is generated by the supporting MES system based on the detection results of the module component unit, and is stored in the target database table after being bound by the identity information of the module component unit by the supporting MES system. The detection results of the module component unit are sent to the supporting MES system by the target cloud server. The target cloud server is the cloud server corresponding to the first target device. The target cloud server is used to receive and store the image of the module component unit uploaded by the supporting MES system and run the target detection model to verify and detect the stored image. The target detection model is configured to detect the appearance defect type and / or the size information of the components introduced by the processing steps performed by the first target device. The first target device is the processing and assembly equipment that performs the Nth processing step.

7. The method according to claim 6, characterized in that, The second target device is configured to process only one module component unit at a time. The target detection model is configured to detect the size information of components introduced by the processing steps performed by the first target device, and at least based on the identity information of the module component unit, obtain the processing strategy of the module component unit from the target database table, and execute the processing strategy of the module component unit, including: Based on the identification information of the module component unit, the processing strategy of the module component unit and the size information in the detection results of the module component unit are obtained from the target database table; Based on the dimensional information in the detection results of the module component unit, the processing strategy of the module component unit is executed.

8. The method according to claim 6, characterized in that, The second target device is configured to process one module component unit on a vehicle at a time. The identification information of the module component unit is the acupoint identifier of the module component unit. At least based on the identification information of the module component unit, the device retrieves the processing strategy of the module component unit from the target database table and executes the processing strategy of the module component unit, including: Based on the vehicle's identification information and the acupoint identifier of each module component unit in the vehicle, the processing strategy of the module component unit is obtained from the target database table, and the processing strategy of the module component unit is executed. The processing strategy of the module component unit is one of the following: processing or skipping. Processing means processing the module component unit, and skipping means not processing the module component unit.

9. A detection and processing system for module component units, characterized in that, The system includes: Multiple processing and assembly equipment, said processing and assembly equipment being used to implement the processing method of the module component unit as described in any one of claims 6 to 8; Multiple cameras, each corresponding to a processing and assembly device, are positioned on the discharge side of the corresponding processing and assembly device. A matching MES system is provided, wherein the matching MES system is used to implement the detection method of the module component unit as described in any one of claims 1 to 5; A cloud server, which corresponds one-to-one with the processing and assembly equipment, is used to store and run a detection model. The detection model is configured to detect the type and / or size information of appearance defects introduced by the processing steps performed by the processing and assembly equipment corresponding to the cloud server. The database includes multiple database tables, each corresponding to a processing and assembly device.

10. The system according to claim 9, characterized in that, The detection model includes at least one defect sub-detection model. One of the defect sub-detection models is configured to detect a type of appearance defect introduced by the processing procedure performed by the cloud server corresponding to the processing and assembly equipment, as well as the level of the appearance defect type. The defect sub-detection model is obtained by training a neural network using multiple sets of first training data. Each set of first training data includes images of module component units acquired in historical time periods, and the appearance defect type corresponding to the images of the module component units.

11. The system according to claim 9, characterized in that, The detection model includes at least one size sub-detection model. One of the size sub-detection models is configured to detect the size information of components introduced by the processing steps performed by the processing and assembly equipment corresponding to the cloud server. The size sub-detection model is obtained by training a neural network using multiple sets of second training data. Each set of second training data includes historical data: images of module component units, parameters of the camera corresponding to the processing and assembly equipment, and size information of components.

Images