A method for establishing a visual defect detection model for a liquid crystal display screen and a defect detection method

Abstract

The present invention discloses a method for establishing a visual defect detection model for a liquid crystal display screen, which is established according to a backbone network module, a neck module and a decoupling head module connected in sequence; the neck module adds an improved CBAM module, and at the same time introduces a fine-grained separable convolution module to replace the CBS module located between the CSPLayer module and the Concat module; the image of the liquid crystal display screen is input into the backbone network module to extract features at all levels; the features at all levels are input into the neck module to obtain three neck output features; the three neck output features are sent to each decoupling head module for prediction, and the position, type and confidence information of the target information in the liquid crystal display screen image are obtained. The present invention can not only effectively improve the accuracy of Mura defect detection of liquid crystal displays, but also improve the efficiency of display screen sorting.

Description

Technical Field

[0001] The invention relates to a method for establishing a visual defect detection model for a liquid crystal display screen and a defect detection method, and belongs to the field of industrial product visual recognition. Background Art

[0002] TFT-LCD is one of the best-selling display products at present, and is widely used in electronic products such as flat-panel TVs, computer monitors, smart phones and tablets. The polymer polarizers attached to both sides of the liquid crystal layer are important components of the TFT-LCD panel because they have the function of adjusting the light path. However, even if the polarizer is produced on the assembly line of a dust-free workshop, Mura defects such as bubbles and scratches caused by dust and the like cannot be avoided. These defects have the characteristics of uneven local brightness, low contrast, blurred edges, uncertain size, uneven background, etc., which reduces the quality of the TFT-LCD panel and further affects the user's sensory experience. Therefore, before the polarization is integrated into the liquid crystal panel, accurate detection and positioning of any Mura defects in it has become a key issue for manufacturers to ensure the quality of display inspection.

[0003] In view of this, the present invention is proposed. Summary of the invention

[0004] The present invention provides a method for establishing a liquid crystal display screen visual defect detection model for constructing a liquid crystal display screen visual defect detection model, further provides a liquid crystal display screen visual defect detection method for liquid crystal display screen defect detection, and further provides a liquid crystal display screen visual defect detection sorting system to provide support for sorting liquid crystal display screens with defect detection on an assembly line.

[0005] The technical solution of the present invention is:

[0006] According to the first aspect of the present invention, a method for establishing a visual defect detection model for a liquid crystal display screen is provided, which is established based on a backbone network module, a neck module and a decoupling head module connected in sequence; the neck module adds an improved CBAM module and introduces a fine-grained separable convolution module to replace the CBS module located between the CSPLayer module and the Concat module; the LCD screen image is input into the backbone network module to extract features at all levels; the features at all levels are input into the neck module to obtain three neck output features; the three neck output features are sent to each decoupling head module for prediction to obtain the position, type and confidence information of the target information in the LCD screen image.

[0007] The backbone network module introduces the SPPA module to replace the SPP module located between the CBS module and the CSPLayer module in the traditional backbone network module; the SPPA module structure consists of two CBFR modules, AdapAvgPool and one CBSG module, and the two CBFR modules are CBFR1 and CBFR2; the output of CBFR1 is multiplied by the output of AdapAvgPool, added to the output of CBFR2, and then input into the CBSG module.

[0008] The improved CBAM module includes four modules: ACBAM1 module, ACBAM2 module, ACBAM3 module and ACBAM4 module. The ACBAM1 module is used to connect the first output of the backbone network module, the ACBAM2 module is used to connect the second output of the backbone network module, the ACBAM3 module is used to connect the third output of the backbone network module, and the neck module is connected to the Concat of the output of the ACBAM2 module and then connected to the ACBAM4 module; the ACBAM1 module, the ACBAM2 module, the ACBAM3 module and the ACBAM4 module have the same structure and are described by the ACBAM1 module: the channel attention and the spatial attention branches are connected in parallel to obtain the first neck splicing feature one and the first neck splicing feature two, the first neck splicing feature one and the first neck splicing feature two are concat spliced, and then a convolution is performed to obtain the first neck intermediate feature map.

[0009] The channel attention and spatial attention branches are connected in parallel to obtain the first neck splicing feature 1 and the first neck splicing feature 2, specifically: the channel attention inputs the feature map F input Perform global maximum pooling and global average pooling respectively to obtain two first feature maps; the two first feature maps are respectively subjected to a 1×1 convolution, a ReLU activation function, and a 1×1 convolution operation, and then summed; the result of the summation operation is activated by a Sigmoid function to generate a channel attention map; the channel attention map is then combined with the input feature map F input Multiply to get the first neck splicing feature 1; spatial attention will input feature map F input Perform global maximum pooling and global average pooling respectively to obtain two second feature maps; perform Concat splicing operation on the two second feature maps in the channel dimension, and after the Concat splicing operation, perform a 7×7 convolution operation and a Sigmoid function activation operation to generate a spatial attention map; the spatial attention map is then combined with the input feature map F input Multiply them to obtain the first neck joint feature 2.

[0010] The fine-grained separable convolution module includes two modules: FGSCM1 module and FGSCM2 module. The output of CSPLayer module is used as the input of the fine-grained separable convolution module, and the output of the fine-grained separable convolution module is used as the input of Concat module. The FGSCM1 module and FGSCM2 module have the same structure, and the FGSCM1 module is used for explanation: the CBS module is decomposed into depth convolution and point-by-point convolution. Among them, the convolution kernel size of the depth-wise separable convolution is 2×2 and the step size is 2. The point-by-point convolution uses a one-dimensional convolution kernel with a size of 1×1 to perform convolution processing on each channel of the intermediate feature map, and linearly combines the information of different channels to generate the final output feature map.

[0011] According to a second aspect of the present invention, a method for detecting visual defects of a liquid crystal display is provided, comprising: acquiring an image of the liquid crystal display in an unpowered state; calling a liquid crystal display visual defect model to detect the image of the liquid crystal display in the unpowered state to obtain a first detection result; acquiring an image of the liquid crystal display in a powered state; calling a liquid crystal display visual defect model to detect the image of the liquid crystal display in the powered state to obtain a second detection result; when the first detection result and / or the second detection result is identified as having defects, it is considered that there is a defect.

[0012] According to a third aspect of the present invention, a liquid crystal display visual defect detection and sorting system is provided, comprising: a first defect detection module 1, for acquiring an image of a liquid crystal display in an unpowered state; for calling a liquid crystal display visual defect model to detect the image of the liquid crystal display in an unpowered state, to obtain a first detection result; a second defect detection module 3, for acquiring an image of the liquid crystal display in a powered state; for calling a liquid crystal display visual defect model to detect the image of the liquid crystal display in a powered state, to obtain a second detection result; and an automatic sorting module 4, for conveying the defective liquid crystal display to a first conveying component when the first detection result and / or the second detection result is identified as defective, otherwise conveying the liquid crystal display to a second conveying component.

[0013] The beneficial effects of the present invention are as follows: the present invention constructs a liquid crystal display visual defect detection model based on liquid crystal display defect detection, and the liquid crystal display visual defect detection model embeds the newly constructed SPPA module in the backbone network module so that the network can focus more on the significant target area in the Mura defect, which can improve the network's detection ability for small-sized defect targets and reduce missed detection or false detection caused by small target size; by embedding the two modules of ACBAM and FGSCM in the Neck part, the context information in the feature map can be more effectively extracted, and the problems of abnormal pixel brightness, color distortion and texture discontinuity in the area where the Mura defect exists can be alleviated. Furthermore, through comparative experiments, it can be seen that the present invention can not only effectively improve the accuracy of liquid crystal display Mura defect detection, but also improve the efficiency of display screen sorting. Furthermore, in the LCD screen visual defect detection and sorting system, the first defect detection module is used to detect the surface defects of the screen in the unpowered state; the second defect detection module is used to detect the defects of the LCD screen under the signal, and the automatic sorting module is used to automatically sort the screen based on the detection results. Screens with qualified quality will be sorted to the lower layer of the transport module, and screens with unqualified quality will be transported to the upper layer of the conveying module by the automatic sorting module. On the factory production line, the demand and cost of labor are reduced, and the subjectivity and fatigue problems of manual inspection, which may lead to missed inspections or false inspections, are avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram of the overall structure of the LCD screen visual defect detection and sorting system;

[0015] Figure 2 is a structural diagram of the first defect detection module;

[0016] Figure 3 It is the module structure diagram of the centering clamping mechanism;

[0017] Figure 4 This is the structural diagram of the testing organization;

[0018] Figure 5 is a structural diagram of the second defect detection module;

[0019] Figure 6 This is the structure diagram of the automatic sorting module;

[0020] Figure 7 It is the structure diagram of the transmission module;

[0021] Figure 8 This is the structural diagram of the live tray module;

[0022] Fig. 9 This is the structural diagram of the lifting module;

[0023] Fig.10 This is a partial structural diagram of the automatic sorting module;

[0024] Fig.11 This is the structure diagram of the power supply module;

[0025] Fig.12 This is the structural diagram of the power part of the centering clamping mechanism module;

[0026] Fig.13 It is a flow chart of the detection method of the present invention;

[0027] Fig.14 A network structure diagram of a liquid crystal display screen visual defect detection model provided by the present invention;

[0028] Fig.15 This is the SPPA module structure diagram;

[0029] Fig.16 It is the structure diagram of ACBAM module;

[0030] Fig.17 This is the FGSCM module structure diagram;

[0031] Fig.18 This is the result of the ablation experiment;

[0032] Fig.19 This is a comparison chart of the effect of the attention mechanism;

[0033] Fig. 20 This is a comparison chart of the activation function effects;

[0034] The numbers in the figure are: 1-first defect detection module, 2-centering clamping mechanism module, 3-second defect detection module, 4-automatic sorting module, 5-charged pallet module, 6-transmission module, 7-tail base, 8-conveyor frame, 9-combined speed regulator, 10-motor fixing base, 11-chain, 12-induction motor I, 13-drive roller, 14-head base, 15-conveyor belt I, 16-standard cylinder, 17-cylinder fixing plate, 1 8-centering rod, 19-centering claw, 20-centering plate, 21-centering rotating arm, 22-countersunk screw, 23-slide rail I, 24-slide rail II, 25-servo motor II, 26-slider I, 27-industrial camera, 28-slider II, 29-servo motor I, 30-column, 31-fixed block, 32-roller, 33-signal module, 34-power supply module, 35-photoelectric switch, 36-transmission motor, 37-motor reducer, 38- Motor bracket, 39-lifting frame, 40-lifting module, 41-lifting frame, 42-baffle, 43-servo motor IV, 44-unpowered roller, 45-conveyor, 46-power interface, 47-display tray, 48-display to be tested, 49-signal interface, 50-bearing seat, 51-reel cover, 52-transmission conveyor belt, 53-large pulley, 54-intermediate shaft, 55-reel, 56-induction motor II, 57-induction motor deceleration , 58-small pulley, 59-wire rope, 60-coupling II, 61-conveying roller, 62-lifting ear, 63-conveyor belt II, 64-power cord, 65-cylinder fixing bracket, 66-double-axis cylinder, 67-power box connector, 68-power connector, 69-hexagonal nut, 70-centering rotating center shaft, 71 coupling I, 72-planetary reducer, 73-servo motor III, 74-center bearing I, 75-washer I, 76-hexagonal bolt. DETAILED DESCRIPTION

[0035] The invention will be further described below in conjunction with the accompanying drawings and embodiments, but the content of the invention is not limited to the scope of the embodiments.

[0036] Example 1: Figure 13-17 As shown, according to the first aspect of an embodiment of the present invention, a method for establishing a visual defect detection model for a liquid crystal display screen is provided, which is established based on a backbone network module, a neck module and a decoupling head module connected in sequence; the neck module adds an improved CBAM module and introduces a fine-grained separable convolution module to replace the CBS module located between the CSPLayer module and the Concat module; the LCD display screen image is input into the backbone network module to extract features at all levels; the features at all levels are input into the neck module to obtain three neck output features; the three neck output features are sent to each decoupling head module for prediction to obtain the position, type and confidence information of the target information in the LCD display screen image.

[0037] Furthermore, the backbone network module introduces an SPPA module to replace the SPP module located between the CBS module and the CSPLayer module in the traditional backbone network module; the SPPA module structure is composed of two CBFR modules, AdapAvgPool and one CBSG module, and the two CBFR modules are CBFR1 and CBFR2; the output of CBFR1 is multiplied by the output of AdapAvgPool, added to the output of CBFR2, and then input into the CBSG module, and connected to the CSPLayer module of the last level in the traditional backbone network module through the CBSG module.

[0038] Furthermore, the improved CBAM module includes four modules: ACBAM1 module, ACBAM2 module, ACBAM3 module, and ACBAM4 module. The ACBAM1 module is used to connect the first output of the backbone network module, the ACBAM2 module is used to connect the second output of the backbone network module, the ACBAM3 module is used to connect the third output of the backbone network module, and the neck module is connected to the Concat of the output of the ACBAM2 module and then connected to the ACBAM4 module; the ACBAM1 module, the ACBAM2 module, the ACBAM3 module, and the ACBAM4 module have the same structure and are described by the ACBAM1 module: the channel attention and the spatial attention branches are connected in parallel to obtain the first neck splicing feature one and the first neck splicing feature two, the obtained first neck splicing feature one and the first neck splicing feature two are subjected to the Concat splicing operation, the number of channels is restored through a convolution, and the first neck intermediate feature map is obtained.

[0039] Furthermore, the channel attention and the spatial attention branches are connected in parallel to obtain the first neck splicing feature 1 and the first neck splicing feature 2, specifically: the channel attention inputs the feature map F input Perform global maximum pooling and global average pooling respectively to obtain two first feature maps; the two first feature maps are respectively subjected to a 1×1 convolution, a ReLU activation function, and a 1×1 convolution operation, and then summed; the result of the summation operation is activated by a Sigmoid function to generate a channel attention map; the channel attention map is then combined with the input feature map F input Multiply to get the first neck splicing feature 1; spatial attention will input feature map F input Perform global maximum pooling and global average pooling respectively to obtain two second feature maps; perform Concat splicing operation on the two second feature maps in the channel dimension, and after the Concat splicing operation, perform a 7×7 convolution operation and a Sigmoid function activation operation to generate a spatial attention map; the spatial attention map is then combined with the input feature map F input Multiply them to obtain the first neck joint feature 2.

[0040] Furthermore, the fine-grained separable convolution module includes two modules: FGSCM1 module and FGSCM2 module. The output of the CSPLayer module is used as the input of the fine-grained separable convolution module, and the output of the fine-grained separable convolution module is used as the input of the Concat module. The FGSCM1 module and the FGSCM2 module have the same structure, and the FGSCM1 module is used for illustration: the CBS module is decomposed into depth convolution and point-by-point convolution; wherein, the convolution kernel size of the depth-wise separable convolution is 2×2 with a step size of 2, and the point-by-point convolution uses a one-dimensional convolution kernel with a size of 1×1 to perform convolution processing on each channel of the intermediate feature map, and linearly combines the information of different channels to generate the final output feature map.

[0041] like Figure 13-20 As shown, according to the second aspect of an embodiment of the present invention, a method for detecting visual defects of a liquid crystal display is provided, including: obtaining an image of the liquid crystal display in an unpowered state; calling a liquid crystal display visual defect model to detect the image of the liquid crystal display in the unpowered state to obtain a first detection result; obtaining an image of the liquid crystal display in a powered state; calling a liquid crystal display visual defect model to detect the image of the liquid crystal display in the powered state to obtain a second detection result; when the first detection result and / or the second detection result is identified as having defects, it is considered that there is a defect.

[0042] Specifically used for detecting Mura defects in liquid crystal displays.

[0043] like Figure 1-20 As shown, according to the third aspect of an embodiment of the present invention, a liquid crystal display visual defect detection and sorting system is provided, including: a first defect detection module 1, used to obtain an image of the liquid crystal display in an unpowered state; used to call a liquid crystal display visual defect model to detect the image of the liquid crystal display in the unpowered state, and obtain a first detection result; a second defect detection module 3, used to obtain an image of the liquid crystal display in a powered state; call a liquid crystal display visual defect model to detect the image of the liquid crystal display in the powered state, and obtain a second detection result; an automatic sorting module 4, used to convey the defective liquid crystal display to the first conveying component when the first detection result and / or the second detection result is identified as defective, otherwise the liquid crystal display is conveyed to the second conveying component.

[0044] Specifically, if Figure 1-12As shown, a visual defect detection and sorting system for a liquid crystal display screen optionally used in an assembly line is provided as follows, comprising a first surface defect detection module 1, a centering clamping mechanism module 2, a second defect detection module 3, an automatic sorting module 4, a charged tray module 5, and a conveying module 6; wherein the charged tray module 5 is used to load the liquid crystal display screen produced on the assembly line, and then the loaded display screen is transported on the assembly line to the second defect detection module 3 for charged signal detection, the first surface defect detection module 1 is used to detect surface scratches and other defects on the display screen in the charged tray module 5 when no power is supplied, and the centering clamping mechanism module Block 2 is used to automatically lift and clamp the charged tray module 5 when it is transported to a fixed position. Then the second defect detection module 3 detects the power-on signal of the display screen. The automatic sorting module 4 is used to sort the LCD screen after defect detection. The LCD screen with quality problems (that is, the defects detected by the first defect detection module 1 or the second defect detection module 3 on the surface are regarded as having quality problems, that is, unqualified) are lifted to the upper layer of the conveying module 6 for subsequent maintenance processing, and the qualified LCD screen will be transmitted to the lower layer of the conveying module 6 for subsequent processes.

[0045] Furthermore, if Figure 2As shown, the first defect detection module 1 can be provided to include a conveying mechanism and a first detection mechanism, wherein the conveying mechanism includes a tail base 7, a conveyor frame 8, a combined speed regulator 9, a motor fixing seat 10, a chain 11, an induction motor I 12, a drive roller 13, a head base 14, and a conveyor belt I 15; the conveying mechanism is used to realize the transportation of the charged pallet module 5, the induction motor I 12 is fixed on the motor fixing seat 10, the motor fixing seat 10 is connected to the conveyor frame 8, the conveyor frame 8 is equipped with a tail base 7, a head base 14 for fixing the drive roller 13, the combined speed regulator 9 is used to adjust the conveying speed of the induction motor I 12, the output shaft end of the induction motor I 12 drives the chain 11 by chain transmission, the chain 11 drives the drive roller 13 to roll, and then the conveyor belt I 15 realizes the transportation function. The display screen is transmitted to the first detection mechanism on the assembly line, and the image is collected by the industrial camera and transmitted to the controller, and the LCD visual defect model is called by the controller to determine whether there are surface defects such as scratches on the surface of the display screen. The first detection mechanism includes two linear motion modules and a controller. Module I includes a fixed block 31, a column 30, a slider I26, and a servo motor I29. The servo motor I29 is fixed to the column 30 by a nut, and the slider I26 can realize linear motion on the column 30 by electric drive. Module II includes a slider II28, a slide rail II24, a servo motor II25, and an industrial camera 27. The principle is the same as that of module I. Two modules I are fixed to the two ends of the conveying mechanism frame by welding. Module II is fixed to the fixed block 31 of the slider I of module I by a nut and driven by a servo motor I29. Module II can realize vertical movement in the two modules I, so that module II and the camera thereon can move in the vertical direction, so that the industrial camera 27 can adjust the appropriate distance to collect image information. The industrial camera 27 is fixed on the slider II28 and can move horizontally on the slide rail II24 through the drive of the servo motor II25, so that the industrial camera 27 can collect images of the display screen in 2 degrees of freedom. After the surface first defect detection module 1 completes the detection, it will be transmitted to the second defect detection module 3 for further defect detection.

[0046] Furthermore, if Figure 3 , 12As shown, a centering clamping mechanism module 2 can be provided, including a standard cylinder 16, a cylinder fixing plate 17, a centering pull rod 18, a centering clamping claw 19, a centering plate 20, a centering rotating arm 21, a countersunk screw 22, a slide rail I 23, a hexagonal nut 69, a centering rotating central shaft 70, a coupling I 71, a planetary reducer 72, a servo motor III 73, a center bearing I 74, a washer I 75, and a hexagonal bolt 76; the output end of the servo motor III 73 coaxially extends the planetary reducer 72, and together they constitute a reduction servo motor, which is arranged in a linear manner. The planetary reducer 72 is fixedly connected to the cylinder fixing plate 17 by nuts, the output end of the planetary reducer 72 coaxially extends the coupling Ⅰ71, the coupling Ⅰ71 coaxially extends the centering rotating center shaft 70, the center bearing Ⅰ74 coaxially penetrates the centering rotating center shaft 70, the centering rotating arm 21 is fixed to the centering rotating center shaft 70 by the hexagonal nut 69, the servo motor Ⅲ73 is powered and rotated to realize the rotation of the centering rotating arm 21, the washer Ⅰ75 is placed between the centering rotating arm 21 and the hexagonal nut 69, and the end of the centering rotating arm 21 is fixed to the centering rotating center shaft 70 by the hexagonal nut 69. One end of the centering rod 18 is fixed by a hexagonal bolt 76 and a hexagonal nut 69, and the end of the centering rod 18 is fixedly connected to the centering plate 20 by a bolt and nut connection method. A plurality of sets of centering jaws 19 arranged oppositely are fixedly connected to the two centering plates 20 by countersunk screws 22, and each set of centering jaws 19 arranged oppositely moves relatively close to or away from each other along the axial direction of the conveying reel 32 based on the interval channel between the conveying reels 32 of the second defect detection module 3. When the rotating arm 21 rotates, the centering rod 18 pulls the centering plates 20 on both sides. 0 synchronously moves linearly on the slide rail Ⅰ 23, so that the centering clamping claw 19 realizes reciprocating motion to clamp the live tray module 5. When the live tray module 5 is transported to the power transmission position, the four groups of standard cylinders 16 under the cylinder fixing plate 17 will simultaneously lift the centering clamping mechanism module 2 to the conveying path of the second defect detection module 3. The live tray module 5 is clamped and fixed by the centering clamping claw 19, which is convenient for the power supply module on the second defect detection module 3 to power the live tray module 5 and then perform the next defect detection.

[0047] Furthermore, if Figure 5 , 11As shown, a second defect detection module 3 can be set up including: a conveying reel 32, a signal module 33, a power supply module 34, a photoelectric switch 35, a transmission motor 36, a motor reducer 37 and a second detection mechanism. The working principle is the same as the first defect detection module 1 on the surface; the cylinder fixing bracket 65 is fixed on the transmission device and together with the dual-axis cylinder 66, the power cord 64, the power connector box 67, and the power connector 68 form a power supply module, and the transmission motor 36 coaxially extends the motor reducer 37 to drive the conveying reel 32 through the belt drive at the output end to form a transmission device. There are cylinder fixing brackets 65 on both sides of the transmission device fixed by welding. The end of the cylinder fixing bracket 65 is fixedly connected to the double-axis cylinder 66. There is a power connector box 67 at the end of the double-axis cylinder 66. The power connector box 67 is equipped with a power connector 68 connected to the power line 64. When the charged pallet module 5 is transported to the transmission device, it will be transported in a straight line. When the LCD screen is transmitted to the bottom of the power supply module 34, it is detected by the photoelectric switch 35. Then the centering clamping mechanism module 2 is lifted by the standard cylinder 16, and the centering clamp is driven by the servo motor III 73 to fix the charged pallet module 5. At the same time, the double-axis cylinder 66 presses the power connector 68 into the pallet power transmission port to power the LCD screen to be tested. Similarly, the signal module 33, the detection mechanism in the second defect detection module 3 is the same as the detection mechanism in the first defect detection module 1. When the LCD screen to be tested is transmitted to the bottom of the detection mechanism of the second defect detection module 3, it will be powered on and tested for defects with signals. "

[0048] Furthermore, if Figure 6 , 9 As shown in Figure 10, the automatic sorting module 4 may include: a motor bracket 38, a lifting frame 39, a lifting module 40, a lifting frame 41, a baffle 42, a servo motor IV 43, a coupling II 60, a conveying roller 61, a lifting lug 62, and a conveyor belt II 63. The servo motor IV 43, the coupling II 60, the conveying roller 61, and the conveyor belt II 63 constitute a conveying device. The servo motor IV 43 coaxially extends the coupling II 60, and the end of the coupling II 60 is fixedly connected to the conveying roller 61. As the conveying roller 61 rotates, the conveyor belt II 63 is driven to rotate to convey the charged pallet module 5. One or more small conveying devices are fixed to the baffle 42 through the motor bracket 38 (two are shown in the figure. If two are used), the conveying device is fixed to the baffle 42 through the motor bracket 38. Figure 1The structure of the invention can be effectively used for cooperation by adopting only one), the baffle 42 is welded on the lifting frame 39, and is lifted by the lifting module 40 thereon, and the lifting frame 41 is used to support the entire structural device, the lifting module 40 includes a bearing seat 50, a drum cover 51, a transmission belt 52, a large pulley 53, an intermediate shaft 54, a drum 55, a lifting mechanism induction motor II 56, a large pulley 57, an induction motor reducer 57, a small pulley 58, and a wire rope 59; wherein the lifting mechanism induction motor II 56 coaxially extends the induction motor reducer 57, and the end of the induction motor reducer 57 coaxially extends the small pulley 58, the small pulley 58 is connected to the large pulley 53 through the transmission belt 52, and the large pulley 53 coaxially runs through the center of the intermediate shaft 54, and the two ends of the intermediate shaft 54 ​​are coaxially extended with a drum 55, and the drum covers 51 are coaxially extended on both sides of the drum 55 for fixing, and the two ends of the intermediate shaft 54 ​​are coaxially extended with a bearing seat 50. When the drum 55 rolls and drives the wire rope 59 to contract, the lifting ear 62 welded on the lifting frame 39 connected to the end of the wire rope 59 and the small conveying device therein can be pulled to achieve vertical up and down movement. Display screens with qualified product quality will directly enter the lower layer of the conveying module 6 through the conveyor belt, and display screens with quality problems will be lifted by the lifting mechanism in the automatic sorting module 4 to the upper layer of the conveying module 6 for transmission, thereby realizing the sorting function.

[0049] Furthermore, if Figure 8 As shown, a live tray module 5 can be set up to include: a power interface 46, a display tray 47, a liquid crystal display screen 48 to be detected, and a signal interface 49; the liquid crystal display screen 48 to be detected produced on the production line is placed on the display tray 47 by the staff, on which a fixed position power connector 46 is provided, and the liquid crystal display screen is placed in the display tray 47, and a line is arranged inside the display tray 47 to connect the signal interface 49 and the power interface 46 for power supply. When the signal interface 49 and the power interface 46 are powered on, the liquid crystal display screen 48 to be detected will have a real-time signal display on the display tray 47, so that the display screen can perform live signal defect detection in the second defect detection module 3.

[0050] Furthermore, if Figure 7 As shown, the conveying module 6 may include: an unpowered roller 44 and a conveying frame 45. The unpowered roller 44 is a conveying component, which is arranged in two layers, and specifically, the unpowered roller 44 and the conveying frame 45 form a simple double-layer sorting and conveying device through bolts. After the display screen passes a series of tests, it will be sorted by the automatic sorting module 4. The LCD screen with qualified quality will be transported to the next process through the lower layer of the conveying module 6, and the LCD screen with unqualified product quality will be transported by the automatic sorting module 4 to the upper layer of the conveying module 6 for subsequent maintenance and processing.

[0051] The training method of the liquid crystal display screen visual defect detection model established by the present invention is given as follows, and the specific steps of the method are as follows:

[0052] Step 1, constructing a liquid crystal display visual defect image dataset; dividing the liquid crystal display visual defect image dataset into a training dataset and a verification dataset;

[0053] Step 2: Use annotation tools to annotate the training data set and the validation data set to obtain the training set and the validation set;

[0054] Step 3, constructing a LCD display visual defect detection model based on the backbone network (Backbone) module, the neck (Neck) module and the decoupling head (YoloHead) module;

[0055] Step 4: Modify the hyperparameters in the configuration file; call the training set and configuration file to train the LCD display visual defect detection model, and obtain the candidate weights after the training;

[0056] Step 5: Evaluate the performance of the candidate weights using the validation set to quantify the performance of the candidate weights, load the optimal weights into the LCD visual defect detection model, and obtain a deep LCD visual defect detection model after loading the optimal weights;

[0057] Step 6: Input the newly acquired LCD screen defect image to be detected into the LCD screen visual defect detection model loaded with the optimal weights for detection to obtain a prediction result.

[0058] Furthermore, the following implementation process is given:

[0059] The step 1 constructs a dataset, and the constructed LCD visual defect image dataset contains 1448 valid defect images with a size of 2590×1942. Among them, there are 248 White spot mura images, 849 Black spot mura images, 185 Line mura images, and 318 Foreign matter images. After subdividing these defect-crossing images, 152 images containing multiple defects (from the same class or multiple classes) are summarized.

[0060] The specific steps of the step 2 labeling are as follows: use LabelImg labeling software to label the defective targets in the visual inspection data set, and the labeling includes: the horizontal and vertical coordinates of the center position of the target information, the length and width of the bounding box, and the foreign matter category; when labeling, it is necessary to determine the category of the Mura defect of the LCD display, and name it as White spot mura, Black spot mura, Line mura, and Foreign matter defect; and divide the labeled data set into 1172 training images, 131 verification images, and 145 test images.

[0061] The construction process of the LCD display visual defect detection model is as follows:

[0062] like Fig.14 As shown in the figure, the present invention improves the SPP module located between the CBS module and the CSPLayer module in the Backbone in the YOLOX backbone network module and names it as the SPPA module. The SPPA module improves the network's detection capability for small targets, such as Fig.15 As shown, the SPPA module structure consists of two CBFR modules (CBFR1 and CBFR2), AdapAvgPool (AdaptiveAverage Pooling) and one CBSG module; the specific process is: the second output feature feat2 obtained by the backbone network module is input into the CBS module for feature integration. This integration method helps to capture the key features of the Mura defect, and then the defect features are passed to the SPPA module for further processing. Since the CBFR module helps to extract the detail information of the Mura defect in the display screen and alleviate the gradient disappearance problem, after the feature image is input into the CBFR1 module in parallel, a 1×1 convolution with a step size of 1 can be used for feature integration; at the same time, AdapAvgPool is used to fix the feature size, because the adaptive pooling operation can adapt to input feature maps of different sizes, which helps to capture the global context information of the Mura defect; the feature information processed by the CBFR1 module and the AdapAvgPool is multiplied, and then added with the feature information processed by the CBFR2 module, and finally input into the CBSG module for a 1×1 convolution operation with a step size of 1. As shown Fig.15As shown in the figure, the CBFR module consists of convolution Conv, BN (BatchNormalization) and FReLU connected in sequence. Among them, BN can effectively avoid model overfitting while improving the convergence performance of the model. In order to effectively solve the problem of missing detail information and gradient disappearance of small targets, this paper adopts FReLU activation function. Compared with ReLU and SiLU, FReLU introduces learnable parameters and can adaptively adjust the shape of the activation function, thereby providing greater flexibility and feature representation capabilities. In contrast, the shapes of ReLU and SiLU are fixed and cannot adapt to complex feature distributions; Fig. 20 Experimental results show that compared with ReLU and SiLU, FReLU achieves better results in mAP. Compared with the SPP module, the SPPA module can better utilize the spatial information in the input image and has better adaptability to input images of different sizes, which helps to improve the robustness and generalization ability of the model. By setting a fixed output size, feature maps of different input sizes can be adapted. Adaptive pooling helps capture the global contextual information of the input feature map and convert it into a fixed-size feature representation, which helps to improve the model's perception of Mura defects on the display and enhances the model's expressiveness.

[0063] The ACBAM module is added to the Neck module. We combine the advantages of channel attention and spatial attention, improve the CBAM module and name it the ACBAM module. Fig.16 As shown in FIG. 1 , the present invention connects the channel attention and spatial attention branches in parallel, which can comprehensively utilize the key information of the channel and space, and help improve the network's ability to perceive and distinguish the Mura defects of the display screen. The channel attention and spatial attention branches are both connected by global average pooling and global maximum pooling to make up for the disadvantage of losing too much defect information due to single pooling. The channel attention of the present invention first inputs the feature map F input (Input feature) performs global maximum pooling and global average pooling respectively to obtain two H×W×1 feature maps. Secondly, the two feature maps are respectively subjected to a 1×1 convolution to realize the change of the number of channels, and a ReLU activation function is used to increase the nonlinear expression ability of the neural network. Then, they are respectively subjected to a 1×1 convolution to restore the number of channels and then summed; finally, after the Sigmoid function activation operation, the channel attention map Mc is generated. Spatial attention first takes the input feature map F input(Input feature) performs global maximum pooling and global average pooling respectively to obtain two H×W×1 feature maps; secondly, the two feature maps are concat spliced ​​in the channel dimension; then, a 7×7 convolution operation is performed to adjust the number of channels to obtain an H×W×1 feature map, which is then activated by a Sigmoid function to generate a spatial attention map Ms. Finally, Mc (Channel attention map Mc) and Ms (Spatial attention map Ms) are respectively compared with the input feature map F input (Input feature) is multiplied to obtain two neck splicing feature maps F Mcoutput 、F Msoutput Perform the Concat operation to obtain the feature map F concat (concat feature), and then a 1×1 convolution is performed to restore the number of channels to obtain the output feature map F output (output feature) (i.e. the middle feature map of the neck), its formula is:

[0064] F Mcoutput =M c (F input )×F input

[0065] F Msoutput =M s (F input )×F input

[0066] F concat =Cat(F Mcoutput ,F Msoutput )

[0067] F output =Conv 1×1 (F concat )

[0068] Based on the above channel attention and spatial attention branches, the feature map F is obtained by parallel connection operation, concat concatenation and convolution. output The key display Mura defect information is retained, and a feature representation with rich semantic information is generated, which improves the accuracy and robustness of Mura defect detection and can be better used for prediction in the subsequent YoloHead detection part.

[0069] In the neck module network, the downsampling ordinary convolution module CBS module located between the CSPLayer module and the Concat module is replaced with a fine-grained separable convolution FGSCM module. This paper uses the FGSCM module to enhance the feature extraction capability of the display screen Mura defects, effectively alleviating the problem of missing detail information due to the large network receptive field, so that the network can capture more refined feature information to improve the accuracy and robustness of Mura defect detection. Depth-separable convolution is one of the commonly used operations in CNN. By decomposing the standard convolution into two steps of depth convolution and point-by-point convolution, the amount of calculation and the amount of parameters are effectively reduced. In the FGSCM module, the present invention reduces the convolution kernel of the depth-separable convolution from 3×3 to 2×2, so that each input channel in the depth convolution stage can be independently convolved with the corresponding 2×2 convolution kernel. The output channels are linearly combined through point-by-point convolution, and the change in the size of the convolution kernel will make the receptive field smaller and capture more refined features in the local area. Deep convolution independently performs convolution operations on each channel of the input feature map. Each channel uses a two-dimensional convolution kernel to perform convolution operations with the feature map of the corresponding channel, which means that the feature map of each channel has a separate convolution kernel for processing. Deep convolution produces an intermediate feature map with the same number of channels as the input feature map. This can improve the feature extraction ability of the model and enhance the model's understanding and representation of the input image. We apply point-by-point convolution to linearly combine the channels of the intermediate feature map, use a one-dimensional convolution kernel of size 1×1 to convolve each channel of the intermediate feature map, and linearly combine the information of different channels to generate the final output feature map. Deep convolution is responsible for processing feature maps in the channel dimension and capturing the spatial feature relationship within the channel. Point-by-point convolution integrates and fuses information from different channels to enhance the expressiveness of features. The constructed fine-grained separable convolution FGSCM can not only reduce the amount of model parameters, but also because the convolution kernel is changed from 3×3 to 2×2, it can capture more refined features in local areas. FGSCM module such as Fig.17 shown.

[0070] The specific steps of step 4 are as follows: The graphics card model for experimental training in the present invention is NVIDIA GeForce RTX 4070Ti, the pre-trained weights provided by the COCO data set are loaded into the LCD visual defect detection model framework constructed by the present invention, and the parameters of the network model are fine-tuned using the display screen Mura defect data set, and the weight parameters with the best detection effect are obtained by continuous parameter adjustment and iterative training. The image input size during all network iterative training is 640×640, the training process is a non-freezing trunk training method and a cosine annealing learning rate is used during the training process, and half-precision training is turned on, using the Adam optimizer, setting the parameter batch size to 16, the learning rate to 0.0001, the momentum, weight decay coefficient and confidence threshold to 0.937, 0 and 0.4 respectively, and the training epoch is 210.

[0071] The specific steps of step 5 are as follows: input the validation set into the trained deep learning network model to perform performance evaluation on all the enhanced training weight parameters obtained in step 4, and screen out the optimal enhanced training weight parameters; wherein the validation set used for quantitative performance evaluation is the validation set in the liquid crystal display Mura defect image dataset, and the evaluation criteria can be judged by average precision, frame rate, missed detection rate, false detection rate and accuracy rate.

[0072] The present invention verifies the performance of the improved network architecture of YOLOX by means of ablation experiments. In order to verify the influence of SPPA module, ACBAM module and FGSCM module on network performance, the present invention sets 8 different combination modes. The experimental results of each combination mode are shown in the figure. Fig.18 shown.

[0073] like Fig.18 As shown in the figure, after replacing the SPP module in the original YOLOX network with the SPPA module (i.e. Add-Sa), the map accuracy decreased by 6.45%, which shows that the improvement caused the depth of the original network to increase, resulting in the loss of feature details. In addition, the detection accuracy of Line mura and White spot mura defects decreased after the improvement, but the accuracy of Black spot mura and Foreign matter defects remained almost unchanged, indicating that the improved YOLOX network over-suppressed the background of large target Line mura and low-contrast White spot mura, resulting in reduced detection accuracy.

[0074] Compared with the YOLOX network with the SPPA module added alone (i.e. Add-Sa), the mAP accuracy increased by 10.68% after the ACBAM module was added (YOLO X+SPPA module+ACBAM module, i.e. Add-Sa-Ac), the accuracy of Black spot mura increased by 12.49%, and the accuracy of White spot mura increased by 24.17%. This shows that after the SPPA module is combined with the ACBAM module, the background over-suppression is effectively alleviated. Therefore, SPPA can make the model pay more attention to important features and learn the importance weights of different positions in the feature map. The way of weighting features can increase the model's perception of key features and improve the model's expression ability and performance. This feature fusion method that promotes accuracy improvement shows that the ACBAM module inherits the advantages of channel attention and spatial attention. That is, it makes up for the loss of too much information in a single pooling, can more effectively utilize the detail information between the upper and lower layers of the feature map, and improve the model's expression ability for small targets. However, after the ACBAM module is added to the YOLOX algorithm alone (i.e. Add-Ac), the mAP value is 0.08% lower than the original YOLOX algorithm. This shows that the ACBAM module does not cause the loss of detailed information. Combining the SPPA module and the ACBAM module and embedding them into the network can increase the depth of the network model while paying more attention to small targets.

[0075] After adding the FGSCM module (YOLOX+SPPA module+ACBAM module+FGSCM module, i.e. Add-Sa-Ac-F), the mAP accuracy increased by 0.16% compared with the "YOLOX+SPPA module+ACBAM module". At the same time, the model size was reduced by about 2.56M, the detection accuracy of Black spot mura was improved by 0.48%, and the detection accuracy of White spot mura was improved by 0.14%. This shows that the model improves the accuracy while reducing the receptive field, making the network pay more attention to special small targets.

[0076] After replacing the SPPA module with SPP, only the ACBAM+FGSCM module (YOLOX+ACBAM module+FGSCM module, i.e. Add-S-Ac-F) is added. Compared with "Add-Sa-Ac-F", the accuracy of Black spot mura is reduced by 1.68%, and the accuracy of White spot mura is reduced by 5.1%. This also verifies the effect of the combination of the SPPA module and the CBAM module on small targets, and the FPS will be reduced by about 25.95. After removing the ACBAM module, only the SPPA+FGSCM module (i.e. Add-Sa-F) is added, and the accuracy is not significantly improved compared with the baseline YOLOX network. When only the FGSCM module (i.e. Add-SF) is added, the detection accuracy of Black spot mura is improved by 9.26% compared with the baseline YOLOX network, and the detection accuracy of White spot mura with low contrast is slightly reduced by 0.77%. This also verifies that the FGSCM module is very effective in focusing on special areas of small targets. These research results show that YOLO-SPPAM is effective in detecting display defects, and greatly improves the detection accuracy of the model while ensuring that the network has a high detection speed.

[0077] In addition, the present invention also compares different attention mechanism modules, and verifies that the attention mechanism ACBAM module proposed in the present invention has more advantages in improving detection accuracy. We select three excellent attention mechanisms, SENet, ECANet, and GAMNet, to replace ACBAM. Fig.19 The results of the comparison of three indicators show that the ACBAM module proposed in the present invention has the highest mAP, reaching 97.11%. Although the model scale of the present invention is larger than SENet, ECANet and GAMNet, the speed reaches 110.23FPS, which can fully meet the high detection speed requirements of factory assembly line production.

[0078] By applying the above technical solution, it can be known that the LCD visual defect detection model disclosed by the present invention successfully solves a difficult problem in the current automated assembly line operation of LCDs: LCD Mura defects are small in size, low in contrast, irregular in shape, and appear at random locations, so traditional detection methods are difficult. The present invention combines deep learning methods to perform defect detection and analysis, outputs the test results, and then sorts them through an automatic sorting module, and finally successfully screens out unqualified products. The present invention can not only effectively improve the accuracy of LCD Mura defect detection, but also improve the efficiency of display screen sorting.

[0079] The specific implementation modes of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for establishing a visual defect detection model for a liquid crystal display screen, characterized in that: The invention is established based on a backbone network module, a neck module and a decoupling head module connected in sequence; the neck module adds an improved CBAM module and introduces a fine-grained separable convolution module to replace the CBS module located between the CSPLayer module and the Concat module; the LCD screen image is input into the backbone network module to extract features at all levels; the features at all levels are input into the neck module to obtain three neck output features; The output features of the three necks are sent to each decoupling head module for prediction to obtain the location, type and confidence information of the target information in the LCD screen image; The improved CBAM module includes four modules: ACBAM1 module, ACBAM2 module, ACBAM3 module, and ACBAM4 module. The ACBAM1 module is used to connect the first output of the backbone network module, the ACBAM2 module is used to connect the second output of the backbone network module, and the ACBAM3 module is used to connect the third output of the backbone network module. The neck module is connected to the Concat of the output of the ACBAM2 module and then connected to the ACBAM4 module. The ACBAM1 module, the ACBAM2 module, the ACBAM3 module, and the ACBAM4 module have the same structure and are described by the ACBAM1 module: the channel attention and the spatial attention branches are connected in parallel to obtain the first neck splicing feature 1 and the first neck splicing feature 2, and the obtained first neck splicing feature 1 and the first neck splicing feature 2 are subjected to a Concat splicing operation, and then a convolution is performed to obtain the first neck intermediate feature map. The channel attention and spatial attention branches are connected in parallel to obtain the first neck splicing feature 1 and the first neck splicing feature 2, specifically: the channel attention inputs the feature map F input Perform global maximum pooling and global average pooling respectively to obtain two first feature maps; the two first feature maps are respectively subjected to a 1×1 convolution, a ReLU activation function, and a 1×1 convolution operation, and then summed; the result of the summation operation is activated by a Sigmoid function to generate a channel attention map; the channel attention map is then combined with the input feature map F input Multiply to get the first neck splicing feature 1; spatial attention will input feature map F input Perform global maximum pooling and global average pooling respectively to obtain two second feature maps; perform Concat splicing operation on the two second feature maps in the channel dimension, and after the Concat splicing operation, perform a 7×7 convolution operation and a Sigmoid function activation operation to generate a spatial attention map; the spatial attention map is then combined with the input feature map F input Multiplying is performed to obtain the first neck joint feature 2; The fine-grained separable convolution module includes two modules: FGSCM1 module and FGSCM2 module. The output of CSPLayer module is used as the input of fine-grained separable convolution module, and the output of fine-grained separable convolution module is used as the input of Concat module. The FGSCM1 module and FGSCM2 module have the same structure, and the FGSCM1 module is used for explanation: the CBS module is decomposed into depth convolution and point-by-point convolution. Among them, the convolution kernel size of the depth-separable convolution is 2×2 and the step size is 2. The point-by-point convolution uses a one-dimensional convolution kernel with a size of 1×1 to perform convolution processing on each channel of the intermediate feature map, and linearly combines the information of different channels to generate the final output feature map. The backbone network module introduces the SPPA module to replace the SPP module located between the CBS module and the CSPLayer module in the traditional backbone network module; the SPPA module structure consists of two CBFR modules, AdapAvgPool and one CBSG module, and the two CBFR modules are CBFR1 and CBFR2; the output of CBFR1 is multiplied by the output of AdapAvgPool, added to the output of CBFR2, and then input into the CBSG module.

2. A method for detecting visual defects of a liquid crystal display screen, characterized in that: include: Get the LCD screen image when it is not powered; Calling the LCD visual defect detection model described in claim 1 to detect the image of the LCD in an unpowered state to obtain a first detection result; Get the LCD screen image under power supply state; Calling a liquid crystal display visual defect detection model to detect an image of the liquid crystal display in a power-on state to obtain a second detection result; When the first detection result and / or the second detection result is identified as having a defect, it is considered that a defect exists.

3. A liquid crystal display screen visual defect detection and sorting system, characterized in that: include: A first defect detection module (1) is used to obtain an image of a liquid crystal display screen in an unpowered state; Used to call the LCD visual defect detection model according to claim 1 to detect the image of the LCD in an unpowered state to obtain a first detection result; A second defect detection module (3) is used to obtain an image of the liquid crystal display screen in a power supply state; Calling the LCD screen visual defect detection model to detect the LCD screen image in the power-on state to obtain a second detection result; The automatic sorting module (4) is used to convey the defective liquid crystal display screen to the first conveying component when the first detection result and / or the second detection result is identified as defective, and otherwise convey the liquid crystal display screen to the second conveying component.

Images