Product appearance defect detection system and detection equipment based on multi-modal model

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, and discloses a product appearance defect detection device based on a multi-modal model, the product appearance defect detection device comprises a base and an image recognition device, the image recognition device has multiple operation modes of vision, infrared and 3D scanning, and an auxiliary support and a support ring frame are respectively mounted at the top and in the middle of the base. According to the product appearance defect detection system and detection equipment based on the multi-modal model, an arranged image recognition device is driven by a corresponding third meshing output part, a second gear sleeve plate and a movable annular plate, and the image recognition device rotates and is adjusted around a composite detection supporting part; according to the invention, the image recognition device is integrated with visual data, infrared data, 3D scanning data and other modal data, so that the whole equipment can capture and analyze product appearance information in more dimensions, and even defects which are sensitive to light or hidden can be accurately recognized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and specifically to a product appearance defect detection system and detection equipment based on a multimodal model. Background Art

[0002] Traditional product appearance inspection generally uses the naked eye to identify, so human factors may lead to inconsistent measurement standards, and long-term inspection may lead to misjudgment due to visual fatigue. With the in-depth development and cooperation of computer technology and optical, mechanical, electrical and other technologies, researchers in related fields have developed image recognition technology, which has the characteristics of fast and accurate detection and can identify targets and objects of various different modes. As a prior art, image recognition devices based on image recognition technology have long been put into use. For example, the patent document with application number 202210937799.1 discloses that "an embodiment of the present invention provides a PCB board detection device and image recognition method based on image recognition. The method includes: if a detection instruction is received, an initial image of a PCB board acquired by an image acquisition device is obtained; pixel dissolution of the initial image is performed to determine the circuit line contour corresponding to the initial image; a non-continuous area is screened from the circuit line contour according to a preset contour screening rule; a regional image of the non-continuous area in the initial image is obtained; and uniformity recognition is performed on each regional image to obtain a recognition result of whether the initial image meets a preset uniformity condition."

[0003] However, when applying to use the existing product appearance defect detection system, it was found that the existing system usually relies on a single image recognition technology, and has problems such as low recognition accuracy and poor detection of complex defects. In recent years, with the rapid development of deep learning technology, large multimodal models have demonstrated powerful capabilities in processing a variety of media data. Therefore, after research and experiments, the applicant will provide a product appearance defect detection system and detection equipment based on a multimodal model to solve the problems existing in the existing technology. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The present invention provides a product appearance defect detection system and detection equipment based on a multimodal model, which solves the problems raised in the above background technology.

[0006] (II) Technical solution

[0007] The present invention provides the following technical solutions: a product appearance defect detection device based on a multimodal model, comprising a base, an image recognition device, the image recognition device having multiple operation modes of vision, infrared, and 3D scanning, an auxiliary bracket and a support ring bracket are respectively installed on the top and the middle of the base, a slide rail is fixedly connected to the top of the auxiliary bracket, a detection platform frame is slidably connected to the slide rail, a clearance groove is arranged inside the detection platform frame, a composite detection support component is installed in the clearance groove, and a transparent detection support space is arranged on the top of the composite detection support component;

[0008] A movable annular plate is mounted on the inner side of the support ring frame through a bearing sleeve, the image recognition device is installed on the inner wall at the top of the movable annular plate and aligned with the composite detection support component, a second gear sleeve is mounted on the surface of one side of the movable annular plate, and the bottom of the second gear sleeve is transmission-connected to a third meshing output component installed on the top of the base, and the image recognition device can be rotated and adjusted around the composite detection support component under the transmission of the third meshing output component, the second gear sleeve and the movable annular plate.

[0009] Preferably, a guide groove capable of engaging with a slide rail is provided at the bottom of the detection platform frame to ensure stability during the subsequent movement, and weight-reducing grooves are provided at the front and rear ends of the middle portion of the detection platform frame.

[0010] Preferably, the composite detection support component includes a transparent plate, a combined ring frame, a first gear sleeve, a first transmission gear and a first brake servo motor, the transparent plate is mounted inside the combined ring frame through a bearing, the top of the first gear sleeve is installed on the bottom of the inner ring structure of the bearing, the middle part of the first transmission gear is transmission-connected to the output end of the first brake servo motor, and the first transmission gear is meshingly connected to the first gear sleeve, a support frame is installed between the shell surface of the first brake servo motor and the rear end surface of the transparent plate, and the combined ring frame is mounted inside the give way groove.

[0011] Preferably, the third meshing output component includes a third transmission gear and a third brake servo motor. The middle part of the third transmission gear is transmission-connected to the output end of the third brake servo motor, and the third transmission gear is meshingly connected to the bottom of the second gear sleeve. A support seat is installed between the shell surface of the third transmission gear and the top surface of the middle part of the base. The third meshing output component is combined with the second gear sleeve so that the image recognition device and the movable annular plate have the function of automatic rotation adjustment, which meets the needs of multi-angle and multi-position image data collection and recognition of products.

[0012] Preferably, the rear end of the give way slot is an open structure, and the number of the give way slots is five and is equidistantly distributed on the top of the detection platform frame, and the number of the composite detection support components is the same as the number of the give way slots, thereby satisfying the use effect of batch detection of products.

[0013] Preferably, the detection platform frame is a hollow structure, and ten assembly grooves are opened on the top of the detection platform frame. The ten assembly grooves are arranged in groups of two on the outside of the five makeshift grooves, and the assembly grooves are communicated with the corresponding makeshift groove spaces, providing auxiliary space for the assembly of the detection platform frame and the composite detection support component.

[0014] Preferably, two combination plates are fixedly connected to the surface of the combination ring frame, and the combination ring frame and the two combination plates on its own structure are respectively clamped in the corresponding make way grooves and two assembly grooves, and the interior of the detection platform frame is fixedly sleeved with an electromagnet that can be magnetically connected to the corresponding combination plate, thereby ensuring the stability of the composite detection support component after the detection platform frame is assembled.

[0015] Preferably, a spur-tooth plate is fixedly connected to the bottom surface of the front end of the detection platform frame, and both ends of the spur-tooth plate are transmission-connected with a second meshing output component, and the two second meshing output components each include a second transmission gear and a second brake servo motor, and the output end of the second brake servo motor is transmission-connected to the middle part of the second transmission gear, and the two second transmission gears are respectively meshingly connected to the two ends of the spur-tooth plate, so as to provide reciprocating power for batch use of multiple composite detection support components on the detection platform frame and improve the automation operation performance of the overall device.

[0016] Preferably, an auxiliary cleaning component is provided at the top of the front end of the detection platform frame, and the auxiliary cleaning component includes a cleaning bracket and a cleaning sponge bar. The rear end of the cleaning bracket is fixedly connected to the surface of the top of the front end of the detection platform frame, and the rear end of the cleaning sponge bar is fixedly connected to the front end of the cleaning bracket. The auxiliary cleaning component is located on the trajectory of the image recognition device that relies on the rotation adjustment of the movable annular plate, and can then assist in cleaning the lens of the image recognition device during the rotation and movement of the image recognition device.

[0017] A product appearance defect detection system based on a multimodal model includes the following steps:

[0018] S1. Data collection and processing

[0019] The product to be tested placed on the top of the composite detection support component is rotated and adjusted around the composite detection support component by the third meshing output component, the second gear sleeve plate and the movable annular plate, and the product appearance images are collected from multiple angles and multiple positions to form image and video data, and then data preprocessing and feature extraction are performed, and finally the feature information of different modalities of the hyperspectral image, depth image and RGB image of the product to be tested are fused together to form a comprehensive feature representation;

[0020] Data collection is the first key step in the data processing process, which aims to collect and organize raw data and provide necessary input data for subsequent data preprocessing, feature extraction, feature fusion, and defect detection. This process ensures the accuracy and effectiveness of subsequent data processing and analysis.

[0021] Data preprocessing: In the product appearance defect detection system based on multimodal large models, the data preprocessing link is responsible for performing a series of processing operations on the collected raw data to improve data quality and ensure the accuracy and effectiveness of subsequent feature extraction and defect detection steps. These processing operations include but are not limited to noise suppression, image enhancement, image normalization, data standardization, data balancing, etc., which aim to eliminate or reduce noise, redundancy and irrelevant information in the raw data, highlight key features, and adjust data distribution to make it more in line with the input requirements of the model. Through data preprocessing, the detection performance and stability of the system can be significantly improved;

[0022] Feature extraction refers to extracting key information or attributes from preprocessed image data. These key information or attributes are called features, which are further converted into feature vectors for subsequent analysis, comparison, and processing of defect detection algorithms. Feature extraction is a key step in image processing, computer vision, and machine learning. It can capture the essential and representative elements in image data, thereby simplifying data representation and improving algorithm performance and accuracy. In product appearance defect detection systems based on multimodal large models, feature extraction is an important bridge connecting image data and defect detection algorithms;

[0023] Feature fusion is the process of effectively integrating feature information from different modalities. By combining features from different sources, more comprehensive and richer information can be provided for subsequent defect detection algorithms. In the product appearance defect detection system based on multimodal large models, feature fusion technology can fuse feature information from different modalities such as hyperspectral images, depth images, and RGB images to form a comprehensive feature representation, thereby improving the accuracy and robustness of defect detection.

[0024] S2. Multimodal large model construction

[0025] Multimodal large model construction is a deep learning model that can process multiple media data such as text, images, and videos. The specific construction includes model architecture design and pre-training;

[0026] The model architecture design is the process of building a suitable network structure based on specific data characteristics and task requirements, aiming to provide a stable and efficient basic framework for subsequent model training. This step is crucial to ensure that the model can accurately capture key information in the data and improve the accuracy and efficiency of defect detection. In the product appearance defect detection system based on multimodal large models, the model architecture design needs to comprehensively consider the diversity and complexity of the data and the specific requirements of the detection task in order to design the optimal model structure;

[0027] Pre-training aims to enable the model to learn general representation capabilities. By training on large-scale datasets, the model can learn rich prior knowledge and provide strong support for fine-tuning the model for subsequent specific tasks. This general representation capability helps the model converge faster on new and unseen tasks and improves performance.

[0028] S3. Model fine-tuning

[0029] Model fine-tuning specifically includes data set preparation, model loading, model training, and model evaluation and optimization. Data set preparation is the basis of model fine-tuning, providing necessary data support for model training. Model loading is the starting point of model fine-tuning, providing existing knowledge and capabilities as a reference. Model training is the core link of model fine-tuning, through which the model learns the characteristics of specific tasks. Model evaluation and optimization are the guarantee links of model fine-tuning, ensuring the stability and accuracy of the model in practical applications.

[0030] S4. Structure display and feedback

[0031] The product appearance defect detection system will eventually visualize the detected defects, such as marking the defect location, displaying the defect type, etc. At the same time, it will generate a test report to provide timely feedback to production personnel.

[0032] Beneficial Effects

[0033] The present invention has the following beneficial effects:

[0034] 1. The product appearance defect detection system and detection equipment based on the multimodal model, through the set image recognition device, under the transmission of the corresponding third meshing output component, the second gear sleeve plate and the movable annular plate, the image recognition device rotates and adjusts around the composite detection support component, and then performs multi-angle and multi-position image collection and recognition detection. The image recognition device integrates multiple modal data such as vision, infrared, 3D scanning, etc., so that the overall equipment can capture and analyze more dimensional product appearance information, even light-sensitive or hidden defects can be accurately identified, significantly reducing the false detection and missed detection rates, and improving the overall detection accuracy to a new level.

[0035] 2. The product appearance defect detection system and detection equipment based on the multimodal model can still maintain stable and reliable detection performance in complex environments such as different lighting conditions, product materials, color backgrounds, etc., reducing the stringent requirements on external conditions and enhancing the versatility and robustness of the system.

[0036] 3. The product appearance defect detection system and detection equipment based on multimodal models realizes a fully automated process from data collection to defect identification and classification, greatly shortening the detection cycle and reducing labor costs. At the same time, it improves the consistency and repeatability of detection, which is suitable for the high-speed quality control needs of large-scale production lines. Combined with advanced deep learning frameworks, the system can continuously learn according to real-time detection feedback, automatically optimize the detection model, adapt to changes in product design and the emergence of new defect patterns, and maintain the cutting-edge nature and long-term effectiveness of detection technology.

[0037] 4. The product appearance defect detection system and detection equipment based on multimodal models can significantly improve the product quality monitoring capabilities of the manufacturing industry, reduce the outflow of defective products, and enhance consumer trust. At the same time, it can provide data support for production process optimization and cost control, accelerate the intelligent transformation of enterprises, and promote the improvement of the overall quality management level of the industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a front view schematic diagram of the present invention;

[0039] Figure 2 It is a rear view schematic diagram of the present invention;

[0040] Figure 3 is an enlarged schematic diagram of the auxiliary bracket of the present invention;

[0041] Figure 4 is an enlarged schematic diagram of the support ring frame of the present invention;

[0042] Figure 5 It is a top view schematic diagram of the detection platform frame of the present invention;

[0043] Figure 6It is a top view schematic diagram of the composite detection support component of the present invention;

[0044] Figure 7 It is a bottom view schematic diagram of the composite detection support component of the present invention;

[0045] Figure 8 is an enlarged schematic diagram of the second meshing output component of the present invention;

[0046] Fig. 9 is an enlarged schematic diagram of the third meshing output component of the present invention;

[0047] Fig.10 It is a partially enlarged schematic diagram of the auxiliary cleaning component of the present invention.

[0048] In the figure: 1. base; 2. image recognition device; 3. auxiliary bracket; 4. slide rail; 5. support ring frame; 6. movable ring plate; 7. detection platform frame; 8. clearance groove; 9. composite detection support component; 91. transparent plate; 92. combination ring frame; 93. first gear sleeve plate; 94. first transmission gear; 95. first brake servo motor; 10. spur plate; 11. second meshing output component; 111. second transmission gear; 112. second brake servo motor; 12. assembly groove; 13. electromagnet; 14. auxiliary cleaning component; 141. cleaning bracket; 142. cleaning sponge strip; 15. second gear sleeve plate; 16. third meshing output component; 161. third transmission gear; 162. third brake servo motor; 17. combination plate. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Embodiment 1

[0051] See also Figure 1-7 , Fig. 9, a product appearance defect detection device based on a multimodal model, comprising a base 1, an image recognition device 2, the image recognition device 2 has multiple operation modes of vision, infrared, and 3D scanning, an auxiliary bracket 3 and a support ring frame 5 are respectively installed on the top and the middle of the base 1, a slide rail 4 is fixedly connected to the top of the auxiliary bracket 3, and a detection platform frame 7 is slidably connected to the slide rail 4, and a guide groove that can be engaged with the slide rail 4 is opened at the bottom of the detection platform frame 7 to ensure the stability of the subsequent movement process, and the front and rear ends of the middle part of the detection platform frame 7 are opened with weight reduction grooves, and a clearance groove 8 is arranged inside the detection platform frame 7, and a composite detection support component 9 is installed in the clearance groove 8, and a transparent detection support space is arranged on the top of the composite detection support component 9;

[0052] The composite detection support component 9 includes a transparent plate 91, a combination ring frame 92, a first gear sleeve plate 93, a first transmission gear 94 and a first brake servo motor 95. The transparent plate 91 is sleeved inside the combination ring frame 92 through a bearing. The top of the first gear sleeve plate 93 is installed at the bottom of the inner ring structure of the bearing. The middle part of the first transmission gear 94 is transmission-connected with the output end of the first brake servo motor 95, and the first transmission gear 94 is meshingly connected with the first gear sleeve plate 93. A support frame is installed between the shell surface of the first brake servo motor 95 and the rear end surface of the transparent plate 91. The combination ring frame 92 is sleeved inside the clearance groove 8.

[0053] The inner side of the support ring frame 5 is sleeved with a movable annular plate 6 through a bearing, the image recognition device 2 is installed on the inner wall of the top of the movable annular plate 6 and aligned with the composite detection support component 9, the surface of one side of the movable annular plate 6 is sleeved with a second gear sleeve plate 15, the bottom of the second gear sleeve plate 15 is transmission-connected to a third meshing output component 16 installed on the top of the base 1, and the image recognition device 2 can be rotated and adjusted around the composite detection support component 9 under the transmission of the third meshing output component 16, the second gear sleeve plate 15 and the movable annular plate 6;

[0054] The third meshing output component 16 includes a third transmission gear 161 and a third brake servo motor 162. The middle part of the third transmission gear 161 is transmission-connected to the output end of the third brake servo motor 162, and the third transmission gear 161 is meshingly connected to the bottom of the second gear sleeve 15. A support seat is installed between the shell surface of the third transmission gear 161 and the top surface of the middle part of the base 1. The third meshing output component 16 is combined with the second gear sleeve 15, so that the image recognition device 2 and the movable annular plate 6 have the function of automatic rotation adjustment, which meets the needs of multi-angle and multi-position image data collection and recognition of products.

[0055] Preferably, the rear end of the give way slot 8 is an open structure, and the number of the give way slots 8 is five and they are evenly distributed on the top of the detection platform frame 7, and the number of the composite detection support components 9 is the same as the number of the give way slots 8, thereby satisfying the use effect of batch detection of products.

[0056] Preferably, the detection platform frame 7 is a hollow structure, and ten assembly grooves 12 are opened on the top of the detection platform frame 7. The ten assembly grooves 12 are arranged in groups of two on the outside of the five makeshift grooves 8, and the assembly grooves 12 are spatially connected with the corresponding makeshift grooves 8, providing auxiliary space for the assembly of the detection platform frame 7 and the composite detection support component 9.

[0057] Preferably, two combination plates 17 are fixedly connected to the surface of the combination ring frame 92, and the combination ring frame 92 and the two combination plates 17 on its own structure are respectively clamped in the corresponding make way groove 8 and the two assembly grooves 12, and the interior of the detection platform frame 7 is fixedly sleeved with an electromagnet 13 that can be magnetically connected to the corresponding combination plate 17, thereby ensuring the stability of the composite detection support component 9 after the detection platform frame 7 is assembled.

[0058] Preferably, a spur plate 10 is fixedly connected to the bottom surface of the front end of the detection platform frame 7, and both ends of the spur plate 10 are transmission-connected with second meshing output components 11. The two second meshing output components 11 both include a second transmission gear 111 and a second brake servo motor 112. The output end of the second brake servo motor 112 is transmission-connected to the middle part of the second transmission gear 111. The two second transmission gears 111 are respectively meshingly connected to the two ends of the spur plate 10, so as to provide reciprocating power for the batch use of multiple composite detection support components 9 on the detection platform frame 7 and improve the automation operation performance of the overall device.

[0059] Preferably, an auxiliary cleaning component 14 is provided at the top of the front end of the detection platform frame 7, and the auxiliary cleaning component 14 includes a cleaning bracket 141 and a cleaning sponge bar 142. The rear end of the cleaning bracket 141 is fixedly connected to the surface of the top of the front end of the detection platform frame 7, and the rear end of the cleaning sponge bar 142 is fixedly connected to the front end of the cleaning bracket 141. The auxiliary cleaning component 14 is located on the trajectory of the image recognition device 2 that is rotated and adjusted by the movable annular plate 6, and can then assist in cleaning the lens of the image recognition device 2 during the rotation and movement of the image recognition device 2.

[0060] S1. Data collection and processing

[0061] The composite detection support component 9 is rotated and adjusted under the transmission of the third meshing output component 16, the second gear sleeve plate 15 and the movable annular plate 6, and the product appearance images of the product to be tested placed on the top of the composite detection support component 9 are collected from multiple angles and multiple positions to form image and video data, and then data preprocessing and feature extraction are performed, and finally the feature information of different modes of the hyperspectral image, depth image and RGB image of the product to be tested are fused together to form a comprehensive feature representation;

[0062] During the adjustment and use of the image recognition device 2, the product to be detected is placed on the top of the transparent plate 91, and then the image recognition device 2 and the third brake servo motor 162 in the third meshing output component 16 are started, and the third brake servo motor 162 drives the third transmission gear 161 to rotate synchronously, and then the third transmission gear 161 meshes with the second gear sleeve plate 15, and finally the movable annular plate 6 drives the image recognition device 2 to surround the product to be detected on the top of the transparent plate 91. At the same time, the image recognition device 2 collects image data of the product to be detected at different angles and positions, which includes the image recognition device 2 collecting images from the transparent plate 91 toward the bottom of the detected product;

[0063] When the workpiece to be inspected is a multi-faceted structure, the first brake servo motor 95 can be started after the image recognition device 2 rotates for one circle, and the first brake servo motor 95 drives the first transmission gear 94 to rotate synchronously, and then the first transmission gear 94 meshes with the first gear sleeve plate 93, and finally the transparent plate 91 drives the workpiece to be inspected to automatically rotate the angle adjustment, and then the above steps are repeated until the image data of the product to be inspected is collected in all directions;

[0064] Data collection is the first key step in the data processing process, which aims to collect and organize raw data and provide necessary input data for subsequent data preprocessing, feature extraction, feature fusion, and defect detection. This process ensures the accuracy and effectiveness of subsequent data processing and analysis.

[0065] Data preprocessing: In the product appearance defect detection system based on multimodal large models, the data preprocessing link is responsible for performing a series of processing operations on the collected raw data to improve data quality and ensure the accuracy and effectiveness of subsequent feature extraction and defect detection steps. These processing operations include but are not limited to noise suppression, image enhancement, image normalization, data standardization, data balancing, etc., which aim to eliminate or reduce noise, redundancy and irrelevant information in the raw data, highlight key features, and adjust data distribution to make it more in line with the input requirements of the model. Through data preprocessing, the detection performance and stability of the system can be significantly improved;

[0066] Feature extraction refers to extracting key information or attributes from preprocessed image data. These key information or attributes are called features. They are further converted into feature vectors for subsequent analysis, comparison, and processing of defect detection algorithms. Feature extraction is a key step in image processing, computer vision, and machine learning. It can capture the essential and representative elements in image data, thereby simplifying data representation and improving algorithm performance and accuracy. In product appearance defect detection systems based on multimodal large models, feature extraction is an important bridge connecting image data and defect detection algorithms.

[0067] Feature fusion is the process of effectively integrating feature information from different modalities. By combining features from different sources, more comprehensive and richer information can be provided for subsequent defect detection algorithms. In the product appearance defect detection system based on multimodal large models, feature fusion technology can fuse feature information from different modalities such as hyperspectral images, depth images, and RGB images to form a comprehensive feature representation, thereby improving the accuracy and robustness of defect detection.

[0068] S2. Multimodal large model construction

[0069] Multimodal large model construction is a deep learning model that can process multiple media data such as text, images, and videos. The specific construction includes model architecture design and pre-training;

[0070] The model architecture design is the process of building a suitable network structure based on specific data characteristics and task requirements, aiming to provide a stable and efficient basic framework for subsequent model training. This step is crucial to ensure that the model can accurately capture key information in the data and improve the accuracy and efficiency of defect detection. In the product appearance defect detection system based on multimodal large models, the model architecture design needs to comprehensively consider the diversity and complexity of the data and the specific requirements of the detection task in order to design the optimal model structure;

[0071] Pre-training aims to enable the model to learn general representation capabilities. By training on large-scale datasets, the model can learn rich prior knowledge and provide strong support for fine-tuning the model for subsequent specific tasks such as product appearance defect detection. This general representation capability helps the model converge faster and improve performance on new and unseen tasks.

[0072] S3. Model fine-tuning

[0073] Model fine-tuning specifically includes data set preparation, model loading, model training, and model evaluation and optimization. Data set preparation is the basis of model fine-tuning, providing necessary data support for model training. Model loading is the starting point of model fine-tuning, providing existing knowledge and capabilities as a reference. Model training is the core link of model fine-tuning, through which the model learns the characteristics of specific tasks. Model evaluation and optimization are the guarantee links of model fine-tuning, ensuring the stability and accuracy of the model in practical applications.

[0074] S4. Structure display and feedback

[0075] The product appearance defect detection system will eventually visualize the detected defects, such as marking the defect location, displaying the defect type, etc. At the same time, it will generate a test report to provide timely feedback to production personnel.

[0076] Embodiment 2

[0077] See also Figure 1-10 , a product appearance defect detection device based on a multimodal model, comprising a base 1, an image recognition device 2, the image recognition device 2 has multiple operation modes of vision, infrared, and 3D scanning, an auxiliary bracket 3 and a support ring frame 5 are respectively installed on the top and the middle of the base 1, a slide rail 4 is fixedly connected to the top of the auxiliary bracket 3, and a detection platform frame 7 is slidably connected to the slide rail 4, and a guide groove that can be engaged with the slide rail 4 is opened at the bottom of the detection platform frame 7 to ensure the stability of the subsequent movement process, and the front and rear ends of the middle part of the detection platform frame 7 are opened with weight reduction grooves, and a clearance groove 8 is arranged inside the detection platform frame 7, and a composite detection support component 9 is installed in the clearance groove 8, and a transparent detection support space is arranged on the top of the composite detection support component 9;

[0078] The composite detection support component 9 includes a transparent plate 91, a combination ring frame 92, a first gear sleeve plate 93, a first transmission gear 94 and a first brake servo motor 95. The transparent plate 91 is sleeved inside the combination ring frame 92 through a bearing. The top of the first gear sleeve plate 93 is installed at the bottom of the inner ring structure of the bearing. The middle part of the first transmission gear 94 is transmission-connected with the output end of the first brake servo motor 95, and the first transmission gear 94 is meshingly connected with the first gear sleeve plate 93. A support frame is installed between the shell surface of the first brake servo motor 95 and the rear end surface of the transparent plate 91. The combination ring frame 92 is sleeved inside the clearance groove 8.

[0079] The inner side of the support ring frame 5 is sleeved with a movable annular plate 6 through a bearing, the image recognition device 2 is installed on the inner wall of the top of the movable annular plate 6 and aligned with the composite detection support component 9, the surface of one side of the movable annular plate 6 is sleeved with a second gear sleeve plate 15, the bottom of the second gear sleeve plate 15 is transmission-connected to a third meshing output component 16 installed on the top of the base 1, and the image recognition device 2 can be rotated and adjusted around the composite detection support component 9 under the transmission of the third meshing output component 16, the second gear sleeve plate 15 and the movable annular plate 6;

[0080] The third meshing output component 16 includes a third transmission gear 161 and a third brake servo motor 162. The middle portion of the third transmission gear 161 is transmission-connected to the output end of the third brake servo motor 162, and the third transmission gear 161 is meshingly connected to the bottom of the second gear sleeve 15. A support seat is installed between the shell surface of the third transmission gear 161 and the top surface of the middle portion of the base 1. The third meshing output component 16 is combined with the second gear sleeve 15, so that the image recognition device 2 and the movable annular plate 6 have the function of automatic rotation adjustment, which meets the image data collection and recognition of the product at multiple angles and multiple positions;

[0081] The rear end of the clearance slot 8 is an open structure, and the number of the clearance slots 8 is five and equidistantly distributed on the top of the detection platform frame 7. The number of the composite detection support components 9 is the same as the number of the clearance slots 8, thereby meeting the use effect of batch detection of products. The detection platform frame 7 is a hollow structure, and ten assembly slots 12 are opened on the top of the detection platform frame 7. The ten assembly slots 12 are arranged in groups of two on the outside of the five clearance slots 8, and the assembly slots 12 are connected to the corresponding clearance slots 8 to provide auxiliary space for the assembly of the detection platform frame 7 and the composite detection support component 9;

[0082] The surface of the combined ring frame 92 is fixedly connected with two combined plates 17, and the combined ring frame 92 and the two combined plates 17 on its own structure are respectively clamped in the corresponding clearance groove 8 and the two assembly grooves 12, and the internal fixed sleeve of the detection platform frame 7 is an electromagnet 13 that can be magnetically connected with the corresponding combined plate 17 to ensure the stability of the composite detection support component 9 after the detection platform frame 7 is assembled;

[0083] A spur plate 10 is fixedly connected to the bottom surface of the front end of the detection platform frame 7, and both ends of the spur plate 10 are transmission-connected with second meshing output components 11. The two second meshing output components 11 each include a second transmission gear 111 and a second brake servo motor 112. The output end of the second brake servo motor 112 is transmission-connected to the middle of the second transmission gear 111. The two second transmission gears 111 are meshingly connected to the two ends of the spur plate 10, respectively, to provide reciprocating power for batch use of multiple composite detection support components 9 on the detection platform frame 7, thereby improving the automation performance of the overall device.

[0084] An auxiliary cleaning component 14 is provided at the top of the front end of the detection platform frame 7, and the auxiliary cleaning component 14 includes a cleaning bracket 141 and a cleaning sponge bar 142. The rear end of the cleaning bracket 141 is fixedly connected to the surface of the top of the front end of the detection platform frame 7, and the rear end of the cleaning sponge bar 142 is fixedly connected to the front end of the cleaning bracket 141. The auxiliary cleaning component 14 is located on the trajectory of the image recognition device 2 that is rotated and adjusted by the movable annular plate 6, and can then assist in cleaning the lens of the image recognition device 2 during the rotation and movement of the image recognition device 2.

[0085] S1. Data collection and processing

[0086] The composite detection support component 9 is rotated and adjusted under the transmission of the third meshing output component 16, the second gear sleeve plate 15 and the movable annular plate 6, and the product appearance images of the product to be tested placed on the top of the composite detection support component 9 are collected from multiple angles and multiple positions to form image and video data, and then data preprocessing and feature extraction are performed, and finally the feature information of different modes of the hyperspectral image, depth image and RGB image of the product to be tested are fused together to form a comprehensive feature representation;

[0087] During the adjustment and use of the image recognition device 2, the product to be detected is placed on the top of the transparent plate 91, and then the image recognition device 2 and the third brake servo motor 162 in the third meshing output component 16 are started, and the third brake servo motor 162 drives the third transmission gear 161 to rotate synchronously, and then the third transmission gear 161 meshes with the second gear sleeve plate 15, and finally the movable annular plate 6 drives the image recognition device 2 to surround the product to be detected on the top of the transparent plate 91. At the same time, the image recognition device 2 collects image data of the product to be detected at different angles and positions, which includes the image recognition device 2 collecting images from the transparent plate 91 toward the bottom of the detected product;

[0088] When the workpiece to be inspected is a multi-faceted structure, the first brake servo motor 95 can be started after the image recognition device 2 rotates for one circle, and the first brake servo motor 95 drives the first transmission gear 94 to rotate synchronously, and then the first transmission gear 94 meshes with the first gear sleeve plate 93, and finally the transparent plate 91 drives the workpiece to be inspected to automatically rotate the angle adjustment, and then the above steps are repeated until the image data of the product to be inspected is collected in all directions;

[0089] When multiple products to be tested need to be tested in batches, multiple composite testing support components 9 and the testing platform frame 7 are modularly installed. Specifically, the composite testing support component 9 is set in the corresponding give way slot 8. At the same time, the combination plate 17 provided on the surface of the composite testing support component 9 is clamped in the corresponding assembly slot 12. Then, the electromagnet 13 is started, and the electromagnet 13 magnetically connects the combination plate 17 to ensure the stability of the composite testing support component 9 assembled on the testing platform frame 7. The remaining composite testing support components 9 are also assembled according to the above steps.

[0090] After the assembly of the multiple composite detection support components 9 is completed, the multiple products to be inspected are placed one by one on the top of the transparent plate 91 of the multiple composite detection support components 9, and then the two second brake servo motors 112 are synchronously started, so that the second brake servo motors 112 are engaged with the transmission spur plate 10 through the second transmission gear 111, and then the detection platform frame 7 is automatically displaced and adjusted on the top of the auxiliary bracket 3 under the guidance of the slide rail 4, so that the multiple composite detection support components 9 and the corresponding products to be inspected arrive in the detection area of ​​the image recognition device 2 in turn, and then the above-mentioned image recognition device 2 is repeated according to the steps of collecting image data in all directions;

[0091] In order to further ensure the operating accuracy of the image recognition device 2, the cleaning sponge strip 142 is supported by the cleaning bracket 141 and is located on the track of the image recognition device 2 rotating and adjusting by the movable annular plate 6, so that the lens of the image recognition device 2 can be auxiliary cleaned during the rotation and movement of the image recognition device 2;

[0092] Select any composite detection support component 9 and place the standard component on top of it, and then repeat the above-mentioned steps of collecting omnidirectional image data by the image recognition device 2 to obtain the detection data of the control group.

[0093] Data collection is the first key step in the data processing process, which aims to collect and organize raw data and provide necessary input data for subsequent data preprocessing, feature extraction, feature fusion, and defect detection. This process ensures the accuracy and effectiveness of subsequent data processing and analysis.

[0094] Data preprocessing: In the product appearance defect detection system based on multimodal large models, the data preprocessing link is responsible for performing a series of processing operations on the collected raw data to improve data quality and ensure the accuracy and effectiveness of subsequent feature extraction and defect detection steps. These processing operations include but are not limited to noise suppression, image enhancement, image normalization, data standardization, data balancing, etc., which aim to eliminate or reduce noise, redundancy and irrelevant information in the raw data, highlight key features, and adjust data distribution to make it more in line with the input requirements of the model. Through data preprocessing, the detection performance and stability of the system can be significantly improved;

[0095] Feature extraction refers to extracting key information or attributes from preprocessed image data. These key information or attributes are called features. They are further converted into feature vectors for subsequent analysis, comparison, and processing of defect detection algorithms. Feature extraction is a key step in image processing, computer vision, and machine learning. It can capture the essential and representative elements in image data, thereby simplifying data representation and improving algorithm performance and accuracy. In product appearance defect detection systems based on multimodal large models, feature extraction is an important bridge connecting image data and defect detection algorithms.

[0096] Feature fusion is the process of effectively integrating feature information from different modalities. By combining features from different sources, more comprehensive and richer information can be provided for subsequent defect detection algorithms. In the product appearance defect detection system based on multimodal large models, feature fusion technology can fuse feature information from different modalities such as hyperspectral images, depth images, and RGB images to form a comprehensive feature representation, thereby improving the accuracy and robustness of defect detection.

[0097] S2. Multimodal large model construction

[0098] Multimodal large model construction is a deep learning model that can process multiple media data such as text, images, and videos. The specific construction includes model architecture design and pre-training;

[0099] The model architecture design is the process of building a suitable network structure based on specific data characteristics and task requirements, aiming to provide a stable and efficient basic framework for subsequent model training. This step is crucial to ensure that the model can accurately capture key information in the data and improve the accuracy and efficiency of defect detection. In the product appearance defect detection system based on multimodal large models, the model architecture design needs to comprehensively consider the diversity and complexity of the data and the specific requirements of the detection task in order to design the optimal model structure;

[0100] Pre-training aims to enable the model to learn general representation capabilities. By training on large-scale datasets, the model can learn rich prior knowledge and provide strong support for fine-tuning the model for subsequent specific tasks such as product appearance defect detection. This general representation capability helps the model converge faster and improve performance on new and unseen tasks.

[0101] S3. Model fine-tuning

[0102] Model fine-tuning specifically includes data set preparation, model loading, model training, and model evaluation and optimization. Data set preparation is the basis of model fine-tuning, providing necessary data support for model training. Model loading is the starting point of model fine-tuning, providing existing knowledge and capabilities as a reference. Model training is the core link of model fine-tuning, through which the model learns the characteristics of specific tasks. Model evaluation and optimization are the guarantee links of model fine-tuning, ensuring the stability and accuracy of the model in practical applications.

[0103] S4. Structure display and feedback

[0104] The product appearance defect detection system will eventually visualize the detected defects, such as marking the defect location, displaying the defect type, etc. At the same time, it will generate a test report to provide timely feedback to production personnel.

[0105] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0106] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A product appearance defect detection device based on a multimodal model, comprising a base (1) and an image recognition device (2), wherein the image recognition device (2) has multiple operation modes including vision, infrared, and 3D scanning, and an auxiliary bracket (3) and a support ring frame (5) are respectively installed on the top and the middle of the base (1), and a slide rail (4) is fixedly connected to the top of the auxiliary bracket (3), and a detection platform frame (7) is slidably connected to the slide rail (4), and a clearance groove (8) is arranged inside the detection platform frame (7), and a composite detection support component (9) is installed in the clearance groove (8), and a transparent detection support space is arranged on the top of the composite detection support component (9); A movable annular plate (6) is sleeved on the inner side of the support ring frame (5) via a bearing, the image recognition device (2) is mounted on the inner wall at the top of the movable annular plate (6) and aligned with the composite detection support component (9), a second gear sleeve (15) is sleeved on the surface of one side of the movable annular plate (6), the bottom of the second gear sleeve (15) is transmission-connected to a third meshing output component (16) mounted on the top of the base (1), and the image recognition device (2) can be rotated and adjusted around the composite detection support component (9) under the transmission of the third meshing output component (16), the second gear sleeve (15) and the movable annular plate (6).

2. The product appearance defect detection device based on a multimodal model according to claim 1, characterized in that: A guide groove capable of being engaged with the slide rail (4) is provided at the bottom of the detection platform frame (7), and weight-reducing grooves are provided at the front and rear ends of the middle portion of the detection platform frame (7).

3. The product appearance defect detection device based on a multimodal model according to claim 1, characterized in that: The composite detection support component (9) comprises a transparent plate (91), a combined ring frame (92), a first gear sleeve plate (93), a first transmission gear (94) and a first brake servo motor (95); the transparent plate (91) is sleeved inside the combined ring frame (92) through a bearing; the top of the first gear sleeve plate (93) is installed at the bottom of the inner ring structure of the bearing; the middle of the first transmission gear (94) is transmission-connected to the output end of the first brake servo motor (95); and the first transmission gear (94) is meshingly connected to the first gear sleeve plate (93); a support frame is installed between the shell surface of the first brake servo motor (95) and the rear end surface of the transparent plate (91); and the combined ring frame (92) is sleeved inside the clearance groove (8).

4. The product appearance defect detection device based on a multimodal model according to claim 1, characterized in that: The third meshing output component (16) comprises a third transmission gear (161) and a third brake servo motor (162); the middle portion of the third transmission gear (161) is transmission-connected to the output end of the third brake servo motor (162); the third transmission gear (161) is meshingly connected to the bottom of the second gear sleeve (15); and a support seat is installed between the shell surface of the third transmission gear (161) and the top surface of the middle portion of the base (1).

5. The product appearance defect detection device based on a multimodal model according to claim 1, characterized in that: The rear end of the clearance groove (8) is an open structure, and the number of the clearance grooves (8) is five and is evenly distributed on the top of the detection platform frame (7), and the number of the composite detection support components (9) is the same as the number of the clearance grooves (8).

6. The product appearance defect detection device based on a multimodal model according to claim 1, characterized in that: The detection platform frame (7) is a hollow structure, and ten assembly grooves (12) are opened on the top of the detection platform frame (7). The ten assembly grooves (12) are arranged in groups of two on the outside of the five clearance grooves (8), and the assembly grooves (12) are spatially connected with the corresponding clearance grooves (8).

7. The product appearance defect detection device based on a multimodal model according to claim 3, characterized in that: The surface of the combined ring frame (92) is fixedly connected to two combined plates (17), and the combined ring frame (92) and the two combined plates (17) on its own structure are respectively clamped in the corresponding clearance grooves (8) and the two assembly grooves (12), and the interior of the detection platform frame (7) is fixedly sleeved with an electromagnet (13) that can be magnetically connected to the corresponding combined plates (17).

8. The product appearance defect detection device based on a multimodal model according to claim 1, characterized in that: A spur-toothed plate (10) is fixedly connected to the bottom surface of the front end of the detection platform frame (7), and both ends of the spur-toothed plate (10) are transmission-connected to second meshing output components (11), and the two second meshing output components (11) each include a second transmission gear (111) and a second brake servo motor (112), and the output end of the second brake servo motor (112) is transmission-connected to the middle of the second transmission gear (111), and the two second transmission gears (111) are respectively meshingly connected to the two ends of the spur-toothed plate (10).

9. The product appearance defect detection device based on a multimodal model according to claim 1, characterized in that: An auxiliary cleaning component (14) is arranged at the top of the front end of the detection platform frame (7), and the auxiliary cleaning component (14) comprises a cleaning bracket (141) and a cleaning sponge bar (142). The rear end of the cleaning bracket (141) is fixedly connected to the surface of the top of the front end of the detection platform frame (7), and the rear end of the cleaning sponge bar (142) is fixedly connected to the front end of the cleaning bracket (141).

10. A product appearance defect detection system based on a multimodal model according to any one of claims 1 to 9, characterized in that: The steps include: S1. Data collection and processing The composite detection support component (9) is rotated and adjusted under the transmission of the third meshing output component (16), the second gear sleeve plate (15) and the movable annular plate (6), and the product appearance images of the product to be tested placed on the top of the composite detection support component (9) are collected from multiple angles and multiple positions to form image and video data, and then data preprocessing and feature extraction are performed, and finally the feature information of different modes of the hyperspectral image, depth image and RGB image of the product to be tested are fused together to form a comprehensive feature representation; S2. Multimodal large model construction Multimodal large model construction is a deep learning model that can process multiple media data such as text, images, and videos. The specific construction includes model architecture design and pre-training; S3. Model fine-tuning Model fine-tuning specifically includes data set preparation, model loading, model training, and model evaluation and optimization; S4. Structure display and feedback The product appearance defect detection system will eventually visualize the detected defects and generate a test report to provide timely feedback to production personnel.

Images