Multimodal optical detection and sorting method and system for spherical lenses

Abstract

The present invention relates to the field of visual inspection technology, and specifically discloses a multimodal optical inspection and sorting method and system for spherical lenses. The method comprises the steps of: obtaining appearance data information of lenses to be sorted, the appearance data information including curvature information, surface defect information, and coating uniformity information; placing the appearance data information into a pre-trained quality scoring model to generate a quality score, so that a spherical lens sorting system sorts the lenses to be sorted according to the quality score; the method integrates multimodal information such as curvature, surface, and coating, and utilizes a deep learning model for comprehensive evaluation. This method can overcome the limitation of traditional methods that only focus on a single indicator, and can comprehensively consider multiple quality factors of spherical lenses to generate an objective quality score, thereby achieving a comprehensive and accurate evaluation of the quality of spherical lenses and improving the efficiency and accuracy of sorting.

Description

Technical Field

[0001] The present application relates to the field of visual inspection technology, and in particular to a multimodal optical inspection and sorting method and system for spherical lenses. Background Art

[0002] In the field of optical manufacturing, spherical lenses are key components whose quality directly affects the performance and stability of optical systems. Therefore, quality inspection and sorting of spherical lenses during production are crucial.

[0003] Currently, spherical lens production lines primarily utilize two methods for quality inspection and sorting: manual inspection and traditional machine vision. Manual inspection requires workers to rely on their eyes and experience to inspect each lens individually. This is not only extremely inefficient and difficult to meet the demands of large-scale production, but also results are highly dependent on workers' expertise and subjective judgment, making missed inspections and false detections highly likely, leading to inconsistent product quality.

[0004] While existing machine vision inspection technology has improved inspection efficiency to a certain extent, its functionality is relatively limited, mostly focusing solely on lens thickness detection. However, spherical lens quality issues extend beyond thickness deviations. Surface defects (such as scratches, stains, and bubbles) and deviations in curvature parameters can also severely impact optical performance. Traditional machine vision inspection, unable to fully cover these critical quality indicators, struggles to accurately assess the overall quality of lenses. This leads to poor sorting results and an inability to effectively distinguish between qualified and unqualified products. This makes it difficult to ensure high-quality spherical lens output, hindering further development of the industry.

[0005] There is currently no effective technical solution to the above problems. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal optical inspection and sorting method and system for spherical lenses, so as to comprehensively consider various quality factors of spherical lenses and generate objective quality scores to drive sorting behavior.

[0007] In a first aspect, the present application provides a multimodal optical detection and sorting method for spherical lenses, which is applied in a spherical lens sorting system. The method comprises the following steps:

[0008] S1. Acquire appearance data information of the lens to be sorted, wherein the appearance data information includes curvature information, surface defect information, and coating uniformity information;

[0009] S2. Putting the appearance data information into a pre-trained quality scoring model to generate a quality score, so that the spherical lens sorting system sorts the lenses to be sorted according to the quality score;

[0010] The process of generating a quality score by the quality scoring model includes:

[0011] A1. Converting the curvature information, the surface defect information, and the coating uniformity information into curvature features, defect features, and uniformity features of the same dimension;

[0012] A2. Using the uniformity feature as the query subject and the curvature feature and the defect feature as the query objects, a cross-modal association relationship is established in combination with the attention mechanism to generate cross-modal features.

[0013] A3. Fusion of the curvature feature, defect feature, uniformity feature, and cross-modal feature to generate a fusion feature;

[0014] A4. Generate the quality score according to the fusion features based on the dual-task collaborative mechanism.

[0015] The multimodal optical inspection and sorting method for spherical lenses disclosed in this application integrates multimodal information such as curvature, surface, and coating, and utilizes a deep learning model for comprehensive evaluation. This method can overcome the limitations of traditional methods that focus on only a single indicator, and can comprehensively consider multiple quality factors of spherical lenses to generate an objective quality score, thereby achieving a comprehensive and accurate evaluation of the quality of spherical lenses and improving the efficiency and accuracy of sorting.

[0016] In the multimodal optical inspection and sorting method for spherical lenses, step A1 performs feature conversion based on a heterogeneous modal encoder, and the heterogeneous modal encoder includes a curvature encoder, a surface defect encoder, and a coating encoder.

[0017] This design allows each encoder to be optimized according to the characteristics of its corresponding modal data, such as adopting a network structure or processing method suitable for the data type.

[0018] The multimodal optical detection and sorting method for spherical lenses, wherein the curvature encoder includes a 2D CNN layer and a Bi-GRU network, and the Bi-GRU network is used to capture the forward and backward path dependencies under the action of the 2D CNN layer extracting local curvature mutation features based on the curvature information to convert the curvature information into 128-dimensional curvature features.

[0019] The multimodal optical inspection and sorting method for spherical lenses, wherein the surface defect encoder includes a ResNet-18 improved module with the first convolution layer replaced by a 7×7 kernel and an SPP layer. The SPP layer is used to adapt to defects of different sizes under the action of the ResNet-18 improved module to extract the macroscopic morphology of the defects to convert the surface defect information into 128-dimensional defect features.

[0020] The multimodal optical inspection and sorting method for spherical lenses, wherein the coating encoder includes a graph convolutional network and a three-layer GraphSAGE network. The three-layer GraphSAGE network is used to aggregate neighborhood information while the graph convolutional network constructs the thickness point cloud as a topological graph and defines the thickness gradient by edge features to convert the coating uniformity information into a 128-dimensional uniformity feature.

[0021] The multimodal optical detection and sorting method for spherical lenses, wherein step A3 comprises:

[0022] A31. generating weight coefficients for the curvature feature, defect feature, and uniformity feature based on the reliability of each modal data;

[0023] A32. After weighting the curvature feature, defect feature, and uniformity feature based on the weight coefficient, the weighted features are fused with the cross-modal feature to generate a fused feature.

[0024] The multimodal optical detection and sorting method for spherical lenses, wherein step A31 includes:

[0025] A311. Predicting variance information corresponding to the curvature feature, defect feature, and uniformity feature for characterizing reliability;

[0026] A312. Generate the weight coefficient based on inverse proportional weighting of the variance information.

[0027] The multimodal optical inspection and sorting method for spherical lenses, wherein in step A4, the dual-task collaborative mechanism includes a main branch mechanism for the regression task and an auxiliary branch mechanism for the interpretation task that are executed simultaneously, the main branch mechanism is used to output the quality score based on the fusion feature, and the auxiliary branch mechanism is used to output an interpretable modal importance score and defect heat map based on the fusion feature and the curvature feature, defect feature, and uniformity feature.

[0028] In a second aspect, the present application further provides a spherical lens sorting system, comprising a controller and an execution module, wherein the controller is used to control the execution module to sort the lenses to be sorted according to the spherical lens multimodal optical detection and sorting method provided in the first aspect.

[0029] The spherical lens sorting system of the present application controls the execution module to classify and sort the lenses to be sorted based on the quality score generated by the multimodal optical inspection and sorting method for spherical lenses provided in the first aspect. The scoring process integrates multimodal information such as curvature, surface and coating, and uses a deep learning model for comprehensive evaluation. It can comprehensively consider various quality factors of spherical lenses, generate objective quality scores, achieve comprehensive and accurate evaluation of the quality of spherical lenses, and effectively improve the sorting accuracy of the execution module.

[0030] The spherical lens sorting system further comprises a spectral component, the spectral component comprises a spectral ellipsometer, and the controller is further configured to utilize the spectral component to collect polarization phase changes of the lenses to be sorted, so as to obtain the coating uniformity information based on the polarization phase changes.

[0031] As can be seen from the above, the present application provides a multimodal optical inspection and sorting method and system for spherical lenses. Among them, the multimodal optical inspection and sorting method for spherical lenses of the present application integrates multimodal information such as curvature, surface and coating, and uses a deep learning model for comprehensive evaluation. This method can overcome the limitation of traditional methods that only focus on a single indicator, and can comprehensively consider multiple quality factors of spherical lenses to generate objective quality scores, thereby achieving a comprehensive and accurate evaluation of the quality of spherical lenses and improving the efficiency and accuracy of sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of the multimodal optical detection and sorting method for spherical lenses provided in an embodiment of the present application.

[0033] Figure 2 Flowchart for generating quality scores for the quality scoring model.

[0034] Figure 3 Schematic diagram of the electrical control structure of the spherical lens sorting system provided in an embodiment of the present application.

[0035] Reference numerals: 201, controller; 202, execution module; 203, interference component; 204, camera component; 205, spectral component. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0037] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0038] First, please refer to Figure 1 and Figure 2 Some embodiments of the present application provide a multimodal optical detection and sorting method for spherical lenses, which is applied in a spherical lens sorting system. The method includes the following steps:

[0039] S1. Obtaining appearance data information of the lens to be sorted, where the appearance data information includes curvature information, surface defect information, and coating uniformity information;

[0040] S2. Putting the appearance data information into a pre-trained quality scoring model to generate a quality score, so that the spherical lens sorting system can sort the lenses to be sorted according to the quality score;

[0041] The process of generating quality scores by the quality scoring model includes:

[0042] A1. Convert curvature information, surface defect information, and coating uniformity information into curvature features, defect features, and uniformity features of the same dimension;

[0043] A2. Using uniformity features as the query subject and curvature and defect features as the query objects, we establish cross-modal associations using an attention mechanism to generate cross-modal features.

[0044] A3, fuse curvature features, defect features, uniformity features and cross-modal features to generate fusion features;

[0045] A4. Generate quality scores based on fused features.

[0046] Specifically, this method aims to address the problem that traditional inspection methods are unable to comprehensively evaluate the quality of spherical lenses and achieve efficient and automated sorting. Step S1 collects curvature information, surface defect information, and coating uniformity information of the lens to be sorted, thereby acquiring multimodal data reflecting different quality dimensions of the lens. This information represents different data types for evaluating lens quality, overcoming the limitation of traditional methods that only focus on a single parameter. Step S2 inputs this multimodal data into a pre-trained quality scoring model, which generates a quality score. The spherical lens sorting system performs sorting based on this score, achieving automated sorting based on comprehensive quality assessment.

[0047] More specifically, the process of generating a quality score by the quality scoring model is divided into several sub-steps, among which step A1 can use a heterogeneous modal encoder to process different types of appearance data information, and convert curvature information, surface defect information and coating uniformity information into curvature features, defect features and uniformity features of the same dimension respectively. This conversion unifies the original data in different forms into a processable feature representation, thereby enabling subsequent unified processing of data of different modalities. These features exist in the form of feature vectors; step A2 analyzes and captures the interactions between different quality aspects, improving the model's ability to identify complex quality problems; step A3 merges all extracted individual features and their relationship features to form a comprehensive representation of fusion features, laying the foundation for a comprehensive evaluation of lens quality; step A4 converts the fusion feature into a numerical value as a quantitative representation of the overall quality of the lens, directly guiding the sorting operation. The entire process realizes the comprehensive processing and evaluation of multimodal quality information of spherical lenses, generates quality scores for sorting, and solves the problem that traditional methods cannot comprehensively evaluate the overall quality of lenses.

[0048] More specifically, after obtaining the curvature, surface defect, and coating uniformity information of the lens to be sorted, this appearance data is fed into a pre-trained quality scoring model. This model first performs feature conversion on the different types of appearance data using a heterogeneous modality encoder, encoding the curvature, surface defect, and coating uniformity information into curvature, defect, and uniformity features of the same dimensions, respectively. This results in a unified representation of data from different modalities. Furthermore, the model uses the uniformity feature as the query subject and the curvature and defect features as the query objects, integrating an attention mechanism to establish cross-modal associations and generate cross-modal features. This process captures the mutual influence between different quality dimensions. Subsequently, the curvature, defect, uniformity, and cross-modal features are fused to generate a comprehensive fused feature. This fusion can be achieved through various methods, such as simple concatenation or weighted summation. This fused feature integrates the lens's performance across various quality dimensions and the relationships between these dimensions, forming a comprehensive description of the lens's overall quality status. Ultimately, the model generates a quality score based on the fused features. This quality score is a quantitative value that represents the overall quality level of the lens. After receiving this quality score, the spherical lens sorting system sorts the lenses into different categories (e.g., qualified, repairable, and non-repairable) according to pre-set sorting rules.

[0049] The multimodal optical inspection and sorting method for spherical lenses in the embodiments of the present application integrates multimodal information such as curvature, surface, and coating, and utilizes a deep learning model for comprehensive evaluation. This method can overcome the limitations of traditional methods that focus on only a single indicator, comprehensively consider multiple quality factors of spherical lenses, generate objective quality scores, and achieve a comprehensive and accurate assessment of spherical lens quality, thereby improving sorting efficiency and accuracy.

[0050] In some preferred embodiments, step A1 performs feature conversion based on a heterogeneous modal encoder, where the heterogeneous modal encoder includes a curvature encoder, a surface defect encoder, and a coating encoder.

[0051] Specifically, the heterogeneous modal encoder consists of independent encoders designed for different modal data. The curvature encoder is specifically designed to process curvature information, the surface defect encoder is specifically designed to process surface defect information, and the coating encoder is specifically designed to process coating uniformity information. This design allows each encoder to be optimized based on the characteristics of its corresponding modal data, such as by adopting a network structure or processing method suitable for that data type. By equipping each modal data with a dedicated encoder, the scheme can more effectively extract representative features from heterogeneous raw data. Each encoder converts the data of its respective modality into feature vectors of the same dimension (curvature features, defect features, uniformity features). This conversion process retains the key information of each modality while unifying them into a comparable and fusible representation space. This solves the problem of effectively extracting and unifying features when processing heterogeneous data in step A1, lays the foundation for subsequent cross-modal association and feature fusion, and thus improves the accuracy of quality scoring.

[0052] In some preferred embodiments, the curvature encoder includes a 2D CNN layer and a Bi-GRU network, and the Bi-GRU network is used to capture the forward and backward path dependencies under the action of the 2D CNN layer extracting local curvature mutation features based on curvature information to convert the curvature information into 128-dimensional curvature features.

[0053] Specifically, the 2D CNN layer is responsible for processing the input curvature information and extracting local features in the data through convolution operations, especially the mutation features of the curvature. Preferably, the convolution kernel width of the 2D CNN layer is 5 and the stride is 2. It can scan the curvature data with a specific receptive field and sampling method to identify local anomalies or changes. The Bi-GRU network receives the output of the 2D CNN layer and further processes these local features. Bi-GRU is a bidirectional recurrent neural network that can capture the forward and backward dependencies in the data sequence. Curvature information has spatial or temporal sequentiality. The Bi-GRU network can understand this sequential structure and associate local features at different positions to obtain a more comprehensive representation of the curvature error.

[0054] More specifically, in this embodiment, first, the curvature information is input into the 2D CNN layer, which scans and processes the curvature information to extract features that reflect local changes in curvature, such as mutation features such as a sudden increase or decrease in curvature, thereby achieving effective identification of local curvature mutation features. Subsequently, the output of the 2D CNN layer is input into the Bi-GRU network. The Bi-GRU network can simultaneously process forward and reverse sequence information and capture the dependencies between curvature information at different positions, such as the continuity or periodicity of curvature change trends. The curvature encoder converts the original curvature information into a 128-dimensional curvature feature vector by combining the local features extracted by 2D CNN and the sequence dependencies captured by Bi-GRU, thereby achieving effective encoding of the curvature information. This feature vector comprehensively reflects the local characteristics and overall structure of the curvature error, providing accurate input for the subsequent quality scoring model and improving the accuracy of the quality score.

[0055] In some preferred embodiments, the surface defect encoder includes a ResNet-18 improved module in which the first convolution layer is replaced with a 7×7 kernel and an SPP layer. The SPP layer is used to adapt to defects of different sizes under the action of the ResNet-18 improved module to extract the macroscopic morphology of the defects to convert the surface defect information into 128-dimensional defect features.

[0056] Specifically, the ResNet-18 improvement module replaces the 3×3 convolutions in the first layer of ResNet-18 with 7×7 convolutions. This large-size convolution kernel can capture a wider range of initial features in the input image. The SPP layer generates a fixed-length feature vector by pooling feature maps of different scales. Ultimately, the features processed by the SPP layer are converted into a 128-dimensional defect feature vector and output. By combining the improved ResNet-18 module to extract macromorphology and the SPP layer to adapt to different scales, the surface defect encoder can more comprehensively and accurately convert raw surface defect information into defect feature vectors, thereby improving the accuracy of subsequent quality scoring models.

[0057] In some preferred embodiments, the coating encoder includes a graph convolutional network and a three-layer GraphSAGE network. The three-layer GraphSAGE network is used to aggregate neighborhood information while the graph convolutional network constructs the thickness point cloud as a topological graph and defines the thickness gradient by the edge features to convert the coating uniformity information into a 128-dimensional uniformity feature.

[0058] Specifically, coating uniformity information usually exists in the form of thickness point clouds with a spatial topological structure. The graph convolutional network is used to convert thickness point cloud data into graph structured data, where points serve as nodes, connections between points form edges, and thickness differences or gradients between points are defined as edge features. The three-layer GraphSAGE network is used to process the constructed graph structured data and learn the representation of nodes by aggregating the neighborhood information of each node layer by layer. The GraphSAGE network captures local and broader features of coating uniformity through aggregation operations. Ultimately, the GraphSAGE network outputs a fixed-dimensional feature vector, namely the uniformity feature. The graph convolutional network is responsible for converting the original point cloud data into a graph structure suitable for processing by the GraphSAGE network. The GraphSAGE network is responsible for feature learning and aggregation on the graph structure, converting variable-length graph data into fixed-length feature vectors.

[0059] More specifically, a graph convolutional network is used to process thickness point clouds. Points in the point cloud are constructed as nodes of a graph, and edges are established based on the spatial relationships or distances between points. The thickness gradients between adjacent points are calculated as edge features. This converts the raw thickness point cloud data into a topological graph with node and edge features. This topological graph is then processed by a three-layer GraphSAGE network. The GraphSAGE network defines an aggregation function that aggregates information from each node's neighborhood and updates the node's representation based on its own features. The first layer of the GraphSAGE network aggregates information from direct neighbors, the second layer aggregates information from two-hop neighbors, and the third layer aggregates information from three-hop neighbors, thereby gradually expanding the receptive field and capturing more comprehensive local and global structural information. After three layers of aggregation, each node acquires a feature representation that incorporates information about its neighborhood. Ultimately, these node feature representations are combined into a fixed-dimensional vector, preferably set to 128, which serves as the output uniformity feature. This processing method effectively utilizes the spatial structure and thickness gradient information of point cloud data, and converts variable-length graph structure data into fixed-length feature vectors, providing standardized input for subsequent feature fusion.

[0060] In some preferred embodiments, step A2 establishes a cross-modal association relationship based on a multi-head attention mechanism, where the multi-head attention mechanism satisfies:

[0061] (1)

[0062] Among them, head j is the calculation result of the jth attention head, that is, the jth cross-modal association relationship, Q is the query subject, K, V are the queried objects, that is, the concatenation vector of curvature features and defect features, is the learnable projection matrix of the j-th attention head with respect to Q, is the learnable projection matrix of the j-th attention head with respect to K, is the learnable projection matrix of the jth attention head with respect to V, d k is the projection dimension of K, T is the transposed matrix label, and in this embodiment of the present application, the multi-head attention mechanism is a four-head attention mechanism.

[0063] Specifically, this technical solution improves the process of establishing cross-modal association relationships by introducing a multi-head attention mechanism. When establishing the association between uniformity features, curvature features, and defect features, the multi-head attention mechanism can project the query subject Q and the query objects K and V into multiple different low-dimensional subspaces through different linear transformations. In each subspace, attention calculations are performed independently and in parallel to obtain the output head of multiple attention heads. j . Each attention head can learn and focus on different correlation patterns or information aspects in the input features. For example, one head may focus on the correlation between uniformity and the magnitude of curvature error, while another head may focus on the correlation between uniformity and defect distribution. This technical solution explicitly adopts a four-head attention mechanism, which means that four independent attention calculations are performed in parallel. Finally, the outputs of these four attention heads are spliced together and then subjected to a linear transformation to generate the final cross-modal features.

[0064] More specifically, the above processing method aims to solve the problem that a single attention mechanism may not be able to fully capture cross-modal correlation information in different dimensions or different aspects, thereby limiting the expressive power of cross-modal features. By splicing the outputs of multiple attention heads together, a more comprehensive and richer cross-modal correlation representation can be obtained. The features that have undergone attention interaction already contain semantic associations between modalities, so that the generated cross-modal features can more finely reflect the complex interactions between uniformity features, curvature features, and defect features, providing more discriminative information for the subsequent generation of fusion features, thereby improving the accuracy of quality scores.

[0065] In some preferred embodiments, the multi-head attention mechanism has learned gating weights that satisfy:

[0066] (2)

[0067] Among them, g is the gate value, satisfying g∈[0,1], and the final attention output of the multi-head attention mechanism is g·concat(head1,...,head n ), where n is the total number of attention heads, σ is the activation function, and MLP is a multi-layer perceptron.

[0068] Specifically, the gating weight processes the concatenated vectors of the query subject and the queried object through a multi-layer perceptron. The output of the multi-layer perceptron is mapped to a range of 0 to 1 using an activation function, thereby generating a gating value. This gating value is used to control the amount of information flow. Specifically, the final output of the multi-head attention mechanism is obtained by multiplying the gating value by the standard output of the multi-head attention mechanism. When the gating value approaches 0, the information flow decreases; when the gating value approaches 1, the information flow increases. Thus, the information flow is dynamically adjusted based on the input features.

[0069] More specifically, in this embodiment, after the multi-head attention mechanism calculates the results of each attention head and splices them together, they are not directly used as the final cross-modal features. The gating value is multiplied by the spliced output of the multi-head attention mechanism. Thus, the gating value dynamically adjusts the intensity of the attention output according to the specific content of the query subject and the queried object. When the input feature combination indicates that the cross-modal association is valid, the gating value is close to 1, and the information is almost completely passed. When the input feature combination indicates that the cross-modal association may contain noise, the gating value is close to 0, and the information is greatly attenuated. This enables the generated cross-modal features to better reflect the effective association between modalities, improves the quality of feature fusion, and thereby improves the accuracy of the quality score.

[0070] In some preferred embodiments, step A3 includes:

[0071] A31. Generate weight coefficients for curvature features, defect features, and uniformity features based on the reliability of each modal data;

[0072] A32. After weighting the curvature features, defect features, and uniformity features based on the weight coefficient, they are fused with the cross-modal features to generate fused features.

[0073] Specifically, the reliability of different modal data may vary, depending on the acquisition method, environmental factors, or sensor performance. For example, surface defect detection may be susceptible to changes in illumination, while curvature measurements may be more sensitive to vibration. Simply averaging or concatenating these features may not fully utilize highly reliable data and may be negatively impacted by less reliable data. Therefore, this solution incorporates a weighting mechanism based on data reliability into the fusion process. First, weight coefficients are calculated based on the reliability of each modal data (corresponding to curvature, defect, and uniformity features). Highly reliable data has a larger weight coefficient, while low-reliability data has a smaller weight coefficient. These weight coefficients are then used to fuse the curvature, defect, uniformity, and cross-modal features. This weighting method allows the fused features to prioritize information from the more reliable modalities, thereby more accurately characterizing the overall quality of the lens. The resulting fused features are then used to generate a final quality score based on a dual-task collaborative mechanism, guiding the sorting system. Thus, by weighting the fusion based on data reliability, the quality of the fused features is improved, thereby enhancing the accuracy of the quality score and the reliability of the sorting process.

[0074] In some preferred embodiments, step A31 includes:

[0075] A311, predict the variance information corresponding to the curvature feature, defect feature, and uniformity feature for characterizing reliability;

[0076] A312. Generate weight coefficients based on inverse proportional weighting of variance information.

[0077] Specifically, variance characterizes the reliability of the data; smaller variance indicates higher reliability and a larger weight coefficient. Therefore, reliable data influences the fusion results, improving the accuracy of the fused features. This processing approach provides a mechanism for dynamically adjusting the weights of each modal feature based on the uncertainty of the data itself. This makes the generation of weight coefficients data-driven, overcomes the problems that may arise from simple averaging or fixed weights, and improves the effectiveness of multimodal feature fusion.

[0078] In some preferred embodiments, in step A311, the variance is generated based on a preset feedforward neural network model, which satisfies:

[0079] (3)

[0080] in, is the variance information of the i-th modal data, MLP i is the feedforward neural network model used for the i-th modal data, h i is the i-th modal data.

[0081] Specifically, the above processing method specifies the use of a preset feedforward neural network model to generate the variance information of the i-th modal data. For each modal data (curvature feature, defect feature, uniformity feature), there is a corresponding feedforward neural network model. i The introduction of makes the prediction of variance no longer an arbitrary or undefined process, but is achieved through a learnable model. The model can learn how to generate variance information reflecting its reliability based on the input feature data during the training process.

[0082] More specifically, the above approach provides a structured, model-based variance prediction method. This method trains a feedforward neural network model, enabling it to learn and output variance information that represents the reliability of input modal features. This makes the variance prediction process controllable and stable, providing reliable input for subsequent variance-based weight calculations, thereby improving the accuracy of feature fusion and ultimately enhancing the quality score generation.

[0083] In some preferred embodiments, in step A312, the weight coefficient is calculated based on the following formula:

[0084] (4)

[0085] Among them, r is a hyperparameter, w i is the weight coefficient of the i-th modal data.

[0086] Specifically, the method receives the variance information used to characterize the reliability of each modal data As input, The variance is converted into a numerical value that is positively correlated with reliability: the smaller the variance (the higher the reliability), the larger the value. The hyperparameter r is set based on usage requirements and is used to adjust the influence of the variance on the final weight. A larger r value indicates a more significant effect of the variance on the weight. The sum of the exponential terms of all modes is used to normalize the calculated weights to ensure that the sum of the weight coefficients of all modes is 1. Therefore, formula (4) converts the variance information representing unreliability into a weight coefficient for feature fusion. Data with high reliability obtains a higher weight, and data with low reliability obtains a lower weight. This provides a quantitative basis based on data reliability for subsequent feature weighted fusion and improves the effectiveness of fusion features.

[0087] In some preferred embodiments, in step A32, the fusion feature satisfies:

[0088] (5)

[0089] Among them, y is the fusion feature, MLP is the multi-layer perceptron, and concat(h1,h2,h3) is the cross-modal feature.

[0090] Specifically, the fusion feature y consists of two parts. The first part is , represents the weighted sum of curvature features, defect features, and uniformity features (h_i), and the weight coefficient w i It reflects the reliability of the corresponding modal data. The second part is MLP (concat (h1, h2, h3)), which represents the cross-modal features generated in step A2. This fusion method combines the weighted original features based on reliability and the processing of cross-modal features to generate a fused feature. This processing method uses the cross-modal associations generated in step A2, combined with the reliability of each modality itself, to dynamically balance the contributions of attention features and original features to perform feature fusion. That is, it combines the establishment of inter-modal associations at the semantic level and the implementation of quantitative weighting at the feature level to construct fused features, jointly realizing high-precision, interpretable, and interference-resistant multimodal data fusion. The generated fused features can more comprehensively and accurately characterize the appearance data information of the lens to be sorted, overcoming the limitations of simple weighted summation in capturing the complex associations between different modal information, and providing a basis for the accuracy of the subsequent quality scoring model.

[0091] In some preferred embodiments, step A4 generates a quality score based on a dual-task collaborative mechanism, wherein the dual-task collaborative mechanism includes a main branch mechanism for the regression task and an auxiliary branch mechanism for the interpretation task that are executed simultaneously, the main branch mechanism is used to output a quality score based on the fusion features, and the auxiliary branch mechanism is used to output an interpretable modal importance score and a defect heat map based on the fusion features and curvature features, defect features, and uniformity features.

[0092] Specifically, this step uses a dual-task collaborative mechanism to generate interpretable indicators (modal importance score and defect heat map) while outputting the product quality score (main branch), thereby achieving simultaneous optimization of prediction accuracy and decision credibility.

[0093] More specifically, the design principle of this step is to achieve the unity of accuracy and interpretability through dual-branch collaborative optimization. Its core is to simultaneously complete quality prediction (regression task) and decision basis analysis (explanation task) through shared feature representation. Among them, the main branch mechanism uses a 3-layer MLP to output a quality score (0-100) based on the fused features and uses the Huber loss function to resist outliers. The Huber loss function is used to δ Abnormal samples (such as outliers caused by sensor failure) are penalized linearly to avoid gradient explosion.

[0094] Based on the above, we can see that the quality scoring model is a multimodal fusion deep learning model based on an attention mechanism. Its core architecture combines a heterogeneous modality encoder, a cross-modal association module, and a dynamic feature fusion mechanism. This quality scoring model encodes multimodal data such as curvature, defects, and coating uniformity into 128-dimensional feature vectors using a curvature encoder (2D CNN + Bi-GRU), a surface defect encoder (improved ResNet-18 + SPP), and a coating encoder (GCN + GraphSAGE). Subsequently, the quality scoring model uses a four-head attention mechanism to establish cross-modal associations between uniformity features, curvature features, and defect features to generate interactive features. Gating weights (dynamic adjustment of the Multi-Layered Loop) are introduced to optimize information flow. Finally, through weighted fusion based on variance prediction and a dual-task collaborative mechanism (regression scoring and interpretability analysis), this quality scoring model achieves a comprehensive assessment of spherical lens quality, achieving both high accuracy and interpretability.

[0095] It should be noted that the quality scoring model can be obtained through strategy training that combines supervised learning with multi-task collaborative training. The training process includes data preparation, loss function setting, dynamic optimization, and end-to-end training. These processes are commonly used training processes for deep learning models and will not be elaborated here.

[0096] It should be noted that the score output by the quality scoring model is 0-100 points, among which 90-100 points, 70-89 points and 0-69 points correspond to the qualified products, repairable defective products and non-repairable defective products mentioned above, respectively.

[0097] Second, please refer to Figure 3 Some embodiments of the present application also provide a spherical lens sorting system, including a controller 201 and an execution module 202, wherein the controller 201 is used to control the execution module 202 to sort the lenses to be sorted according to the spherical lens multimodal optical detection and sorting method provided in the first aspect.

[0098] Specifically, controller 201 receives and processes the quality scores generated by the multimodal optical inspection and sorting method for spherical lenses provided in the first aspect. Controller 201 can also be configured to execute the multimodal optical inspection and sorting method for spherical lenses provided in the first aspect to generate lens quality scores and then determine the destination of the lenses to be sorted based on the quality scores. Execution module 202, based on instructions from controller 201, performs physical operations, such as grabbing, moving, or pushing the lenses, and placing them in corresponding sorting areas or containers. Controller 201 is the core of the system, responsible for converting abstract quality scores into concrete physical control signals. Its technical contribution lies in connecting the method logic with physical execution. Execution module 202 is the system's actuator, responsible for completing the actual physical sorting actions. Its technical contribution lies in providing a physical means for achieving automated sorting. Through the collaborative operation of controller 201 and execution module 202, this system converts the scoring results of the inspection and sorting method into actual physical sorting actions, thereby achieving automated inspection and sorting of spherical lenses.

[0099] More specifically, the controller 201 can control the execution module 202 to classify and sort the lenses to be sorted according to a preset score interval, such as classifying them into superior lenses, good superior lenses, qualified superior lenses, and unqualified superior lenses.

[0100] The spherical lens sorting system of the embodiment of the present application controls the execution module 202 to classify and sort the lenses to be sorted based on the quality scores generated by the multimodal optical inspection and sorting method for spherical lenses provided in the first aspect. The scoring process integrates multimodal information such as curvature, surface, and coating, and uses a deep learning model for comprehensive evaluation. It can comprehensively consider various quality factors of spherical lenses, generate objective quality scores, achieve comprehensive and accurate evaluation of the quality of spherical lenses, and effectively improve the sorting accuracy of the execution module 202.

[0101] In some preferred embodiments, the spherical lens sorting system further includes an interference component 203, which is a laser interferometer or a white light interferometer;

[0102] The controller 201 is configured to utilize the interference component 203 to scan the surface height data of the lens to be sorted to obtain curvature information.

[0103] Specifically, the interferometer component 203 employs a laser interferometer or a white-light interferometer, which can measure optical surface height data in a non-contact manner. The controller 201 receives the surface height data acquired by the interferometer component 203. Based on this surface height data, the controller 201 performs calculations to obtain curvature information for the lens to be sorted. The controller 201 uses this component to scan and perform calculations, converting raw measurement data into curvature information. By adding the interferometer component 203 and its functionality, the system gains the ability to acquire curvature information.

[0104] In some preferred embodiments, the spherical lens sorting system further includes a camera assembly 204 , and the controller 201 is further configured to use the camera assembly 204 to capture images of the lenses to be sorted, so as to obtain surface defect information based on the images.

[0105] Specifically, camera assembly 204 can be an industrial camera. Controller 201 is electrically connected to camera assembly 204 and controls it to perform image acquisition. Controller 201 receives the captured image data, processes and analyzes it, and identifies and extracts information related to surface defects. This provides the necessary surface defect data for subsequent sorting methods.

[0106] In some preferred embodiments, the spherical lens sorting system further includes a spectral component 205, which includes a spectral ellipsometer. The controller 201 is further configured to utilize the spectral component 205 to collect polarization phase changes of the lenses to be sorted, so as to obtain coating uniformity information based on the polarization phase changes.

[0107] Specifically, controller 201 is electrically connected to spectroscopic component 205 to control the spectroscopic ellipsometer to perform measurement tasks. The spectroscopic ellipsometer is configured to perform optical measurements on the lenses to be sorted, collecting data on polarization phase changes caused by the lens surface coating. Controller 201 receives the polarization phase change data output by the spectroscopic ellipsometer and executes a data processing algorithm to extract information from this data that characterizes the coating uniformity. Thus, spectroscopic component 205 provides the physical measurement method, while controller 201 provides the data collection and information processing capabilities. Together, they enable the acquisition of coating uniformity information, providing the necessary data input for subsequent quality scoring and sorting.

[0108] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0110] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0111] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal optical detection and sorting method for spherical lenses, applied in a spherical lens sorting system, characterized in that: The method comprises the following steps: S1. Acquire appearance data information of the lens to be sorted, wherein the appearance data information includes curvature information, surface defect information, and coating uniformity information; S2. Putting the appearance data information into a pre-trained quality scoring model to generate a quality score, so that the spherical lens sorting system sorts the lenses to be sorted according to the quality score; The process of generating a quality score by the quality scoring model includes: A1. Converting the curvature information, the surface defect information, and the coating uniformity information into curvature features, defect features, and uniformity features of the same dimension; A2. Using the uniformity feature as the query subject and the curvature feature and the defect feature as the query objects, a cross-modal association relationship is established in combination with the attention mechanism to generate cross-modal features. A3. Fusion of the curvature feature, defect feature, uniformity feature, and cross-modal feature to generate a fusion feature; A4. The quality score is generated according to the fusion features based on a dual-task collaborative mechanism. The dual-task collaborative mechanism includes a main branch mechanism for the regression task and an auxiliary branch mechanism for the interpretation task that are executed simultaneously. The main branch mechanism is used to output the quality score according to the fusion features, and the auxiliary branch mechanism is used to output an interpretable modal importance score and defect heat map according to the fusion features and the curvature features, defect features, and uniformity features.

2. The multimodal optical detection and sorting method for spherical lenses according to claim 1, characterized in that: Step A1 performs feature conversion based on a heterogeneous modal encoder, wherein the heterogeneous modal encoder includes a curvature encoder, a surface defect encoder, and a coating encoder.

3. The multimodal optical detection and sorting method for spherical lenses according to claim 2, wherein: The curvature encoder includes a 2D CNN layer and a Bi-GRU network. The Bi-GRU network is used to capture forward and backward path dependencies under the action of the 2D CNN layer extracting local curvature mutation features based on the curvature information to convert the curvature information into 128-dimensional curvature features.

4. The multimodal optical detection and sorting method for spherical lenses according to claim 2, characterized in that: The surface defect encoder includes a ResNet-18 improved module with the first convolution layer replaced by a 7×7 kernel and an SPP layer. The SPP layer is used to adapt to defects of different sizes under the action of the ResNet-18 improved module to extract the macroscopic morphology of the defects to convert the surface defect information into 128-dimensional defect features.

5. The multimodal optical detection and sorting method for spherical lenses according to claim 2, characterized in that: The coating encoder includes a graph convolutional network and a three-layer GraphSAGE network. The three-layer GraphSAGE network is used to aggregate neighborhood information while the graph convolutional network constructs the thickness point cloud as a topological graph and defines the thickness gradient with edge features to convert the coating uniformity information into a 128-dimensional uniformity feature.

6. The multimodal optical detection and sorting method for spherical lenses according to claim 1, characterized in that: Step A3 includes: A31. generating weight coefficients for the curvature feature, defect feature, and uniformity feature based on the reliability of each modal data; A32. After weighting the curvature feature, defect feature, and uniformity feature based on the weight coefficient, the weighted features are fused with the cross-modal feature to generate a fused feature.

7. The multimodal optical detection and sorting method for spherical lenses according to claim 6, characterized in that: Step A31 includes: A311. Predicting variance information corresponding to the curvature feature, defect feature, and uniformity feature for characterizing reliability; A312. Generate the weight coefficient based on inverse proportional weighting of the variance information.

8. A spherical lens sorting system, characterized in that: The system comprises a controller and an execution module, wherein the controller is used to control the execution module to sort the lenses to be sorted according to the multimodal optical detection and sorting method for spherical lenses according to any one of claims 1 to 7.

9. The spherical lens sorting system according to claim 8, characterized in that: The spherical lens sorting system also includes a spectral component, which includes a spectral ellipsometer. The controller is also used to use the spectral component to collect polarization light phase changes about the lenses to be sorted, so as to obtain the coating uniformity information based on the polarization light phase changes.

Images