Image sensor automatic detection method and system, and electronic equipment

Abstract

The invention provides an image sensor automatic detection method and system, and electronic equipment. The method comprises the following steps: obtaining a trained feature fusion model; inputting original image data shot by an image sensor to be detected into the feature fusion model; the feature fusion model extracts image features from the original image data, performs feature fusion on the extracted image features to obtain a fusion value, and performs normalization processing on the fusion value to obtain a detection result; and performing defect classification according to a detection result. The original image data shot by the image sensor is automatically analyzed and processed through the feature fusion model, then the defect classification result is obtained, accurate defect classification can be obtained without manual participation, the detection efficiency is high, and the problem of how to improve the accuracy and efficiency of image sensor detection is solved.

Description

Technical Field

[0001] The present invention relates to the field of automatic detection technology, and in particular to an image sensor automatic detection method and system, and electronic equipment. Background Art

[0002] Image sensor is a device that converts optical images into electronic signals. It is widely used in digital cameras and other electronic optical devices. Early image sensors used analog signals, such as camera tubes. With the rapid development of digital technology, semiconductor manufacturing technology and the Internet, the commonly used image sensor products are mainly divided into two types: CCD image sensors and CMOS image sensors.

[0003] In order to ensure the imaging quality of image sensors in applications, a series of functional tests are usually required before leaving the factory, such as resolution test, dark noise test, full well capacity test, signal-to-noise ratio test, gain test, etc.

[0004] However, the existing image sensor imaging defect detection process mainly relies on manual work: the image sensor shoots the target scene, the shot image is transmitted to the display device for display, and the inspector observes the image displayed on the display device with the naked eye to determine whether the image sensor has imaging defects, and classifies the image sensor according to the determined imaging defect type. This existing detection method is extremely dependent on manual experience. If manual experience is insufficient, it is easy to cause misjudgment of defects; moreover, the human eye observation method is easy to cause visual fatigue, which can also lead to false detection and missed detection. In short, the existing image sensor imaging defect detection method is highly subjective, has poor accuracy, and is inefficient, and cannot meet the detection needs of image sensors. Summary of the invention

[0005] The object of the present invention is to provide an image sensor automatic detection method and system, and electronic equipment to solve the problem of how to improve the accuracy and efficiency of image sensor detection.

[0006] In order to solve the above technical problems, the present invention provides an automatic detection method of an image sensor, comprising: Obtain a trained feature fusion model; Inputting the original image data captured by the image sensor to be detected into the feature fusion model; The feature fusion model extracts image features from the original image data, performs feature fusion on the extracted image features to obtain a fusion value, and normalizes the fusion value to obtain a detection result; According to the detection results, the image sensor to be detected performs defect classification.

[0007] Optionally, in the automatic detection method of the image sensor, the feature fusion model includes a feature extraction module, a feature fusion module and a normalization module; the feature extraction module is used to extract image features from the original image data; the feature fusion module is used to perform feature fusion on the image features extracted by the feature extraction module to obtain fused features, and convert the fused features into fusion values; the normalization module is used to normalize the fusion values ​​obtained by the feature fusion module to obtain detection results.

[0008] Optionally, in the automatic detection method of the image sensor, the feature extraction module uses a convolutional neural network to extract image features from the original image data; the feature fusion module uses an image pyramid to fuse the extracted image features to obtain fused features; and the normalization module uses a normalized exponential function to normalize the fusion value.

[0009] Optionally, in the automatic detection method of the image sensor, the method for obtaining the trained feature fusion model includes training the feature fusion model, including: Creating a training image dataset, wherein the training image dataset includes a defect image dataset and a normal image dataset; Using a feature extraction module, extracting features from a training image data set to obtain an image feature set, wherein the image feature set includes a defect image feature set and a normal image feature set; Using the feature fusion module, the defect image feature set and the normal image feature set are fused to obtain a fusion value; The normalization module is used to normalize the fusion value to obtain the training result.

[0010] Optionally, in the automatic detection method of the image sensor, the method for training the constructed feature fusion model includes: A complementary image feature set is obtained by using the defect image feature set and the normal image feature set; The feature fusion module is used to fuse the defect image feature set, the normal image feature set and the supplementary image feature set to obtain a fusion value.

[0011] Optionally, in the automatic detection method of the image sensor, the method of obtaining the supplementary image feature set by using the defect image feature set and the normal image feature set includes: Randomly extracting a number of defect image features and a number of normal image features from the image feature set to obtain a random feature subset; The maximum value in the random feature subset is used as the first reference value, and the minimum value in the random feature subset is used as the second reference value; Calculating a deviation value using the first reference value and the second reference value; A complementary image feature set is generated according to the deviation value and the image feature set.

[0012] Optionally, in the automatic detection method of the image sensor, the deviation value B is expressed as:

[0013] Where n represents the number of image features in the random feature subset, represents the i-th image feature in the random feature subset, represents the mean of image features in a random feature subset; The supplementary image feature set It is expressed as:

[0014] in, represents the first reference value, Indicates the second reference value.

[0015] Optionally, in the image sensor automatic detection method, the original image data includes the grayscale value of each pixel in the original image.

[0016] In order to solve the above technical problems, the present invention further provides an image sensor automatic detection system, which is used to implement the image sensor automatic detection method as described in any one of the above items, and the image sensor automatic detection system comprises: A data acquisition module is used to acquire the original image data captured by the image sensor to be detected; The defect detection module includes a feature fusion model, which is used to extract image features from the original image data; to perform feature fusion on the extracted image features to obtain a fusion value; and to perform normalization processing on the fusion value to obtain a detection result; The defect classification module is used to classify defects of the image sensor to be detected according to the detection results obtained by the defect detection module.

[0017] In order to solve the above technical problems, the present invention also provides an electronic device, comprising a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image sensor automatic detection method as described in any one of the above items is executed.

[0018] The image sensor automatic detection method and system and electronic device provided by the present invention include: obtaining a trained feature fusion model; inputting the original image data taken by the image sensor to be detected into the feature fusion model; the feature fusion model extracts image features from the original image data, and performs feature fusion on the extracted image features to obtain a fusion value, and normalizes the fusion value to obtain a detection result; and classifies defects according to the detection result. The original image data taken by the image sensor is automatically analyzed and processed by the feature fusion model to obtain a defect classification result. Accurate defect classification can be obtained without human intervention, and the detection efficiency is high, which solves the problem of how to improve the accuracy and efficiency of image sensor detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of the automatic detection method of the image sensor provided in this embodiment; Figure 2 This is a structural block diagram of the image sensor automatic detection system provided in this embodiment. DETAILED DESCRIPTION

[0020] The image sensor automatic detection method and system, and electronic device proposed in the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the accompanying drawings is often part of the actual structure. In particular, the emphasis that each drawing needs to show is different, and sometimes different proportions are used.

[0021] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects in order to describe the embodiments of the present invention, rather than to describe a specific order or sequence. It should be understood that the structures used in this way can be interchanged under appropriate circumstances. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] This embodiment provides an automatic detection method for an image sensor. Figure 1 As shown, including: S1, obtain the trained feature fusion model; S2, inputting the original image data captured by the image sensor to be detected into the feature fusion model; S3, the feature fusion model extracts image features from the original image data, performs feature fusion on the extracted image features to obtain a fusion value, and normalizes the fusion value to obtain a detection result; S4, according to the detection result, the image sensor to be detected performs defect classification.

[0023] The image sensor automatic detection method provided in this embodiment automatically analyzes and processes the original image data taken by the image sensor through a feature fusion model to obtain a defect classification result. Accurate defect classification can be obtained without human intervention, and the detection efficiency is high, which solves the problem of how to improve the accuracy and efficiency of image sensor detection.

[0024] Specifically, in this embodiment, step S1, the method for obtaining the trained feature fusion model includes: S11, construct a feature fusion model.

[0025] In this embodiment, the feature fusion model includes a feature extraction module, a feature fusion module and a normalization module; the feature extraction module is used to extract image features from original image data; the feature fusion module is used to perform feature fusion on the image features extracted by the feature extraction module to obtain fusion features, and convert the fusion features into fusion values; the normalization module is used to normalize the fusion values ​​obtained by the feature fusion module to obtain detection results.

[0026] Specifically, in order to achieve better feature fusion results, the feature extraction module uses a convolutional neural network to extract image features from original image data; the feature fusion module uses an image pyramid to fuse the extracted image features to obtain fused features; and the normalization module uses a normalized exponential function to normalize the fusion value.

[0027] Among them, convolutional neural network (CNN) is a type of feedforward neural network that includes convolution calculation and has a deep structure, and is one of the representative algorithms of deep learning. Convolutional neural network has the ability of representation learning, and can classify input information in a translation-invariant manner according to its hierarchical structure. Convolutional neural network mainly includes input layer, hidden layer and output layer. In this embodiment, the input layer may include a normalization layer for normalizing the input original image data to the interval [0,1], thereby improving the learning efficiency and performance of the convolutional neural network. The hidden layer may include a convolutional layer, a pooling layer and a fully connected layer, wherein a plurality of convolutional layers with different convolutional kernels may be included, so as to effectively extract a variety of features, for example, two convolutional layers with convolutional kernels of 2×2 and 3×3 may be selected to extract features from the original image data; the pooling layer may set the pooling function according to actual needs, so as to perform feature selection and information filtering on the image features extracted by the convolutional layer, thereby improving the processing efficiency of the convolutional neural network while ensuring that the effective features are retained; the fully connected layer is used to perform nonlinear combination of the extracted features to obtain the output of the convolutional neural network, thereby completing the learning goal. The output layer outputs the extracted features using a logistic function or a normalized exponential function (softmax).

[0028] And, the image pyramid is a kind of multi-scale representation of the image, which is an effective but conceptually simple structure to interpret the image at multiple resolutions. Each pass of the image pyramid consists of three consecutive steps: first, the approximation of the reduced resolution of the input image features is calculated; then, the output of the previous step is interpolated and filtered to generate the predicted image features with the same resolution as the input image features; finally, the difference between the input image features and the predicted image features is calculated, so as to reconstruct the input image features to complete the feature fusion. Specifically, it can be implemented using the Laplacian pyramid.

[0029] After the image pyramid completes feature fusion, in this embodiment, a multivariate feature algorithm and a weighted average algorithm can be used to obtain a fusion value corresponding to the fused feature. Through the multivariate feature algorithm and the weighted average algorithm (multi-feature fusion algorithm), data features of different modalities can be merged in the same feature space to obtain a more comprehensive and richer feature representation, thereby effectively improving the accuracy of classification and recognition.

[0030] And, the normalized exponential function (softmax) can "compress" a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that the range of each element is between (0,1) and the sum of all elements is 1. Usually, the normalized exponential function is used as the output of the classification model to achieve the classification of the results.

[0031] In practical applications, those skilled in the art can reasonably set the model structure and parameter settings of the feature fusion model according to actual needs. This application will not go into details and will not limit the model structure and parameter settings of the feature fusion model. Other feature fusion models that do not violate the purpose of this application should also fall within the scope of protection of this application.

[0032] S12, training the constructed feature fusion model.

[0033] In order to improve the robustness of the constructed feature fusion model and the accuracy of the output results, in this embodiment, the constructed feature fusion model also needs to be trained.

[0034] Specifically, in this embodiment, the method for training the constructed feature fusion model includes: First, a training image dataset P is prepared, wherein the training image dataset P includes a defective image dataset P1 and a normal image dataset P2. In practical applications, the defective image dataset P1 may include a plurality of original image data with defects, for example, n original image data with defects; and the normal image dataset P2 may include a plurality of original image data without defects, for example, m original image data without defects.

[0035] In this embodiment, the original image data includes the grayscale value of each pixel in the original image. That is, if the size of an original image is 256×256, it contains 65536 original image data (grayscale values). As the original image for training, the pixels with defects have been marked, so the defective image data and normal image data can be obtained according to the marking.

[0036] In practical applications, one original image can be used as an image data set, or multiple original images can be used to form an image data set, which is not limited in this application.

[0037] Then, the feature extraction module is used to extract features from the training image dataset P to obtain the image feature set , where the image feature set includes the defect image feature set and normal image feature set Specifically, the image features extracted from the defective image data are defective image features, and the image features extracted from the normal image data are normal image features. In other words, the defective image feature set The number of defect image features in the defect image dataset P1 is consistent with the number of defect image data in the defect image dataset P1; the normal image feature set The number of normal image features in is consistent with the number of normal image data in the normal image dataset P2.

[0038] Through the convolutional neural network in the feature extraction module, various image features such as edge features and bending features can be extracted.

[0039] Then, the feature fusion module is used to fusion the defect image feature set. and normal image feature set The feature fusion is performed to obtain a fusion feature, and a multi-feature fusion calculation is performed on the fusion feature to obtain a fusion value. Specifically, the multi-feature fusion calculation may include a weighted average calculation of multi-features.

[0040] Finally, the normalization module is used to normalize the fusion value to obtain the training result.

[0041] Compare the training output results with the labeled results to confirm the training effect of the feature fusion model. If the result accuracy exceeds the preset threshold, the feature fusion model training is considered complete, otherwise continue training until the result accuracy meets the standard.

[0042] Preferably, in order to improve the sensitivity of the feature fusion model to unknown defects and to confirm whether the feature fusion model can effectively identify unknown defects, in this embodiment, the method for training the constructed feature fusion model also includes: using the defect image feature set and the normal image feature set to obtain a supplementary image feature set; using the feature fusion module to perform feature fusion on the defect image feature set, the normal image feature set and the supplementary image feature set to obtain a fusion value.

[0043] Specifically, in this embodiment, first, from the image feature set A number of defect image features and a number of normal image features are randomly extracted to obtain a random feature subset; then, the maximum value in the random feature subset is used as the first reference value, and the minimum value in the random feature subset is used as the second reference value; then, the deviation value is calculated using the first reference value and the second reference value, where the deviation value B is expressed as:

[0044] Where n represents the number of image features in the random feature subset, represents the i-th image feature in the random feature subset, represents the mean value of image features in the random feature subset; finally, a supplementary image feature set is generated according to the deviation value and the image feature set, where the supplementary image feature set It is expressed as:

[0045] in, represents the first reference value, Indicates the second reference value.

[0046] In practical applications, in the process of generating the supplementary feature image set, all the defect image features and normal image features in the image feature set can be used. The more image features are extracted, the richer the generated supplementary feature image set is, and the better the feature fusion model trained with the defect image feature set, the normal image feature set and the supplementary image feature set will be able to recognize unknown defects.

[0047] Furthermore, in this embodiment, in step S2, the original image data captured by the image sensor to be detected is input into the feature fusion model.

[0048] Specifically, the size of the original image captured by the input image sensor to be detected should be consistent with the image size used for training the fusion thermal resistance model, so that the feature fusion model has better classification performance.

[0049] Also, in this embodiment, in step S3, the feature fusion model extracts image features from the original image data, and performs feature fusion on the extracted image features to obtain a fusion value, and normalizes the fusion value to obtain a detection result, wherein the processing process of the feature fusion model on the original image data is consistent with the process during training, and this application will not repeat it.

[0050] Furthermore, in this embodiment, in step S4, the image sensor to be inspected performs defect classification according to the inspection result.

[0051] Specifically, the feature fusion model can eventually output a defect classification result of the image sensor to be detected, and then the defect classification of the image sensor to be detected can be performed according to the detection result output by the feature fusion model.

[0052] The automatic detection method for image sensors provided in this embodiment automatically analyzes and processes the original image data taken by the image sensor through a feature fusion model to obtain defect classification results. Accurate defect classification can be obtained without human intervention, and the detection efficiency is high, which solves the problem of how to improve the accuracy and efficiency of image sensor detection. The automatic detection method for image sensors provided in this embodiment uses the existing defect image feature set and normal image feature set to obtain a supplementary image feature set, thereby enriching the training samples, effectively improving the feature fusion model's ability to identify unknown defects, and thus improving the accuracy of image sensor defect classification detection.

[0053] This embodiment also provides an image sensor automatic detection system for implementing the above-mentioned image sensor automatic detection method, such as Figure 2 As shown, the image sensor automatic detection system includes: A data acquisition module is used to acquire the original image data captured by the image sensor to be detected; The defect detection module includes a feature fusion model, which is used to extract image features from the original image data; to perform feature fusion on the extracted image features to obtain a fusion value; and to perform normalization processing on the fusion value to obtain a detection result; The defect classification module is used to classify defects of the image sensor to be detected according to the detection results obtained by the defect detection module.

[0054] In practical applications, the data acquisition module can be various data interfaces, so that the original image data captured by the image sensor can be directly read from the image sensor. The defect detection module can be a computer program based on a computer, which includes a feature fusion model, so that the original image data can be analyzed and processed to obtain the detection results; preferably, the defect detection module can also display the detection results on a display, and can also generate corresponding operation instructions based on the detection results. The defect classification module can be a device such as a manipulator with a picking function and a communication function, so that the image sensor installation detection results currently detected are classified and transported to the target position by receiving the operation instructions output by the defect detection module, so as to realize different classification and placement of image sensors with different defects. In this way, the automatic defect detection and classification of the image sensor can be efficiently realized without human intervention, thereby improving the accuracy and efficiency of image sensor defect detection.

[0055] The image sensor automatic detection system provided in this embodiment automatically analyzes and processes the original image data taken by the image sensor through the feature fusion model in the defect detection module, and then obtains the defect classification result. It can obtain accurate defect classification without human intervention, and has high detection efficiency, which solves the problem of how to improve the accuracy and efficiency of image sensor detection.

[0056] Furthermore, this embodiment further provides an electronic device, comprising a memory, a processor, and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image sensor automatic detection method as described above is executed.

[0057] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. In addition, the different parts between the various embodiments can also be used in combination with each other, and the present invention is not limited to this.

[0058] The image sensor automatic detection method and system, and electronic device provided in this embodiment include: obtaining a trained feature fusion model; inputting the original image data taken by the image sensor to be detected into the feature fusion model; the feature fusion model extracts image features from the original image data, and performs feature fusion on the extracted image features to obtain a fusion value, and normalizes the fusion value to obtain a detection result; and classifies defects according to the detection result. The original image data taken by the image sensor is automatically analyzed and processed by the feature fusion model to obtain a defect classification result. Accurate defect classification can be obtained without human intervention, and the detection efficiency is high, which solves the problem of how to improve the accuracy and efficiency of image sensor detection.

[0059] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. An automatic detection method for an image sensor, characterized in that: include: Obtain a trained feature fusion model; Inputting the original image data captured by the image sensor to be detected into the feature fusion model; The feature fusion model extracts image features from the original image data, performs feature fusion on the extracted image features to obtain a fusion value, and normalizes the fusion value to obtain a detection result; According to the detection results, the defects of the image sensor to be inspected are classified.

2. The automatic detection method of image sensor according to claim 1, characterized in that: The feature fusion model includes a feature extraction module, a feature fusion module and a normalization module; the feature extraction module is used to extract image features from original image data; the feature fusion module is used to perform feature fusion on the image features extracted by the feature extraction module to obtain fusion features, and convert the fusion features into fusion values; the normalization module is used to normalize the fusion values ​​obtained by the feature fusion module to obtain detection results.

3. The automatic detection method of image sensor according to claim 2, characterized in that: The feature extraction module uses a convolutional neural network to extract image features from original image data; the feature fusion module uses an image pyramid to fuse the extracted image features to obtain fused features; and the normalization module uses a normalized exponential function to normalize the fused values.

4. The automatic detection method of image sensor according to claim 2, characterized in that: The method for obtaining a trained feature fusion model includes training the feature fusion model, including: Creating a training image dataset, wherein the training image dataset includes a defect image dataset and a normal image dataset; Using a feature extraction module, extracting features from a training image data set to obtain an image feature set, wherein the image feature set includes a defect image feature set and a normal image feature set; Using the feature fusion module, the defect image feature set and the normal image feature set are fused to obtain a fusion value; The normalization module is used to normalize the fusion value to obtain the training result.

5. The automatic detection method of image sensor according to claim 4, characterized in that: The method for training the constructed feature fusion model also includes: A complementary image feature set is obtained by using the defect image feature set and the normal image feature set; The feature fusion module is used to fuse the defect image feature set, the normal image feature set and the supplementary image feature set to obtain a fusion value.

6. The automatic detection method of image sensor according to claim 5, characterized in that: The method for obtaining a supplementary image feature set by using a defect image feature set and a normal image feature set comprises: Randomly extracting a number of defect image features and a number of normal image features from the image feature set to obtain a random feature subset; The maximum value in the random feature subset is used as the first reference value, and the minimum value in the random feature subset is used as the second reference value; Calculating a deviation value using the first reference value and the second reference value; A complementary image feature set is generated according to the deviation value and the image feature set.

7. The automatic detection method of image sensor according to claim 6, characterized in that: The deviation value B is expressed as: Where n represents the number of image features in the random feature subset, represents the i-th image feature in the random feature subset, represents the mean of image features in a random feature subset; The supplementary image feature set It is expressed as: in, represents the first reference value, Indicates the second reference value.

8. The automatic detection method of image sensor according to claim 1, characterized in that: The original image data includes the grayscale value of each pixel in the original image.

9. An automatic detection system for an image sensor, used to implement the automatic detection method for an image sensor according to any one of claims 1 to 8, characterized in that: The image sensor automatic detection system comprises: A data acquisition module is used to acquire the original image data captured by the image sensor to be detected; The defect detection module includes a feature fusion model, which is used to extract image features from the original image data; to perform feature fusion on the extracted image features to obtain a fusion value; and to perform normalization processing on the fusion value to obtain a detection result; The defect classification module is used to classify defects of the image sensor to be detected according to the detection results obtained by the defect detection module.

10. An electronic device, characterized in that: The method comprises a memory, a processor and an executable program stored in the memory and capable of being run by the processor; when the processor runs the executable program, the image sensor automatic detection method according to any one of claims 1 to 8 is executed.

Images