Mobile phone screen backlight foreign matter defect diagnosis method and system

Abstract

The invention provides a mobile phone screen backlight foreign matter defect diagnosis method and system, belongs to the field of artificial intelligence, and is used for improving the robustness of foreign matter defect diagnosis and detection. The method comprises the steps that the electronic equipment obtains a first image and a second image obtained by shooting a mobile phone screen, the mobile phone screen is in a backlight state when being shot, and the first image and the second image are different in light and shadow state; the electronic device performs synchronous convolution on the first image and the second image through a dual-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; the electronic device synchronously fuses the first feature sequence and the second feature sequence through a fusion layer of a neural network model to obtain a fused feature sequence; the electronic device processes the fused feature sequence through a feature processing layer of the neural network model to obtain a processing result, and the processing result indicates whether the foreign matter defect exists in the mobile phone screen in the backlight state or not.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for diagnosing foreign matter defects in the backlight of a mobile phone screen. Background Art

[0002] With the rapid development of technology, smartphones have become an indispensable part of people's daily lives. As one of its core components, the quality of the mobile phone screen directly affects the user's visual and usage experience. Therefore, quality control of mobile phone screens is particularly important. Among them, the detection of screen backlight foreign matter defects is a key link. Traditional methods for detecting screen backlight foreign matter defects mainly rely on manual visual inspection or simple automated equipment. However, these methods have problems such as strong subjectivity, low efficiency, insufficient accuracy, and high cost. To address the above problems, the recent development of artificial intelligence technology, especially deep learning technology, has provided new solutions for screen backlight foreign matter defect detection.

[0003] However, how to ensure the robustness of neural network models during detection is a current research issue. Summary of the Invention

[0004] An embodiment of the present application provides a method for diagnosing foreign matter defects in the backlight of a mobile phone screen, so as to improve the robustness of defect detection.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a method for diagnosing foreign matter defects in the backlight of a mobile phone screen is provided, which is applied to an electronic device. The method includes: the electronic device acquires a first image and a second image obtained by shooting the mobile phone screen, the mobile phone screen is in a backlight state when being shot, and the light and shadow states of the first image and the second image are different; the electronic device synchronously convolves the first image and the second image through a dual-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; the electronic device synchronously fuses the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence; the electronic device processes the fused feature sequence through a feature processing layer of the neural network model to obtain a processing result, and the processing result indicates whether there is a foreign matter defect on the mobile phone screen in the backlight state.

[0007] Optionally, the difference in light and shadow conditions between the first image and the second image means that: the first image is an image taken when the mobile phone screen is highlighted, the second image is an image taken when the mobile phone screen is not illuminated, the first area in the first image is the highlighted area, and the first area is a partial area in the first image.

[0008] Optionally, the electronic device performs synchronous convolution on the first image and the second image respectively through the dual-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image, including: the electronic device extracts a first sub-image containing only the first area from the first image, and extracts a second sub-image containing only the second area from the second image, and the position of the second area in the second image is the same as the position of the first area in the first image; the electronic device performs a first convolution on the first sub-image through the first convolution channel of the neural network model, and performs a second convolution on the second image through the second convolution channel of the neural network model to obtain a first feature sequence and a second feature sequence, respectively. Synchronous convolution means that the position of each convolution performed in the first convolution is the same as the position of each convolution performed in the second convolution, and the size of the convolution kernel used to perform the convolution in the first convolution is the same as the size of the convolution kernel used to perform the convolution in the second convolution.

[0009] Optionally, the electronic device synchronously fuses the first feature sequence and the second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence, including: the electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence.

[0010] Optionally, the first feature sequence includes K1 first features, and the second feature sequence includes K2 second features, K1 and K2 are integers with the same value and greater than 10; each at least one first feature is the i-th first feature to the j-th first feature among the K1 first features, and each at least one second feature is the i-th second feature to the j-th second feature among the K2 second features; when the values ​​of i and j are the same, i is an integer ranging from 1 to K1 / K2, and / represents an or relationship; when the values ​​of i and j are different, i is an integer ranging from 1 to K1-j / K2-j, and j is an integer ranging from j to K1 / K2; the i-th first feature to the j-th first feature are the first features obtained by performing the i-th convolution to the j-th convolution in the first convolution, and the i-th second feature to the j-th second feature are the second features obtained by performing the i-th convolution to the j-th convolution in the second convolution.

[0011] Optionally, the electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence, including: the electronic device inputs the i-th first feature to the j-th first feature into the first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtains the i-th fused feature to the j-th fused feature output by the output end of the fusion layer; if the values ​​of i and j are the same, then when i traverses 1 to K1 / K2, the fused feature sequence is obtained; if the values ​​of i and j are different, then when i traverses integers from 1 to K1-j / K2-j and j traverses K1 / K2, the fused feature sequence is obtained; the fusion layer is a partial sub-neuronal network in the feature processing layer, and the neuronal network in the fusion layer is a star connection structure.

[0012] Optionally, when i and j are the same, the i-th first feature to the j-th first feature is the i-th first feature, and the i-th second feature to the j-th second feature is the i-th second feature; the fusion layer includes the first neuron to the sixth neuron, a total of 6 neurons, and the connection relationship of the 6 neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron one by one, and the fourth neuron, the fifth neuron and the sixth neuron are connected to each other; on this basis, the electronic device inputs the i-th first feature to the j-th first feature into the first input end of the fusion layer, and the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtains the output end output of the fusion layer The process of merging the i-th fused feature to the j-th fused feature includes: the electronic device inputs the i-th first feature to the input end of the first neuron and inputs the i-th second feature to the input end of the second neuron, and obtains the i-th fused feature output by the output end of the third neuron; wherein, the input end of the first neuron is the first input end, and the input end of the first neuron is not connected to any neuron among the six neurons except the first neuron; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected to any neuron among the six neurons except the second neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron among the six neurons except the third neuron.

[0013] Optionally, j=i+1, the i-th first feature to the j-th first feature are the i-th first feature and the i+1-th first feature, and the i-th second feature to the i+1-th second feature are the i-th second feature; the fusion layer includes the first neuron to the ninth neuron, a total of 9 neurons, and the connection relationship of the 9 neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron one-to-one, and the fourth neuron, the fifth neuron and the sixth neuron are connected to each other, the fourth neuron, the fifth neuron and the sixth neuron are connected to the seventh neuron, the eighth neuron and the ninth neuron one-to-one, and the seventh neuron, the eighth neuron and the ninth neuron are connected to each other. On this basis, the electronic device inputs the i-th first feature to the j-th first feature into the first input end of the fusion layer, and the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtains the i-th fused feature to the j-th fused feature output by the output end of the fusion layer, including: the electronic device inputs the i-th first feature to the j-th first feature A feature is input to the input end of the first neuron, the i-th second feature is input to the input end of the second neuron, the i+1-th first feature is input to the input end of the seventh neuron, and the i-th second feature is input to the input end of the eighth neuron, to obtain the i-th fused feature output by the output end of the third neuron; wherein, the input end of the first neuron and the input end of the seventh neuron are the first input end, and the input end of the first neuron is not connected to any neuron among the 9 neurons except the first neuron, and the input end of the seventh neuron is not connected to any neuron among the 9 neurons except the seventh neuron; the input end of the second neuron and the input end of the eighth neuron are the second input end, and the input end of the second neuron is not connected to any neuron among the 9 neurons except the second neuron, and the input end of the eighth neuron is not connected to any neuron among the 9 neurons except the eighth neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron among the 9 neurons except the third neuron.

[0014] In a second aspect, a mobile phone screen backlight foreign body defect diagnosis system is provided, characterized in that the system includes an electronic device configured to execute the method described in the first aspect.

[0015] In a third aspect, a computer-readable storage medium is provided, comprising: a computer program or instructions; when the computer program or instructions are run on a computer, the computer is caused to execute the method described in the first aspect.

[0016] In summary, the above method and system have the following technical effects: by using different light and shadow states to shoot the mobile phone screen in the backlight state respectively, a first image and a second image are obtained, whereby the electronic device synchronously convolves the first image and the second image respectively through the dual-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image, so that the first feature sequence and the second feature sequence can be synchronously fused through the fusion layer of the neural network model to obtain a fused feature sequence. In this way, the electronic device processes the fused feature sequence through the feature processing layer of the neural network model, so that different light and shadow states can also be taken into account during processing, thereby improving the robustness of the diagnosis and detection of foreign body defects in the backlight of the mobile phone screen, that is, the reliability and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a process for diagnosing foreign matter defects in a mobile phone screen backlight provided in an embodiment of the present application;

[0018] Figure 2 Schematic diagram of the application of the method for diagnosing foreign matter defects in the backlight of a mobile phone screen provided in the embodiment of the present application Figure 1 ;

[0019] Figure 3 Schematic diagram of the application of the method for diagnosing foreign matter defects in the backlight of a mobile phone screen provided in the embodiment of the present application Figure 2 ;

[0020] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solution in this application will be described below with reference to the accompanying drawings.

[0022] This application will present various aspects, embodiments, or features in the context of systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0023] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0024] In the embodiments of the present application, "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, the meanings to be expressed are matched. In addition, the " / " mentioned in this application can be used to represent an "or" relationship. The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. It is known to those skilled in the art that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0025] For example, Figure 1 The present invention provides a flowchart of a method for diagnosing foreign matter defects in a mobile phone screen backlight. The method can be applied to electronic devices.

[0026] like Figure 1 As shown, the process of the mobile phone screen backlight foreign body defect diagnosis method is as follows:

[0027] S101: The electronic device obtains a first image and a second image obtained by photographing a mobile phone screen.

[0028] The mobile phone screen is in a backlight state when it is photographed, and the light and shadow states of the first image and the second image are different. For example, the light and shadow states of the first image and the second image are different, which means that the first image is an image taken when the mobile phone screen is highlighted, and the second image is an image taken when the mobile phone screen is not illuminated (that is, the mobile phone screen is directly photographed without adding any light, and the light source at this time is provided by the backlight state of the mobile phone screen). The first area in the first image is the highlighted area, and the first area is a partial area in the first image. As an example, the backlight state of the mobile phone screen can be that the mobile phone screen displays a dark color, such as dark blue or dark gray. At this time, the designated area of ​​the mobile phone screen (that is, the first area mentioned above) can be illuminated by lighting. At this time, the first area will superimpose light on the dark color, forming a certain reflection, so that some foreign matter defects may be more obvious. For example, Figure 2 As shown, the first area may be a rectangular area.

[0029] S102, the electronic device performs synchronous convolution on the first image and the second image respectively through the dual-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image.

[0030] The electronic device can first extract a first sub-image containing only the first area from the first image, and extract a second sub-image containing only the second area from the second image, where the position of the second area in the second image is the same as the position of the first area in the first image. For example, because the first area is illuminated and has a higher brightness, the electronic device can grayscale the first image and then determine the pixels at the edge of the first area through pixel difference evaluation, thereby extracting the first sub-image containing only the first area from the first image. Since the first image and the second image are both based on the same angle, such as an image taken from an angle looking down at a mobile phone screen, and the image resolutions are also the same, the electronic device can extract the area with the same coordinate position, i.e., the second area, from the second image based on the coordinate position of the first area in the first image.

[0031] The neural network model may be a convolutional neural network model (CNN).

[0032] The electronic device can then perform a first convolution on the first sub-image through the first convolution channel of the neural network model, and perform a second convolution on the second image through the second convolution channel of the neural network model, to obtain a first feature sequence and a second feature sequence, respectively. It can be understood that synchronous convolution means that the position of each convolution performed in the first convolution is the same as the position of each convolution performed in the second convolution, and the size of the convolution kernel used to perform the convolution in the first convolution is the same as the size of the convolution kernel used to perform the convolution in the second convolution. For example, if the size of the first convolution kernel of the first convolution is 2*2, then the size of the second convolution kernel of the second convolution is also 2*2, and the step size of the convolution is 1. On this basis, the first convolution of the first convolution is to convolve the four pixels in the first sub-image with coordinate positions of (1,1), (1,2), (2,1), and (2,2) through the first convolution kernel. Then the first convolution of the second convolution is also to convolve the four pixels in the second sub-image with coordinate positions of (1,1), (1,2), (2,1), and (2,2) through the second convolution kernel, which is the so-called synchronous convolution. After that, the second convolution of the first convolution is to convolve the four pixels in the first sub-image with coordinate positions of (1,2), (1,3), (2,2), and (2,3) through the first convolution kernel. Then the first convolution of the second convolution is also to convolve the four pixels in the second sub-image with coordinate positions of (1,2), (1,3), (2,2), and (2,3) through the second convolution kernel, which is the so-called synchronous convolution, and so on. Each convolution obtains a feature sequence, from which the first feature sequence and the second feature sequence can be obtained respectively.

[0033] S103, the electronic device synchronously fuses the first feature sequence and the second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence.

[0034] The electronic device may fuse each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through a fusion layer of a neural network model to obtain a fused feature sequence. For example, the first feature sequence includes K1 first features, and the second feature sequence includes K2 second features, where K1 and K2 are integers having the same value and greater than 10. Each at least one first feature is the i-th first feature to the j-th first feature among the K1 first features, and each at least one second feature is the i-th second feature to the j-th second feature among the K2 second features; when the values ​​of i and j are the same, i is an integer ranging from 1 to K1 / K2, and " / " represents an or relationship; when the values ​​of i and j are different, i is an integer ranging from 1 to K1-j / K2-j, and j is an integer ranging from j to K1 / K2; the i-th first feature to the j-th first feature are the first features obtained by performing the i-th convolution to the j-th convolution in the first convolution, and the i-th second feature to the j-th second feature are the second features obtained by performing the i-th convolution to the j-th convolution in the second convolution, that is, the above-mentioned synchronous convolution.

[0035] Specifically, the electronic device can input the i-th first feature to the j-th first feature into the first input end of the fusion layer, and the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtain the i-th fused feature to the j-th fused feature output by the output end of the fusion layer; if the values ​​of i and j are the same, then when i traverses 1 to K1 / K2, the fused feature sequence is obtained; if the values ​​of i and j are different, then when i traverses integers from 1 to K1-j / K2-j and j traverses K1 / K2, the fused feature sequence is obtained; the fusion layer is a partial sub-neural network in the feature processing layer, and the neuronal network in the fusion layer is a star-connected structure, which is specifically introduced in two ways below.

[0036] Method 1:

[0037] When i and j are the same, the i-th first feature to the j-th first feature is the i-th first feature, and the i-th second feature to the j-th second feature is the i-th second feature; the fusion layer includes the first neuron to the sixth neuron, a total of 6 neurons. Figure 3As shown in (a), the connection relationship of the six neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron in a one-to-one correspondence, and the fourth neuron, the fifth neuron and the sixth neuron are connected to each other. This connection method of the six neurons constitutes a star structure. The advantage is that the input features can be fully fused through the connection relationship between the fourth neuron, the fifth neuron and the sixth neuron. It should be understood that the six neurons forming the star structure can be directly copied from the fully connected layer of the neural network model, that is, the fully connected layer of the neural network model, such as the six neurons of the star structure in the fully connected layer that has been trained to convergence, are copied and then configured into the fusion layer. The function of any one of the six neurons can be expressed as follows:

[0038]

[0039] Among them, x n is a characteristic sequence, ω mn is the fully connected weight matrix (i.e., the weight trained to convergence), b m For bias.

[0040] At this point, since these six neurons have already been trained to convergence in the fully connected layer, the feature fusion performed at the front end can be coupled with the feature processing performed by the fully connected layer at the back end, further improving the robustness of the model. Furthermore, since there are only a small number of neurons (six), the processing does not result in normalization, but rather in feature fusion.

[0041] On this basis, the electronic device can input the i-th first feature into the input end of the first neuron and input the i-th second feature into the input end of the second neuron to obtain the i-th fused feature output by the output end of the third neuron; wherein, the input end of the first neuron is the first input end, and the input end of the first neuron is not connected to any neuron among the six neurons except the first neuron; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected to any neuron among the six neurons except the second neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron among the six neurons except the third neuron.

[0042] Method 2:

[0043] j=i+1, the i-th first feature to the j-th first feature are the i-th first feature and the i+1-th first feature, and the i-th second feature to the i+1-th second feature are the i-th second feature. The fusion layer includes the first neuron to the ninth neuron, a total of 9 neurons. Figure 3As shown in (b), the connection relationship of the 9 neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron in a one-to-one correspondence, the fourth neuron, the fifth neuron and the sixth neuron are connected to each other, the fourth neuron, the fifth neuron and the sixth neuron are connected to the seventh neuron, the eighth neuron and the ninth neuron in a one-to-one correspondence, and the seventh neuron, the eighth neuron and the ninth neuron are connected to each other. Similar to method 1, it should also be understood that the star structure composed of these 9 neurons can be directly copied from the fully connected layer of the neural network model, that is, the fully connected layer of the neural network model, such as the 9 neurons of the star structure in the fully connected layer that has been trained to convergence, is copied and then configured into the fusion layer. Since these 9 neurons have already been trained to convergence in the fully connected layer, the feature fusion at the front end can be coupled with the feature processing performed by the fully connected layer at the back end, further improving the robustness of the model. In addition, since the number of neurons is small, that is, 9, there will be no normalization effect in the processing process, but feature fusion.

[0044] On this basis, the electronic device can input the i-th first feature to the input end of the first neuron, input the i-th second feature to the input end of the second neuron, input the i+1-th first feature to the input end of the seventh neuron, input the i-th second feature to the input end of the eighth neuron, and obtain the i-th fused feature output by the output end of the third neuron;

[0045] The input of the first neuron and the input of the seventh neuron constitute the first input, and the input of the first neuron is not connected to any of the nine neurons except the first neuron, and the input of the seventh neuron is not connected to any of the nine neurons except the seventh neuron; the input of the second neuron and the input of the eighth neuron constitute the second input, and the input of the second neuron is not connected to any of the nine neurons except the second neuron, and the input of the eighth neuron is not connected to any of the nine neurons except the eighth neuron; the output of the third neuron constitutes the output of the fusion layer, and the output of the third neuron is not connected to any of the nine neurons except the third neuron. At this point, because the seventh neuron is connected to the fourth neuron, the i-th first feature, after being fused first, can be fused again with the i+1-th first feature, thereby achieving a second-superposition deep fusion through the neuron connection structure. Similarly, because the eighth neuron is connected to the fifth neuron, the i-th second feature, after being fused first, can be fused again with the i+1-th second feature, thereby achieving a second-superposition deep fusion through the neuron connection structure, thereby achieving a better fusion effect.

[0046] S104, the electronic device processes the fused feature sequence through the feature processing layer of the neural network model to obtain a processing result.

[0047] The feature processing layer may include a pooling layer and a fully connected layer, and the specific design may adopt the existing design and will not be described in detail.

[0048] The processing result indicates whether there is a foreign matter defect on the mobile phone screen in the backlight state.

[0049] In summary: By using different light and shadow conditions to shoot the mobile phone screen in the backlight state, a first image and a second image are obtained. As a result, the electronic device performs synchronous convolution on the first image and the second image respectively through the dual-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image, and then the first feature sequence and the second feature sequence can be synchronously fused through the fusion layer of the neural network model to obtain a fused feature sequence. In this way, the electronic device processes the fused feature sequence through the feature processing layer of the neural network model, so that different light and shadow conditions can also be taken into account during processing, thereby improving the robustness of the diagnosis and detection of foreign body defects in the backlight of the mobile phone screen, that is, the reliability and stability.

[0050] It should also be understood that the first area is the area occupying about half of the upper part of the mobile phone screen. That is to say, the method of the embodiment of the present application cannot complete the foreign matter defect diagnosis and detection of the entire mobile phone screen in one execution. It needs to be executed again. The highlighted area this time can be Figure 2 The other areas except the first area, that is, the area around the lower half of the mobile phone screen, are inspected to complete the foreign body defect diagnosis and detection of the entire mobile phone screen.

[0051] Combination of the above Figure 1-Figure 3 The following describes in detail a method for diagnosing foreign matter defects in a mobile phone screen backlight provided by an embodiment of the present application. A mobile phone screen backlight foreign matter defect diagnosis system for executing the method provided by an embodiment of the present application is described in detail.

[0052] The system includes electronic equipment configured to:

[0053] The electronic device obtains a first image and a second image obtained by photographing a mobile phone screen. The mobile phone screen is in a backlight state when photographed, and the light and shadow states of the first image and the second image are different; the electronic device synchronously convolves the first image and the second image through the dual-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; the electronic device synchronously fuses the first feature sequence and the second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence; the electronic device processes the fused feature sequence through the feature processing layer of the neural network model to obtain a processing result, and the processing result indicates whether there is a foreign matter defect on the mobile phone screen in the backlight state.

[0054] Optionally, the difference in light and shadow conditions between the first image and the second image means that: the first image is an image taken when the mobile phone screen is highlighted, the second image is an image taken when the mobile phone screen is not illuminated, the first area in the first image is the highlighted area, and the first area is a partial area in the first image.

[0055] Optionally, the electronic device performs synchronous convolution on the first image and the second image respectively through the dual-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image, including: the electronic device extracts a first sub-image containing only the first area from the first image, and extracts a second sub-image containing only the second area from the second image, and the position of the second area in the second image is the same as the position of the first area in the first image; the electronic device performs a first convolution on the first sub-image through the first convolution channel of the neural network model, and performs a second convolution on the second image through the second convolution channel of the neural network model to obtain a first feature sequence and a second feature sequence, respectively. Synchronous convolution means that the position of each convolution performed in the first convolution is the same as the position of each convolution performed in the second convolution, and the size of the convolution kernel used to perform the convolution in the first convolution is the same as the size of the convolution kernel used to perform the convolution in the second convolution.

[0056] Optionally, the electronic device synchronously fuses the first feature sequence and the second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence, including: the electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence.

[0057] Optionally, the first feature sequence includes K1 first features, and the second feature sequence includes K2 second features, K1 and K2 are integers with the same value and greater than 10; each at least one first feature is the i-th first feature to the j-th first feature among the K1 first features, and each at least one second feature is the i-th second feature to the j-th second feature among the K2 second features; when the values ​​of i and j are the same, i is an integer ranging from 1 to K1 / K2, and / represents an or relationship; when the values ​​of i and j are different, i is an integer ranging from 1 to K1-j / K2-j, and j is an integer ranging from j to K1 / K2; the i-th first feature to the j-th first feature are the first features obtained by performing the i-th convolution to the j-th convolution in the first convolution, and the i-th second feature to the j-th second feature are the second features obtained by performing the i-th convolution to the j-th convolution in the second convolution.

[0058] Optionally, the electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence, including: the electronic device inputs the i-th first feature to the j-th first feature into the first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtains the i-th fused feature to the j-th fused feature output by the output end of the fusion layer; if the values ​​of i and j are the same, then when i traverses 1 to K1 / K2, the fused feature sequence is obtained; if the values ​​of i and j are different, then when i traverses integers from 1 to K1-j / K2-j and j traverses K1 / K2, the fused feature sequence is obtained; the fusion layer is a partial sub-neuronal network in the feature processing layer, and the neuronal network in the fusion layer is a star connection structure.

[0059] Optionally, when i and j are the same, the i-th first feature to the j-th first feature is the i-th first feature, and the i-th second feature to the j-th second feature is the i-th second feature; the fusion layer includes the first neuron to the sixth neuron, a total of 6 neurons, and the connection relationship of the 6 neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron one by one, and the fourth neuron, the fifth neuron and the sixth neuron are connected to each other; on this basis, the electronic device inputs the i-th first feature to the j-th first feature into the first input end of the fusion layer, and the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtains the output end output of the fusion layer The process of merging the i-th fused feature to the j-th fused feature includes: the electronic device inputs the i-th first feature to the input end of the first neuron and inputs the i-th second feature to the input end of the second neuron, and obtains the i-th fused feature output by the output end of the third neuron; wherein, the input end of the first neuron is the first input end, and the input end of the first neuron is not connected to any neuron among the six neurons except the first neuron; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected to any neuron among the six neurons except the second neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron among the six neurons except the third neuron.

[0060] Optionally, j=i+1, the i-th first feature to the j-th first feature are the i-th first feature and the i+1-th first feature, and the i-th second feature to the i+1-th second feature are the i-th second feature; the fusion layer includes the first neuron to the ninth neuron, a total of 9 neurons, and the connection relationship of the 9 neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron one-to-one, and the fourth neuron, the fifth neuron and the sixth neuron are connected to each other, the fourth neuron, the fifth neuron and the sixth neuron are connected to the seventh neuron, the eighth neuron and the ninth neuron one-to-one, and the seventh neuron, the eighth neuron and the ninth neuron are connected to each other. On this basis, the electronic device inputs the i-th first feature to the j-th first feature into the first input end of the fusion layer, and the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtains the i-th fused feature to the j-th fused feature output by the output end of the fusion layer, including: the electronic device inputs the i-th first feature to the j-th first feature A feature is input to the input end of the first neuron, the i-th second feature is input to the input end of the second neuron, the i+1-th first feature is input to the input end of the seventh neuron, and the i-th second feature is input to the input end of the eighth neuron, to obtain the i-th fused feature output by the output end of the third neuron; wherein, the input end of the first neuron and the input end of the seventh neuron are the first input end, and the input end of the first neuron is not connected to any neuron among the 9 neurons except the first neuron, and the input end of the seventh neuron is not connected to any neuron among the 9 neurons except the seventh neuron; the input end of the second neuron and the input end of the eighth neuron are the second input end, and the input end of the second neuron is not connected to any neuron among the 9 neurons except the second neuron, and the input end of the eighth neuron is not connected to any neuron among the 9 neurons except the eighth neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron among the 9 neurons except the third neuron.

[0061] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. For example, the electronic device may be a terminal device, or a chip (system) or other component or assembly that can be provided in the terminal device. Figure 4 As shown, electronic device 400 may include a processor 401. Optionally, electronic device 400 may further include a memory 402 and / or a transceiver 403. Processor 401 is coupled to memory 402 and transceiver 403, for example, via a communication bus. Furthermore, electronic device 400 may be a chip, such as one including processor 401. In this case, the transceiver may be the chip's input / output interface.

[0062] The following combination Figure 4 The components of the electronic device 400 are described in detail below:

[0063] The processor 401 is the control center of the electronic device 400 and can be a single processor or a collective term for multiple processing elements. For example, the processor 401 can be one or more central processing units (CPUs), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0064] Optionally, the processor 401 can execute various functions of the electronic device 400 by running or executing the software program stored in the memory 402 and calling the scientific data stored in the memory 402, such as executing the above Figure 1 The following figure shows the method for diagnosing foreign body defects in the backlight of a mobile phone screen.

[0065] In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0066] In a specific implementation, as an embodiment, the electronic device 400 may also include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing scientific data (e.g., computer programs or instructions).

[0067] The memory 402 is used to store the software program for executing the solution of the present application, and the execution is controlled by the processor 401. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0068] Alternatively, the memory 402 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or scientific data structures and can be accessed by a computer, but is not limited thereto. The memory 402 may be integrated with the processor 401 or exist independently and accessed through the interface circuit ( Figure 4 (not shown) is coupled to the processor 401, which is not specifically limited in this embodiment of the present application.

[0069] Transceiver 403 is used for communication with other electronic devices. For example, if electronic device 400 is a terminal device, transceiver 403 can be used to communicate with a network device or another terminal device. For another example, if electronic device 400 is a network device, transceiver 403 can be used to communicate with a terminal device or another network device.

[0070] Optionally, the transceiver 403 may include a receiver and a transmitter ( Figure 4 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0071] Optionally, the transceiver 403 may be integrated with the processor 401 or may exist independently and communicate with the electronic device 400 through an interface circuit ( Figure 4 (not shown) is coupled to the processor 401, which is not specifically limited in this embodiment of the present application.

[0072] It is understandable that Figure 4 The structure of the electronic device 400 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0073] In addition, the technical effects of the electronic device 400 can refer to the technical effects of the methods described in the above method embodiments, and will not be repeated here.

[0074] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0075] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0076] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions can be transmitted from one website, computer, server or scientific data center to another website, computer, server or scientific data center via a wired (such as infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a scientific data storage device such as a server or scientific data center that contains one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, hard disk, tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0077] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0078] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0079] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0080] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0081] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0083] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0085] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0086] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for diagnosing foreign matter defects in the backlight of a mobile phone screen, characterized in that: Applied to electronic equipment, the method includes: The electronic device acquires a first image and a second image obtained by photographing a mobile phone screen, the mobile phone screen being in a backlight state when photographed, and the first image and the second image having different light and shadow states; The electronic device performs synchronous convolution on the first image and the second image respectively through a dual-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; The electronic device synchronously fuses the first feature sequence and the second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence; The electronic device processes the fused feature sequence through the feature processing layer of the neural network model to obtain a processing result, and the processing result indicates whether there is a foreign matter defect on the mobile phone screen in the backlight state.

2. The method according to claim 1, characterized in that The difference in light and shadow conditions between the first image and the second image means that: the first image is an image taken when the mobile phone screen is highlighted, the second image is an image taken when the mobile phone screen is not illuminated, the first area in the first image is the highlighted area, and the first area is a partial area in the first image.

3. The method according to claim 2, characterized in that The electronic device performs synchronous convolution on the first image and the second image respectively through a dual-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image, including: The electronic device extracts a first sub-image containing only the first region from the first image, and extracts a second sub-image containing only the second region from the second image, where a position of the second region in the second image is the same as a position of the first region in the first image; The electronic device performs a first convolution on the first sub-image through the first convolution channel of the neural network model, and performs a second convolution on the second image through the second convolution channel of the neural network model, to obtain the first feature sequence and the second feature sequence respectively. The synchronous convolution means that the position of each convolution performed in the first convolution is the same as the position of each convolution performed in the second convolution, and the size of the convolution kernel used to perform the convolution in the first convolution is the same as the size of the convolution kernel used to perform the convolution in the second convolution.

4. The method according to claim 3, characterized in that The electronic device synchronously fuses the first feature sequence and the second feature sequence through the fusion layer of the neural network model to obtain a fused feature sequence, including: The electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain the fused feature sequence.

5. The method according to claim 4, characterized in that The first feature sequence includes K1 first features, and the second feature sequence includes K2 second features, where K1 and K2 are integers having the same value and greater than 10; each at least one first feature is the i-th first feature to the j-th first feature among the K1 first features, and each at least one second feature is the i-th second feature to the j-th second feature among the K2 second features; when i and j have the same value, i is an integer ranging from 1 to K1 / K2, and / represents an OR relationship; When the values ​​of i and j are different, i is an integer ranging from 1 to K1-j / K2-j, and j is an integer ranging from j to K1 / K2; the i-th first feature to the j-th first feature are the first features obtained by performing the i-th convolution to the j-th convolution in the first convolution, and the i-th second feature to the j-th second feature are the second features obtained by performing the i-th convolution to the j-th convolution in the second convolution.

6. The method according to claim 5, characterized in that The electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain the fused feature sequence, including: The electronic device inputs the i-th first feature to the j-th first feature to the first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature to the second input end of the fusion layer, and obtains the i-th fused feature to the j-th fused feature output by the output end of the fusion layer; if the values ​​of i and j are the same, then when i traverses 1 to K1 / K2, the fused feature sequence is obtained; if the values ​​of i and j are different, then when i traverses integers from 1 to K1-j / K2-j and j traverses K1 / K2, the fused feature sequence is obtained; the fusion layer is a partial sub-neural network in the feature processing layer, and the neural network in the fusion layer is a star connection structure.

7. The method according to claim 6, characterized in that When i and j are the same, the i-th first feature to the j-th first feature is the i-th first feature, and the i-th second feature to the j-th second feature is the i-th second feature; the fusion layer includes the first neuron to the sixth neuron, a total of 6 neurons, and the connection relationship of the 6 neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron in a one-to-one correspondence, and the fourth neuron, the fifth neuron and the sixth neuron are connected to each other; on this basis, the electronic device inputs the i-th first feature to the j-th first feature to the first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature to the second input end of the fusion layer, and obtains the i-th fused feature to the j-th fused feature output by the output end of the fusion layer, including: The electronic device inputs the i-th first feature into the input end of the first neuron and inputs the i-th second feature into the input end of the second neuron, to obtain the i-th fused feature outputted by the output end of the third neuron; Among them, the input end of the first neuron is the first input end, and the input end of the first neuron is not connected to any neuron among the six neurons except the first neuron; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected to any neuron among the six neurons except the second neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron among the six neurons except the third neuron.

8. The method according to claim 6, characterized in that j=i+1, the i-th first feature to the j-th first feature are the i-th first feature and the i+1-th first feature, and the i-th second feature to the i+1-th second feature are the i-th second feature; the fusion layer includes the first neuron to the ninth neuron, a total of 9 neurons, and the connection relationship of the 9 neurons is: the first neuron, the second neuron and the third neuron are connected to the fourth neuron, the fifth neuron and the sixth neuron in a one-to-one correspondence, the fourth neuron, the fifth neuron and the sixth neuron are connected to each other, the fourth neuron, the fifth neuron and the sixth neuron are connected to the seventh neuron, the eighth neuron and the ninth neuron in a one-to-one correspondence, and the seventh neuron, the eighth neuron and the ninth neuron are connected to each other. On this basis, the electronic device inputs the i-th first feature to the j-th first feature into the first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature into the second input end of the fusion layer, and obtains the i-th fused feature to the j-th fused feature output by the output end of the fusion layer, including: The electronic device inputs the i-th first feature to the input end of the first neuron, inputs the i-th second feature to the input end of the second neuron, inputs the (i+1)-th first feature to the input end of the seventh neuron, and inputs the i-th second feature to the input end of the eighth neuron, to obtain the i-th fused feature output by the output end of the third neuron; Among them, the input end of the first neuron and the input end of the seventh neuron are the first input ends, and the input end of the first neuron is not connected to any neuron among the 9 neurons except the first neuron, and the input end of the seventh neuron is not connected to any neuron among the 9 neurons except the seventh neuron; the input end of the second neuron and the input end of the eighth neuron are the second input ends, and the input end of the second neuron is not connected to any neuron among the 9 neurons except the second neuron, and the input end of the eighth neuron is not connected to any neuron among the 9 neurons except the eighth neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron among the 9 neurons except the third neuron.

9. A mobile phone screen backlight foreign body defect diagnosis system, characterized in that: The system comprises an electronic device configured to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium comprising: A computer program or instruction; when the computer program or instruction is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

Images