A mobile phone screen backlight foreign matter defect diagnosis method and system

Abstract

The application provides a mobile phone screen backlight foreign matter defect diagnosis method and system, belonging to the field of artificial intelligence, to improve the robustness of foreign matter defect diagnosis. The method comprises: an electronic device acquires a first image and a second image obtained by shooting a mobile phone screen, the mobile phone screen is in a backlight state when being shot, and the light and shadow states of the first image and the second image are different; the electronic device synchronously convolves the first image and the second image through a double-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; the electronic device synchronously fuses the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence; and the electronic device processes the fused feature sequence through a feature processing layer of the neural network model to obtain a processing result, the processing result indicating whether there is a foreign matter defect in the mobile phone screen under the backlight state.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a mobile phone screen backlight foreign matter defect diagnosis method and system. BACKGROUND

[0002] With the rapid development of technology, smart phones have become an indispensable part of people's daily life. The quality of the mobile phone screen, as one of its core components, directly affects the user's visual experience and use experience. Therefore, the quality control of the mobile phone screen is particularly important. Among them, the detection of screen backlight foreign matter defects is a key link. Traditional screen backlight foreign matter defect detection methods mainly rely on manual visual inspection or simple automated equipment. However, these methods have the problems of strong subjectivity, low efficiency, insufficient accuracy, high cost, etc. In order to solve the above problems, in recent years, the development of artificial intelligence technology, especially deep learning technology, has provided a new solution for screen backlight foreign matter defect detection.

[0003] However, how to ensure the robustness of the neural network model during detection is a current research problem. SUMMARY

[0004] The mobile phone screen backlight foreign matter defect diagnosis method provided by the embodiments of the present application can improve the robustness of defect detection.

[0005] To achieve the above object, the technical scheme is as follows:

[0006] In a first aspect, a mobile phone screen backlight foreign matter defect diagnosis method is provided, applied to an electronic device, and the method comprises: the electronic device acquires a first image and a second image obtained by photographing a mobile phone screen, the mobile phone screen is in a backlight state when photographed, and the light and shadow states of the first image and the second image are different; the electronic device synchronously convolves the first image and the second image through a double-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; the electronic device synchronously fuses the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence; and the electronic device processes the fused feature sequence through a feature processing layer of the neural network model to obtain a processing result, the processing result indicating whether there is a foreign matter defect in the mobile phone screen in the backlight state.

[0007] Optionally, the light and shadow states of the first image and the second image are different, that is, the first image is an image photographed under the condition of highlighting the mobile phone screen, the second image is an image photographed under the condition of non-highlighting the mobile phone screen, and a first region in the first image is a highlighted region, and the first region is a partial region in the first image.

[0008] Optionally, the electronic device performs synchronous convolution on the first image and the second image through a double-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image, including: the electronic device extracts a first sub-image containing only the first region from the first image and a second sub-image containing only the second region from the second image, the position of the second region in the second image being the same as the position of the first region in the first image; the electronic device performs first convolution on the first sub-image through a first convolution channel of the neural network model and performs second convolution on the second image through a second convolution channel of the neural network model to obtain the first feature sequence and the second feature sequence respectively, the synchronous convolution being that the position of each convolution in the first convolution is the same as the position of each convolution in the second convolution, and the size of the convolution kernel used for convolution in the first convolution is the same as the size of the convolution kernel used for convolution in the second convolution.

[0009] Optionally, the electronic device performs synchronous fusion on the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence, including: the electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain the fused feature sequence.

[0010] Optionally, the first feature sequence includes K1 first features, and the second feature sequence includes K2 second features, K1 and K2 being integers with the same value and greater than 10; each at least one first feature is an i-th first feature to a j-th first feature in the K1 first features, and each at least one second feature is an i-th second feature to a j-th second feature in the K2 second features; in the case that the values of i and j are the same, i is an integer with a value traversing from 1 to K1 / K2, / represents the relationship of or; in the case that the values of i and j are different, i is an integer with a value traversing from 1 to K1-j / K2-j, and j is an integer with a value traversing from j to K1 / K2; the i-th first feature to the j-th first feature is a first feature obtained by performing i-th convolution to j-th convolution in the first convolution, and the i-th second feature to the j-th second feature is a second feature obtained by performing i-th convolution to j-th convolution in the second convolution.

[0011] Optionally, the electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence, including: the electronic device inputs the i th first feature to the j th first feature to a first input end of the fusion layer, and inputs the i th second feature to the j th second feature to a second input end of the fusion layer to obtain the i th fused feature to the j th fused feature output by an output end of the fusion layer; if the values of i and j are the same, the i is iterated under the condition that 1 to K1 / K2, and the fused feature sequence is obtained; if the values of i and j are different, the i is iterated under the condition that 1 to K1-j / K2-j, and the j is iterated under the condition that K1 / K2, and the fused feature sequence is obtained; the fusion layer is part of a sub-neuron network in the feature processing layer, and the neuron network in the fusion layer is a star connection structure.

[0012] Optionally, in the case that i and j are the same, the i th first feature to the j th first feature is the i th first feature, and the i th second feature to the j th second feature is the i th second feature; the fusion layer includes first to sixth neurons, a total of 6 neurons, and the connection relationship of the 6 neurons is: the first neuron, the second neuron and the third neuron are connected with the fourth neuron, the fifth neuron and the sixth neuron one by one, and the fourth neuron, the fifth neuron and the sixth neuron are connected with each other; on this basis, the electronic device inputs the i th first feature to the j th first feature to a first input end of the fusion layer, and inputs the i th second feature to the j th second feature to a second input end of the fusion layer to obtain the i th fused feature to the j th fused feature output by an output end of the fusion layer, including: the electronic device inputs the i th first feature to an input end of the first neuron and inputs the i th second feature to an input end of the second neuron to obtain the i th fused feature output by an output end of the third neuron; wherein, the input end of the first neuron is the first input end, and the input end of the first neuron is not connected with any neuron in the 6 neurons except the first neuron; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected with any neuron in the 6 neurons except the second neuron; and the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected with any neuron in the 6 neurons except the third neuron.

[0013] Optionally, j = i + 1, the ith first feature to the jth first feature is the ith first feature and the ith + 1th first feature, and the ith second feature to the ith + 1th second feature is the ith second feature; the fusion layer includes first neurons to ninth neurons, a total of 9 neurons, and the connection relationship of the 9 neurons is that the first neuron, the second neuron and the third neuron are connected with the fourth neuron, the fifth neuron and the sixth neuron one by one, the fourth neuron, the fifth neuron and the sixth neuron are connected with each other, the fourth neuron, the fifth neuron and the sixth neuron are connected with the seventh neuron, the eighth neuron and the ninth neuron one by one, and the seventh neuron, the eighth neuron and the ninth neuron are connected with each other; on this basis, the electronic device inputs the ith first feature to the jth first feature to the first input end of the fusion layer, and inputs the ith second feature to the jth second feature to the second input end of the fusion layer, to obtain the ith fused feature to the jth fused feature output by the output end of the fusion layer, including: the electronic device inputs the ith first feature to the input end of the first neuron, inputs the ith second feature to the input end of the second neuron, inputs the ith + 1th first feature to the input end of the seventh neuron, and inputs the ith second feature to the input end of the eighth neuron, to obtain the ith fused feature output by the output end of the third neuron; wherein the input end of the first neuron and the input end of the seventh neuron are the first input end, and the input end of the first neuron is not connected with any neuron in the 9 neurons except the first neuron, and the input end of the seventh neuron is not connected with any neuron in the 9 neurons except the seventh neuron; the input end of the second neuron and the input end of the eighth neuron are the second input end, and the input end of the second neuron is not connected with any neuron in the 9 neurons except the second neuron, and the input end of the eighth neuron is not connected with any neuron in the 9 neurons except the eighth neuron; and the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected with any neuron in the 9 neurons except the third neuron.

[0014] In a second aspect, a mobile phone screen backlight foreign matter defect diagnosis system is provided, and the system includes an electronic device configured to perform the method of the first aspect.

[0015] In a third aspect, a computer readable storage medium is provided, including: a computer program or instructions; when the computer program or instructions run on a computer, the computer is caused to perform the method of the first aspect.

[0016] In summary, the method and system have the following technical effects: by adopting different light and shadow states to capture the mobile phone screen in the backlight state to obtain the first image and the second image, the electronic device performs synchronous convolution on the first image and the second image through the double-channel convolution layer of the neural network model to obtain the first feature sequence of the first image and the second feature sequence of the second image, so that the first feature sequence and the second feature sequence can be synchronously fused through the fusion layer of the neural network model to obtain the fused feature sequence. In this way, the electronic device processes the fused feature sequence through the feature processing layer of the neural network model, so that different light and shadow states can also be considered during processing, thereby improving the robustness of the mobile phone screen backlight foreign matter defect diagnosis and detection, that is, the reliability and stability. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a mobile phone screen backlight foreign matter defect diagnosis method provided by an embodiment of the present application is shown in the figure.

[0018] Figure 2 An application diagram of the mobile phone screen backlight foreign matter defect diagnosis method provided by the embodiment of the present application is shown in the figure. Figure 1

[0019] Figure 3 An application diagram of the mobile phone screen backlight foreign matter defect diagnosis method provided by the embodiment of the present application is shown in the figure. Figure 2

[0020] Figure 4 A structural diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0021] The technical solutions in the present application will be described below with reference to the accompanying drawings.

[0022] The present application will present various aspects, embodiments or features around a system that can include multiple devices, components, modules, etc. It should be understood and appreciated that each system can include additional devices, components, modules, etc., and / or can not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. In addition, combinations of these solutions can also be used.

[0023] In addition, in the embodiments of the present application, the words "example", "for example", etc. are used to represent an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner.

[0024] ​​In the embodiments of the present application, "of", "corresponding" and "relevant" can be used interchangeably at times. It should be pointed out that when the distinction is not emphasized, the meaning expressed is matching. In addition, the " / " mentioned in the present application can be used to represent the relationship of "or". The network architecture and business scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of network architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0025] Exemplary, Figure 1 The flowchart of the mobile phone screen backlight foreign matter defect diagnosis method provided by the embodiments of the present application is shown. The method can be applied to electronic devices.

[0026] As Figure 1 shown, the flow of the mobile phone screen backlight foreign matter defect diagnosis method is as follows:

[0027] S101, the electronic device obtains a first image and a second image obtained by shooting a mobile phone screen.

[0028] The mobile phone screen is in a backlight state when being shot, and the light and shadow states of the first image and the second image are different. For example, the light and shadow states of the first image and the second image are different, which means that the first image is an image shot under the condition of highlighting the mobile phone screen, and the second image is an image shot under the condition of not highlighting the mobile phone screen (i.e. without adding any light, directly shooting the mobile phone screen, and the light source is provided by the backlight state of the mobile phone screen), the first area in the first image is the highlighted area, and the first area is a part of the first image. An example, the backlight state of the mobile phone screen can be that the mobile phone screen displays a deep color, such as dark blue or dark gray, etc. At this time, the specified area (i.e. the above-mentioned first area) of the mobile phone screen can be highlighted by highlighting, at this time, the first area will superimpose light on the basis of the deep color, and the form will be reflected, so that some foreign matter defects may be more obvious. For example, as Figure 2 shown, the first area can be a rectangular area.

[0029] S102, the electronic device synchronously convolves the first image and the second image through the double-channel convolution layer of the neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image.

[0030] The electronic device can first extract a first sub-image containing only the first region from the first image, and extract a second sub-image containing only the second region from the second image, the position of the second region in the second image being the same as the position of the first region in the first image. For example, the first region is brighter due to being illuminated, and the electronic device can perform grayscale processing on the first image, and then determine the pixels of the first region edge through pixel difference evaluation, thereby extracting the first sub-image containing only the first region from the first image. Since the first image and the second image are images captured based on the same angle, such as the angle of looking down at the screen of the mobile phone, and the resolution of the images is the same, the electronic device can extract the region at the same coordinate position, i.e., the second region, from the second image based on the coordinate position of the first region in the first image.

[0031] The neural network model can be a convolutional neural network model (CNN).

[0032] The electronic device can further perform first convolution on the first sub-image through a first convolution channel of the neural network model, and perform second convolution on the second image through a second convolution channel of the neural network model, to obtain a first feature sequence and a second feature sequence, respectively. It can be understood that synchronous convolution refers to the same position of each convolution in the first convolution and the same position of each convolution in the second convolution, and the size of the convolution kernel used to perform convolution in the first convolution is the same as the size of the convolution kernel used to perform convolution in the second convolution. For example, the size of the first convolution kernel of the first convolution is 2*2, and the size of the second convolution kernel of the second convolution is also 2*2, and the step of convolution is 1. On this basis, the first convolution of the first convolution is performed on the 4 pixel points with coordinate positions (1, 1), (1, 2), (2, 1), (2, 2) in the first sub-image through the first convolution kernel, and the first convolution of the second convolution is also performed on the 4 pixel points with coordinate positions (1, 1), (1, 2), (2, 1), (2, 2) in the second sub-image through the second convolution kernel, i.e., so-called synchronous convolution. Then, the second convolution of the first convolution is performed on the 4 pixel points with coordinate positions (1, 2), (1, 3), (2, 2), (2, 3) in the first sub-image through the first convolution kernel, and the first convolution of the second convolution is also performed on the 4 pixel points with coordinate positions (1, 2), (1, 3), (2, 2), (2, 3) in the second sub-image through the second convolution kernel, i.e., so-called synchronous convolution, and so on. Each convolution obtains a feature sequence, and thus the first feature sequence and the second feature sequence are obtained.

[0033] In S103, the electronic device synchronously fuses the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence.

[0034] The electronic device can fuse each of at least one first feature in the first feature sequence and each of at least one second feature in the corresponding second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence. For example, the first feature sequence includes K1 first features, and the second feature sequence includes K2 second features, K1 and K2 are integers with the same value and greater than 10. Each of at least one first feature is the i-th first feature to the j-th first feature in the K1 first features, and each of at least one second feature is the i-th second feature to the j-th second feature in the K2 second features; in the case of the same value of i and j, i is an integer with a value traversing 1 to K1 / K2, and " / " represents the relationship of or; in the case of different values of i and j, i is an integer with a value traversing 1 to K1-j / K2-j, and j is an integer with a value traversing j to K1 / K2; the i-th first feature to the j-th first feature is a first feature obtained by performing the i-th convolution to the j-th convolution in the first convolution, and the i-th second feature to the j-th second feature is a second feature obtained by performing the i-th convolution to the j-th convolution in the second convolution, that is, the above-mentioned synchronous convolution.

[0035] Specifically, the electronic device can input the i-th first feature to the j-th first feature to the first input end of the fusion layer, and input the i-th second feature to the j-th second feature to the second input end of the fusion layer, to obtain the i-th fused feature to the j-th fused feature output by the output end of the fusion layer; if the values of i and j are the same, the fused feature sequence is obtained in the case that i traverses 1 to K1 / K2; if the values of i and j are different, the fused feature sequence is obtained in the case that i traverses 1 to K1-j / K2-j, and j traverses K1 / K2; the fusion layer is part of the sub-neuron network in the feature processing layer, and the neuron network in the fusion layer is a star connection structure, which will be introduced in detail in the following two ways.

[0036] Method 1:

[0037] In the case of the same i and j, the i-th first feature to the j-th first feature is the i-th first feature, and the i-th second feature to the j-th second feature is the i-th second feature; the fusion layer includes first to sixth neurons, a total of 6 neurons. As Figure 3As shown in (a) in FIG. 6, the connection relationship of the six neurons is that the first neuron, the second neuron, and the third neuron are connected with the fourth neuron, the fifth neuron, and the sixth neuron one by one, and the fourth neuron, the fifth neuron, and the sixth neuron are connected with each other. This connection mode of the six neurons constitutes a star structure, which has the advantage that the input features can be fully fused through the connection relationship between the fourth neuron, the fifth neuron, and the sixth neuron. It should be understood that the six neurons constituting the star structure can be directly copied from the fully connected layer of the neural network model, that is, the six neurons of the star structure are copied from the fully connected layer of the neural network model, specifically the fully connected layer trained to convergence, and then configured into the fusion layer. The function of any one of the six neurons can be expressed as follows:

[0038]

[0039] wherein x n is a feature sequence, ω mn is a fully connected weight matrix (i.e., a weight trained to convergence), b m is a bias.

[0040] At this time, since the six neurons have been trained to convergence in the fully connected layer, the front-end feature fusion can be coupled with the feature processing performed by the fully connected layer at the back end, thereby further improving the robustness of the model. In addition, since the number of neurons is not large, i.e., six, the normalization effect is not achieved in the processing process, but feature fusion is achieved.

[0041] On this basis, the electronic device can input the ith first feature to the input end of the first neuron and input the ith second feature to the input end of the second neuron to obtain the ith fused feature output by the output end of the third neuron; wherein the input end of the first neuron is the first input end, and the input end of the first neuron is not connected with any neuron except the first neuron among the six neurons; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected with any neuron except the second neuron among the six neurons; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected with any neuron except the third neuron among the six neurons.

[0042] Mode 2:

[0043] j = i + 1, the ith first feature to the jth first feature is the ith first feature and the ith + 1th first feature, and the ith second feature to the ith + 1th second feature is the ith second feature. The fusion layer includes the first neuron to the ninth neuron, a total of nine neurons. As shown in Figure 3As shown in (b) in FIG. 9, the connection relationship of the 9 neurons is that the first neuron, the second neuron and the third neuron are connected with the fourth neuron, the fifth neuron and the sixth neuron one by one, the fourth neuron, the fifth neuron and the sixth neuron are connected with each other, and the fourth neuron, the fifth neuron and the sixth neuron are connected with the seventh neuron, the eighth neuron and the ninth neuron one by one, and the seventh neuron, the eighth neuron and the ninth neuron are connected with each other. Similar to the mode 1, it should also be understood that the 9 neurons constituting the star structure can be directly copied from the full connection layer of the neural network model, that is, the full connection layer of the neural network model, specifically the full connection layer trained to convergence, is copied for the 9 neurons of the star structure, and then configured into the fusion layer. Since the 9 neurons have been trained to convergence in the full connection layer, the feature fusion in the front end can be coupled with the feature processing in the back end performed by the full connection layer, further improving the robustness of the model. In addition, since the number of neurons is not large, that is, 9, the normalization effect will not be achieved in the processing process, but feature fusion.

[0044] On this basis, the electronic device can input the i th first feature to the input end of the first neuron, input the i th second feature to the input end of the second neuron, input the i+1 th first feature to the input end of the seventh neuron, and input the i th second feature to the input end of the eighth neuron, to obtain the i th fused feature output by the output end of the third neuron.

[0045] Among them, the input end of the first neuron and the input end of the seventh neuron are the first input end, and the input end of the first neuron is not connected with any neuron in the 9 neurons except the first neuron, and the input end of the seventh neuron is not connected with any neuron in the 9 neurons except the seventh neuron; the input end of the second neuron and the input end of the eighth neuron are the second input end, and the input end of the second neuron is not connected with any neuron in the 9 neurons except the second neuron, and the input end of the eighth neuron is not connected with any neuron in the 9 neurons except the eighth neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected with any neuron in the 9 neurons except the third neuron. At this time, since the seventh neuron is connected with the fourth neuron, the i th first feature can be fused again with the i+1 th first feature in the case of being fused first, so as to realize the depth fusion of secondary superposition through the connection structure of the neurons, and similarly, since the eighth neuron is connected with the fifth neuron, the i th second feature can be fused again with the i+1 th second feature in the case of being fused first, so as to realize the depth fusion of secondary superposition through the connection structure of the neurons, and further realize better fusion effect.

[0046] S104, the electronic device processes the fused feature sequence through a feature processing layer of the neural network model to obtain a processing result.

[0047] The feature processing layer can include a pooling layer and a fully connected layer, and specific designs can be used without repetition.

[0048] The processing result indicates whether there is a foreign matter defect on the mobile phone screen in the backlight state.

[0049] In summary: by using different light and shadow states to capture the mobile phone screen in the backlight state to obtain the first image and the second image, the electronic device performs synchronous convolution on the first image and the second image through the double-channel convolution layer of the neural network model to obtain the first feature sequence of the first image and the second feature sequence of the second image, so that the first feature sequence and the second feature sequence can be synchronously fused through the fusion layer of the neural network model to obtain the fused feature sequence. In this way, the electronic device processes the fused feature sequence through the feature processing layer of the neural network model, so that different light and shadow states can also be considered during processing, thereby improving the robustness of the mobile phone screen backlight foreign matter defect diagnosis and detection, that is, the reliability and stability.

[0050] It should also be understood that the above-mentioned first area is an area occupying about half of the top of the mobile phone screen, that is, one execution of the method of the present application cannot complete the foreign matter defect diagnosis and detection of the entire mobile phone screen, and needs to be executed again. The area highlighted this time can be Figure 2 the other area except the first area, that is, the area occupying about half of the bottom of the mobile phone screen, so as to complete the foreign matter defect diagnosis and detection of the entire mobile phone screen.

[0051] The above Figures 1-3 The method of diagnosing the foreign matter defect of the mobile phone screen in the backlight state provided by the embodiments of the present application is described in detail. The following describes a mobile phone screen backlight foreign matter defect diagnosis system for executing the method provided by the embodiments of the present application.

[0052] The system includes an electronic device, and the system is configured to:

[0053] The electronic device obtains a first image and a second image captured by a mobile phone screen, the mobile phone screen is in a backlit state when being captured, and the first image and the second image are different in light and shadow state; the electronic device performs synchronous convolution on the first image and the second image through a double-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; the electronic device performs synchronous fusion on the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence; and the electronic device processes the fused feature sequence through a feature processing layer of the neural network model to obtain a processing result, the processing result indicating whether there is a foreign matter defect in the mobile phone screen in the backlit state.

[0054] Optionally, the first image and the second image are different in light and shadow state, that is, the first image is an image captured in the case of highlighting the mobile phone screen, and the second image is an image captured in the case of non-highlighting the mobile phone screen, and the first region in the first image is a highlighted region, and the first region is a partial region in the first image.

[0055] Optionally, the electronic device performs synchronous convolution on the first image and the second image through the double-channel convolution layer of the neural network model to obtain the first feature sequence of the first image and the second feature sequence of the second image, including: the electronic device extracts a first sub-image containing only the first region from the first image, and extracts a second sub-image containing only a second region from the second image, the position of the second region in the second image being the same as that of the first region in the first image; the electronic device performs first convolution on the first sub-image through a first convolution channel of the neural network model, and performs second convolution on the second image through a second convolution channel of the neural network model to obtain the first feature sequence and the second feature sequence respectively, the synchronous convolution being that the position of each convolution in the first convolution is the same as that of each convolution in the second convolution, and the size of the convolution kernel used for convolution in the first convolution is the same as that of the convolution kernel used for convolution in the second convolution.

[0056] Optionally, the electronic device performs synchronous fusion on the first feature sequence and the second feature sequence through the fusion layer of the neural network model to obtain the fused feature sequence, including: the electronic device fuses each at least one first feature in the first feature sequence and each at least one second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain the fused feature sequence.

[0057] Optionally, the first feature sequence includes K1 first features, the second feature sequence includes K2 second features, K1 and K2 are integers with the same value and greater than 10; each of the at least one first feature is an i-th first feature to a j-th first feature in the K1 first features, each of the at least one second feature is an i-th second feature to a j-th second feature in the K2 second features; in the case that i and j have the same value, i is an integer with a value traversing from 1 to K1 / K2, / represents the relationship of or; in the case that i and j have different values, i is an integer with a value traversing from 1 to K1-j / K2-j, j is an integer with a value traversing from j to K1 / K2; the i-th first feature to the j-th first feature is a first feature obtained by performing an i-th convolution to a j-th convolution in the first convolution, and the i-th second feature to the j-th second feature is a second feature obtained by performing an i-th convolution to a j-th convolution in the second convolution.

[0058] Optionally, the electronic device fuses each of the at least one first feature in the first feature sequence and each of the at least one second feature in the corresponding second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence, including: the electronic device inputs the i-th first feature to the j-th first feature to a first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature to a second input end of the fusion layer to obtain an i-th fused feature to a j-th fused feature output by an output end of the fusion layer; if i and j have the same value, the fused feature sequence is obtained in the case that i traverses from 1 to K1 / K2; if i and j have different values, the fused feature sequence is obtained in the case that i traverses from 1 to K1-j / K2-j, and j traverses from K1 / K2; the fusion layer is part of the sub-neural network in the feature processing layer, and the neural network in the fusion layer is a star connection structure.

[0059] Optionally, in the case that i and j are the same, the ith first feature to the jth first feature is the ith first feature, and the ith second feature to the jth second feature is the ith second feature; the fusion layer includes first neurons to sixth neurons, a total of 6 neurons, and the connection relationship of the 6 neurons is that the first neuron, the second neuron, and the third neuron are connected with the fourth neuron, the fifth neuron, and the sixth neuron one by one, and the fourth neuron, the fifth neuron, and the sixth neuron are connected with each other; on this basis, the electronic device inputs the ith first feature to the jth first feature to the first input end of the fusion layer, and inputs the ith second feature to the jth second feature to the second input end of the fusion layer, to obtain the ith fused feature to the jth fused feature output by the output end of the fusion layer, including: the electronic device inputs the ith first feature to the input end of the first neuron and inputs the ith second feature to the input end of the second neuron, to obtain the ith fused feature output by the output end of the third neuron; wherein the input end of the first neuron is the first input end, and the input end of the first neuron is not connected with any neuron in the 6 neurons except the first neuron; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected with any neuron in the 6 neurons except the second neuron; and the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected with any neuron in the 6 neurons except the third neuron.

[0060] Optionally, j = i + 1, the ith first feature to the jth first feature is the ith first feature and the i + 1th first feature, and the ith second feature to the i + 1th second feature is the ith second feature; the fusion layer includes first neurons to ninth neurons, a total of 9 neurons, and the connection relationship of the 9 neurons is that the first neuron, the second neuron and the third neuron are connected with the fourth neuron, the fifth neuron and the sixth neuron one by one, the fourth neuron, the fifth neuron and the sixth neuron are connected with each other, the fourth neuron, the fifth neuron and the sixth neuron are connected with the seventh neuron, the eighth neuron and the ninth neuron one by one, and the seventh neuron, the eighth neuron and the ninth neuron are connected with each other, on the basis of which, the electronic device inputs the ith first feature to the jth first feature to the first input end of the fusion layer, and inputs the ith second feature to the jth second feature to the second input end of the fusion layer, to obtain the ith fused feature to the jth fused feature output by the output end of the fusion layer, including: the electronic device inputs the ith first feature to the input end of the first neuron, inputs the ith second feature to the input end of the second neuron, inputs the i + 1th first feature to the input end of the seventh neuron, and inputs the ith second feature to the input end of the eighth neuron, to obtain the ith fused feature output by the output end of the third neuron; wherein the input end of the first neuron and the input end of the seventh neuron are the first input end, and the input end of the first neuron is not connected with any neuron in the 9 neurons except the first neuron, and the input end of the seventh neuron is not connected with any neuron in the 9 neurons except the seventh neuron; the input end of the second neuron and the input end of the eighth neuron are the second input end, and the input end of the second neuron is not connected with any neuron in the 9 neurons except the second neuron, and the input end of the eighth neuron is not connected with any neuron in the 9 neurons except the eighth neuron; and the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected with any neuron in the 9 neurons except the third neuron.

[0061] Figure 4 The structure schematic diagram of the electronic device provided in the embodiments of the present application is shown. Exemplarily, the electronic device can be a terminal device, or a chip (system) or other components or assemblies which can be arranged in the terminal device. As shown in the figure, Figure 4 the electronic device 400 can include a processor 401. Optionally, the electronic device 400 can also include a memory 402 and / or a transceiver 403. The processor 401 is coupled with the memory 402 and the transceiver 403, which can be connected through a communication bus. In addition, the electronic device 400 can also be a chip, which includes the processor 401, at this time, the transceiver can be an input / output interface of the chip.

[0062] The following will be described in combination withFigure 4 The various constituent components of the electronic device 400 are described in detail as follows:

[0063] The processor 401 is the control center of the electronic device 400, and can be one processor or a collective term of multiple processing elements. For example, the processor 401 is one or more central processing units (CPUs), application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0064] Optionally, the processor 401 can execute various functions of the electronic device 400 by running or executing software programs stored in the memory 402 and calling scientific data stored in the memory 402, such as executing the above-mentioned Figure 1 diagnosis method for foreign matter defects of a mobile phone screen backlight.

[0065] In a specific implementation, as an embodiment, the processor 401 can include one or more CPUs, such as the CPU0 and CPU1 shown in Figure 4

[0066] In a specific implementation, as an embodiment, the electronic device 400 can also include multiple processors. Each of the processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing scientific data (such as computer programs or instructions).

[0067] The memory 402 is configured to store software programs for implementing the schemes of the present application, and the processor 401 is configured to control the execution. The specific implementation manner can refer to the above-mentioned method embodiments, and will not be described here again.

[0068] ​Optionally, the memory 402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or scientific data structures and accessible by a computer, but not limited thereto. The memory 402 may be integrated with the processor 401 or may exist independently and be accessible through the interface circuit of the electronic device 400. Figure 4 (Not shown in the image) is coupled to processor 401, and this embodiment of the application does not specifically limit this.

[0069] Transceiver 403 is used for communication with other electronic devices. For example, if electronic device 400 is a terminal device, transceiver 403 can be used to communicate with a network device or with another terminal device. As another example, if electronic device 400 is a network device, transceiver 403 can be used to communicate with a terminal device or with another network device.

[0070] Alternatively, transceiver 403 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0071] Alternatively, the transceiver 403 can be integrated with the processor 401, or it can exist independently and be connected via the interface circuit of the electronic device 400. Figure 4 (Not shown in the image) is coupled to processor 401, and this embodiment of the application does not specifically limit this.

[0072] Understandable, Figure 4 The structure of the electronic device 400 shown does not constitute a limitation on the electronic device. Actual electronic devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0073] Furthermore, the technical effects of the electronic device 400 can be referred to the technical effects of the methods described in the above method embodiments, and will not be repeated here.

[0074] It should be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0075] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DRRAM).

[0076] The above-described embodiments can be implemented in part or in whole through software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When loaded and executed by a computer, the computer instructions or computer programs can produce the processes or functions described above in accordance with the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer program or instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, such as from a website site, a computer, a server, or a scientific data center to another website site, a computer, a server, or a scientific data center through a wired (e.g., infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a scientific data storage device such as a server, scientific data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0077] It should be understood that the term "and / or" used herein is merely an association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it, but it can also represent an "and / or" relationship, which can be understood in the context before and after it.

[0078] In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0079] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0080] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0081] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0082] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0083] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0084] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0085] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0086] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A mobile phone screen backlight foreign matter defect diagnosis method, characterized in that, The method is applied to an electronic device, and the method comprises: The electronic device acquires a first image and a second image obtained by photographing a mobile phone screen, the mobile phone screen is in a backlit state when being photographed, and the light and shadow states of the first image and the second image are different; The electronic device performs synchronous convolution on the first image and the second image through a double-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image; The electronic device performs synchronous fusion on the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence; The electronic device processes the fused feature sequence through a feature processing layer of the neural network model to obtain a processing result, and the processing result indicates whether there is a foreign matter defect in the mobile phone screen in the backlit state; The light and shadow states of the first image and the second image are different, that is, the first image is an image photographed in the case of highlighting the mobile phone screen, the second image is an image photographed in the case of non-highlighting the mobile phone screen, a first region in the first image is a region highlighted, and the first region is a partial region in the first image; The electronic device performs synchronous convolution on the first image and the second image through a double-channel convolution layer of a neural network model to obtain a first feature sequence of the first image and a second feature sequence of the second image, comprising: The electronic device extracts a first sub-image containing only the first region from the first image, and extracts a second sub-image containing only a second region from the second image, the position of the second region in the second image being the same as the position of the first region in the first image; The electronic device performs first convolution on the first sub-image through a first convolution channel of the neural network model, and performs second convolution on the second sub-image through a second convolution channel of the neural network model to obtain the first feature sequence and the second feature sequence respectively, the synchronous convolution means that the position of each convolution in the first convolution is the same as the position of each convolution in the second convolution, and the size of the convolution kernel used for convolution in the first convolution is the same as the size of the convolution kernel used for convolution in the second convolution; The electronic device performs synchronous fusion on the first feature sequence and the second feature sequence through a fusion layer of the neural network model to obtain a fused feature sequence, comprising: The electronic device fuses each first feature in the first feature sequence and each second feature in the corresponding second feature sequence through the fusion layer of the neural network model to obtain the fused feature sequence; The first feature sequence includes K1 first features, and the second feature sequence includes K2 second features, K1 and K2 are integers with the same value and greater than 10; each of the at least one first feature is an i-th first feature to a j-th first feature in the K1 first features, and each of the at least one second feature is an i-th second feature to a j-th second feature in the K2 second features; in the case that i and j have the same value, i is an integer traversing 1 to K1 / K2, and / represents the relationship of or; in the case that i and j have different values, i is an integer traversing 1 to K1-j / K2-j, and j is an integer traversing j to K1 / K2; the i-th first feature to the j-th first feature is a first feature obtained by performing i-th convolution to j-th convolution in the first convolution, and the i-th second feature to the j-th second feature is a second feature obtained by performing i-th convolution to j-th convolution in the second convolution; The electronic device fuses each of the at least one first feature in the first feature sequence and each of the at least one second feature in the corresponding second feature sequence through a fusion layer of the neural network model to obtain the fused feature sequence, including: The electronic device inputs the i-th first feature to the j-th first feature to a first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature to a second input end of the fusion layer to obtain i-th fused features to j-th fused features output by an output end of the fusion layer; if i and j have the same value, the fused feature sequence is obtained in the case that i traverses 1 to K1 / K2; if i and j have different values, the fused feature sequence is obtained in the case that i traverses 1 to K1-j / K2-j and j traverses j to K1 / K2; the fusion layer is part of a sub-neural network in the feature processing layer, and the neural network in the fusion layer is a star connection structure; In the case that i and j are the same, the i-th first feature to the j-th first feature is an i-th first feature, and the i-th second feature to the j-th second feature is an i-th second feature; the fusion layer includes first to sixth neurons, a total of 6 neurons, and the connection relationship of the 6 neurons is that the first, second and third neurons are connected with the fourth, fifth and sixth neurons one by one, and the fourth, fifth and sixth neurons are connected with each other; the electronic device inputs the i-th first feature to the j-th first feature to a first input end of the fusion layer, and inputs the i-th second feature to the j-th second feature to a second input end of the fusion layer to obtain i-th fused features to j-th fused features output by an output end of the fusion layer, including: The electronic device inputs the i-th first feature to the input end of the first neuron and inputs the i-th second feature to the input end of the second neuron, to obtain the i-th fused feature output by the output end of the third neuron; The input end of the first neuron is the first input end, and the input end of the first neuron is not connected to any neuron except the first neuron among the six neurons; the input end of the second neuron is the second input end, and the input end of the second neuron is not connected to any neuron except the second neuron among the six neurons; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected to any neuron except the third neuron among the six neurons.

2. The method of claim 1, wherein, j=i+1, the i-th first feature to the j-th first feature is the i-th first feature and the i+1-th first feature, and the i-th second feature to the j-th second feature is the i-th second feature and the i+1-th second feature; the fusion layer includes first to ninth neurons, a total of 9 neurons, and the connection relationship of the 9 neurons is: the first, second and third neurons are respectively connected to the fourth, fifth and sixth neurons, the fourth, fifth and sixth neurons are connected to each other, the fourth, fifth and sixth neurons are respectively connected to the seventh, eighth and ninth neurons, and the seventh, eighth and ninth neurons are connected to each other; the electronic device inputs the i-th first feature to the first input end of the fusion layer, and inputs the i-th second feature to the second input end of the fusion layer, to obtain the i-th fused feature to the j-th fused feature output by the output end of the fusion layer, including: The electronic device inputs the i-th first feature to the input end of the first neuron, inputs the i-th second feature to the input end of the second neuron, inputs the i+1-th first feature to the input end of the seventh neuron, and inputs the i+1-th second feature to the input end of the eighth neuron, to obtain the i-th fused feature output by the output end of the third neuron; The input end of the first neuron and the input end of the seventh neuron are the first input end, and the input end of the first neuron is not connected with any neuron in the nine neurons except the first neuron, and the input end of the seventh neuron is not connected with any neuron in the nine neurons except the seventh neuron; the input end of the second neuron and the input end of the eighth neuron are the second input end, and the input end of the second neuron is not connected with any neuron in the nine neurons except the second neuron, and the input end of the eighth neuron is not connected with any neuron in the nine neurons except the eighth neuron; the output end of the third neuron is the output end of the fusion layer, and the output end of the third neuron is not connected with any neuron in the nine neurons except the third neuron.

3. A mobile phone screen backlight foreign matter defect diagnosis system, characterized by, The system comprises an electronic device configured to perform the method of claim 1 or 2.

4. A computer-readable storage medium comprising: Computer program or instructions; when the computer program or instructions run on the computer, make the computer perform the method of claim 1 or 2.

Images