Broken screen detection and training method of convolutional neural network for broken screen detection

Abstract

The present invention discloses a broken screen detection and a training method for a convolutional neural network for broken screen detection. The training method for a convolutional neural network for broken screen detection comprises: obtaining multiple sample images containing the screen of an electronic device; and training a pre-established convolutional neural network based on the multiple sample images to optimize the convolution kernel in each convolution layer included in the convolutional neural network and the channel weights in each first sub-model; wherein the multiple convolution layers included in the convolutional neural network correspond to the multiple first sub-models respectively; wherein, for each sample image, each convolution layer is used to extract features of the screen portion in the sample image to generate a multi-channel feature map; the first sub-model corresponding to the convolution layer is used to multiply the multi-channel feature map by the corresponding channel weights respectively to generate a new multi-channel feature map, and the new multi-channel feature map is input into the next convolution layer for feature extraction.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a broken screen detection method, a device, a computer device, and a computer-readable storage medium for training a convolutional neural network for broken screen detection. Background Art

[0002] In recent years, an increasing number of insurance companies have introduced mobile phone screen insurance services. During the underwriting phase, customers submit photos of their insured phone's screen, and the insurance company performs screen crack detection on the back end. With the advancement of computer vision technology, machine learning algorithms based on neural networks are gradually replacing manual recognition for screen crack detection.

[0003] However, the neural networks commonly used today have many limitations. For example, because the cracks on a broken phone screen are often not very obvious, machine learning algorithms have poor detection effects under most shooting angles and environmental color overlays, and may even mistakenly identify a screen with actual cracks as a normal screen.

[0004] It should be noted that the above information disclosed in the background technology section is only used to enhance the understanding of the background of the present invention, and therefore it may include information that does not constitute the prior art known to ordinary technicians in this field. Summary of the Invention

[0005] In view of this, the present invention provides a broken screen detection and a training method, apparatus, computer equipment and computer-readable storage medium for a convolutional neural network for broken screen detection.

[0006] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0007] According to one aspect of the present invention, a method for training a convolutional neural network for broken screen detection is provided, comprising: acquiring a plurality of sample images including a screen of an electronic device; and training a pre-established convolutional neural network based on the plurality of sample images to optimize the convolution kernels in each convolutional layer included in the convolutional neural network and the channel weights in each first sub-model; wherein the plurality of convolutional layers included in the convolutional neural network correspond to a plurality of first sub-models respectively; wherein, for each sample image, each convolutional layer is used to perform feature extraction on the screen portion in the sample image to generate a multi-channel feature map; the first sub-model corresponding to the convolutional layer is used to multiply the multi-channel feature map by the corresponding channel weights respectively to generate a new multi-channel feature map, and input the new multi-channel feature map into the next convolutional layer for feature extraction.

[0008] According to one embodiment of the present invention, training a pre-established convolutional neural network based on the multiple sample images includes: globally pooling the multi-channel feature maps generated by each convolutional layer to generate multi-channel feature data; upsampling and / or downsampling the multi-channel feature data to obtain upsampled feature data and / or downsampled feature data; and training the first sub-model corresponding to the convolutional layer based on the upsampled feature data and / or the downsampled feature data.

[0009] According to one embodiment of the present invention, the convolutional neural network also includes: a second sub-model and a fully connected layer; the second sub-model is used to deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer to generate a feature vector; the method also includes: determining the weight matrix of the fully connected layer based on the feature vector.

[0010] According to one embodiment of the present invention, for each sample image, the second sub-model is used to perform deformation processing on the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer, and generate multiple first column vectors accordingly; determine the product matrix of the transposed matrix of the feature matrix corresponding to the multiple first column vectors and the feature matrix; and deform the product matrix to generate a second column vector.

[0011] According to one embodiment of the present invention, the second sub-model is further configured to perform the following operation on each element in the second column vector to generate a third column vector corresponding to the second column vector: determining the first square root of the absolute value of the element, and dividing the first square root by the absolute value of the second column vector. norm; wherein, the first square root has the same sign as the element; determining the weight matrix of the fully connected layer according to the eigenvector includes: determining the weight matrix of the fully connected layer according to the third column vector.

[0012] According to one embodiment of the present invention, the convolutional neural network also includes: a plurality of group normalization layers corresponding to the plurality of first sub-modules respectively; each group normalization layer is used to perform normalization processing on the multi-channel feature maps generated by the corresponding convolution layer based on at least one channel grouping before the corresponding first sub-module multiplies the multi-channel feature maps generated by the corresponding convolution layer by the corresponding channel weights.

[0013] According to one embodiment of the present invention, the convolutional neural network is established based on the pre-training results of the ResNet 50 network, and the pre-training samples of the ResNet 50 network include: multiple images in the ImageNet dataset.

[0014] According to another aspect of the present invention, a broken screen detection method is provided, comprising: obtaining an image to be detected containing a screen of an electronic device; performing the following operations on a screen portion in the image to be detected through a trained convolutional neural network; wherein the convolutional neural network comprises: multiple convolutional layers and multiple first sub-models corresponding to the multiple convolutional layers respectively: performing feature extraction on the screen portion through each convolutional layer to generate a multi-channel feature map; and multiplying the multi-channel feature map by corresponding channel weights through the first sub-model corresponding to the convolutional layer to generate a new multi-channel feature map, and inputting the new multi-channel feature map into the next convolutional layer for feature extraction; and judging whether the screen portion is broken based on the result of the operation.

[0015] According to one embodiment of the present invention, the convolutional neural network also includes: a second sub-model and a fully connected layer; judging whether the screen portion is broken according to the result of the operation includes: deforming the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer through the second sub-model to generate a feature vector; multiplying the feature vector with the weight matrix of the fully connected layer to generate an output vector of the fully connected layer; determining a first probability that the screen portion belongs to the first category and a second probability that the screen portion belongs to the second category according to the output vector; wherein the first category is "broken" and the second category is "unbroken"; and when the first probability is greater than the second probability, determining that the screen portion is broken.

[0016] According to one embodiment of the present invention, the convolutional neural network also includes: multiple group normalization layers corresponding to the multiple first sub-modules respectively; before the multi-channel feature maps are multiplied by the corresponding channel weights respectively through the first sub-model corresponding to the convolution layer, the operation also includes: through the group normalization layer corresponding to the first sub-model, based on at least one channel grouping, the multi-channel feature maps are normalized respectively.

[0017] According to another aspect of the present invention, a training device for a convolutional neural network for broken screen detection is provided, comprising: a sample acquisition module for acquiring a plurality of sample images including a screen of an electronic device; and a network training module for training a pre-established convolutional neural network based on the plurality of sample images to optimize the convolution kernels in each convolutional layer and the channel weights in each first sub-model included in the convolutional neural network; wherein the plurality of convolutional layers included in the convolutional neural network correspond to a plurality of first sub-models respectively; wherein, for each sample image, each convolutional layer is used to perform feature extraction on the screen portion in the sample image to generate a multi-channel feature map; the first sub-model corresponding to the convolutional layer is used to multiply the multi-channel feature map by the corresponding channel weights respectively to generate a new multi-channel feature map, and input the new multi-channel feature map into the next convolutional layer for feature extraction.

[0018] According to another aspect of the present invention, a broken screen detection device is provided, including: an image acquisition module for acquiring an image to be detected containing a screen of an electronic device; a feature extraction module for performing the following operations on the screen portion in the image to be detected through a trained convolutional neural network; wherein the convolutional neural network includes: multiple convolutional layers and multiple first sub-models corresponding to the multiple convolutional layers respectively: through each convolutional layer, feature extraction is performed on the screen portion to generate a multi-channel feature map; and through the first sub-model corresponding to the convolutional layer, the multi-channel feature map is multiplied by the corresponding channel weights to generate a new multi-channel feature map, and the new multi-channel feature map is input into the next convolutional layer for feature extraction; and a broken screen judgment module for judging whether the screen portion is broken according to the result of the operation.

[0019] According to another aspect of the present invention, a computer device is provided, comprising: a memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor implements any of the above methods when executing the executable instructions.

[0020] According to yet another aspect of the present invention, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, any one of the above methods is implemented.

[0021] According to the training method of the convolutional neural network for broken screen detection provided by the present invention, the trained convolutional neural network can effectively utilize the importance of each channel of the feature map to ensure that inconspicuous broken screen cracks can be accurately identified at any shooting angle and under any environmental color superposition.

[0022] The broken screen detection method provided by the present invention can accurately identify non-obvious cracks that are easily affected by the shooting angle and the superposition of environmental colors, thereby accurately determining whether the screen is broken.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0025] Figure 1 The figure is a flowchart of a method for training a convolutional neural network for broken screen detection according to an exemplary embodiment.

[0026] Figure 2 The figure is a flowchart of another method for training a convolutional neural network for broken screen detection according to an exemplary embodiment.

[0027] Figure 3 The figure is a flow chart showing a broken screen detection method according to an exemplary embodiment.

[0028] Figure 4 FIG. 4 is a flow chart showing another method for detecting a broken screen according to an exemplary embodiment.

[0029] Figure 5 1 is a block diagram of a convolutional neural network training device for screen breakage detection according to an exemplary embodiment.

[0030] Figure 6 is a block diagram showing a broken screen detection device according to an exemplary embodiment.

[0031] Figure 7 The figure is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures indicate identical or similar parts, and thus repeated descriptions thereof will be omitted.

[0033] In addition, the described features, structures or characteristics can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solutions of the present invention can be practiced while omitting one or more of the specific details, or other methods, devices, steps, etc. can be adopted. In other cases, well-known structures, methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present invention.

[0034] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0035] As mentioned above, the solution provided by the present invention uses computer vision technology to identify broken screens. To facilitate understanding, the concept of "computer vision" involved in the present invention is first explained below:

[0036] Computer vision (CV) is a science that studies how to make machines "see". More specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, track, and measure targets, and further performs image processing so that the computer processing becomes an image more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, simultaneous positioning and mapping, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition. In the solution of the present invention, it mainly involves the application of image recognition technology in the scene of electronic device broken screen detection.

[0037] To address the numerous limitations of currently used neural networks, the present invention proposes a novel convolutional neural network training method, which will be described in detail below using various embodiments of the present invention.

[0038] Figure 1 FIG is a flow chart showing a method for training a convolutional neural network for screen crack detection according to an exemplary embodiment. Figure 1 The training method of the convolutional neural network for broken screen detection shown can be applied to the server side of an insurance company, for example.

[0039] refer to Figure 1 , the training method 10 of the convolutional neural network for broken screen detection includes:

[0040] In step S102, a plurality of sample images including the screen of the electronic device are acquired.

[0041] Among them, multiple sample images can be divided into a first category of sample images and a second category of sample images. The electronic device screens in the first category of sample images have gaps or cracks, and the electronic device screens in the second category of sample images are intact.

[0042] In step S104, the pre-established convolutional neural network is trained based on the multiple sample images to optimize the convolution kernel in each convolution layer included in the convolutional neural network and the channel weight in each first sub-model.

[0043] The convolutional neural network includes multiple convolutional layers corresponding to multiple first sub-models. For each sample image, each convolutional layer is used to extract features from the screen portion of the sample image to generate a multi-channel feature map. The first sub-model corresponding to the convolutional layer is used to multiply the multi-channel feature map by the corresponding channel weights to generate a new multi-channel feature map, which is then input into the next convolutional layer for feature extraction.

[0044] According to the training method of the convolutional neural network for broken screen detection provided in an embodiment of the present invention, the trained convolutional neural network can effectively utilize the importance of each channel of the feature map to ensure that inconspicuous broken screen cracks can be accurately identified at any shooting angle and under any environmental color superposition.

[0045] It should be clearly understood that the present invention describes how to make and use specific examples, but the principles of the present invention are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present invention, these principles can be applied to many other embodiments.

[0046] In accordance with the above, Figure 2 FIG2 is a flowchart of another method for training a convolutional neural network for screen crack detection according to an exemplary embodiment. Figure 1 The difference of the method 10 shown is that Figure 2 The method shown further provides a specific method for optimizing the channel weights of each layer in the convolutional neural network, that is, further provides an embodiment of the above step S104. Figure 2 The training method of the convolutional neural network for screen crack detection shown can also be applied to the server side of an insurance company, for example.

[0047] refer to Figure 2 , step S104 may include:

[0048] In step S1042, global pooling is performed on the multi-channel feature maps generated by each convolutional layer to generate multi-channel feature data.

[0049] For example, global average pooling or global maximum pooling can be performed on the multi-channel feature maps respectively, and multi-channel feature data in the form of row vectors can be generated accordingly.

[0050] In step S1044, up-sample and / or down-sample the multi-channel feature data to obtain up-sampled feature data and / or down-sampled feature data.

[0051] In step S1046, the first sub-model corresponding to the convolutional layer is trained according to the up-sampled feature data and / or the down-sampled feature data.

[0052] Correspondingly, the channel weights in the first sub-model can also be expressed as row vectors, with the dimension of the row vector equal to the number of channels in the feature map, where each dimension represents the importance of the corresponding channel in the feature map. During network training, the channel weights are continuously optimized and updated, allowing the convolutional neural network to increasingly focus on channels with prominent screen crack features. Therefore, the convolutional neural network trained by this method can be used to accurately identify subtle cracks that are susceptible to the effects of camera angle and the superposition of ambient colors.

[0053] There are many ways for electronic device screens to shatter, such as tiny cracks that occur on the edges or corners of the screen when it falls to the ground. The neural networks commonly used currently have great difficulty identifying such cracks.

[0054] In some embodiments of the present invention, the convolutional neural network may further include a second sub-model and a fully connected layer. The second sub-model is configured to deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer to generate a feature vector. Accordingly, method 10 may further include determining a weight matrix for the fully connected layer based on the feature vector.

[0055] Specifically, for each sample image, the second sub-model is used to deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer, generate multiple first column vectors, determine the product matrix of the transpose matrix of the feature matrix corresponding to the multiple first column vectors and the feature matrix, and deform the product matrix to generate a second column vector (feature vector).

[0056] The above-mentioned role of the second sub-model enables the convolutional neural network to focus on the location of the crack and the characteristics of the crack. Therefore, the convolutional neural network trained by the present invention can also be used to accurately identify small cracks occurring at the edges or corners of the screen.

[0057] In some embodiments, the second sub-model is further configured to determine the first square root of the absolute value of each element in the second column vector (with the same sign as the element), and to divide the first square root by the absolute value of the second column vector. norm to generate a third column vector (eigenvector) corresponding to the second column vector.

[0058] Compared with training with the second column vector as the feature vector, training with the third column vector obtained by the above operation as the feature vector can more significantly improve the classification accuracy of the convolutional neural network.

[0059] Commonly used neural networks currently suffer from the problem that their classifier performance is significantly affected by the number of training samples. When training and updating convolutional neural networks online, a small number of input samples can lead to reduced classification accuracy or instability.

[0060] In this regard, in some embodiments of the present invention, the convolutional neural network may further include: a plurality of group normalization layers corresponding to the plurality of first submodules, respectively. Each group normalization layer is configured to perform normalization (standardization) on the multi-channel feature maps generated by the corresponding convolutional layer based on at least one channel grouping before the corresponding first submodule multiplies the multi-channel feature maps generated by the corresponding convolutional layer by the corresponding channel weights.

[0061] Due to the presence of the group normalization layer, the convolutional neural network's normalization of the feature maps of the input data is independent of the amount of input data. Therefore, when training and updating the convolutional neural network, the present invention uses any size of training sample without affecting the final classification accuracy of the convolutional neural network.

[0062] Furthermore, in some embodiments of the present invention, a convolutional neural network may be built based on the pre-training results of a ResNet 50 network. The pre-training samples may include, for example, multiple images from the ImageNet dataset. It should be noted that the present invention is not limited to the ResNet 50 network; it can be replaced by other commonly used backbone networks such as VGG and InceptionNet.

[0063] As mentioned above, the backbone network can be pre-trained on large classification datasets such as ImageNet, and then the network structure of the backbone network can be improved by adding the above-mentioned first submodule, second submodule and group normalization layer to establish the convolutional neural network used in the present invention.

[0064] The pre-established convolutional neural network is continuously optimized through repeated iterative training. Due to the presence of the first submodule, the second submodule, and the group normalization layer, the convolutional neural network of the present invention achieves performance improvements in multiple aspects as compared to currently used neural networks. The following, using the example of detecting a broken screen on an electronic device, will provide a detailed explanation through various embodiments of the present invention.

[0065] Figure 3 FIG. 1 is a flow chart showing a method for detecting a broken screen according to an exemplary embodiment. Figure 3 The broken screen detection method shown can also be applied to a server of an insurance company, for example.

[0066] refer to Figure 3 , the broken screen detection method 30 includes:

[0067] In step S302, an image to be detected including the screen of the electronic device is acquired.

[0068] In step S304, the trained convolutional neural network performs the following operations on the screen portion in the image to be detected:

[0069] In step S3042, features of the screen portion are extracted through each convolutional layer to generate a multi-channel feature map.

[0070] In step S3044, the multi-channel feature map is multiplied by the corresponding channel weights through the first sub-model corresponding to the convolutional layer to generate a new multi-channel feature map, and the new multi-channel feature map is input into the next convolutional layer for feature extraction (loop step S3042).

[0071] In step S306, based on the result of the above operation, it is determined whether the screen is partially broken.

[0072] The broken screen detection method provided in accordance with the embodiment of the present invention can accurately identify non-obvious cracks that are easily affected by the shooting angle and the superposition of environmental colors, thereby accurately determining whether the screen is broken.

[0073] It should be clearly understood that the present invention describes how to make and use specific examples, but the principles of the present invention are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present invention, these principles can be applied to many other embodiments.

[0074] In some embodiments, before step S3044, the above operation may further include: performing normalization processing on the multi-channel feature maps based on at least one channel grouping through a group normalization layer corresponding to the first sub-model.

[0075] In accordance with the above, Figure 4 FIG is a flow chart showing another method for detecting a broken screen according to an exemplary embodiment. Figure 3 The difference of the method 30 shown is that Figure 4 The method shown further provides a specific method for determining screen breakage, that is, further provides an embodiment of the above step S306. Figure 4 The broken screen detection method shown can also be applied to a server of an insurance company, for example.

[0076] refer to Figure 4 , step S306 may include:

[0077] In step S3062, the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer is deformed by the second sub-model in the trained convolutional neural network to generate a feature vector.

[0078] In step S3064, the feature vector is multiplied by the weight matrix of the fully connected layer to generate an output vector of the fully connected layer.

[0079] In step S3066, a first probability that the screen portion belongs to the first category and a second probability that the screen portion belongs to the second category are determined based on the output vector.

[0080] Among them, the first category is "broken" and the second category is "unbroken".

[0081] In step S3068, when the first probability is greater than the second probability, it is determined that the screen is partially broken.

[0082] The broken screen detection method provided in an embodiment of the present invention can accurately identify small cracks occurring at the edges or corners of the screen, thereby further improving the accuracy of broken screen detection.

[0083] Those skilled in the art will appreciate that all or part of the steps implementing the above embodiments can be implemented as a computer program executed by a CPU. When executed by the CPU, the computer program performs the functions defined in the method provided by the present invention. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0084] Furthermore, it should be noted that the aforementioned figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the aforementioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0085] The following are embodiments of the apparatus of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the apparatus embodiments of the present invention, please refer to the method embodiments of the present invention.

[0086] Figure 5 1 is a block diagram of a convolutional neural network training device for screen breakage detection according to an exemplary embodiment.

[0087] refer to Figure 5 The convolutional neural network training device 50 for screen breakage detection includes: a sample acquisition module 502 and a network training module 504.

[0088] The sample acquisition module 502 is used to acquire a plurality of sample images including the screen of the electronic device.

[0089] The network training module 504 is used to train the pre-established convolutional neural network based on multiple sample images to optimize the convolution kernel in each convolution layer included in the convolutional neural network and the channel weights in each first sub-model.

[0090] The convolutional neural network includes multiple convolutional layers corresponding to multiple first sub-models. For each sample image, each convolutional layer is used to extract features from the screen portion of the sample image to generate a multi-channel feature map. The first sub-model corresponding to the convolutional layer is used to multiply the multi-channel feature map by the corresponding channel weights to generate a new multi-channel feature map, which is then input into the next convolutional layer for feature extraction.

[0091] According to the training device for a convolutional neural network for broken screen detection provided in an embodiment of the present invention, the trained convolutional neural network can effectively utilize the importance of each channel of the feature map to ensure that inconspicuous broken screen cracks can be accurately identified at any shooting angle and under any environmental color superposition.

[0092] Figure 6 is a block diagram showing a broken screen detection device according to an exemplary embodiment.

[0093] refer to Figure 6 The broken screen detection device 60 includes: an image acquisition module 602 , a feature extraction module 604 and a broken screen judgment module 606 .

[0094] The image acquisition module 602 is used to acquire an image to be detected including the screen of the electronic device.

[0095] The feature extraction module 604 is used to perform the following operations on the screen portion in the image to be detected using the convolutional neural network trained by the apparatus 50:

[0096] Through each convolution layer, the features of the screen part are extracted to generate a multi-channel feature map;

[0097] Through the first sub-model corresponding to the convolutional layer, the multi-channel feature map is multiplied by the corresponding channel weight to generate a new multi-channel feature map, and the new multi-channel feature map is input into the next convolutional layer for feature extraction.

[0098] The broken screen determination module 606 is used to determine whether the screen is partially broken based on the results of the above operations.

[0099] The broken screen detection device provided in accordance with the embodiment of the present invention can accurately identify non-obvious cracks that are easily affected by the shooting angle and the superposition of environmental colors, thereby accurately determining whether the screen is broken.

[0100] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0101] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0102] Figure 7 is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. It should be noted that, Figure 7 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0103] like Figure 7 As shown, the computer device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 are also stored in the RAM 803. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0104] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read therefrom can be installed into the storage section 808 as needed.

[0105] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809 and / or installed from a removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the device of the present invention are performed.

[0106] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The units involved in the embodiments of the present invention may be implemented in software or in hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor includes a sending unit, an acquisition unit, a determination unit, and a first processing unit. The names of these units do not, in some cases, limit the units themselves. For example, the sending unit may also be described as a "unit that sends a request for obtaining an image to the connected server."

[0109] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently without being incorporated into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device includes:

[0110] Acquire multiple sample images including the screen of an electronic device; train a pre-established convolutional neural network based on the multiple sample images to optimize the convolution kernels in each convolution layer and the channel weights in each first sub-model included in the convolutional neural network; wherein the multiple convolution layers included in the convolutional neural network correspond to the multiple first sub-models respectively; for each sample image, each convolution layer is used to extract features of the screen portion in the sample image to generate a multi-channel feature map; the first sub-model corresponding to the convolution layer is used to multiply the multi-channel feature map by the corresponding channel weights to generate a new multi-channel feature map, and input the new multi-channel feature map into the next convolution layer for feature extraction; and

[0111] An image to be detected containing the screen of an electronic device is obtained; the following operations are performed on the screen portion in the image to be detected using a trained convolutional neural network: features of the screen portion are extracted through each convolutional layer to generate a multi-channel feature map; the multi-channel feature map is multiplied by the corresponding channel weights through the first sub-model corresponding to the convolutional layer to generate a new multi-channel feature map, and the new multi-channel feature map is input into the next convolutional layer for feature extraction; and based on the results of the above operations, whether the screen portion is broken is determined.

[0112] While the exemplary embodiments of the present invention have been specifically illustrated and described above, it should be understood that the present invention is not limited to the detailed structures, configurations, or implementations described herein; rather, the present invention is intended to encompass various modifications and equivalent arrangements within the spirit and scope of the appended claims.

Claims

1. A convolutional neural network training method for screen crack detection, characterized in that: include: Acquiring a plurality of sample images including a screen of an electronic device; as well as Training a pre-established convolutional neural network based on the multiple sample images to optimize the convolution kernel in each convolution layer and the channel weights in each first sub-model included in the convolutional neural network; wherein the multiple convolution layers included in the convolutional neural network correspond to the multiple first sub-models respectively; For each sample image, each convolutional layer is used to extract features from the screen portion of the sample image to generate a multi-channel feature map; the first sub-model corresponding to the convolutional layer is used to multiply the multi-channel feature map by the corresponding channel weights to generate a new multi-channel feature map, and the new multi-channel feature map is input into the next convolutional layer for feature extraction; In which, the convolutional neural network also includes: a second sub-model and a fully connected layer; the second sub-model is used to deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer to generate a feature vector; based on the feature vector, the weight matrix of the fully connected layer is determined; for each sample image, the second sub-model is used to deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer, respectively, to generate multiple first column vectors; determine the product matrix of the transposed matrix of the feature matrix corresponding to the multiple first column vectors and the feature matrix; and deform the product matrix to generate a second column vector.

2. The method according to claim 1, characterized in that Training a pre-established convolutional neural network according to the plurality of sample images includes: Perform global pooling on the multi-channel feature maps generated by each convolutional layer to generate multi-channel feature data; Upsampling and / or downsampling the multi-channel feature data to obtain upsampled feature data and / or downsampled feature data; and The first sub-model corresponding to the convolutional layer is trained according to the up-sampled feature data and / or the down-sampled feature data.

3. The method according to claim 1, characterized in that The second sub-model is further configured to perform the following operations on each element in the second column vector to generate a third column vector corresponding to the second column vector: determining the first square root of the absolute value of the element and dividing the first square root by the absolute value of the second column vector. norm; wherein the first square root has the same sign as the element; Determining the weight matrix of the fully connected layer according to the eigenvector includes: determining the weight matrix of the fully connected layer according to the third column vector.

4. The method according to claim 1, wherein The convolutional neural network also includes: multiple group normalization layers corresponding to the multiple first sub-models respectively; each group normalization layer is used to normalize the multi-channel feature maps generated by the corresponding convolution layer based on at least one channel grouping before the corresponding first sub-model multiplies the multi-channel feature maps by the corresponding channel weights.

5. The method according to any one of claims 1 to 4, characterized in that The convolutional neural network is established based on the pre-training results of the ResNet 50 network, and the pre-training samples of the ResNet 50 network include: multiple images in the ImageNet dataset.

6. A broken screen detection method, characterized in that: include: Acquire an image to be detected containing a screen of an electronic device; The following operations are performed on the screen portion of the image to be detected using a trained convolutional neural network, wherein the convolutional neural network includes: a plurality of convolutional layers and a plurality of first sub-models corresponding to the plurality of convolutional layers respectively: Through each convolution layer, extract features of the screen portion to generate a multi-channel feature map; and Multiplying the multi-channel feature maps by corresponding channel weights through the first sub-model corresponding to the convolutional layer to generate new multi-channel feature maps, and inputting the new multi-channel feature maps into the next convolutional layer for feature extraction; and According to the result of the operation, determining whether the screen portion is broken; In which, the convolutional neural network also includes: a second sub-model and a fully connected layer; according to the result of the operation, judging whether the screen part is broken includes: through the second sub-model, deforming the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer to generate a feature vector; multiplying the feature vector with the weight matrix of the fully connected layer to generate an output vector of the fully connected layer; according to the output vector, determining the first probability that the screen part belongs to the first category and the second probability that it belongs to the second category; wherein, the first category is "broken" and the second category is "unbroken"; and when the first probability is greater than the second probability, determining that the screen part is broken.

7. The method according to claim 6, characterized in that The convolutional neural network further includes: a plurality of group normalization layers corresponding to the plurality of first sub-models respectively; Before multiplying the multi-channel feature maps by the corresponding channel weights respectively through the first sub-model corresponding to the convolutional layer, the operation also includes: normalizing the multi-channel feature maps respectively based on at least one channel grouping through the group normalization layer corresponding to the first sub-model.

8. A convolutional neural network training device for screen crack detection, characterized in that: include: A sample acquisition module, configured to acquire a plurality of sample images including the screen of an electronic device; as well as a network training module, configured to train a pre-established convolutional neural network based on the plurality of sample images to optimize the convolution kernels in each convolutional layer and the channel weights in each first sub-model included in the convolutional neural network; wherein the plurality of convolutional layers included in the convolutional neural network respectively correspond to the plurality of first sub-models; For each sample image, each convolutional layer is used to extract features from the screen portion of the sample image to generate a multi-channel feature map; the first sub-model corresponding to the convolutional layer is used to multiply the multi-channel feature map by the corresponding channel weights to generate a new multi-channel feature map, and the new multi-channel feature map is input into the next convolutional layer for feature extraction; In which, the convolutional neural network also includes: a second sub-model and a fully connected layer; the second sub-model is used to deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer to generate a feature vector; based on the feature vector, the weight matrix of the fully connected layer is determined; for each sample image, the second sub-model is used to deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer, respectively, to generate multiple first column vectors; determine the product matrix of the transposed matrix of the feature matrix corresponding to the multiple first column vectors and the feature matrix; and deform the product matrix to generate a second column vector.

9. A broken screen detection device, characterized in that: include: An image acquisition module, used to acquire an image to be detected including the screen of an electronic device; A feature extraction module is configured to perform the following operations on the screen portion of the image to be detected using a trained convolutional neural network, wherein the convolutional neural network includes: a plurality of convolutional layers and a plurality of first sub-models corresponding to the plurality of convolutional layers: Through each convolution layer, extract features of the screen portion to generate a multi-channel feature map; and Multiplying the multi-channel feature maps by corresponding channel weights through the first sub-model corresponding to the convolutional layer to generate new multi-channel feature maps, and inputting the new multi-channel feature maps into the next convolutional layer for feature extraction; and a broken screen determination module, configured to determine whether the screen is partially broken based on the result of the operation; In which, the convolutional neural network also includes: a second sub-model and a fully connected layer; the screen judgment module is also used to, through the second sub-model, deform the multi-channel feature map generated by the first sub-model corresponding to the last convolutional layer to generate a feature vector; multiply the feature vector with the weight matrix of the fully connected layer to generate an output vector of the fully connected layer; according to the output vector, determine the first probability that the screen part belongs to the first category and the second probability that it belongs to the second category; wherein, the first category is "broken" and the second category is "unbroken"; and when the first probability is greater than the second probability, it is determined that the screen part is broken.

10. A computer device comprising: A memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the executable instructions.

11. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images