Image-based equipment fault identification method and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of communication, in particular to an image-based equipment fault identification method and electronic equipment. The method comprises the following steps: the electronic equipment inputs a first image into a first neural network model to obtain identification information of the first image; the identification information comprises the type of the first image, and the type of the first image indicates that the first image is a normal image or an abnormal image; under the condition that the first image is an abnormal image, the identification information further comprises a noise type of the first image, and the noise type is at least one noise type in M preset noise types; the M preset noise types of noise appear in the first image due to the failure of shooting equipment for shooting the first image; the first neural network model has the capability of outputting identification information of a corresponding image according to the input image; if the first image is an abnormal image, the electronic equipment sends the first image and abnormal information of the first image to a server; the abnormal information comprises a noise type of the first image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technology, and in particular to an image-based device fault identification method and electronic device. Background Art

[0002] Today's terminal devices generally have photo and video recording functions. Users can use terminal devices to take pictures or videos and record their lives anytime and anywhere.

[0003] When an electronic device fails, the electronic device will capture an abnormal image including noise. When the electronic device is affected by the surrounding environment, it will also capture an abnormal image including noise. When the user finds that the abnormal image including noise is captured, he will apply for after-sales repair. In the related art, the user needs to discover that the electronic device captures an abnormal image including noise and describe the abnormal phenomenon to the after-sales service.

[0004] However, the error rate of manual identification of abnormal images is high, and problems are easily missed. Moreover, even if all problems can be accurately identified manually, it may not be possible to clearly describe the abnormal phenomena in the abnormal images. In this way, the problems of the electronic equipment cannot be accurately located, and after-sales service cannot provide users with accurate and timely repair services, which will reduce the user's after-sales experience. Summary of the invention

[0005] The embodiments of the present application provide an image-based device fault identification method and electronic device, which can provide a server with abnormal images for accurately locating device problems, thereby improving the accuracy of device problem locating and further improving the user's after-sales experience.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a method for device fault identification based on an image is provided, which is applied to an electronic device. The method comprises:

[0008] The electronic device uses a first neural network model to recognize a first image and obtains recognition information of the first image; the recognition information includes the type of the first image, and the type of the first image indicates whether the first image is a normal image or an abnormal image; if the first image is an abnormal image, the recognition information also includes the noise type of the first image, and the noise type is at least one of M preset noise types; the noise of the M preset noise types appears in the first image due to a malfunction of a shooting device used to shoot the first image; wherein M is an integer greater than or equal to 1; the first neural network model has the ability to output recognition information of a corresponding image based on an input image; thereafter, if the first image is an abnormal image, the electronic device sends the first image and abnormal information of the first image to a server; the abnormal information includes the noise type of the first image.

[0009] In the present application, the first image may be a normal image or an abnormal image. That is, the first neural network can identify whether the first image is a normal image, and can also identify whether the first image is an abnormal image including at least one of the M types of noise caused by electronic failure. Afterwards, if the first image is an abnormal image including noise caused by electronic failure, the electronic device can upload the abnormal first image and the noise type of the first image to the server. The first image and the noise type of the first image are used to locate the fault type of the electronic device. In this way, when the after-sales staff receives a problem of an electronic device reported by the user, they can directly obtain the first image of the electronic device from the server, and provide remote maintenance assistance to the user based on the first image and the noise type of the first image. There is no need for human subjective identification of abnormal images, which can avoid user recognition errors. And the noise of the first image can be quantified to avoid unclear user descriptions. It can also avoid the problem that users and after-sales staff cannot find it in time because the noise is too hidden in the image, which improves the user's after-sales experience as a whole.

[0010] In a possible implementation of the first aspect, the abnormal information further includes log information when the photographing device photographs the first image. The log information is used to locate the cause of the failure of the photographing device.

[0011] In a possible implementation of the first aspect, the log information includes working status information of a radio frequency module in a shooting device and call stack information of a camera APP in an electronic device when the shooting device shoots a first image. The shooting device may be an electronic device. The shooting device may also be another electronic device, and the electronic device may obtain a first image shot by another electronic device, and identify whether the other electronic device is faulty based on the first image.

[0012] Before the electronic device sends the first image and abnormal information of the first image to the server, the electronic device may read the log information of the first image from the log information of the backup electronic device, or read the log information of the first image from the log information of other electronic devices.

[0013] Alternatively, the first image includes log information of the first image; the log information of the first image is acquired and stored in the first image when the shooting device shoots the first image. The first image may include image data and file data, wherein the file data may be shooting information, and the log information may be stored in the file data of the first image. The first image sent by another electronic device to the electronic device includes the log information of the first image.

[0014] In a possible implementation of the first aspect, the server may receive the first image and abnormal information of the first image sent by the electronic device, and send a maintenance suggestion to the electronic device based on the first image and the abnormal information of the first image. The electronic device may receive the maintenance suggestion and display the maintenance suggestion.

[0015] In a possible implementation of the first aspect, a gallery APP, a camera APP, or a fault reporting APP of an electronic device may provide a first interface for human-computer interaction. In response to a user trigger, the electronic device may execute this solution. Specifically, the mobile phone may display a first interface, which may include a first button. In response to a user's first operation on the first button on the first interface, the electronic device inputs the first image into a first neural network model to identify the type of the first image.

[0016] In a possible implementation of the first aspect, a gallery APP, a camera APP, or a fault reporting APP of the electronic device may display a second interface, and the second interface displays identification information of the first image. The identification information may be, for example, a normal image or an abnormal image. If the first image is an abnormal image, the identification information further includes at least one noise type among the preset noise types in M.

[0017] If the first image is an abnormal image, the second interface may include a second button, and in response to the user clicking the second button, the electronic device uploads the first image and identification information of the first image to the server.

[0018] In a possible implementation manner of the first aspect, if the first image is a normal image, the second interface may also include a second button, and in response to the user clicking the second button, the electronic device uploads the first image and identification information of the first image to the server.

[0019] In a possible implementation manner of the first aspect, the electronic device may display a third interface, where the third interface displays maintenance suggestions.

[0020] In a possible implementation manner of the first aspect, the electronic device may display the maintenance suggestion in the form of a notification message or a pop-up window.

[0021] In a possible implementation of the first aspect, before the electronic device inputs the first image into the first neural network model, it can pre-derivate the first image to obtain a first gradient image. The first gradient image includes key features of the first image and noise features corresponding to noise in the first image; wherein the key features include features of pixels in the first image whose adjacent pixel changes are greater than a preset change threshold. The first gradient image includes fewer features than the first image, and inputting the first gradient into the first neural network model to identify the type of the first image can improve the operation speed and improve the operation efficiency.

[0022] In a possible implementation of the first aspect, since the first gradient image only includes key images and / or noise features, areas in the first image where pixel values ​​change more smoothly are ignored. Therefore, the electronic device uploading the first gradient image to the server can achieve a desensitizing effect and protect the privacy of the user. Before the electronic device inputs the first image into the first neural network model, it can pre-derivate the first image to obtain a first gradient image. When uploading, upload the first gradient image. Alternatively, the electronic device inputs the first image into the first neural network model to identify the type of the first image. Before uploading, perform image differentiation on the first image to obtain a first gradient image and upload the first gradient image.

[0023] In a possible implementation of the first aspect, before the electronic device inputs the first image into the first neural network model, the electronic device obtains the latest model parameters of the first neural network model from the server; the electronic device updates the first neural network model using the latest model parameters. Since the server is constantly training the first neural network model, the accuracy and precision of the first neural network model are continuously improved. Before using the first neural network model, the electronic device can use the latest model parameters of the first neural network model to update the first neural network model, improve the precision of the first neural network model in the electronic device, and ensure better recognition results.

[0024] In a possible implementation of the first aspect, the first image is an abnormal image, and the first image is used by the server as a negative sample to train a first neural network model; the first image is an abnormal image, and the electronic device sends the first image and the type of the first image to the server; the first image is used by the server as a positive sample to train the first neural network model.

[0025] In a second aspect, an image-based device fault identification method is provided, which is applied to a server. The method includes: the server receives a first image and abnormal information of the first image sent by an electronic device to the server; the abnormal information includes a noise type of the first image; the noise type is at least one noise type among M preset noise types; at least one noise among the M preset noise types appears in the first image due to a fault in a shooting device that shoots the first image; wherein M is an integer greater than or equal to 1; the server sends a maintenance suggestion to the electronic device according to the first image and the abnormal information of the first image.

[0026] In the present application, the server can receive the first image and the abnormal information of the first image sent by the electronic device, and provide maintenance suggestions for the electronic device based on the first image and the abnormal information of the first image. It can realize remote maintenance of the electronic device and provide a convenient maintenance method for the user.

[0027] In a possible implementation manner of the second aspect, the abnormal information further includes log information when the photographing device photographs the first image, and the log information is used to locate a cause of the failure of the photographing device.

[0028] In a possible implementation manner of the second aspect, the log information includes working status information of a radio frequency module in the shooting device and call stack information of a camera APP in the electronic device when the shooting device shoots the first image.

[0029] In a possible implementation of the second aspect, the server uses the first image and the noise type of the first image to train the first neural network model, so that the trained first neural network model has the ability to output the noise type of the corresponding image according to the input image. The server can use the first image to train the first neural network model to improve the precision and accuracy of the first neural network model.

[0030] In a possible implementation of the second aspect, after training, the server updates the model parameters of the first neural network model. The server electronic device sends the updated latest model parameters. Before using the first neural network model, the electronic device can use the latest model parameters to update the first neural network model.

[0031] In a possible implementation of the second aspect, the server may receive a first gradient image of a first image sent by an electronic device, wherein the first gradient image is obtained by derivation of the first image; the first gradient image includes a key feature of the first image, or the first gradient image includes the key feature of the first image and a noise feature corresponding to noise in the first image; wherein the key feature includes a feature of a pixel in the first image whose adjacent pixel change is greater than a preset change threshold.

[0032] In a third aspect, an image-based equipment fault identification system is provided, the system comprising an electronic device and a server; the electronic device is used to execute any method as in the first aspect; the server is used to execute any method as in the second aspect.

[0033] In a fourth aspect, an electronic device is provided, comprising: a memory, a display screen, one or more processors, and a communication module; the memory, the communication module, the display screen and the processor are coupled; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the electronic device executes any one of the methods of the first aspect.

[0034] In a fifth aspect, the present application provides a chip system, which can be applied to an electronic device including a memory. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The interface circuit is used to receive a signal from the above-mentioned memory and send the signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device executes the method as in the first aspect and any possible design thereof.

[0035] In a sixth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and / or the second aspect and any possible design thereof.

[0036] In a seventh aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and / or the second aspect and any possible design thereof.

[0037] It can be understood that the beneficial effects that can be achieved by the electronic device of the third aspect to the computer program product of the seventh aspect mentioned above can be referred to the beneficial effects in the first aspect and / or the second aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of multiple types of first images provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0040] Figure 3 A system architecture diagram of an image-based device failure system provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a display interface of a gallery APP provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a display interface of another gallery APP provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of a display interface of a camera APP provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of a display interface of a fault analysis APP provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of a display interface of another fault analysis APP provided in an embodiment of the present application;

[0046] Fig. 9 A schematic diagram of a flow chart of an image-based device fault identification method provided in an embodiment of the present application;

[0047] Fig.10 A schematic diagram of a flow chart of another method for identifying device faults based on images provided in an embodiment of the present application;

[0048] Fig.11 A schematic diagram of a display interface of an electronic device provided in an embodiment of the present application;

[0049] Fig.12 A schematic diagram of a first gradient image provided in an embodiment of the present application;

[0050] Fig.13 A schematic diagram of a first image preprocessing process provided in an embodiment of the present application;

[0051] Fig.14 A schematic diagram of the interaction process between an electronic device and a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] Today's terminal devices generally have photo and video recording functions. Users can use terminal devices to take pictures or videos and record their lives anytime and anywhere.

[0053] A terminal device such as a mobile phone may include at least one camera. Exemplarily, in response to a user clicking a photo button in a camera application (application, APP) photo interface, the camera may capture an original image. The original image may be a RAW image captured by the camera. Afterwards, an image signal processor in the mobile phone may process the RAW image to obtain a processed first image. Finally, the mobile phone may display the first image in a preview interface of the camera APP. In this way, the user may preview the first image captured by the mobile phone.

[0054] When a mobile phone fails, the mobile phone will shoot abnormally. Specifically, the mobile phone will capture a first image including noise. Among them, mobile phone failures include hardware failures and software failures. Hardware failures can be, for example, device failures, such as camera hardware failures. Hardware failures can also be device interference, such as the image sensor of the camera being electromagnetically interfered by other devices inside the mobile phone. Exemplarily, the image sensor of the camera fails, the mobile phone receives a call while shooting, and the mobile phone downloads data in the background while shooting, all of which can cause the mobile phone to capture a first image including noise. Software failures can be, for example, missing software data of the camera APP, disordered code operation of the camera APP, and bugs in the software system operation. Among them, the first image including noise is an abnormal image, and the first image not including noise is a normal image.

[0055] Noise refers to unnecessary or redundant interference information in the first image. Noise may include stripe noise, point noise, and mosaic noise. Among them, the noise appears randomly at any position in one or more first images. The first image including stripe noise may include color stripes or black and white stripes, and the stripes may be straight lines or curves. The stripes may be displayed in an array in the entire area or a part of the area of ​​the first image, and the stripes may also be displayed disorderly in a part of the area of ​​the first image. The dots may be color dots or black and white dots. The first image including point noise may include color, black or white dots, and the dots may be displayed in an array in the entire area or a part of the area of ​​the first image, or may be displayed disorderly and chaotically in the entire area or a part of the area of ​​the first image. In the first image including mosaic noise, mosaics may be displayed in the entire area or a part of the area of ​​the first image. For example, Figure 1 shown. Figure 1 Shown are a normal first image, a first image including stripe noise, a first image including point noise, a first image including mosaic noise, and another first image including stripe noise.

[0056] In some cases, when the mobile phone is affected by the surrounding environment during shooting, the first image captured may also include stripes, dots or mosaics. Take stripes as an example. For example, when a user shoots a bright or strong scene in a dimly lit environment, highlight overflow may occur, resulting in stripes in the first image. Usually, stripes appear near the light source in the first image, and the stripes spread outward from the light source. This is a normal phenomenon. The first image is a normal image, and the mobile phone is not faulty.

[0057] That is to say, when the mobile phone fails, the first image including noise captured by the mobile phone is a real abnormal image, and the maintenance personnel need to further analyze and locate the cause of the mobile phone failure. The first image including noise captured by the mobile phone under the influence of the surrounding light source is not a real abnormal image. The first image is a normal image, and the mobile phone does not need to be repaired.

[0058] Usually, when a user finds that the first image captured by the mobile phone includes noise, the user cannot distinguish whether the noise is caused by a mobile phone failure or the influence of the surrounding light source. Users generally call the after-sales service hotline to apply for after-sales maintenance and describe the abnormal phenomenon to the after-sales staff. The user may not be able to objectively describe the abnormal phenomenon clearly, resulting in the after-sales staff being unable to obtain the real abnormal phenomenon and unable to provide accurate maintenance suggestions to the user. In addition, some noise is more obvious in the first image, and the user can intuitively identify the noise in the first image. Some noise is relatively hidden in the first image, and the user may not be able to identify the noise in the first image, which may lead to the omission of mobile phone failure problems, and the mobile phone cannot be repaired in time, leaving hidden dangers for users to use the mobile phone.

[0059] In summary, users cannot distinguish the shooting anomaly caused by the phone failure, nor can they objectively describe the noise in the first image, which makes it impossible for after-sales personnel to accurately provide repair suggestions to users. In addition, when the noise in the first image is relatively hidden, users cannot find the noise in time, which may lead to the omission of the phone failure problem, and the phone cannot be repaired in time, resulting in a poor after-sales experience for users.

[0060] To this end, an embodiment of the present application provides an image-based device fault identification method, which helps users to timely and accurately identify abnormal images including noise caused by electronic equipment failures, and upload abnormal images to a server. Specifically, the electronic device can input a first image into a first neural network model to obtain identification information of the first image. The identification information includes the type of the first image. In an embodiment of the present application, the types of images may include two types: normal images and abnormal images. If the first image includes at least one of the M types of noise caused by an electronic failure, the first image is an abnormal image; if the first image does not include any of the M types of noise caused by an electronic failure, the first image is a normal image. If the first image is an abnormal image, the identification information also includes the noise type of the first image. The noise type of the first image is the noise type of at least one of the M types of noise caused by an electronic failure included in the first image.

[0061] That is to say, the first image may be a normal image or an abnormal image. That is, the first neural network can identify whether the first image is a normal image, and can also identify whether the first image is an abnormal image including at least one of the M types of noise caused by electronic failure. Afterwards, if the first image is an abnormal image including noise caused by electronic failure, the electronic device can upload the abnormal first image and the noise type of the first image to the server. The first image and the noise type of the first image are used to locate the fault type of the electronic device.

[0062] In this way, when after-sales personnel receive feedback from users about a problem with an electronic device, they can directly obtain the first image of the electronic device from the server and provide remote repair assistance to the user based on the first image and the noise type of the first image. There is no need for human subjective identification of abnormal images, which can avoid user recognition errors. It can also quantify the noise of the first image to avoid unclear user descriptions. It can also avoid problems that users and after-sales personnel cannot discover in time because the noise is too hidden in the image, thereby improving the user's after-sales experience as a whole.

[0063] The method provided in the embodiments of the present application can be applied to electronic devices with data processing capabilities. The above-mentioned electronic devices may include servers, mobile phones, tablet computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smart watches, smart bracelets, etc.), vehicle-mounted devices, virtual reality devices, etc., and the embodiments of the present application do not impose any restrictions on this. In the embodiments of the present application, the above-mentioned electronic devices are electronic devices that can run operating systems and install applications. Optionally, the operating system running on the electronic device may be system, system, System, etc.

[0064] For example, please refer to Figure 2 , which shows a schematic diagram of the structure of an electronic device 200. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, an audio module 230, a speaker 230A, a microphone 230B, a display screen 240, a communication module 250, a power module 260, an input device 270, a sensor module 280, a camera 290, etc. Among them, the sensor module 280 may include a pressure sensor, a touch sensor, etc.

[0065] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0066] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 200 may also include one or more processors 210.

[0067] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0068] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.

[0069] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0070] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0071] The internal memory 221 may be used to store one or more computer programs, which may include instructions. The processor 210 may execute the instructions stored in the internal memory 221, thereby enabling the electronic device 200 to execute the application running method provided in some embodiments of the present application, as well as various applications and data management, etc.

[0072] The electronic device 200 can implement audio functions through the audio module 230, the speaker 230A, the microphone 230B, and the application processor, etc. For example, music playing, recording, etc. The audio module 230 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 230 can also be used to encode and decode audio signals. In some embodiments, the audio module 230 can be arranged in the processor 210, or some functional modules of the audio module 230 can be arranged in the processor 210.

[0073] The speaker 230A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0074] The microphone 230B, also called a "microphone" or "microphone", is used to convert a sound signal into an electrical signal. A user can make a sound by approaching the microphone 230B with his or her mouth, and input the sound signal into the microphone 230B.

[0075] The communication function of the electronic device 200 can be realized through the antenna 1, the antenna 2 and the communication module 250.

[0076] The communication module 250 can provide solutions for wireless communications including cellular, Wi-Fi, Bluetooth (BT), wireless data transmission modules (e.g., 433MHz, 868MHz, 915MHz) applied to the electronic device 200. The communication module 250 can be one or more devices integrating at least one communication processing module. The communication module 250 receives electromagnetic waves via antenna 1 or antenna 2, filters and frequency modulates the electromagnetic wave signals, and sends the processed signals to the processor 210. The communication module 250 can also receive the signal to be sent from the processor 210, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through antenna 1 or antenna 2.

[0077] The electronic device 200 implements the display function through a GPU, a display screen 240, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 240 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0078] The display screen 240 is used to display images, videos, etc. The display screen 240 includes a display panel. In some embodiments, the electronic device 200 may include 1 or N display screens 240, where N is a positive integer greater than 1. In the embodiment of the present application, the display screen 240 may be used to display a UI and receive user operations on the UI. In some embodiments, a pressure sensor, a touch sensor, etc. are provided on the display screen 240.

[0079] The power module 260 can be used to supply power to various components included in the electronic device 200. In some embodiments, the power module 260 can be a battery, such as a rechargeable battery.

[0080] The input device 270 may include a keyboard, a mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc. into the electronic device 200, thereby issuing commands to the electronic device 200, inputting data, etc. The mouse is an indicator for the vertical and horizontal coordinates of the display system of the electronic device 200, and is used to input instructions, etc. to the electronic device 200. Among them, the input device 270 can be connected to the electronic device 200 through a wired connection, for example, the input device 270 is connected to the electronic device 200 through a GPIO interface, a USB interface, etc. The input device 270 can also be connected to the electronic device 200 wirelessly, for example, the input device 270 is connected to the electronic device 200 through Bluetooth, infrared, etc.

[0081] The electronic device 200 can realize the shooting function through an ISP, a camera 290, a video codec, a GPU, a display screen 240, and an application processor. The ISP is used to process the data fed back by the camera 290. The camera 290 is used to capture static images or videos. In some embodiments, the electronic device 200 includes 1 or N cameras 290, where N is a positive integer greater than 1. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 290 is at a frequency point selection, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc. The video codec is used to compress or decompress digital video.

[0082] Take the above electronic device 200 as a mobile phone as an example. The software system of the electronic device 200 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 200.

[0083] Figure 3 2 is a system architecture diagram of an image-based device fault identification system according to an embodiment of the present invention. The system architecture diagram includes a software structure block diagram of an electronic device 200.

[0084] Exemplarily, the server includes an image management module, a model training module, and a model conversion and integration module.

[0085] The image management module is used to construct training samples for training the first neural network. The training samples include training samples pre-configured in the service and abnormal images uploaded by the electronic device 200. The abnormal image can be a first image or a first gradient image. The abnormal image can be used as a negative sample to train the first neural network model. Optionally, the image management module can also obtain a normal image from the electronic device 200, and the normal image can be used as a positive sample to train the first neural network model.

[0086] Optionally, the electronic device 200 may upload the first image to the cloud, and the server may obtain the first image from the cloud.

[0087] For example, the training samples pre-configured in the service can be images with noise such as stripes, dots, mosaics, etc., taken by the manufacturer of the electronic device 200 when the electronic device 200 is subject to radio frequency interference and bus interference anomalies. The image management module can also identify the noise type of the first image uploaded by the electronic device 200 side, and classify and store the first image according to the anomaly type.

[0088] The model training module is used to construct the first neural network model and train the first neural network model. Specifically, the model training module can construct the first neural network model, automatically iterate the first neural network model using training samples, and save model parameters. The model parameters may include the number of nodes and node weights. The model training module can periodically and automatically train the first neural network model, and gradually improve the accuracy of the first neural network model as the number of training samples increases.

[0089] The model conversion and integration module can convert the first neural network into a preset format, which can be a format that can be integrated and recognized by the electronic device 200. Exemplarily, the format of the first neural network model before conversion can be, for example, a .H5 file, and the format after conversion can be, for example, a .tflite file. Optionally, the model conversion and integration module can also push the latest updated parameters of the first neural network model through differential upgrade after the electronic device 200 leaves the factory, and the electronic device 200 can use the updated parameters to update the first neural network model.

[0090] The layered architecture of the electronic device 200 divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime), the system library, and the kernel layer.

[0091] The electronic device 200 may include a fault analysis and reporting APP, an image recognition module and a memory. The fault analysis and reporting APP is included in the application framework layer of the electronic device 200. The image recognition module is included in the application framework layer of the electronic device 200. The memory is the hardware of the electronic device 200.

[0092] Among them, the fault analysis and reporting APP can be an APP that includes abnormality identification, abnormality analysis and abnormality reporting functions, such as Figure 3 The fault analysis and reporting APP includes a UI interface. The first image recognition, recognition result indication, and abnormal image reporting can be realized in response to various operations of the user on the UI interface.

[0093] The image recognition module is used to identify the type of the first image. Specifically, the image recognition module can update the first neural network model, and the update process can include model parameter loading. The image recognition model can also preprocess the first image, and the preprocessing includes image derivation, and the preprocessing also includes one or more of a first format conversion, rotation, storage, scaling, and a second format conversion. The image recognition module can use the first gradient image of the first image as the input of the first neural network model, run the first neural network model, and obtain the recognition result of the first image.

[0094] like Figure 3 As shown, the application framework layer may also include an input system (not shown), a view system (not shown), a notification manager (not shown), a camera service, and a multimedia service.

[0095] In an embodiment of the present application, the input system is used to drive the touch screen of the monitoring electronic device 200 through the display screen, and convert the touch parameters generated by the touch operation input by the touch screen into usable events, and pass them to the upper-level fault analysis and reporting APP.

[0096] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to construct a display interface of an application. The display interface can be composed of one or more views. For example, in an embodiment of the present application, the visual system can be used to construct a first interface, which includes a first image and a first button. The visual system can also be used to construct a second interface, which includes a first image and identification information of the first image. The visual system can also be used to construct a third interface, in which maintenance suggestions are displayed.

[0097] The notification manager enables the fault analysis and reporting APP to display maintenance suggestions in the status bar, which can be used to communicate maintenance suggestions. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, or a notification that appears on the display in the form of a dialog window.

[0098] The camera service provides an interface that can be called by upper-level applications (such as the camera APP) to realize image capture and preview based on the application's call. The multimedia service provides an interface that can be called by upper-level applications (such as the gallery APP) to realize video or audio playback based on the application's call.

[0099] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system. The core library contains two parts: one is the function that the Java language needs to call, and the other is the Android core library.

[0100] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0101] The kernel layer is the layer between hardware and software. The kernel layer can include display driver, audio driver, sensor driver, camera driver, audio driver, touch screen driver, etc.

[0102] The hardware includes a camera, a display screen, a memory, and a transmission module. The memory is used to store pictures and videos taken by the user. The memory can also be used to store the first gradient image of the first image after preprocessing, and the identification information of the first image output by the first neural network model.

[0103] In conjunction with the accompanying drawings, taking the electronic device being a mobile phone as an example, an image-based device fault identification method provided by an embodiment of the present application is introduced. In the embodiment of the present application, the mobile phone can identify the type of a first image captured by the mobile phone. The first image can be a picture captured by the mobile phone or a video image captured by the mobile phone. The mobile phone can also identify the type of a first image captured by other electronic devices. The first image can be a picture captured by other electronic devices or a video image captured by other electronic devices. For example, a tablet computer can send the first image captured by the tablet computer to the mobile phone, and the mobile phone can identify the type of the first image, and then identify whether the tablet computer has a fault.

[0104] The mobile phone may include a gallery application. The gallery application may include display interfaces in various forms, and one of the display interfaces may include an abnormality detection button and a first image. In response to a user operating the abnormality detection button on the interface, the mobile phone may identify the type of the first image. The abnormality detection button is used to trigger the mobile phone to identify the type of the first image. The first image may be taken by the mobile phone, or by other electronic devices.

[0105] For example, the large image preview interface of the gallery application may include an abnormality detection button. Specifically, the mobile phone may display a main interface, which may include an application icon of the gallery application. In response to the user clicking the application icon of the gallery application, the mobile phone may display a Figure 4 The list display interface 401 shown. The list display interface 401 may include thumbnails of one or more pictures. For example, the list display interface 401 may include a thumbnail of picture 1. In response to a user clicking on the thumbnail of picture 1, the mobile phone may jump to a large picture preview interface 402. The large picture preview interface 402 may display picture 1 (first image) in large picture mode. The large picture preview interface 402 may include an abnormality detection button 403 (first button). That is, the large picture preview interface 402 includes a first image and a first button.

[0106] For another example, the list selection interface of the gallery application may include an abnormality detection button. Figure 5 As shown, in response to the user long pressing the thumbnail of picture 1 on the list display interface 401, the mobile phone can display the list selection interface 501. The list selection interface 501 includes thumbnails of multiple pictures, among which the thumbnail of picture 1 has been selected (that is, picture 1 has been selected). The list selection interface 501 can also include an abnormality detection button 502. Since picture 1 has been selected, the mobile phone receives the user clicking the abnormality detection button 502, and the mobile phone can identify the type of picture 1. That is, the list selection interface 501 can include a first image and a first button.

[0107] It should be understood that in response to a user long pressing the thumbnail of any picture on the list display interface 401, the mobile phone can jump from the list display interface 401 to the list selection interface 501. For example, in response to a user long pressing the thumbnail of picture 2 on the list display interface 401, the mobile phone can display the list selection interface 501. At this time, the thumbnail of picture 2 is selected on the list selection interface 501. That is, this first image is picture 2, and in response to the user clicking the abnormal detection button, the mobile phone can identify the type of picture 2. If the mobile phone responds to the user clicking the thumbnail of picture 1 on the list selection interface 501, the mobile phone can reselect picture 1. In this scenario, the user can only select one picture as the first image to be identified at a time. Further, in the list selection interface, the user can select multiple pictures as the first image to be identified at a time. For example, in response to the user clicking the thumbnail of picture 1 and clicking the thumbnail of picture 1 on the list selection interface 501, the mobile phone identifies the types of picture 1 and picture 2.

[0108] That is, the mobile phone can display the first interface of the gallery APP including the first image and the first button. The first button is, for example, an abnormality detection button, which is used to trigger the mobile phone to identify whether the first image is an abnormal image. In this way, when the user browses images in the gallery APP, he can select one or more first images and trigger the mobile phone to identify the type of the one or more first images.

[0109] Furthermore, after taking a photo, the user can also trigger the mobile phone to identify the type of the photo taken. Figure 6 As shown, the mobile phone can display a main interface, which includes a camera application icon. In response to the user clicking the camera application icon, the mobile phone displays a photo preview interface 601. In response to the user clicking the capture button 602, the mobile phone can capture an image to obtain picture 1. The mobile phone can save picture 1 in the gallery application and display a thumbnail of picture 1 in the photo preview interface 601. In response to the user clicking the thumbnail of picture 1, the mobile phone can display a large picture preview interface 402 of picture 1, i.e., the first interface.

[0110] Furthermore, the mobile phone may include a fault analysis APP, which may be a built-in APP of the mobile phone or an APP downloaded and installed in the mobile phone by the user. Figure 7As shown, the main interface of the mobile phone can display the application icon of the fault analysis APP. In response to the user clicking the application icon of the fault analysis APP on the main interface of the mobile phone, the fault analysis APP displays interface 701. Interface 701 may include a selection button 702. In response to the user clicking the selection button 702 on interface 701, the mobile phone displays a list display interface 703 of the fault analysis APP. In response to the user selecting picture 1 on the list display interface 703 of the fault analysis APP, the mobile phone displays a large picture preview interface 704 of the fault analysis APP. The large picture preview interface 704 includes a first image, i.e., picture 1, and an abnormality detection button 705. Or, as Figure 7 As shown, in response to the user selecting picture 1 in the list display interface 703 of the fault analysis APP, the mobile phone displays the list selection interface 706 of the fault analysis APP. In the list selection interface 706, image 1 is selected, and the list selection interface includes an abnormality detection button 705. Similarly, in response to the user selecting multiple pictures in the list display interface 703 of the fault analysis APP, the mobile phone displays the list selection interface 706 of the fault analysis APP, and the multiple pictures are selected in the interface 706.

[0111] Combined with the previous article Figure 4-Figure 7 A method of triggering a mobile phone to display a first interface including a first image and a first button is introduced. Figure 4-Figure 7 In the method shown, the mobile phone can identify the type of one or more first images selected by the user, and the user is required to select the first image to be identified.

[0112] Optionally, the mobile phone does not require the user to select the first image to be identified, and the mobile phone can automatically identify the type of the stored first image. Figure 8 As shown, the main interface of the mobile phone can display the application icon of the fault analysis APP. In response to the user clicking the application icon of the fault analysis APP on the main interface of the mobile phone, the fault analysis APP displays interface 801. Interface 801 includes a one-key identification button 802, which is used to identify the type of images stored in the mobile phone. In response to the user clicking the one-key identification button 802, the mobile phone automatically obtains the first image stored in the mobile phone and identifies the type of the first image one by one.

[0113] Optionally, in response to the user clicking the one-key identification button 802, the mobile phone may display an interface 803, where the interface 803 is used for the user to select the first image stored in the mobile phone by the mobile phone within the target time period. Exemplarily, in response to the user selecting the start date as 2021 / 10 / 1 and the end date as 2021 / 12 / 1 in the interface 803, and clicking the start button, the mobile phone obtains multiple first images stored in the mobile phone within the time period from 2021 / 10 / 1 to 2021 / 12 / 1, and identifies the type of the first images one by one.

[0114] In other embodiments, the mobile phone automatically identifies the type of the first image stored in the mobile phone without user triggering. Exemplarily, the mobile phone automatically identifies the type of the first image taken by the mobile phone at a preset time point, such as 23:00 in the evening. The recognition process can be run in the background of the mobile phone. Alternatively, the mobile phone automatically identifies the type of the first image in the mobile phone in response to the recognition instruction issued by the server. In other embodiments, the mobile phone can automatically identify the type of the first image stored in the mobile phone in response to the first image being captured without user triggering. The first image can be a photo or a frame image in a video, and the embodiment of the present application is not limited to this. In this embodiment, the mobile phone automatically identifies the first image in the mobile phone with the consent of the user. For example, before the mobile phone automatically identifies the first image for the first time, it can notify and remind the user to read the relevant user agreement (notification) and sign the agreement (authorization) including the authorization-related user information, or the user manually starts the automatic recognition function before the mobile phone can automatically recognize the first image.

[0115] The previous article introduces various ways to trigger the mobile phone to recognize the type of the first image. Fig. 9 The present invention provides an image-based device fault identification method. For each first image, the mobile phone can execute Fig. 9 The method shown identifies the type of a first image.

[0116] S901, the mobile phone inputs the first image into the first neural network model to obtain recognition information of the first image.

[0117] The identification information of the first image includes the type of the first image, and the type of the first image indicates whether the first image is a normal image or an abnormal image. Exemplarily, the type of the first image is 1, and the type 1 indicates that the first image is a normal image. The type of the first image is 2, indicating that the first image is an abnormal image.

[0118] If the first image is an abnormal image, the identification information also includes the noise type of the first image. The noise type of the first image may be one or more noise types among M preset noises caused by a failure of the shooting device included in the first image. Wherein, M is an integer greater than or equal to 1. Exemplarily, the M preset noises may be stripe noise, point noise, and mosaic noise. For example, the noise type of the first image may be at least one of stripe noise, point noise, and mosaic noise. That is, the noise type of the first image may be one or more noise types among M preset noises, and the M preset noises appear in the first image due to a failure of the shooting device. In other words, if the first image is an abnormal image, the first image includes one or more noises among M preset noises caused by a failure of the shooting device. The following takes the shooting device being a mobile phone as an example.

[0119] Exemplarily, the identification information may include indication information, which is used to indicate the noise type of the first image. For example, the first neural network model outputs 2a, indicating that the first image is an abnormal image and the noise type is stripe noise. The first neural network model outputs 2b, indicating that the first image is an abnormal image and the noise type is point noise. The first neural network model outputs 2c, indicating that the first image is an abnormal image and the noise type is mosaic noise.

[0120] The mobile phone can use the first neural network model to identify the type of the first image. The first neural network model can be integrated in the mobile phone, or the first neural network model can be deployed on the server. If the first neural network is deployed on the server, the mobile phone can send the first image to the server. The server uses the first neural network model to identify the type of the first image. The server returns the type of the first image to the mobile phone. The following describes this solution by taking the first neural network model deployed in the mobile phone as an example.

[0121] The first neural network has the ability to output identification information of the first image based on the input first image. Exemplarily, the first neural network model can be trained using a normal image that does not include any noise, a normal image that includes one or more noises including stripes, dots, and mosaics caused by the influence of ambient light sources, and an abnormal image that includes one or more noises including stripes, dots, and mosaics caused by a mobile phone failure. For the first image that does not include any noise, the trained first neural network model has the ability to output that the type of the first image is a normal image. For the first image that includes one or more noises including stripes, dots, and mosaics caused by the influence of ambient light sources, the trained first neural network model has the ability to output that the type of the first image is a normal image. For the first image that includes one or more noises including stripes, dots, and mosaics caused by a mobile phone failure, the trained first neural network model has the ability to output that the first image is an abnormal image and the noise type of the first image.

[0122] As a possible implementation, the first neural network model includes multiple sub-neural network sub-models, and a preset noise type includes a corresponding sub-neural network model. For example, neural network model one is used to identify whether the first image includes stripe noise caused by a mobile phone failure; neural network model two is used to identify whether the first image includes mosaic noise caused by a mobile phone failure; neural network model three is used to identify whether the first image includes point noise caused by a mobile phone failure. The mobile phone inputs the first image into the three neural network models in sequence to obtain identification information of the first image. Exemplarily, the first image is input into neural network model one, and neural network model one outputs a probability value one that the first image includes stripe noise caused by a mobile phone failure. If the probability value one is greater than or equal to the preset threshold one, the first image is considered to be an abnormal image and the noise type of the first image is stripe noise. The first image is input into neural network model two, and a probability value two that the first image includes mosaic noise caused by a mobile phone failure is obtained. If the probability value two is greater than or equal to the preset threshold two, the first image is considered to be an abnormal image and the noise type of the first image is mosaic noise. The first image is input into neural network model three, and a probability value three that the first image includes dot noise caused by a mobile phone failure is obtained. If the probability value three is greater than or equal to the preset threshold value three, the first image is considered to be an abnormal image and the noise type of the first image is point noise. Among them, the preset threshold value one, the preset threshold value two, and the preset threshold value three can be equal, for example, all can be 0.5. Among them, if the probability value one of the first image is less than the preset threshold value one, and the probability value two is less than the preset threshold value two, and the probability value three is less than the preset threshold value three, then the type of the first image is a normal image, that is, the first image is a noise-free image.

[0123] As another possible implementation, the first neural network model is a neural network model. The first neural network model can output the probability value of each noise in the M preset noises caused by the mobile phone failure, and the first neural network uses the preset noise type corresponding to the probability value that meets the preset conditions in the M probability values ​​as the noise type of the first image. Exemplarily, the first neural network model can output the probability value of the first image including stripe noise, mosaic noise, and point noise caused by the mobile phone failure. For example, the probability values ​​are 0.05, 0.6, and 0.35 respectively. The preset noise type corresponding to the maximum probability value can be the noise type of the first image, that is, the mosaic noise corresponding to the probability value of 0.6 is the noise type of the first image. Optionally, the noise type corresponding to the probability value exceeding the preset threshold four is the noise type of the first image. For example, the preset threshold four can be 0.3, then the mosaic noise corresponding to the probability value of 0.6 and the point noise corresponding to the probability value of 0.35 are the noise types of the first image. If the probability values ​​of the first image belonging to stripe noise, mosaic noise, and dot noise are all less than the preset threshold five, then the first image is a normal image. For example, the probability values ​​of the first image belonging to stripe noise, mosaic noise, and dot noise are 0.35, 0.35, and 0.3 respectively, and the fifth threshold may be 0.4, so the type of the first image may be a normal image.

[0124] S902: If the first image is an abnormal image, the mobile phone reports the first image and abnormal information of the first image to the server.

[0125] If the first image is an abnormal image including at least one of the M preset noises caused by a mobile phone failure, the mobile phone reports the first image and abnormal information of the first image to the server. The abnormal information of the first image may include the noise type of the first image.

[0126] Optionally, the mobile phone may also report to the server the log information when the camera captures the first image, that is, the abnormal information may also include the log information when the camera captures the first image. The log information is used to locate the cause of the failure of the camera.

[0127] As an example, the log information includes the operation log information of the radio frequency module in the shooting device when the shooting device shoots the first image, and the operation log information may include the working status information of the radio frequency module. Among them, the working status of the radio frequency module includes working and not working. If the radio frequency module is working, the working status information of the radio frequency module also includes information such as the working frequency band and the transmission power. Exemplarily, the log information may be the call stack information of the application running in the shooting device when the shooting device shoots the first image. The call stack information can reflect the call status of the application to the radio frequency module, and can also reflect the working status of the radio frequency module. The log information may also include the call stack information of the camera APP in the mobile phone when the shooting device shoots the first image. The call stack information may reflect the code running status when the camera APP shoots the first image. Optionally, the log information may also include the operation log information of the camera module that shoots the first image. Optionally, the log information may also include the operating parameters of the camera module that shoots the first image. Among them, the operating parameters may include, for example, exposure value, exposure time, aperture value, focal length, etc.

[0128] Optionally, after the mobile phone identifies the first image as an abnormal image and before uploading the image, the mobile phone may read the log information when the shooting device shot the first image from the log information of the shooting device.

[0129] Optionally, when the shooting device shoots the first image, the log information of the shooting moment can be obtained. The log information is stored in the first image. Exemplarily, the first image may include image data and file data. Among them, the file data may include shooting information, that is, the log information can be stored in the file data of the first image.

[0130] The above-mentioned shooting device may be a mobile phone or other electronic devices.

[0131] Optionally, after receiving the first image and abnormal information of the first image sent by the mobile phone, the server may send maintenance suggestions to the mobile phone.

[0132] Furthermore, the server receiving the first image and the abnormal information of the first image and the server sending the maintenance suggestion to the mobile phone may be the same or different. For example, the mobile phone may send the first image and the abnormal information of the first image to the cloud server, and the cloud server may send the maintenance suggestion to the mobile phone. The mobile phone may send the first image and the abnormal information of the first image to the cloud server, and the server 1 may obtain the first image and the abnormal information of the first image from the cloud server and send the maintenance suggestion to the mobile phone.

[0133] S903: The server sends a maintenance suggestion to the mobile phone.

[0134] The server can send maintenance suggestions to the mobile phone. For example, after-sales personnel can obtain abnormal images and abnormal information sent by the mobile phone of the repair user from the server. After-sales personnel can remotely analyze the cause of the user's mobile phone failure and trigger the server to send maintenance suggestions to the mobile phone.

[0135] Alternatively, the server may automatically identify the cause of the user's mobile phone failure and automatically send a repair suggestion to the mobile phone. For example, the server includes a mapping relationship between noise types and repair suggestions, and sends a repair suggestion to the mobile phone based on the noise type of the first image and the mapping relationship. Alternatively, the server may obtain a repair suggestion from recent repair records of other mobile phones based on the first image and the noise type of the first image.

[0136] Alternatively, the server can identify whether the mobile phone has a hardware fault or a software fault based on the first image sent by the mobile phone and the abnormal information of the first image. The server identifies that when the mobile phone takes the first image, the working state of the RF module is working, and the server identifies that the mobile phone has a hardware fault. For example, based on the log information when the mobile phone takes the first image, the server identifies that the call APP in the mobile phone called the RF module when the mobile phone took the first image, and the server identifies that the reason for the abnormality of the first image is that the image sensor in the camera is subject to radio frequency interference. The server can send a maintenance suggestion to the mobile phone, and the maintenance suggestion can be "Hardware abnormality, please go to the maintenance outlet for maintenance in time." The server identifies the abnormal operation of the software in the mobile phone based on the log information, and identifies that the mobile phone has a software fault. For example, based on the log information when the mobile phone takes the first image, the server identifies that the code of the camera APP of the mobile phone taking the first image is running wrong. The server can send a maintenance suggestion to the mobile phone, and the maintenance suggestion can be "Camera software abnormality, please repair the camera software."

[0137] Afterwards, the mobile phone can receive the maintenance suggestions sent by the server and display the maintenance suggestions.

[0138] If the user actively triggers the mobile phone to identify the type of the first image, for example Figure 4-Figure 8 As shown, after receiving the first image and the identification information of the first image, the server can send a maintenance suggestion to the mobile phone, and the mobile phone can display the maintenance suggestion. Fig.11 As shown, the mobile phone may display a third interface, which includes maintenance suggestions. If the mobile phone automatically identifies the type of the first image, the server may receive the first image and the identification information of the first image, and then the server may send maintenance suggestions to the mobile phone, and the mobile phone may display the maintenance suggestions in a notification message or a pop-up window. The mobile phone may execute S904.

[0139] S904, the phone displays a repair suggestion.

[0140] The mobile phone may display the maintenance suggestion in the form of a notification message or a pop-up window, or the mobile phone may display the maintenance suggestion on the interface.

[0141] Taking the example of a mobile phone displaying repair suggestions in the form of a pop-up window, the pop-up window may include a repair button. In response to the user clicking the repair button, the mobile phone can obtain update data for the camera software from the server and use the update data to repair the camera APP.

[0142] Optionally, the abnormality of the first image is caused by abnormality of the mobile phone software. The server can send software update data to the mobile phone while sending the repair suggestion to the mobile phone, and the mobile phone can receive the update data and store it in the mobile phone. In response to the user clicking the repair button in the pop-up window, the mobile phone directly uses the update data to repair the camera APP.

[0143] The embodiment of the present application provides an image-based device fault identification method, in which the electronic device can input a first image into a first neural network model to obtain identification information of the first image, thereby improving the recognition accuracy. Afterwards, if the first image is an abnormal image including noise caused by an electronic fault, the electronic device can upload the abnormal first image and the noise type of the first image to the server. The first image and the noise type of the first image are used to locate the fault type of the electronic device, thereby improving the user's after-sales experience.

[0144] In some embodiments, the mobile phone can automatically execute S901. In other embodiments, the mobile phone can also execute S901 in response to a user trigger. For example, if the user triggers Fig.10 As shown, before the mobile phone executes S901, the mobile phone may first execute S101.

[0145] S101, the mobile phone displays the first interface.

[0146] The first interface can be Figure 4 or Figure 6 The large image preview interface 402 shown, or the first interface is Figure 5 The list selection interface 501 of the gallery APP shown. The first interface can also be Figure 7 The large image preview interface 704 of the fault analysis APP shown, or Figure 7 The list selection interface 706 of the fault analysis APP shown in FIG. Figure 8 Interface 803 of the fault analysis APP is shown.

[0147] The first interface may include a first button, and the first button is used to trigger the mobile phone to identify the type of the first image.

[0148] Optionally, S901 may be S102.

[0149] S102, in response to a first operation of a user on a first button on a first interface, the mobile phone inputs a first image into a first neural network model to obtain recognition information of the first image.

[0150] The first operation may be, for example, a click operation. In response to the user clicking the first button on the first interface, the mobile phone uses the first neural network model to identify the type of the first image. The first image may be selected by the user and displayed on the first interface. The first image may also not be displayed on the first interface. For example, in response to the user clicking the first button on the first interface, the mobile phone uses the first neural network model to identify the type of the first image. Figure 8 In the interface 801 shown in FIG. 1 , the one-key recognition button is clicked, and the mobile phone obtains multiple first images taken by the user and stored in the mobile phone. For each of the first images, the mobile phone executes the following steps: Fig. 9 The method shown.

[0151] Optionally, after S102, the mobile phone may display the recognition result. Exemplarily, the mobile phone may execute S103.

[0152] S103, the mobile phone displays the second interface.

[0153] The second interface may include identification information of the first image, and the identification information may be the type of the first image. If the first image is an abnormal image, the identification information may also include the noise type of the first image.

[0154] Optionally, the second interface may further include the first image, that is, the second interface may display the identification information of the first image and / or the first image. Fig.11 As shown. Fig.11 In the example, interface 101 shows that the first image and the identification information of the first image are normal images. Interface 102 shows that the first image and the identification information of the first image are stripe abnormalities. Further, if the mobile phone recognizes multiple first images at a time, the mobile phone can display the identification result of one first image on the second interface. In response to the user's sliding operation on the second interface, the mobile phone can switch the identification result of the first image.

[0155] Optional, such as Fig.11 As shown, the second interface may further include an upload button, which is used to trigger the mobile phone to upload the first image to the server. In other words, the mobile phone may report the first image and the identification information of the first image to the server in response to the user's upload operation.

[0156] Optionally, if the first image is a normal image, the second interface may not include an upload button; if the first image is an abnormal image, the second interface may include an upload button. Alternatively, regardless of the type of the first image, the second interface includes an upload button. That is, if the first image is a normal image, the mobile phone may upload the first image to the server, or the mobile phone may not upload the first image to the server. If the first image is an abnormal image, the mobile phone may upload the first image to the server.

[0157] If the first image is a normal image, the mobile phone uploads the first image and the type of the first image to the server. If the first image is an abnormal image, the mobile phone uploads the first image and abnormal information of the first image to the server.

[0158] Taking the example of a mobile phone uploading an abnormal image to a server, the method further includes S104. Fig.10 As shown, Fig. 9 S902 (not in Fig.10 shown) may be S104.

[0159] S104: If the first image is an abnormal image, in response to a second operation of the user on the second interface, the mobile phone reports the first image and abnormal information of the first image to the server.

[0160] The second operation may be that the user clicks an upload button on the second interface.

[0161] Exemplary, in response to using Fig.11 After clicking the upload button in the interface 102 shown in the figure, the mobile phone displays the third interface. The third interface can be as follows Fig.11 The third interface includes maintenance suggestions. Fig. 9 As shown, S904 may be S105.

[0162] S105, the mobile phone displays a third interface, which includes maintenance suggestions.

[0163] In some embodiments, before the mobile phone inputs the first image into the first neural network model, it can desensitize the first image. The desensitized first image includes key features of the image and / or noise features corresponding to the noise in the first image. Therefore, inputting the desensitized first image into the first neural network model to identify the type of the first image can improve the computing efficiency.

[0164] In other embodiments, before uploading the first image to the server, the mobile phone may desensitize the first image. If the first neural network is deployed in the mobile phone, after executing S901 to identify the first image and before executing S902, the mobile phone desensitizes the first image, and the mobile phone uploads the desensitized first image to the server. Alternatively, the mobile phone desensitizes the first image before executing S901 to identify the first image, and the mobile phone uses the desensitized first image as the input of the model of the first neural network, identifies the type image of the first image, and uploads the desensitized first image to the server.

[0165] In other embodiments, the mobile phone desensitizes the first image before S901 recognizes the first image, and the mobile phone sends the desensitized first image to the server, and the server uses the desensitized first image as input to the model of the first neural network to identify the type of the first image. The following describes this solution by taking the example of a mobile phone using the first neural network model deployed in the mobile phone to identify the type of the first image.

[0166] The mobile phone performs desensitization processing on the sensitive content in the first image so that it no longer contains sensitive content. The sensitive content may include faces, sensitive text, sensitive patterns, etc. The sensitive text may include one or more personal information such as user name, user ID information, user home address, and user contact information. The desensitization processing includes blurring, mosaic processing, and replacement processing of the sensitive content in the image.

[0167] Exemplarily, after the mobile phone executes S901 to identify the first image, the mobile phone may first identify the sensitive content in the first image. For example, the mobile phone may identify whether the first image includes a face, preset sensitive text, preset sensitive pattern, etc. Afterwards, the mobile phone performs one or more of blurring, mosaicing, and replacement processing on the sensitive content in the first image. The mobile phone may identify the sensitive area in the first image based on the second neural network model. The sensitive area is an area including sensitive content. The second neural network model has the ability to output the sensitive area in the first image. Afterwards, the mobile phone performs one or more of blurring, mosaicing, and replacement processing on the sensitive content in the sensitive area in the first image. Furthermore, in order to distinguish the abnormal part in the first image from the sensitive area after desensitization processing, the mobile phone may upload the position of the sensitive area in the first image when uploading the abnormal information of the first image, and separate the sensitive area after desensitization processing from the noise.

[0168] However, if the noise of the first image is in a sensitive area of ​​the first image, the above method may remove the noise in the first image. For example, in the embodiment of the present application, the mobile phone can perform image derivation on the first image, and use the first gradient image after the derivation of the first image as the image after desensitization processing. The first gradient image includes the noise features in the first image.

[0169] Image derivation, also known as gradient calculation, refers to the derivation of the grayscale value of each pixel in the first image in the x and y directions respectively. Image derivation measures the edge information of the image based on the brightness change of the image. For each pixel, the gradient value of the point is obtained by calculating the change in the grayscale of the surrounding pixels. Exemplarily, image derivation includes calculating the gradient value of the first image in the x direction, calculating the gradient value of the first image in the y direction, and merging the gradient value in the x direction with the gradient value in the y direction to obtain the first gradient image of the first image. For example, the gradient value of pixel one is dx(i,j)+dy(i,j). dx(i,j)=I(i+1,j)-I(i,j), dy(i,j)=I(i,j+1)-I(i,j). Among them, I is the grayscale value of pixel one, and (i,j) is the coordinate of pixel one in the first image. The first gradient image can reflect the change of pixel values ​​in the first image. The larger the pixel value change, the larger the corresponding gradient value. For example, if the grayscale value in the x direction in the first image is (100.90.90.90…), then the gradient value in the x direction is (10.0.0…). The first gradient image can be a black and white image, and the first gradient image displays the grayscale gradient value of each pixel in the first image. The more the grayscale value of a pixel in the first image changes, the larger the gradient value at the pixel in the first gradient image. Specifically, the brighter (whiter) the pixel is in the first gradient image. The less the grayscale value of a pixel in the first image changes, the smaller the gradient value at the pixel in the first gradient image. Specifically, the darker (blacker) the pixel is in the first gradient image. The places where the grayscale value of a pixel in the first image changes greatly are usually at the edge of the first image, so the first gradient image can reflect the image edge of the first image.

[0170] Generally, noise often appears in an image as isolated pixels or pixel blocks that cause strong visual effects. If the first image includes noise, the difference between the noise pixels and the surrounding pixels is relatively large. Therefore, the processed first gradient image also includes noise. For example, if the first image includes a black dot, the grayscale value of the pixel at the black dot varies greatly from the grayscale value of the pixel around the black dot. Therefore, the first gradient image may also display the black dot.

[0171] like Fig.12 As shown, Fig.12 The first image shown in FIG. 1 includes irregular black dots. Fig.12The first gradient image after the first image is derived is shown, and the first gradient image includes an edge image in the first image and irregular black dots in the first image. The process of generating the first gradient image from the first image can be called feature extraction, that is, extracting key features in the first image by deriving the first image. The key features include the change value of the pixel point in the first image, and the key features can reflect the change of the pixel point in the first image. The key features include the features of the pixels in the first image whose adjacent pixel changes are greater than the preset change threshold.

[0172] Furthermore, when the mobile phone executes S901, the mobile phone may pre-process the first image, and the pre-processing includes taking a derivative. For example, Fig.13 As shown, the preprocessing includes derivation, and the preprocessing also includes one or more of the first format conversion, rotation, storage, scaling, and the second format conversion. The first format conversion converts the bitmap of the first image into an image in Mat format. The image in Mat format stores the pixel value of each pixel in the bitmap in a matrix manner. The image in Mat format also includes basic information of the first image, such as the height of the first image, the width of the first image, the number of channels of the first image, etc. Rotation is to rotate the first image to the required aspect ratio preset value. Storage is to temporarily store the rotated first intermediate image. Afterwards, the first intermediate image is derivation-processed, and the second intermediate image after the derivation is scaled, and the second intermediate document is scaled to a preset size to obtain a first gradient image of the first image.

[0173] After obtaining the first gradient image of the first image, the mobile phone can input the first gradient image into the first neural network model to identify the type of the first image. Since the first gradient image includes the key features and noise features of the first image compared to the first image, inputting the first gradient image into the first neural network model to identify the type of the first image can improve computing efficiency.

[0174] Optionally, when uploading the first image to the cloud server, the mobile phone can specifically upload the first gradient image. Since the first gradient image only includes key features and / or noise features, the area where the pixel value changes more smoothly in the first image is ignored, which can achieve a desensitization effect and protect the user's privacy.

[0175] Optionally, when uploading the first gradient image, the mobile phone may display the first gradient image to be uploaded to the user for preview. Fig.11 As shown, after the mobile phone obtains the recognition result of the first image, the mobile phone can display the third interface, and the third interface can also be as follows Fig.11As shown in interface 104 and interface 105 in . Specifically, the first image is a normal image, and the mobile phone displays interface 104. Interface 104 includes the first image, the first gradient image after derivation, that is, the desensitized image. Optionally, the first image is a normal image, and the mobile phone displays the first image and / or the first gradient image when displaying the recognition result of the first image. If the first image is an abnormal image, the mobile phone displays interface 105. Interface 105 includes the first image, the first gradient image after derivation, that is, the desensitized image. In response to the user clicking the upload button on interface 105, the mobile phone can upload the first gradient image. Alternatively, in response to the user clicking the upload button on interface 102, the mobile phone can upload the first gradient image, that is, the mobile phone may not display the first gradient image, but upload the first gradient image.

[0176] The server may receive the first gradient image and the abnormal information of the first gradient image uploaded by the mobile phone, and may send a maintenance suggestion to the mobile phone based on the first gradient image and the abnormal information of the first gradient image.

[0177] Optional, such as Fig.11 As shown, the third interface such as interface 101, interface 102, interface 104, and interface 105 may also include a share button. The share button is used to trigger the mobile phone to share the third interface with other electronic devices. Exemplarily, in response to the user clicking the share button on interface 104, the mobile phone shares the third interface with other electronic devices.

[0178] Optionally, in response to the user's operation 1 on the first gradient image on the third interface, the mobile phone can copy or save the first gradient image. Alternatively, in response to the user's operation 2 on the first gradient image on the third interface, the mobile phone can share the first gradient image with other electronic devices. Operation 2, for example, can be that the user long presses the first gradient image on the third interface, and in response to operation 2, the mobile phone displays a list of electronic devices that can be shared. In response to the user selecting a target electronic device, the mobile phone can send the first gradient image to the target electronic device.

[0179] The above text introduces an abnormality identification method provided by an embodiment of the present application in conjunction with the accompanying drawings. Fig.14 The interaction process between each module in the mobile phone and the server is introduced. Among them, the first neural network model can be preset in the mobile phone before the mobile phone leaves the factory, or, after the mobile phone leaves the factory, the mobile phone downloads the first neural network model from the server and integrates it in the mobile phone. The mobile phone can download the first neural network model from the server automatically or in response to the user's trigger. For example, in response to the user clicking the application icon of the fault analysis APP on the main interface of the mobile phone, the mobile phone downloads the first neural network model from the server and integrates it in the mobile phone.

[0180] The mobile phone can input the first image into the first neural network model to obtain identification information of the first image. After the test, if the first image is an abnormal image, the mobile phone can upload the first image and the abnormal information to the server, and the server can send maintenance suggestions to the mobile phone based on the first image and the identification information of the first image.

[0181] Or, if Fig.14 As shown, the mobile phone can input the first gradient image of the first image into the first neural network model to identify its type and obtain identification information of the first image. If the first image is an abnormal image, the mobile phone can upload the first gradient image and the abnormal information to the server, and the server can send maintenance suggestions to the mobile phone based on the first gradient image and the identification information of the first image.

[0182] The abnormal information includes the noise type of the first image. The noise type is at least one of M preset noise types. The noise of the M preset noise types appears in the first image due to a mobile phone failure; wherein M is an integer greater than or equal to 1. Optionally, the identification information also includes log information when the mobile phone takes the first image. The log information includes the working status information of the radio frequency module in the mobile phone and the call stack information of the camera APP in the mobile phone when the mobile phone takes the first image.

[0183] Furthermore, the server may also use the first gradient image and the noise type to train the first neural network model. Alternatively, the server may also use the first image and the noise type to train the first neural network model. The server may update the model parameters of the first neural network model and send the latest model parameters of the first neural network model to the mobile phone.

[0184] For example, a mobile phone is configured with a first neural network model before leaving the factory. Fig.14 As shown, the interaction process between the mobile phone and the server includes S1401-S1417.

[0185] S1401, the image management module in the server obtains from the cloud server a first gradient image and a noise type of the first image previously uploaded by a mobile phone or other electronic device, wherein the first gradient image is an image obtained by taking the derivative of the first image, and the first image is an abnormal image.

[0186] S1402: The image management module classifies and stores the first gradient image.

[0187] S1403, the model training module in the server obtains training samples from the image management module.

[0188] S1404, the model training module in the server uses the training samples to automatically train the first neural network model and update the model parameters of the first neural network model.

[0189] Exemplarily, the model training module in the server can periodically obtain the latest training sample set from the image management module and use the latest training sample set to train the first neural network model. The latest training samples include the gradient images obtained by the image management module from the cloud and / or the training samples stored by the user in the server.

[0190] S1405, after the training is completed, the model training module in the server triggers the model conversion and integration module to send the latest model parameter numbers to the mobile phone.

[0191] S1406: The transmission module in the mobile phone stores the latest parameters.

[0192] S1407, the fault analysis APP in the mobile phone displays the first interface.

[0193] S1408, the fault analysis APP in the mobile phone responds to the user's first operation on the first interface, triggering the image recognition module in the mobile phone to load the latest model parameters of the first neural network model recently acquired, and update the first neural network model.

[0194] S1409: The image recognition module in the mobile phone pre-processes the first image to obtain a first gradient image.

[0195] The image recognition module may include a message queue for storing a plurality of first images to be processed. The image recognition module in the mobile phone may execute S1409-S1411 one by one for the plurality of first images in the message queue.

[0196] S1410, the image recognition module in the mobile phone uses the first gradient image as input of the first neural network model, runs the first neural network model, and obtains first image recognition information.

[0197] S1411: The image recognition module in the mobile phone stores the first gradient image and the recognition information of the first image in a storage module in the mobile phone.

[0198] S1412, the fault analysis APP in the mobile phone displays the second interface.

[0199] S1414, the fault analysis APP in the mobile phone responds to the second operation of the user on the second interface, and uploads the first gradient image of the currently identified abnormal image and the abnormal information of the abnormal image to the cloud server.

[0200] S1415, the maintenance module in the server obtains the first gradient image of the abnormal image uploaded this time and the abnormal information of the first image from the cloud server. The maintenance module can identify the cause of the failure of the mobile phone according to the abnormal information of the first image, and send maintenance suggestions to the mobile phone accordingly.

[0201] S1416, the maintenance module in the server sends maintenance suggestions to the mobile phone.

[0202] S1417, the mobile phone displays a third interface, which includes maintenance suggestions.

[0203] An image-based device fault identification method provided in an embodiment of the present application can bring convenience to users. For example, when a user is browsing pictures in a gallery app or when a user is taking pictures, if a first image containing noise is found, the user can trigger the mobile phone in the gallery app to execute the method described above to achieve remote maintenance. Afterwards, if the user's mobile phone hardware fails and the user goes to a maintenance outlet for repair, the maintenance personnel can directly obtain the abnormal images and abnormal information uploaded by the user's mobile phone from the server, and provide maintenance services to the user without having to identify whether the mobile phone is faulty again.

[0204] For another example, when a user is repairing a mobile phone at a repair outlet, the user can trigger the mobile phone to execute the method described above to identify abnormal images and noise types of abnormal images caused by mobile phone failures. In this way, the user can directly show the recognition results of the mobile phone to the maintenance personnel, and the maintenance personnel will provide maintenance services to the user based on the recognition results. In this way, there is no need for the maintenance personnel to look through the first image in the user's mobile phone. At the same time, there is no need for the user to describe the abnormal phenomenon of the abnormal image to the maintenance personnel, avoiding the problem of unclear description. If the abnormal image needs to be exported, the maintenance personnel can download the gradient image corresponding to the abnormal image, that is, the desensitized abnormal image, from the server to protect the user's privacy.

[0205] The present application provides an electronic device, which includes: a memory, a communication module, a display screen, and one or more processors. The communication module receives and sends data according to the control of the processor to achieve communication between the electronic device and other electronic devices. The memory and the display screen are coupled to the processor; wherein the memory is used to store computer program code, the computer program code includes computer instructions; the computer instructions are in the above electronic device (such as Figure 2 When the method is executed on an electronic device 200 shown in the figure, the electronic device executes each function or step in the above method embodiment.

[0206] The present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are in the above electronic device (such as Figure 2 When the method is executed on the electronic device 200 shown in the figure, the electronic device executes each function or step in the above method embodiment.

[0207] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step in the above method embodiment.

[0208] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device, a microphone of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0209] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0210] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0211] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0212] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0213] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0214] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for identifying equipment faults based on images, characterized in that: Applied to electronic equipment, the method comprises: The electronic device inputs a first image into a first neural network model to obtain identification information of the first image; the identification information includes a type of the first image, and the type of the first image indicates whether the first image is a normal image or an abnormal image; when the first image is an abnormal image, the identification information also includes a noise type of the first image, and the noise type is at least one noise type among M preset noise types; noise of the M preset noise types appears in the first image due to a malfunction of a shooting device for shooting the first image; wherein M is an integer greater than or equal to 1; the first neural network model has the ability to output identification information of a corresponding image according to an input image; If the first image is the abnormal image, the electronic device sends the first image and abnormal information of the first image to a server; the abnormal information includes a noise type of the first image.

2. The method according to claim 1, characterized in that The abnormal information also includes log information when the shooting device shoots the first image; the log information is used to locate the cause of the failure of the shooting device.

3. The method according to claim 2, characterized in that The log information includes working status information of the radio frequency module in the shooting device and call stack information of the camera APP in the electronic device when the shooting device shoots the first image.

4. The method according to any one of claims 1 to 3, characterized in that: The photographing device is the electronic device.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The electronic device receives a maintenance suggestion from the server; The electronic device displays the maintenance suggestion.

6. The method according to any one of claims 1 to 5, characterized in that: Before the electronic device inputs the first image into the first neural network model, the method further includes: The electronic device displays a first interface, wherein the first interface includes a first button; The electronic device inputs the first image into a first neural network model, including: In response to a user's first operation of a first button on a first interface, the electronic device inputs a first image into a first neural network model.

7. The method according to any one of claims 1 to 6, characterized in that: After the electronic device inputs the first image into the first neural network model, the method further includes: The electronic device displays a second interface, the second interface including identification information of the first image; if the first image is an abnormal image, the second interface includes a second button; The electronic device sending a first image and abnormal information of the first image to a server includes: In response to a second operation performed by the user on the second interface, the electronic device sends the first image and abnormal information of the first image to the server.

8. The method according to claim 5, characterized in that The electronic device displaying the maintenance suggestion includes: the electronic device displays a third interface, and the third interface includes the maintenance suggestion.

9. The method according to any one of claims 1 to 8, characterized in that: The electronic device inputs the first image into a first neural network model, including: The electronic device derives the first image to obtain a first gradient image of the first image; the first gradient image includes a key feature of the first image, or the first gradient image includes the key feature of the first image and a noise feature corresponding to noise in the first image; wherein the key feature includes a feature of a pixel in the first image whose adjacent pixel changes are greater than a preset change threshold; The electronic device inputs the first gradient image of the first image into the first neural network model.

10. The method according to claim 9, characterized in that The electronic device sending a first image and abnormal information of the first image to a server includes: The electronic device sends a first gradient image of the first image and abnormal information of the first image to the server.

11. The method according to claim 4, characterized in that If the first image is the abnormal image, before the electronic device sends the first image and abnormal information of the first image to the server, the method further includes: The electronic device reads the log information of the first image from the log information of the electronic device; or, The first image includes log information of the first image; the log information of the first image is acquired and stored in the first image when the electronic device shoots the first image.

12. The method according to claim 1, characterized in that Before the electronic device inputs the first image into the first neural network model, the method further includes: The electronic device obtains the latest model parameters of the first neural network model from the server; The electronic device updates the first neural network model using the latest model parameters.

13. The method according to claim 1, characterized in that In the case where the first image is the abnormal image, the first image is used by the server as a negative sample to train the first neural network model; The method further comprises: If the first image is the normal image, the electronic device sends the first image and the type of the first image to the server; in the case where the first image is the normal image, the first image is used by the server as a positive sample to train the first neural network model.

14. An image-based device fault identification method, characterized in that: Applied to a server, the method comprises: The server receives a first image and abnormal information of the first image sent by the electronic device to the server; the abnormal information includes a noise type of the first image; the noise type is at least one noise type among M preset noise types; at least one noise among the M preset noise types appears in the first image due to a malfunction of a shooting device for shooting the first image; wherein M is an integer greater than or equal to 1; The server sends a maintenance suggestion to the electronic device according to the first image and abnormal information of the first image.

15. The method according to claim 14, characterized in that The abnormal information also includes log information when the shooting device shoots the first image, and the log information is used to locate the cause of the failure of the shooting device.

16. The method according to claim 15, characterized in that The log information includes working status information of the radio frequency module in the shooting device and call stack information of the camera APP in the electronic device when the shooting device shoots the first image.

17. The method according to claim 14, characterized in that The method further comprises: The server uses the first image and the noise type of the first image to train a first neural network model, so that the trained first neural network model has the ability to output the noise type of the corresponding image according to the input image.

18. The method according to claim 17, characterized in that The method further comprises: After the training, the server updates the model parameters of the first neural network model; The server and the electronic device send the updated latest model parameters.

19. The method according to any one of claims 14 to 18, characterized in that: The first image is specifically a first gradient image of the first image; The first gradient image is obtained by differentiating the first image; the first gradient image includes key features of the first image, or the first gradient image includes the key features of the first image and noise features corresponding to noise in the first image; wherein the key features include features of pixels in the first image whose adjacent pixel changes are greater than a preset change threshold.

20. An image-based equipment fault identification system, characterized in that: The system includes an electronic device and a server; the electronic device is used to execute the method according to any one of claims 1-13; the server is used to execute the method according to any one of claims 14-19.

21. An electronic device, characterized in that: The electronic device comprises: a memory, a display screen, one or more processors, and a communication module; the memory, the communication module, and the display screen are coupled to the processor; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-13.

22. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1-13 or 14-19.

Images