Image-based device fault identification method and electronic device

Abstract

Embodiments of the present application relate to the technical field of communication, in particular to a device fault identification method based on images and an electronic device. The method comprises: inputting, by an electronic device, a first image into a first neural network model to obtain identification information of the first image; the identification information comprises a type of the first image, the type of the first image indicating whether the first image is a normal image or an abnormal image; in the case that the first image is an abnormal image, the identification information further comprises a noise type of the first image, the noise type being at least one of M preset noise types; the noise of the M preset noise types is caused by a fault of a shooting device for shooting the first image; the first neural network model has the ability to output identification information of a corresponding image according to an input image; and if the first image is an abnormal image, the electronic device sends the first image and abnormal information of the first image to a server; the abnormal information comprises the noise type of the first image.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an image-based device fault identification method and electronic device. Background Technology

[0002] Modern mobile devices generally have photo and video recording functions, allowing users to capture pictures or videos and record their lives anytime, anywhere.

[0003] When electronic devices malfunction, they may capture abnormal images that include noise. The capture of images by electronic devices can also be affected by the surrounding environment, resulting in abnormal images including noise. When a user discovers an abnormal image containing noise, they will request after-sales repair. In related technologies, the user is required to discover the abnormal image containing noise themselves and describe the abnormal phenomenon to after-sales service.

[0004] However, human identification of abnormal images has a high false positive rate and is prone to overlooking problems. Furthermore, even if a person can accurately identify all problems, they may not be able to clearly describe the abnormal phenomena in the image. Consequently, it becomes impossible to accurately pinpoint the problem with the electronic device, hindering after-sales service from providing timely and accurate repairs, thus reducing the user's after-sales experience. Summary of the Invention

[0005] This application provides an image-based device fault identification method and electronic device, which can provide the server with abnormal images for accurately locating device problems, thereby improving the accuracy of device problem location and enhancing the user's after-sales experience.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, an image-based device fault identification method is provided, applied to electronic devices. This method includes:

[0008] The electronic device uses a first neural network pattern to recognize a first image and obtain recognition information for the first image. This recognition information includes the type of the first image, which indicates whether the first image is a normal image or an abnormal image. If the first image is an abnormal image, the recognition information also includes the noise type of the first image, which is at least one of M preset noise types. The noise of the M preset noise types appears in the first image due to a malfunction of the imaging device that captured the first image. Here, M is an integer greater than or equal to 1. The first neural network model has the ability to output recognition information for the corresponding image based on the input image. Subsequently, if the first image is an abnormal image, the electronic device sends the first image and its abnormal information to the server. The abnormal information includes the noise type of the first image.

[0009] In this application, the first image may be a normal image or an abnormal image. That is, the first neural network can identify whether the first image is normal, and also whether it is an abnormal image containing at least one of M types of noise caused by electronic malfunction. Subsequently, if the first image is an abnormal image containing noise caused by electronic malfunction, the electronic device can upload the abnormal first image and its noise type to the server. This first image and its noise type are used to locate the type of malfunction in the electronic device. Thus, when after-sales personnel receive feedback from a user about an electronic device problem, they can directly obtain the first image of the electronic device from the server and provide remote repair assistance to the user based on the first image and its noise type. This eliminates the need for subjective human identification of abnormal images, avoiding user misidentification. Furthermore, it quantifies the noise in the first image, preventing unclear user descriptions. It also avoids the problem of users and after-sales personnel failing to detect noise in a timely manner due to it being too hidden in the image, thus improving the overall after-sales experience for users.

[0010] In one possible implementation of the first aspect, the anomaly information also includes log information from when the imaging device captured the first image. This log information is used to locate the cause of the imaging device's malfunction.

[0011] In one possible implementation of the first aspect, the log information includes the operating status information of the radio frequency module in the capturing device and the call stack information of the camera APP in the electronic device when the capturing device captures the first image. The capturing device can be an electronic device. The capturing device can also be other electronic devices, which can acquire the first image captured by other electronic devices and identify whether the other electronic devices are malfunctioning based on the first image.

[0012] Before sending the first image and its abnormal information to the server, the electronic device can read the log information of the first image from the log information of the backup electronic device, or from the log information of other electronic devices.

[0013] Alternatively, the first image may include log information of the first image; the log information of the first image is acquired and stored in the first image when the capturing device captures it. The first image may include image data and file data, wherein the file data may contain capture information, and the log information may be stored in the file data of the first image. The first image sent by other electronic devices to the electronic device includes the log information of the first image.

[0014] In one possible implementation of the first aspect, the server can receive a first image and abnormal information of the first image sent by the electronic device, and send maintenance suggestions to the electronic device based on the first image and the abnormal information of the first image. The electronic device can receive and display the maintenance suggestions.

[0015] In one possible implementation of the first aspect, the electronic device's gallery app, camera app, or fault reporting app can provide a first interface for human-computer interaction. In response to a user's trigger, the electronic device can execute this solution. Specifically, the mobile phone can display a first interface, which may include a first button. In response to a user's first operation on the first button on the first interface, the electronic device inputs a first image into a first neural network model to identify the type of the first image.

[0016] In one possible implementation of the first aspect, the electronic device's gallery app, camera app, or fault reporting app can display a second interface showing the recognition information of the first image. The recognition information may be, for example, a normal image or an abnormal image. If the first image is an abnormal image, the recognition information also includes at least one noise type from the preset noise types in M.

[0017] If the first image is an abnormal image, the second interface may include a second button. In response to the user clicking the second button, the electronic device uploads the first image and its recognition information to the server.

[0018] In one possible implementation of the first aspect, if the first image is a normal image, the second interface may also include a second button. In response to the user clicking the second button, the electronic device uploads the first image and the recognition information of the first image to the server.

[0019] In one possible implementation of the first aspect, the electronic device may display a third interface that shows maintenance suggestions.

[0020] In one possible implementation of the first aspect, the electronic device may display maintenance suggestions in the form of a notification message or a pop-up window.

[0021] In one possible implementation of the first aspect, before inputting the first image into the first neural network model, the electronic device can pre-calculate the image derivative of the first image to obtain a first gradient image. The first gradient image includes key features of the first image and noise features corresponding to noise in the first image; wherein, the key features include the features of pixels in the first image whose changes between adjacent pixels exceed a preset change threshold. The first gradient image contains fewer features than the first image, and inputting the first gradient into the first neural network model to identify the type of the first image can improve computational speed and efficiency.

[0022] In one possible implementation of the first aspect, since the first gradient image only includes key image and / or noise features, areas in the first image where pixel values ​​change relatively smoothly are ignored. Therefore, uploading the first gradient image to the server by the electronic device can achieve a desensitization effect and protect user privacy. Before inputting the first image into the first neural network model, the electronic device can pre-calculate the image derivative of the first image to obtain the first gradient image. The first gradient image is then uploaded during the upload process. Alternatively, the electronic device can input the first image into the first neural network model to identify the type of the first image. Before uploading, the image derivative of the first image is calculated to obtain the first gradient image, which is then uploaded.

[0023] In one possible implementation of the first aspect, before the electronic device inputs the first image into the first neural network model, the electronic device obtains the latest model parameters of the first neural network model from the server; the electronic device then updates the first neural network model using the latest model parameters. Because the server continuously trains the first neural network model, the accuracy and precision of the first neural network model continuously improve. Before using the first neural network model, the electronic device can use the latest model parameters to update the first neural network model, thereby improving the precision of the first neural network model in the electronic device and ensuring better recognition results.

[0024] In one possible implementation of the first aspect, the first image is an anomalous image, and the first image is used by the server as a negative sample to train the first neural network model; the first image is an anomalous image, and the electronic device sends the first image and the type of the first image to the server; the first image is used by the server as a positive sample to train the first neural network model.

[0025] Secondly, an image-based device fault identification method is provided, applied to a server. The method includes: the server receiving a first image and abnormal information of the first image sent by an electronic device; the abnormal information includes a noise type of the first image; the noise type is at least one of M preset noise types; the noise among the M preset noise types is noise that appears in the first image due to a fault in the capturing device that captured the first image; where M is an integer greater than or equal to 1; the server sending maintenance suggestions to the electronic device based on the first image and the abnormal information of the first image.

[0026] In this application, the server can receive a first image and abnormal information of the first image sent by an electronic device, and provide repair suggestions for the electronic device based on the first image and the abnormal information. This enables remote repair of electronic devices, providing users with a convenient repair method.

[0027] In one possible implementation of the second aspect, the anomaly information also includes log information from when the imaging device captured the first image, which is used to locate the cause of the imaging device's malfunction.

[0028] In one possible implementation of the second aspect, the log information includes the working status information of the radio frequency module in the imaging device and the call stack information of the camera APP in the electronic device when the imaging device captures the first image.

[0029] In one possible implementation of the second aspect, the server uses the first image and the noise type of the first image to train a first neural network model, enabling the trained first neural network model to output the noise type corresponding to the input image. The server can use the first image to train the first neural network model, improving the accuracy and precision of the first neural network model.

[0030] In one possible implementation of the second aspect, after training, the server updates the model parameters of the first neural network model. The server electronic device sends the updated model parameters. The electronic device can update the first neural network model with the latest model parameters before using it.

[0031] In one possible implementation of the second aspect, the server can receive a first gradient image of a first image sent by an electronic device. The first gradient image is obtained by differentiating the first image; the first gradient image includes key features of the first image, or the first gradient image includes key features of the first image and noise features corresponding to noise in the first image; wherein the key features include features of pixels in the first image whose changes between adjacent pixels are greater than a preset change threshold.

[0032] Thirdly, an image-based device fault identification system is provided, the system including an electronic device and a server; the electronic device is used to perform the method as described in any of the first aspects; the server is used to perform the method as described in any of the second aspects.

[0033] Fourthly, an electronic device is provided, comprising: a memory, a display screen, one or more processors, and a communication module; the memory, communication module, and display screen are coupled to the processor; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method of any one of the first aspects.

[0034] Fifthly, this application provides a chip system applicable to electronic devices including memory. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The interface circuits are used to receive signals from the aforementioned memory and send the signals to the processors, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device performs the method as described in the first aspect and any of its possible design embodiments.

[0035] Sixthly, this application provides a computer-readable storage medium including computer instructions. When executed on an electronic device, the computer instructions cause the electronic device to perform methods as described in the first aspect and / or the second aspect and any possible design configuration thereof.

[0036] In a seventh aspect, this application provides a computer program product that, when run on a computer, causes the computer to perform a method as described in the first aspect and / or the second aspect and any possible design thereof.

[0037] Understandably, the beneficial effects that the electronic devices of the third aspect to the computer program products of the seventh aspect can achieve can be referenced to the beneficial effects of the first aspect and / or the second aspect and any possible design of them, which will not be repeated here. Attached Figure Description

[0038] Figure 1 A schematic diagram illustrating various types of first images provided in embodiments of this application;

[0039] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0040] Figure 3 A system architecture diagram of an image-based device fault system provided in this application embodiment;

[0041] Figure 4 This is a schematic diagram of the display interface of a gallery app provided in an embodiment of this application;

[0042] Figure 5 A schematic diagram of the display interface of another gallery app provided in this application embodiment;

[0043] Figure 6 A schematic diagram of the display interface of a camera app provided in an embodiment of this application;

[0044] Figure 7 A schematic diagram of the display interface of a fault analysis APP provided in an embodiment of this application;

[0045] Figure 8 A schematic diagram of the display interface of another fault analysis APP provided in this application embodiment;

[0046] Figure 9 A flowchart illustrating an image-based device fault identification method provided in this application embodiment;

[0047] Figure 10 A flowchart illustrating another image-based device fault identification method provided in this application embodiment;

[0048] Figure 11 A schematic diagram of the display interface of an electronic device provided in an embodiment of this application;

[0049] Figure 12 A schematic diagram of a first gradient image provided in an embodiment of this application;

[0050] Figure 13 This is a schematic diagram of a first image preprocessing process provided in an embodiment of this application;

[0051] Figure 14 This is a schematic diagram of the interaction process between an electronic device and a server, provided as an embodiment of this application. Detailed Implementation

[0052] Modern mobile devices generally have photo and video recording functions, allowing users to capture pictures or videos and record their lives anytime, anywhere.

[0053] Terminal devices, such as mobile phones, may include at least one camera. For example, in response to a user tapping the shutter button on the camera app's screen, the camera can capture a raw image. This raw image can be a RAW image captured by the camera. Then, the image signal processor in the phone can process the RAW image to obtain a processed first image. Finally, the phone can display this first image in the camera app's preview interface. In this way, the user can preview the first image captured by the phone.

[0054] When a mobile phone malfunctions, it will produce abnormal shooting results. Specifically, the phone will capture a first image that includes noise. These malfunctions can be hardware or software issues. Hardware malfunctions can be caused by component failures, such as a camera hardware failure. Hardware malfunctions can also be due to interference, such as electromagnetic interference to the camera's image sensor from other components within the phone. For example, a malfunctioning camera image sensor, receiving an incoming call while shooting, or the phone downloading data in the background while shooting can all lead to a first image containing noise. Software malfunctions can be caused by missing software data in the camera app, malfunctioning camera app code, or software system bugs. The first image containing noise is considered an abnormal image, while the first image without noise is considered a normal image.

[0055] Noise refers to unnecessary or redundant interfering information in the first image. Noise can include stripe noise, dot noise, and mosaic noise. Noise randomly appears at any location in one or more first images. A first image including stripe noise can include colored or black-and-white stripes, which can be straight lines or curves. These stripes can be displayed in an array across all or part of the first image, or they may be displayed randomly in a portion of the first image. Dots can be colored or black-and-white dots. A first image including dot noise can include colored, black, or white dots, which can be displayed in an array across all or part of the first image, or they may be displayed randomly in a portion of the first image. In a first image including mosaic noise, mosaics can be displayed across all or part of the first image. For example, as shown... Figure 1 As shown. Figure 1 The images shown are a normal first image, a first image including stripe noise, a first image including dot noise, a first image including mosaic noise, and another first image including stripe noise.

[0056] In some cases, when a mobile phone is affected by the surrounding environment during shooting, the first image captured may also include stripes, dots, or mosaic effects. For example, when a user takes a picture of a brightly lit or intensely lit scene in a dimly lit environment, highlight clipping may occur, resulting in stripes in the first image. Typically, stripes will appear near the light source in the first image, spreading outwards from the light source. This is a normal phenomenon; the first image is normal, and the phone is not malfunctioning.

[0057] In other words, when a mobile phone malfunctions, the first image captured by the phone, including noise, is a truly abnormal image, requiring further analysis by repair personnel to pinpoint the cause of the malfunction. However, if the first image captured by the phone, influenced by ambient light, includes noise, it is not a truly abnormal image; this first image is normal and requires no repair.

[0058] Typically, when users find noise in the first image captured by their phone, they cannot distinguish whether the noise is caused by a phone malfunction or by ambient light. Users usually call customer service to request repairs, describing the problem to the technicians. However, users may not be able to clearly and objectively describe the problem, preventing technicians from understanding the true issue and providing accurate repair advice. Furthermore, some noise is quite obvious in the first image, allowing users to visually identify it. Other noise is more subtle and may go unnoticed, potentially leading to overlooking phone malfunctions and delays in repair, leaving potential problems for the user.

[0059] In summary, users are unable to distinguish between shooting abnormalities caused by phone malfunctions and cannot objectively describe the noise in the first image clearly. This makes it difficult for after-sales personnel to provide accurate repair advice. Furthermore, when the noise in the first image is subtle, users may not notice it in time, potentially leading to overlooking phone malfunctions and delays in repair, resulting in a poor after-sales experience for users.

[0060] Therefore, this application provides an image-based device fault identification method, which helps users to identify abnormal images, including noise, caused by electronic device malfunctions in a timely and accurate manner, and upload the abnormal images to a server. Specifically, the electronic device can input a first image into a first neural network model to obtain identification information for the first image. The identification information includes the type of the first image. In this application embodiment, the image type can include two types: normal image and abnormal image. If the first image includes at least one of M types of noise caused by electronic malfunctions, the first image is an abnormal image; if the first image does not include any of the M types of noise caused by electronic malfunctions, the first image is a normal image. If the first image is an abnormal image, the identification information also includes the noise type of the first image. The noise type of the first image is the noise type of at least one of the M types of noise caused by electronic malfunctions included in the first image.

[0061] In other words, the first image could be a normal image or an abnormal image. That is, the first neural network can identify whether the first image is normal, and also whether it is an abnormal image containing at least one of M types of noise caused by electronic malfunction. Subsequently, if the first image is an abnormal image containing noise caused by electronic malfunction, the electronic device can upload the abnormal first image and its noise type to the server. This first image and its noise type are used to locate the type of malfunction in the electronic device.

[0062] In this way, when after-sales personnel receive feedback from a user about an electronic device problem, they can directly obtain the first image of the device from the server and provide remote repair assistance to the user based on the first image and its noise type. This eliminates the need for subjective human identification of abnormal images, avoiding user misidentification. Furthermore, it allows for the quantification of noise in the first image, preventing unclear user descriptions. It also avoids the problem of noise being too subtle in the image, which users and after-sales personnel might miss in a timely manner, thus improving the overall after-sales experience for users.

[0063] The method provided in this application can be applied to electronic devices with data processing capabilities. These electronic devices may include servers, mobile phones, tablets, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smartwatches, smart bracelets, etc.), in-vehicle devices, virtual reality devices, etc., and this application does not impose any limitations on these. In this application, the aforementioned electronic device is an electronic device capable of running an operating system and installing applications. Optionally, the operating system running on the electronic device may be... system, system, Systems, etc.

[0064] For example, please refer to Figure 2 The diagram illustrates the structure of an electronic device 200. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, an audio module 230, a speaker 230A, a microphone 230B, a display screen 240, a communication module 250, a power module 260, an input device 270, a sensor module 280, a camera 290, etc. The sensor module 280 may include a pressure sensor, a touch sensor, etc.

[0065] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0066] Processor 210 may include one or more processing units. For example, processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, electronic device 200 may also include one or more processors 210.

[0067] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0068] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.

[0069] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0070] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0071] Internal memory 221 can be used to store one or more computer programs, which include instructions. Processor 210 can execute the aforementioned instructions stored in internal memory 221, thereby causing electronic device 200 to perform application operation methods, various applications, and data management, etc., as provided in some embodiments of this application.

[0072] Electronic device 200 can implement audio functions through audio module 230, speaker 230A, microphone 230B, and application processor, such as music playback and recording. Audio module 230 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. Audio module 230 can also be used for encoding and decoding audio signals. In some embodiments, audio module 230 can be located in processor 210, or some functional modules of audio module 230 can be located in processor 210.

[0073] The 230A loudspeaker, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals.

[0074] Microphone 230B, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. Users can speak by bringing their mouth close to microphone 230B, inputting sound signals into microphone 230B.

[0075] The communication function of electronic device 200 can be realized through antenna 1, antenna 2 and communication module 250, etc.

[0076] Communication module 250 can provide solutions for wireless communication applications on electronic device 200, including cellular, Wi-Fi, Bluetooth, and wireless data transmission modules (e.g., 433MHz, 868MHz, 915MHz). Communication module 250 can be one or more devices integrating at least one communication processing module. Communication module 250 receives electromagnetic waves via antenna 1 or antenna 2, filters and frequency-modulates the electromagnetic wave signals, and sends the processed signal to processor 210. Communication module 250 can also receive signals to be transmitted from processor 210, frequency-modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 1 or antenna 2.

[0077] Electronic device 200 implements display functions through a GPU, display screen 240, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 240 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0078] The display screen 240 is used to display images, videos, etc. The display screen 240 includes a display panel. In some embodiments, the electronic device 200 may include one or N display screens 240, where N is a positive integer greater than 1. In this embodiment, the display screen 240 can be used to display a user interface (UI) and receive user operations on the UI. In some embodiments, the display screen 240 is equipped with pressure sensors, touch sensors, etc.

[0079] The power module 260 can be used to supply power to the various components included in the electronic device 200. In some embodiments, the power module 260 can be a battery, such as a rechargeable battery.

[0080] Input device 270 may include a keyboard, mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc., into electronic device 200, thereby issuing commands and inputting data to electronic device 200. The mouse is an indicator for the horizontal and vertical coordinate positioning of the display system on electronic device 200, used to input instructions to electronic device 200. Input device 270 can be connected to electronic device 200 via a wired connection, such as through a GPIO interface or USB interface. Input device 270 can also be connected to electronic device 200 wirelessly, such as through Bluetooth or infrared.

[0081] The electronic device 200 can implement its shooting function through an ISP, a camera 290, a video codec, a GPU, a display 240, and an application processor. The ISP processes data fed back from the camera 290. The camera 290 captures still images or videos. In some embodiments, the electronic device 200 may include one or N cameras 290, where N is a positive integer greater than 1. The digital signal processor processes digital signals, including digital image signals and other digital signals. For example, when the electronic device 290 selects a frequency, the digital signal processor performs Fourier transforms on the frequency energy. The video codec compresses or decompresses digital video.

[0082] Taking the aforementioned electronic device 200 as an example, which is a mobile phone, the software system of the electronic device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 200.

[0083] Figure 3 This is a system architecture diagram of an image-based device fault identification system according to an embodiment of the present invention. The system architecture diagram includes a software structure block diagram of the electronic device 200.

[0084] For example, the server includes an image management module, a model training module, and a model conversion and integration module.

[0085] The image management module is used to construct training samples for training the first neural network. The training samples include pre-configured training samples within the service, as well as abnormal images uploaded by the electronic device 200. The abnormal image can be either the first image or the first gradient image. The abnormal image can be used as a negative sample to train the first neural network model. Optionally, the image management module can also acquire normal images from the electronic device 200, which can be used as positive samples to train the first neural network model.

[0086] Optionally, the electronic device 200 can upload the first image to the cloud, and the server can retrieve the first image from the cloud.

[0087] For example, the training samples pre-configured within the service can be images with noise such as stripes, dots, and mosaics, captured by the manufacturer of electronic device 200 when electronic device 200 is subjected to abnormal radio frequency interference and bus interference. The image management module can also identify the noise type of the first image uploaded by electronic device 200 and classify and store the first image according to the anomaly type.

[0088] The model training module is used to construct and train the first neural network model. Specifically, the model training module can construct the first neural network model, automatically iterate the model using training samples, and save the model parameters. Model parameters may include the number of nodes and node weights. The model training module can periodically and automatically train the first neural network model, gradually improving its accuracy as the number of training samples increases.

[0089] The model conversion and integration module can convert the first neural network into a preset format, which can be integrated into and recognized by the electronic device 200. For example, the format of the first neural network model before conversion could be an .H5 file, and the format after conversion could be a .tflite file. Optionally, the model conversion and integration module can also push the latest updated parameters of the first neural network model through differential upgrades after the electronic device 200 leaves the factory, allowing the electronic device 200 to update the first neural network model using the updated parameters.

[0090] The layered architecture of the electronic device 200 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime, the system libraries, and the kernel layer.

[0091] The electronic device 200 may include a fault analysis and reporting application (APP), an image recognition module, and a memory. The fault analysis and reporting APP is contained within the application framework layer of the electronic device 200. The image recognition module is contained within the application framework layer of the electronic device 200. The memory is the hardware of the electronic device 200.

[0092] Among them, the fault analysis and reporting APP can be an APP that includes functions such as anomaly identification, anomaly analysis, and anomaly reporting, such as Figure 3 The app includes a gallery app, a camera app, and a fault analysis app. The fault analysis and reporting app includes a user interface. It can respond to various user operations on the UI, enabling image recognition, displaying recognition results, and reporting abnormal images.

[0093] The image recognition module is used to identify the type of the first image. Specifically, the image recognition module can update the first neural network model, and the update process may include loading model parameters. The image recognition model can also preprocess the first image, including image differentiation, and one or more of the following: first format conversion, rotation, storage, scaling, and second format conversion. The image recognition module can use the first gradient image of the first image as input to the first neural network model, run the first neural network model, and obtain the recognition result of the first image.

[0094] like Figure 3 As shown, the application framework layer may also include an input system (not shown), a view system (not shown), a notification manager (not shown), a camera service, and a multimedia service.

[0095] In this embodiment, the input system is used to drive the touch screen of the monitoring electronic device 200 through the display screen, and convert the touch parameters generated by the touch operation input by the touch screen into usable events, which are then transmitted to the upper-level fault analysis and reporting APP.

[0096] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to construct the display interface of an application. The display interface can consist of one or more views. For example, in this embodiment, the vision system can be used to construct a first interface, including a first image and a first button. The vision system can also be used to construct a second interface, including a first image and its recognition information. The vision system can also be used to construct a third interface, displaying maintenance suggestions.

[0097] The notification manager allows the fault analysis and reporting app to display repair suggestions in the status bar, which can be used to convey repair recommendations. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, or as dialog boxes on the screen.

[0098] The camera service provides interfaces that can be called by upper-layer applications (such as camera apps), enabling image capture and preview based on these applications' calls. The multimedia service provides interfaces that can be called by upper-layer applications (such as gallery apps), enabling video or audio playback based on these applications' calls.

[0099] The Android Runtime consists of the core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0100] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0101] The kernel layer is the layer between hardware and software. The kernel layer can contain display drivers, audio drivers, sensor drivers, camera drivers, touchscreen drivers, etc.

[0102] The hardware includes a camera, a display screen, a memory, and a transmission module. The memory is used to store images and videos taken by the user. The memory can also be used to store the first gradient image of the first image after preprocessing, and the recognition information of the first image output by the first neural network model.

[0103] The following, with reference to the accompanying drawings and using a mobile phone as an example, describes an image-based device fault identification method provided by an embodiment of this application. In this embodiment, the mobile phone can identify the type of a first image captured by the phone. This first image can be a photograph or a video image captured by the phone. The mobile phone can also identify the type of a first image captured by other electronic devices. This first image can be a photograph or a video image captured by other electronic devices. For example, a tablet computer can send a first image captured by the tablet computer to the mobile phone, and the mobile phone can identify the type of the first image, thereby identifying whether the tablet computer is faulty.

[0104] The mobile phone may include a gallery application. The gallery application may include various display interfaces, one of which may include an anomaly detection button and a first image. In response to a user's interaction with the anomaly detection button on this interface, the mobile phone can identify the type of the first image. The anomaly detection button triggers the mobile phone to identify the type of the first image. This first image may be taken by the mobile phone or by another electronic device.

[0105] For example, the gallery app's large image preview screen may include an anomaly detection button. Specifically, the phone can display a main screen, which may include the gallery app's icon. In response to the user clicking the gallery app's icon, the phone can display something like... Figure 4 The list display interface 401 shown may include thumbnails of one or more images; for example, list display interface 401 may include a thumbnail of image 1. In response to a user clicking on the thumbnail of image 1, the mobile phone can jump to a large image preview interface 402. Large image preview interface 402 may display image 1 (first image) in large image mode. Large image preview interface 402 may include an anomaly detection button 403 (first button). That is, large image preview interface 402 includes a first image and a first button.

[0106] For example, the gallery app's list selection interface could include an anomaly detection button. Specifically, such as... Figure 5 As shown, in response to a user long-pressing the thumbnail of image 1 on the list display interface 401, the mobile phone can display a list selection interface 501. The list selection interface 501 includes thumbnails of multiple images, where the thumbnail of image 1 is selected (i.e., image 1 is selected). The list selection interface 501 may also include an anomaly detection button 502. Since image 1 is selected, the mobile phone receives a notification that the user has clicked the anomaly detection button 502, and the mobile phone can identify the type of image 1. In other words, the list selection interface 501 can include a first image and a first button.

[0107] It should be understood that in response to a user long-pressing the thumbnail of any image on the list display interface 401, the phone can jump from the list display interface 401 to the list selection interface 501. For example, in response to a user long-pressing the thumbnail of image 2 on the list display interface 401, the phone can display the list selection interface 501. At this time, the thumbnail of image 2 is selected on the list selection interface 501. That is, the first image is image 2, and in response to the user clicking the anomaly detection button, the phone can identify the type of image 2. If the phone responds to the user clicking the thumbnail of image 1 on the list selection interface 501, the phone can reselect image 1. In this scenario, the user can only select one image as the first image to be identified at a time. Furthermore, on the list selection interface, the user can select multiple images as the first images to be identified at a time. For example, in response to the user clicking the thumbnail of image 1 on the list selection interface 501, the phone identifies the types of image 1 and image 2.

[0108] In other words, the phone can display the first interface of the gallery app, including the first image and the first button. The first button, for example, is an anomaly detection button, used to trigger the phone to identify whether the first image is an abnormal picture. Thus, when browsing images in the gallery app, the user can select one or more first images and trigger the phone to identify the type of those images.

[0109] Furthermore, after a photo is taken, the user can also trigger the phone to identify the type of the captured image. For example... Figure 6 As shown, the phone can display a main interface, which includes a camera application icon. In response to the user clicking the camera application icon, the phone displays a photo preview interface 601. In response to the user clicking the shutter button 602, the phone can capture an image, obtaining picture 1. The phone can save picture 1 to the gallery application and display a thumbnail of picture 1 on the photo preview interface 601. In response to the user clicking the thumbnail of picture 1, the phone can display a large-image preview interface 402 of picture 1, i.e., the first interface.

[0110] Furthermore, the mobile phone may include a fault analysis app, which can be a pre-installed app or an app downloaded and installed by the user. For example... Figure 7As shown, the main screen of the phone can display the application icon of the fault analysis APP. In response to the user clicking the application icon on the main screen, the fault analysis APP displays interface 701. Interface 701 may include a selection button 702. In response to the user clicking the selection button 702 on interface 701, the phone displays a list display interface 703 for the fault analysis APP. In response to the user selecting image 1 on the list display interface 703, the phone displays a large image preview interface 704 for the fault analysis APP. This large image preview interface 704 includes the first image, i.e., image 1, and an anomaly detection button 705. Alternatively, as... Figure 7 As shown, in response to the user selecting image 1 in the list display interface 703 of the fault analysis app, the phone displays the list selection interface 706 of the fault analysis app. Image 1 is selected in the list selection interface 706, and this list selection interface includes an anomaly detection button 705. Similarly, in response to the user selecting multiple images in the list display interface 703 of the fault analysis app, the phone displays the list selection interface 706 of the fault analysis app, where the multiple images are selected.

[0111] Combined with the preceding text Figures 4-7 This section describes the methods for triggering the display of a first interface on a mobile phone, including a first image and a first button. Figures 4-7 In the method shown, the mobile phone can identify the type of one or more first images selected by the user, and the user needs to select the first image to be identified.

[0112] Optionally, the phone does not require the user to select a first image to be recognized; the phone can automatically identify the type of the stored first image. For example, such as... Figure 8 As shown, the main screen of the phone can display the application icon of the fault analysis APP. In response to the user clicking the application icon of the fault analysis APP on the main screen, the fault analysis APP displays interface 801. Interface 801 includes a one-click recognition button 802, which is used to identify the type of images stored on the phone. In response to the user clicking the one-click recognition button 802, the phone automatically acquires the first image stored on the phone and identifies the type of the first image one by one.

[0113] Optionally, in response to the user clicking the one-click recognition button 802, the mobile phone can display interface 803, which is used by the user to select the first image stored on the mobile phone within a target time period. For example, in response to the user selecting a start date of 2021 / 10 / 1 and an end date of 2021 / 12 / 1 on interface 803 and clicking the start button, the mobile phone acquires multiple first images stored on the mobile phone during the time period from 2021 / 10 / 1 to 2021 / 12 / 1, and identifies the type of each first image.

[0114] In other embodiments, the phone automatically identifies the type of a first image stored on the phone without user intervention. For example, the phone automatically identifies the type of a first image captured at a preset time, such as 11:00 PM. This identification process can run in the background. Alternatively, the phone automatically identifies the type of the first image in the phone in response to a recognition command issued by a server. In other embodiments, the phone can automatically identify the type of a first image stored on the phone in response to capturing a first image, without user intervention. The first image can be a photograph or a frame from a video; this application does not limit this. In this embodiment, the phone automatically identifies the first image in the phone with user consent. For example, before automatically identifying the first image for the first time, the phone can notify and remind the user to read the relevant user agreement (notification) and sign the agreement (authorization), which includes authorization of relevant user information; or, the user can manually activate the automatic identification function before the phone can automatically identify the first image.

[0115] The previous section introduced various ways to trigger a phone to recognize the first image. The following section will combine... Figure 9 This application introduces an image-based device fault identification method provided by an embodiment. For each first image, the mobile phone can perform... Figure 9 The method shown identifies the type of the first image.

[0116] S901, the mobile phone inputs the first image into the first neural network model to obtain the recognition information of the first image.

[0117] The identification information of the first image includes the type of the first image, which indicates whether the first image is a normal image or an abnormal image. For example, the type of the first image is 1, which indicates that the first image is a normal image. The type of the first image is 2, which indicates that the first image is an abnormal image.

[0118] If the first image is an abnormal image, the identification information also includes the noise type of the first image. The noise type of the first image can be one or more types of noise from M preset noises caused by a malfunction of the shooting device. Here, M is an integer greater than or equal to 1. For example, the M preset noises can be stripe noise, dot noise, or mosaic noise. In other words, the noise type of the first image can be at least one of stripe noise, dot noise, or mosaic noise. That is, the noise type of the first image can be one or more of the M preset noises that appear in the first image due to a malfunction of the shooting device. In other words, if the first image is an abnormal image, the first image includes one or more of the M preset noises caused by a malfunction of the shooting device. The following example uses a mobile phone as the shooting device.

[0119] For example, the identification information may include indication information used to indicate the type of noise in the first image. For instance, the first neural network model outputs 2a, indicating that the first image is an abnormal image and the noise type is stripe noise. The first neural network model outputs 2b, indicating that the first image is an abnormal image and the noise type is dot noise. The first neural network model outputs 2c, indicating that the first image is an abnormal image and the noise type is mosaic noise.

[0120] The mobile phone can use a first neural network model to identify the type of the first image. The first neural network model can be integrated into the mobile phone or deployed on a server. If the first neural network is deployed on the server, the mobile phone can send the first image to the server. The server uses the first neural network model to identify the type of the first image. The server then returns the type of the first image to the mobile phone. The following description uses an example of a mobile phone with a first neural network model deployed in it.

[0121] The first neural network has the ability to output recognition information of the first image based on the input first image. For example, the first neural network model can be trained using a normal image without any noise, a normal image including one or more types of noise such as stripes, dots, and mosaics caused by ambient light sources, and an abnormal image including one or more types of noise such as stripes, dots, and mosaics caused by a mobile phone malfunction. For a first image without any noise, the trained first neural network model has the ability to output that the first image is of the normal image type. For a first image including one or more types of noise such as stripes, dots, and mosaics caused by ambient light sources, the trained first neural network model has the ability to output that the first image is of the normal image type. For a first image including one or more types of noise such as stripes, dots, and mosaics caused by a mobile phone malfunction, the trained first neural network model has the ability to output that the first image is an abnormal image and the noise type of the first image.

[0122] As one possible implementation, the first neural network model includes multiple sub-neural network sub-models, and a preset noise type includes a corresponding sub-neural network model. For example, neural network model one is used to identify whether the first image includes stripe noise caused by mobile phone malfunction; neural network model two is used to identify whether the first image includes mosaic noise caused by mobile phone malfunction; and neural network model three is used to identify whether the first image includes dot noise caused by mobile phone malfunction. The mobile phone sequentially inputs the first image into these three neural network models to obtain the recognition information of the first image. For example, the first image is input into neural network model one, and neural network model one outputs a probability value one that the first image includes stripe noise caused by mobile phone malfunction. If probability value one is greater than or equal to a preset threshold one, the first image is considered an abnormal image and the noise type of the first image is stripe noise. The first image is then input into neural network model two to obtain a probability value two that the first image includes mosaic noise caused by mobile phone malfunction. If probability value two is greater than or equal to a preset threshold two, the first image is considered an abnormal image and the noise type of the first image is mosaic noise. The first image is then input into neural network model three to obtain a probability value three that the first image includes dot noise caused by mobile phone malfunction. If probability value three is greater than or equal to preset threshold three, then the first image is considered an abnormal image and the noise type of the first image is point noise. Preset thresholds one, two, and three can be equal, for example, all can be 0.5. If probability value one of the first image is less than preset threshold one, probability value two is less than preset threshold two, and probability value three is less than preset threshold three, then the first image is a normal image, i.e., the first image is a noise-free image.

[0123] As another possible implementation, the first neural network model is a neural network model. The first neural network model can output the probability value of each of M preset noises caused by a mobile phone malfunction in the first image. The first neural network uses the preset noise type corresponding to the probability value that meets preset conditions among the M probability values ​​as the noise type of the first image. For example, the first neural network model can output the probability values ​​of stripe noise, mosaic noise, and dot noise caused by a mobile phone malfunction in the first image. For example, the probability values ​​are 0.05, 0.6, and 0.35 respectively. The preset noise type corresponding to the highest probability value can be the noise type of the first image; that is, the mosaic noise corresponding to the probability value of 0.6 is the noise type of the first image. Optionally, the noise type corresponding to the probability value exceeding a preset threshold four is the noise type of the first image. For example, the preset threshold four can be 0.3; then, the mosaic noise corresponding to the probability value of 0.6 and the dot noise corresponding to the probability value of 0.35 are the noise types of the first image. If the probability values ​​of the first image belonging to stripe noise, mosaic noise, and dot noise are all less than the preset threshold five, then the first image is a normal image. For example, if the probability values ​​of the first image belonging to stripe noise, mosaic noise, and dot matrix noise are 0.35, 0.35, and 0.3 respectively, and the fifth threshold can be 0.4, then the type of the first image can be a normal image.

[0124] S902, if the first image is an abnormal image, the mobile phone reports the first image and the abnormal information of the first image to the server.

[0125] If the first image is an abnormal image containing at least one of M preset noises caused by a mobile phone malfunction, the mobile phone reports the first image and its abnormal information to the server. The abnormal information of the first image may include the noise type of the first image.

[0126] Optionally, the mobile phone will also report log information from when the camera captured the first image to the server; that is, the anomaly information may also include log information from when the camera captured the first image. This log information is used to locate the cause of the camera malfunction.

[0127] As an example, the log information includes the operation log information of the radio frequency (RF) module in the imaging device when the imaging device captures the first image. This operation log information may include the working status information of the RF module. The working status of the RF module includes whether it is working or not. If the RF module is working, the working status information also includes information such as the operating frequency band and transmission power. For example, the log information may be the call stack information of the application running in the imaging device when the imaging device captures the first image. This call stack information can reflect the application's calls to the RF module and also reflect the working status of the RF module. The log information may also include the call stack information of the camera app on the mobile phone when the imaging device captures the first image. This call stack information can reflect the code execution of the camera app when capturing the first image. Optionally, the log information may also include the operation log information of the camera module capturing the first image. Optionally, the log information may also include the operating parameters of the camera module capturing the first image. These operating parameters may include, for example, exposure value, exposure time, aperture value, focal length, etc.

[0128] Optionally, after the phone identifies the first image as an abnormal image, it can read the log information of the camera when it captured the first image from the camera's log information before uploading.

[0129] Optionally, the capturing device may acquire log information at the time of capture when capturing the first image. This log information is stored in the first image. For example, the first image may include image data and file data. The file data may include capture information, i.e., the log information, which can be stored in the file data of the first image.

[0130] The aforementioned shooting device can be a mobile phone or other electronic device.

[0131] Optionally, after receiving the first image and its abnormality information from the mobile phone, the server can send repair suggestions to the mobile phone.

[0132] Furthermore, the server receiving the first image and its anomaly information can be the same as or different from the server sending repair suggestions to the mobile phone. For example, the mobile phone can send the first image and its anomaly information to a cloud server, and the cloud server can send repair suggestions to the mobile phone. Alternatively, the mobile phone can send the first image and its anomaly information to a cloud server, and the server can obtain the first image and its anomaly information from the cloud server and send repair suggestions to the mobile phone.

[0133] S903, the server sends repair suggestions to the mobile phone.

[0134] The server can send repair suggestions to the mobile phone. For example, after-sales personnel can obtain abnormal images and information sent by the user's mobile phone from the server. The after-sales personnel can remotely analyze the cause of the user's mobile phone failure and trigger the server to send repair suggestions to the mobile phone.

[0135] Alternatively, the server can automatically identify the cause of the user's phone malfunction and automatically send repair suggestions to the phone. For example, the server includes a mapping relationship between noise types and repair suggestions, and sends repair suggestions to the phone based on the noise type of the first image and this mapping relationship. Alternatively, the server can obtain repair suggestions from recent repair records of other phones based on the first image and the noise type of the first image.

[0136] Alternatively, the server can identify whether the phone has a hardware or software malfunction based on the first image sent by the phone and any abnormal information within that image. For example, if the server detects that the radio frequency module was working when the phone captured the first image, it identifies a hardware malfunction. For instance, based on log information from when the phone captured the first image, the server might identify that the call app on the phone used the radio frequency module, and the abnormality in the first image might be due to radio frequency interference to the image sensor in the camera. The server can then send a repair suggestion to the phone, such as "Hardware malfunction, please go to a repair center for repair as soon as possible." The server can also identify software malfunctions on the phone based on log information, thus identifying a software malfunction. For instance, based on log information from when the phone captured the first image, the server might identify a code error in the camera app that captured the first image. The server can then send a repair suggestion to the phone, such as "Camera software malfunction, please repair the camera software."

[0137] After that, the phone can receive and display repair suggestions sent by the server.

[0138] If the user actively triggers the phone to recognize the type of the first image, for example... Figures 4-8 As shown, after receiving the first image and its recognition information, the server can send repair suggestions to the mobile phone, which can then display them. Figure 11 As shown, the phone can display a third interface, which includes repair suggestions. If the phone automatically recognizes the type of the first image, the server can receive the first image and its recognition information, and then send repair suggestions to the phone. The phone can display the repair suggestions via a notification message or a pop-up window. The phone can execute S904.

[0139] S904, the phone displays repair suggestions.

[0140] The phone can display repair suggestions via notification messages or pop-ups, or the phone can display repair suggestions on the interface.

[0141] Taking the display of repair suggestions in a pop-up window on a mobile phone as an example, the pop-up window may include a repair button. In response to the user clicking the repair button, the mobile phone can obtain the updated data of the camera software from the server and use the updated data to repair the camera app.

[0142] Optionally, if the anomaly in the first image is due to a software malfunction, the server can send software update data to the phone along with repair suggestions. The phone can receive and store this update data. In response to the user clicking the repair button in the pop-up window, the phone directly uses the updated data to repair the camera app.

[0143] This application provides an image-based device fault identification method. The electronic device can input a first image into a first neural network model to obtain identification information for the first image, thereby improving identification accuracy. Subsequently, if the first image is an abnormal image including noise caused by electronic faults, the electronic device can upload the abnormal first image and its noise type to a server. This first image and its noise type are used to pinpoint the fault type of the electronic device, improving the user's after-sales experience.

[0144] In some embodiments, the mobile phone can automatically execute S901. In other embodiments, the mobile phone can also execute S901 in response to a user trigger. For example, user triggering... Figure 10 As shown, the phone can execute S101 before executing S901.

[0145] S101, the phone displays the first screen.

[0146] The first interface can be Figure 4 or Figure 6 The large image preview interface shown is error 402, or the first interface is... Figure 5 The image shows the gallery app's list selection interface, 501. The first interface can also be... Figure 7 The large preview interface of the fault analysis APP shown is 704, or Figure 7 The fault analysis app's list selection interface 706 is shown. The first interface can also be as follows: Figure 8 The interface of the fault analysis APP shown is 803.

[0147] The first interface may include a first button, which is used to trigger the phone to recognize the type of the first image.

[0148] Optionally, S901 can be S102.

[0149] S102, in response to the user's first operation on the first button on the first interface, the mobile phone inputs the first image into the first neural network model to obtain the recognition information of the first image.

[0150] The first operation could be, for example, a click. In response to a user clicking a first button on the first interface, the phone uses a first neural network model to identify the type of the first image. The first image can be selected by the user and displayed on the first interface. Alternatively, the first image may not be displayed on the first interface; for example, in response to a user clicking a button on the first interface... Figure 8 On the interface 801 shown, clicking the one-click recognition button allows the phone to acquire multiple first images taken by the user and stored on the phone. For each of these first images, the phone performs the following... Figure 9 The method shown.

[0151] Optionally, after S102, the phone can display the recognition result. For example, the phone can execute S103.

[0152] S103, the phone displays the second interface.

[0153] The second interface may include recognition information of the first image, which may include the type of the first image. If the first image is an abnormal image, the recognition information may also include the noise type of the first image.

[0154] Optionally, the second interface may also include the first image; that is, the second interface may display the recognition information of the first image and / or the first image itself. For example, the second interface may be as follows: Figure 11 As shown. In Figure 11 In the first interface 101, the first image and its recognition information are displayed as a normal image. In the second interface 102, the first image and its recognition information are displayed as stripe anomalies. Furthermore, if the phone recognizes multiple first images at once, it can display the recognition result of one first image on the second interface. The phone can switch between the recognition results of the first images in response to the user's swiping operation on the second interface.

[0155] Optional, such as Figure 11 As shown, the second interface may also include an upload button, which triggers the phone to upload the first image to the server. In other words, the phone can respond to the user's upload operation by reporting the first image and its recognition information to the server.

[0156] Optionally, if the first image is a normal image, the second interface may not include an upload button; if the first image is an abnormal image, the second interface may include an upload button. Alternatively, the second interface may include an upload button regardless of the type of the first image. In other words, if the first image is a normal image, the phone may upload the first image to the server, or the phone may not upload the first image to the server. If the first image is an abnormal image, the phone may upload the first image to the server.

[0157] If the first image is a normal image, the mobile phone uploads the first image and its type to the server. If the first image is an abnormal image, the mobile phone uploads the first image and its abnormality information to the server.

[0158] Taking the example of a mobile phone uploading an abnormal image to a server, the method also includes S104. For example... Figure 10 As shown, Figure 9 S902 (not in Figure 10 (As shown) can be S104.

[0159] S104, if the first image is an abnormal image, in response to the user's second operation on the second interface, the mobile phone reports the first image and the abnormal information of the first image to the server.

[0160] The second step is for the user to click the upload button on the second interface.

[0161] For example, in response to use in Figure 11 After clicking the upload button on the interface shown in screen 102, the phone displays the third interface. The third interface can be viewed as follows: Figure 11 The third interface is shown in interface 103. It includes repair suggestions. That is, as shown in... Figure 9 As shown, S904 can be S105.

[0162] S105, the phone displays a third screen, which includes repair suggestions.

[0163] In some embodiments, the mobile phone can perform desensitization processing on the first image before inputting it into the first neural network model. The desensitized first image includes the key features of the image and / or the noise features corresponding to the noise in the first image. Therefore, inputting the desensitized first image into the first neural network model to identify the type of the first image can improve computational efficiency.

[0164] In other embodiments, the mobile phone can de-identify the first image before uploading it to the server. If the first neural network is deployed in the mobile phone, after executing S901 to recognize the first image and before executing S902, the mobile phone de-identifies the first image and uploads the de-identified first image to the server. Alternatively, the mobile phone de-identifies the first image before executing S901 to recognize the first image, uses the de-identified first image as input to the model of the first neural network, identifies the type of the first image, and uploads the de-identified first image to the server.

[0165] In other embodiments, the mobile phone desensitizes the first image before recognizing it in S901. The mobile phone sends the desensitized first image to the server, and the server uses the desensitized first image as input to the model of the first neural network to identify the type of the first image. The following describes this solution using the example of the mobile phone using the first neural network model deployed in the mobile phone to identify the type of the first image.

[0166] The mobile phone performs desensitization processing on the sensitive content in the first image, removing it from the image. Sensitive content can include faces, sensitive text, and sensitive images. Sensitive text can include one or more personal information such as username, user ID information, user home address, and user contact information. Desensitization processing includes blurring, pixelating, and replacing sensitive content in the image.

[0167] For example, after the mobile phone performs S901 to recognize the first image, it can first identify sensitive content in the first image. For instance, the phone can identify whether the first image includes a face, preset sensitive text, preset sensitive patterns, etc. Then, the phone performs one or more of the following processing on the sensitive content in the first image: blurring, mosaicking, or replacement. The phone can identify sensitive regions in the first image based on a second neural network model. Sensitive regions are areas containing sensitive content. The second neural network model has the ability to output the sensitive regions in the first image. Then, the phone performs one or more of the following processing on the sensitive content in the sensitive regions of the first image: blurring, mosaicking, or replacement. Furthermore, to distinguish between abnormal parts in the first image and the desensitized sensitive regions, when uploading abnormal information of the first image, the phone can upload the position of the sensitive regions in the first image, separating the desensitized sensitive regions from noise.

[0168] However, if the noise in the first image is located in a sensitive area of ​​the first image, the above method may remove the noise from the first image. For example, in this embodiment of the application, the mobile phone can perform image differentiation on the first image and use the first gradient image after differentiation as the image after desensitization processing. The first gradient image includes the noise features in the first image.

[0169] Image differentiation, also known as gradient calculation, refers to differentiating the grayscale value of each pixel in a first image along the x and y directions. Image differentiation measures the edge information of an image based on changes in brightness. For each pixel, the gradient value is obtained by calculating the grayscale changes of its surrounding pixels. For example, image differentiation includes calculating the gradient value of the first image in the x-direction, calculating the gradient value of the first image in the y-direction, and merging the gradient values ​​in the x and y directions to obtain the first gradient image of the first image. For example, the gradient value of pixel one is dx(i,j) + dy(i,j). dx(i,j) = I(i+1,j) - I(i,j), dy(i,j) = I(i,j+1) - I(i,j). Where I is the grayscale value of pixel one, and (i,j) is the coordinate of pixel one in the first image. The first gradient image reflects the changes in pixel values ​​in the first image; the larger the change in pixel value, the larger the corresponding gradient value. For example, if the grayscale values ​​in the x-direction of the first image are (100, 90, 90, 90…), then the gradient values ​​in the x-direction are (10, 0, 0…). The first gradient image can be a black and white image, displaying the grayscale gradient values ​​of each pixel in the first image. The greater the variation in grayscale values ​​of a pixel in the first image, the larger the gradient value at that pixel in the first gradient image; specifically, the brighter (whiter) the pixel in the first gradient image. Conversely, the smaller the variation in grayscale values ​​of a pixel in the first image, the smaller the gradient value at that pixel in the first gradient image; specifically, the darker (blacker) the pixel in the first gradient image. Areas with large variations in grayscale values ​​in the first image are usually at the edges of the first image; therefore, the first gradient image can reflect the edges of the first image.

[0170] Typically, noise in an image often manifests as isolated pixels or pixel blocks that cause a strong visual effect. If the first image includes noise, the noise pixels differ significantly from their surrounding pixels. Therefore, the processed first gradient image also includes noise. For example, if the first image includes black dots, the grayscale value of the pixel at the black dot differs greatly from the grayscale values ​​of the pixels surrounding the black dot. Therefore, the first gradient image can also display the black dot.

[0171] like Figure 12 As shown, Figure 12 The first image shown includes irregular black dots. Figure 12The image shown is the first gradient image obtained by differentiating the first image. The first gradient image includes edge images and irregular black dots in the first image. The process of generating the first gradient image from the first image can be called feature extraction, which involves extracting key features from the first image by differentiating it. Key features include the change values ​​of pixels in the first image, reflecting the changes in pixels. Key features include pixels in the first image whose changes between adjacent pixels exceed a preset change threshold.

[0172] Furthermore, when executing S901, the mobile phone can preprocess the first image, including differentiation. For example, as shown... Figure 13 As shown, preprocessing includes differentiation, and also includes one or more of the following: first format conversion, rotation, storage, scaling, and second format conversion. The first format conversion converts the bitmap of the first image into a Mat format image. The Mat format image stores the pixel value of each pixel in the bitmap in a matrix manner. The Mat format image also includes basic information about the first image, such as its height, width, and number of channels. Rotation rotates the first image to a preset aspect ratio. Storage temporarily stores the rotated first intermediate image. Then, differentiation is performed on the first intermediate image, and the differentiated second intermediate image is scaled to a preset size to obtain the first gradient image of the first image.

[0173] After obtaining the first gradient image of the first image, the mobile phone can input the first gradient image into the first neural network model to identify the type of the first image. Since the first gradient image contains the key features and noise features of the first image compared to the first image, inputting the first gradient image into the first neural network model to identify the type of the first image can improve computational efficiency.

[0174] Optionally, when uploading the first image to the cloud server, the mobile phone can specifically upload a first gradient image. Since the first gradient image only includes key features and / or noise features, ignoring areas in the first image where pixel values ​​change relatively smoothly, it can achieve a desensitization effect and protect user privacy.

[0175] Optionally, when uploading the first gradient image, the mobile phone can display a preview of the first gradient image to be uploaded to the user. For example, as shown below... Figure 11 As shown, after obtaining the recognition result of the first image, the phone can display a third interface, which can also be as follows: Figure 11Interfaces 104 and 105 are shown in the diagram. Specifically, if the first image is a normal image, the phone displays interface 104. Interface 104 includes the first image and the desensitized first gradient image (i.e., the image after desensitization). Optionally, if the first image is a normal image, the phone displays the first image and / or the first gradient image when displaying the recognition result of the first image. If the first image is an abnormal image, the phone displays interface 105. Interface 105 includes the first image and the desensitized first gradient image (i.e., the image after desensitization). In response to the user clicking the upload button on interface 105, the phone can upload the first gradient image. Alternatively, in response to the user clicking the upload button on interface 102, the phone can upload the first gradient image; that is, the phone can upload the first gradient image without displaying it.

[0176] The server can receive the first gradient image uploaded by the mobile phone and any anomaly information from the first gradient image. Based on the first gradient image and its anomaly information, the server can send repair suggestions to the mobile phone.

[0177] Optional, such as Figure 11 As shown, the third interface, such as interfaces 101, 102, 104, and 105, may also include a share button. This share button is used to trigger the mobile phone to share the third interface with other electronic devices. For example, in response to a user clicking the share button on interface 104, the mobile phone shares the third interface with other electronic devices.

[0178] Optionally, in response to user operation 1 on the first gradient image on the third interface, the phone can copy or save the first gradient image. Alternatively, in response to user operation 2 on the first gradient image on the third interface, the phone can share the first gradient image with other electronic devices. Operation 2 could be, for example, the user long-pressing the first gradient image on the third interface; in response to operation 2, the phone displays a list of shareable electronic devices. In response to the user selecting a target electronic device, the phone can send the first gradient image to the target electronic device.

[0179] The preceding text, in conjunction with the accompanying drawings, describes an anomaly identification method provided by an embodiment of this application. The following text, in conjunction with... Figure 14 This section describes the interaction process between the various modules in the mobile phone and the server. The first neural network model can be pre-installed in the phone before it leaves the factory, or the phone can download the first neural network model from the server and integrate it after leaving the factory. The phone can automatically download the first neural network model from the server or in response to user triggers. For example, in response to a user clicking the application icon of the fault analysis app on the phone's main screen, the phone downloads the first neural network model from the server and integrates it into the phone.

[0180] The mobile phone can input the first image into the first neural network model to obtain the recognition information of the first image. After testing, if the first image is an abnormal image, the mobile phone can upload the first image and the abnormal information to the server. The server can then send repair suggestions to the mobile phone based on the first image and its recognition information.

[0181] Or, such as Figure 14 As shown, the mobile phone can input the first gradient image of the first image into the first neural network model to identify its type and obtain the recognition information of the first image. If the first image is an abnormal image, the mobile phone can upload the first gradient image and the abnormal information to the server. The server can send repair suggestions to the mobile phone based on the first gradient image and the recognition information of the first image.

[0182] The anomaly information includes the noise type of the first image. The noise type is at least one of M preset noise types. The noise of the M preset noise types appears in the first image due to a mobile phone malfunction; where M is an integer greater than or equal to 1. Optionally, the identification information also includes log information when the mobile phone captured the first image. The log information includes the working status information of the radio frequency module in the mobile phone and the call stack information of the camera APP in the mobile phone when the first image was captured.

[0183] Furthermore, the server can also train the first neural network model using the first gradient image and noise type. Alternatively, the server can also train the first neural network model using the first image and noise type. The server can update the model parameters of the first neural network model and send the latest model parameters of the first neural network model to the mobile phone.

[0184] For example, taking a mobile phone equipped with a first neural network model before it leaves the factory as an example, such as... Figure 14 As shown, the interaction process between the mobile phone and the server includes S1401-S1417.

[0185] S1401, the image management module in the server obtains the first gradient image and the noise type of the first image uploaded by the mobile phone or other electronic device from the cloud server. The first gradient image is the derivative of the first image, and the first image is an anomalous image.

[0186] S1402, the image management module classifies and stores the first gradient image.

[0187] S1403, the model training module in the server obtains training samples from the image management module.

[0188] S1404, The model training module in the server uses training samples to automatically train the first neural network model and update the model parameters of the first neural network model.

[0189] For example, the model training module in the server can periodically retrieve the latest training sample set from the image management module and use the latest training sample set to train the first neural network model. The latest training samples include gradient images retrieved by the image management module from the cloud and / or training samples stored by the user on the server.

[0190] S1405, after training is complete, the model training module in the server triggers the model conversion and integration module to send the latest parameter model numbers to the mobile phone.

[0191] S1406, the transmission module in the mobile phone stores the latest parameters.

[0192] S1407, the fault analysis APP on the mobile phone displays the first interface.

[0193] S1408, the fault analysis APP in the mobile phone responds to the user's first operation on the first interface, triggering the image recognition module in the mobile phone to load the latest model parameters of the recently acquired first neural network model and update the first neural network model.

[0194] S1409, the image recognition module in the mobile phone preprocesses the first image to obtain the first gradient image.

[0195] The image recognition module may include a message queue for storing multiple first images to be processed. The image recognition module in the mobile phone can execute steps S1409-S1411 on each of the multiple first images in the message queue.

[0196] S1410, the image recognition module in the mobile phone takes the first gradient image as the input of the first neural network model, runs the first neural network model, and obtains the first image recognition information.

[0197] S1411, The image recognition module in the mobile phone stores the first gradient image and the recognition information of the first image in the storage module of the mobile phone.

[0198] S1412, the fault analysis APP on the mobile phone displays the second interface.

[0199] S1414, the fault analysis APP on the mobile phone responds to the user's second operation on the second interface and uploads the first gradient image of the currently identified abnormal image and the abnormal information of the abnormal image to the cloud server.

[0200] S1415, the repair module in the server retrieves the first gradient image and its anomaly information from the cloud server. Based on the anomaly information in the first image, the repair module can identify the cause of the phone's malfunction and send corresponding repair suggestions to the phone.

[0201] S1416, The repair module in the server sends repair suggestions to the mobile phone.

[0202] S1417, the phone displays a third screen, which includes repair suggestions.

[0203] This application provides an image-based device fault identification method that offers convenience to users. For example, when a user is browsing images in a gallery app or taking a picture, if they find a first image containing noise, they can trigger the method described above on their phone within the gallery app to achieve remote repair. Subsequently, if the user's phone experiences a hardware failure and they take it to a repair shop, the repair personnel can directly obtain the abnormal image and information uploaded by the user's phone from the server and provide repair services without needing to re-identify whether the phone is faulty.

[0204] For example, when a user takes their phone to a repair shop, they can trigger the method described earlier to identify abnormal images caused by the phone malfunction and the type of noise within those images. The user can then directly show the repair technician the identification results, which will be used to provide repair services. This eliminates the need for the technician to review the initial images on the user's phone, and also avoids the need for the user to describe the abnormal phenomena, thus preventing unclear descriptions. If the abnormal image needs to be exported, the technician can download the corresponding gradient image (an anonymized version) from the server, protecting the user's privacy.

[0205] This application provides an electronic device, comprising: a memory, a communication module, a display screen, and one or more processors. The communication module receives and transmits data under the control of the processors to enable communication between the electronic device and other electronic devices. The memory, display screen, and processors are coupled; wherein, the memory stores computer program code, including computer instructions; the computer instructions are stored in the aforementioned electronic device (e.g., ...). Figure 2 When the electronic device 200 is run, it causes the electronic device to perform the various functions or steps in the above method embodiments.

[0206] This application embodiment also provides a computer storage medium, which includes computer instructions, when the computer instructions are executed in the aforementioned electronic device (such as...). Figure 2 When the electronic device 200 shown is run, it causes the electronic device to perform the various functions or steps in the above method embodiments.

[0207] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps described in the above method embodiments.

[0208] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device, the microphone of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0210] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0211] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0213] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0214] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image-based device fault identification method, characterized in that, Applied to electronic devices, the method includes: The electronic device inputs a first image into a first neural network model to obtain recognition information for the first image. The recognition information includes the type of the first image, which indicates whether the first image is a normal image or an abnormal image. If the first image is an abnormal image, the recognition information also includes the noise type of the first image, which is at least one of M preset noise types. The noise of the M preset noise types appears in the first image due to a malfunction of the imaging device used to capture the first image. Here, M is an integer greater than or equal to 1. The first neural network model has the ability to output recognition information for the corresponding image based on the input image. If the first image is the abnormal image, the electronic device sends the first image and the abnormal information of the first image to the server; the abnormal information includes the noise type of the first image and the log information when the shooting device took the first image; the log information is used to locate the cause of the fault of the shooting device, and the log information includes the working status information of the radio frequency module in the shooting device and the call stack information of the camera APP in the electronic device when the shooting device took the first image.

2. The method according to claim 1, characterized in that, The shooting device is the electronic device.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The electronic device receives maintenance suggestions from the server; The electronic device displays the repair recommendations.

4. The method according to claim 1 or 2, characterized in that, Before the electronic device inputs the first image into the first neural network model, the method further includes: The electronic device displays a first interface, which includes a first button; The electronic device inputs the first image into the first neural network model, including: In response to a user's first operation on a first button on a first interface, the electronic device inputs a first image into a first neural network model.

5. The method according to claim 1 or 2, characterized in that, After the electronic device inputs the first image into the first neural network model, the method further includes: The electronic device displays a second interface, which includes the recognition information of the first image; if the first image is an abnormal image, the second interface includes a second button. The electronic device sends a first image and abnormal information about the first image to the server, including: In response to a second operation by the user on the second interface, the electronic device sends the first image and abnormal information of the first image to the server.

6. The method according to claim 3, characterized in that, The electronic device displays the repair suggestions by displaying a third interface, which includes the repair suggestions.

7. The method according to claim 1 or 2, characterized in that, The electronic device inputs the first image into the first neural network model, including: The electronic device differentiates the first image to obtain a first gradient image of the first image; the first gradient image includes key features of the first image, or the first gradient image includes key features of the first image and noise features corresponding to noise in the first image; wherein, the key features include the features of pixels in the first image whose changes between adjacent pixels are greater than a preset change threshold; The electronic device inputs the first gradient image of the first image into the first neural network model.

8. The method according to claim 7, characterized in that, The electronic device sends a first image and abnormal information about the first image to the server, including: The electronic device sends a first gradient image of the first image and abnormal information of the first image to the server.

9. The method according to claim 2, characterized in that, If the first image is the abnormal image, before the electronic device sends the first image and the abnormal information of the first image to the server, the method further includes: The electronic device reads the log information of the first image from the log information of the electronic device; or, The first image includes log information of the first image; the log information of the first image is acquired and stored in the first image when the electronic device takes the first image.

10. The method according to claim 1, characterized in that, Before the electronic device inputs the first image into the first neural network model, the method further includes: The electronic device obtains the latest model parameters of the first neural network model from the server; The electronic device updates the first neural network model using the latest model parameters.

11. The method according to claim 1, characterized in that, In the case where the first image is the anomalous image, the first image is used by the server as a negative sample to train the first neural network model; The method further includes: If the first image is the normal image, the electronic device sends the first image and the type of the first image to the server; in the case that the first image is the normal image, the first image is used by the server as a positive sample to train the first neural network model.

12. An image-based device fault identification method, characterized in that, Applied to a server, the method includes: The server receives a first image and abnormal information about the first image sent by the electronic device. The abnormal information includes the noise type of the first image and log information from the camera when it captured the first image. The log information is used to locate the cause of the camera malfunction. The log information includes the operating status information of the radio frequency module in the camera and the call stack information of the camera app in the electronic device when it captured the first image. The noise type is at least one of M preset noise types. The noise in the M preset noise types appears in the first image due to a malfunction of the camera that captured the first image. Where M is an integer greater than or equal to 1. The server sends maintenance suggestions to the electronic device based on the first image and the abnormal information in the first image.

13. The method according to claim 12, characterized in that, The method further includes: The server uses the first image and the noise type of the first image to train a first neural network model, so that the trained first neural network model has the ability to output the noise type of the corresponding image based on the input image.

14. The method according to claim 13, characterized in that, The method further includes: After the training, the server updates the model parameters of the first neural network model; The server sends the updated model parameters to the electronic device.

15. The method according to any one of claims 12-14, characterized in that, The first image is specifically the first gradient image of the first image; The first gradient image is obtained by differentiating the first image; the first gradient image includes key features of the first image, or the first gradient image includes key features of the first image and noise features corresponding to noise in the first image; wherein, the key features include the features of pixels in the first image whose changes in adjacent pixels are greater than a preset change threshold.

16. An image-based equipment fault identification system, characterized in that, The system includes an electronic device and a server; the electronic device is used to perform the method as described in any one of claims 1-11; the server is used to perform the method as described in any one of claims 12-15.

17. An electronic device, characterized in that, The electronic device includes: a memory, a display screen, one or more processors, and a communication module; the memory, the communication module, and the display screen are coupled to the processor; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-11.

18. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-11 or 12-15.

Images