Image-based device failure testing method and electronic device

Abstract

Embodiments of the present application relate to the field of device testing, and particularly relate to a device fault testing method based on images and an electronic device. A first electronic device includes N cameras, and a working frequency band of the first electronic device includes M radio frequency bands. The method includes: obtaining M*N groups of first images collected by the N cameras of the first electronic device when the first electronic device works in each of the M radio frequency bands; each of the M*N groups of first images includes at least one second image; inputting at least one second image in each of the M*N groups of first images into a first neural network model to obtain a group of identification information; wherein the group of identification information includes identification information of the at least one second image; the identification information includes a type of the second image, and the type of the second image indicates that the second image is a normal image or an abnormal image; and M*N groups of identification information corresponding to the M*N groups of first images are used to generate a test report.

Description

Technical Field

[0001] This application relates to the field of equipment testing, and more particularly to an image-based equipment fault testing method and electronic device. Background Technology

[0002] Modern mobile devices generally have photo and video recording functions, allowing users to capture pictures or videos and record their lives anytime, anywhere.

[0003] When a terminal device malfunctions, such as when the camera module within the device is subjected to electromagnetic interference, the device may experience image capture problems. Specifically, the device may capture abnormal images that include noise. For instance, if the image sensor in the camera module is subjected to electromagnetic interference from radio frequency signals within the terminal device, the image sensor may capture abnormal images that include noise.

[0004] Typically, the hardware and software modules of a terminal device can be tested in the factory before it leaves the factory to avoid the aforementioned problems. Conventional testing methods do not specifically test for abnormal images captured by the terminal device due to interference from the radio frequency system to the camera module, and therefore cannot effectively identify problems caused by this interference. Furthermore, conventional testing methods rely on human intervention to identify abnormal images, resulting in a high false positive rate. In short, conventional testing methods cannot effectively identify terminal devices with shooting malfunctions, and suffer from low accuracy in identifying faulty devices. Summary of the Invention

[0005] This application provides an image-based device fault testing method and electronic device. A first neural network model is used to identify whether the test images acquired by the device under test (DUT) include noisy or abnormal images, thereby identifying whether the DUT is a faulty device. Furthermore, the test images acquired by the DUT can cover all radio frequency bands and all cameras of the DUT. Fault testing is performed based on a large number of test images, and the accuracy of abnormal image recognition is improved, which helps to improve the accuracy of faulty device identification. To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an image-based device fault testing method is provided. The method includes: acquiring M*N sets of first images captured by N cameras of a first electronic device when the device operates in each of M radio frequency bands; each of the M*N sets of first images includes at least one second image; where M is an integer greater than or equal to 1; N is an integer greater than or equal to 1; inputting at least one second image from each of the M*N sets of first images into a first neural network model to obtain a set of recognition information; wherein the set of recognition information includes recognition information for at least one second image; the recognition information includes the type of the second image, which indicates whether the second image is a normal image or an abnormal image; wherein the second image is a normal image if it does not contain any of Q preset noise types; and an abnormal image if it contains at least one of Q preset noise types; wherein Q is an integer greater than or equal to 1; the first neural network model has the ability to output recognition information for the corresponding image based on the input image; and the M*N sets of recognition information corresponding to the M*N sets of first images are used to generate a test report.

[0007] This application uses a first neural network model to identify the type of at least one second image in each of the M*N groups of first images acquired by a first electronic device. Furthermore, the M*N groups of first images acquired by the first electronic device can cover all radio frequency bands and all cameras of the first electronic device, including a sufficient number of test samples. Fault testing is performed based on a large number of images, and the accuracy of abnormal image recognition is improved, thus contributing to the accuracy of fault device identification.

[0008] In one possible implementation of the first aspect, the M*N sets of first images are M*N sets of first videos, and the second images are image frames from the first videos. The first electronic device can traverse and activate each of the M radio frequency bands, and after activating each radio frequency band, use N cameras in the first electronic device to capture the first videos respectively, thus obtaining the M*N sets of first videos. The second images can be some or all of the image frames from the first videos.

[0009] In one possible implementation of the first aspect, if the second image is an anomalous image, the recognition information also includes the noise type of the second image, where the noise type is at least one of Q preset noise types included in the second image. That is, the first neural network model can output the type of the second image, which can be a normal image or an anomalous image. If the second image is an anomalous image, the first neural network model can also output the noise type of the second image. The noise type is at least one of Q preset noise types.

[0010] In one possible implementation of the first aspect, after obtaining the M*N group recognition information corresponding to the M*N group of first images, a test report can be obtained based on the M*N group recognition information corresponding to the M*N group of first images. Specifically, if the M*N group of first images does not contain any abnormal images, the test report indicates that the test passed and the first electronic device is not faulty. If the M*N group of first images contains at least one abnormal image, the test report indicates that the test failed and the first electronic device is faulty.

[0011] In one possible implementation of the first aspect, the test report indicates a test failure and also includes the radio frequency band in which the radio frequency system of the first electronic device operates when the first electronic device captures the abnormal image, and the camera identifier of the camera that captured the abnormal image in the second image. The radio frequency band and the camera identifier are used to pinpoint the cause of the abnormal image capture by the first electronic device.

[0012] In one possible implementation of the first aspect, the method further includes displaying a test report. For example, the first electronic device displays the test results. Alternatively, the first electronic device establishes a connection with a second electronic device, and the second electronic device displays the test report. The first or second electronic device displays the test report to promptly inform the tester of the test status.

[0013] In one possible implementation of the first aspect, the derivative of the second image can be calculated first to obtain a first gradient image of the second image. The first gradient image includes key features of the second image, or it includes key features of the second image and noise features corresponding to noise in the second image. The key features include pixels in the first image whose adjacent pixel changes are greater than a preset change threshold. Finally, the first gradient image is input into a first neural network model. Compared to the second image, the first gradient image includes fewer features; inputting the second image into the first neural network model helps improve the recognition performance.

[0014] In one possible implementation of the first aspect, a second image and its recognition information can also be sent to the server. Wherein, if the second image is an anomalous image, it is used by the server as a negative sample to train the first neural network model; if the second image is a normal image, it is used by the server as a positive sample to train the first neural network model.

[0015] In one possible implementation of the first aspect, a first gradient image and recognition information of the second image can be sent to the server. Since the first gradient image only includes key images and ignores areas in the second image where pixel values ​​change relatively smoothly, it can achieve a desensitization effect and protect privacy.

[0016] In one possible implementation of the first aspect, the server can continuously train the first neural network model. Before testing, the electronic device can send a download request to the server to request the download of the first neural network model. Based on the server's instructions, the first neural network model is downloaded and saved.

[0017] In a second aspect, an electronic device is provided, comprising: a memory, a display screen, one or more processors, and a communication module; the memory, communication module, and display screen are coupled to the processor; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method of any one of the first aspects.

[0018] Thirdly, this application provides a chip system applicable to electronic devices including memory. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The interface circuits are used to receive signals from the aforementioned memory and send the signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device performs the method as described in the first aspect and any of its possible design embodiments.

[0019] Fourthly, this application provides a computer-readable storage medium comprising computer instructions. When executed on an electronic device, the computer instructions cause the electronic device to perform methods as described in the first aspect and / or the second aspect and any possible design configuration thereof.

[0020] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method as described in the first aspect and any possible design of the first aspect.

[0021] Understandably, the beneficial effects that the electronic devices of the second aspect to the computer program products of the fifth aspect can achieve can be referenced to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the structure of a camera provided in an embodiment of this application;

[0023] Figure 2 This application provides a schematic diagram of the interaction process between an image sensor and a main control chip.

[0024] Figure 3 A schematic diagram illustrating various types of first images provided in embodiments of this application;

[0025] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0026] Figure 5 A system architecture diagram of an image-based device fault detection system provided in this application embodiment;

[0027] Figure 6 A schematic flowchart illustrating an image-based device fault detection method provided in an embodiment of this application;

[0028] Figure 7 A schematic flowchart of another image-based device fault detection method provided in an embodiment of this application;

[0029] Figure 8 A flowchart illustrating yet another image-based device fault detection method provided in this application embodiment;

[0030] Figure 9 A schematic diagram of the display interface of a test APP provided in an embodiment of this application;

[0031] Figure 10 A comparison image of images before and after desensitization provided for an embodiment of this application;

[0032] Figure 11 This is a schematic diagram of the interaction process between an electronic device and a server, provided as an embodiment of this application. Detailed Implementation

[0033] Modern mobile devices generally have photo and video recording functions, allowing users to capture pictures or videos and record their lives anytime, anywhere.

[0034] Terminal devices, such as mobile phones, may include at least one camera. For example, in response to a user tapping the shutter button on the camera app's screen, the camera can capture a raw image. This raw image can be a RAW image captured by the camera. Then, the image signal processor in the phone can process the RAW image to obtain a processed first image. Finally, the phone can display this first image in the camera app's preview interface. In this way, the user can preview the first image captured by the phone.

[0035] A camera may include a lens, a holder, an image sensor, a voice coil motor (VCM), and a flexible printed circuit (FPC). An image sensor, also known as a photosensitive device, is a device that converts optical images into electrical signals. For example, the surface of an image sensor contains hundreds of thousands to millions of photodiodes. When these photodiodes are illuminated, they generate light signals. The image sensor can then convert these light signals into electrical signals and output a RAW image in Bayer format. Some image sensors can also output RAW images in formats such as RGGB, RGBW, CMYK, RYYB, and CMY.

[0036] Figure 1 A schematic diagram of a camera structure is shown. Figure 1 As shown, the image sensor 101 can be soldered onto a printed circuit board (PCB). The PCB1 can be connected to the PCB2 via a flexible circuit board 102. A support 103 is also included around the image sensor, which can be a structural component. A lens 104 is also included above the image sensor, used to converge light. A voice coil motor 105 is used to control the focal length and focusing of the lens.

[0037] Electronic devices may also include an image signal processor (ISP), which is a special type of processor for processing image signals. Taking a Bayer-format RAW image output by an image sensor as an example, the ISP can perform image processing on the Bayer-format RAW image to obtain an image in YUV or RGB format. Image processing includes, but is not limited to, automatic exposure control, automatic gain control, automatic white balance, color correction, gamma correction, and dead pixel removal. The ISP can then transmit the processed image to the central processing unit (CPU). The CPU can store the image and / or display it. Optionally, the mobile phone may also include a digital signal processor (DSP), where the ISP can transmit the processed image to the DSP, which then transmits the image to the CPU.

[0038] The camera, ISP, and DSP can be collectively referred to as the camera module. When a mobile phone is taking a picture, the camera module can be interfered with by the phone's radio frequency system, leading to abnormal shooting conditions. For example, the phone may capture a first image that includes noise. For instance, when the image sensor is subjected to electromagnetic interference from the radio frequency module, it will capture an abnormal image that includes noise. For example, receiving an incoming call or downloading data in the background while the phone is taking a picture can both cause the phone to capture a first image that includes noise. The first image containing noise is an abnormal image, while the first image without noise is a normal image.

[0039] The main reason why the radio frequency system causes electromagnetic interference to the camera module is that the operating frequency of the radio frequency system in certain frequency bands is close to the frequency of the clock signal of the image sensor or the bus signal of the camera module, which causes signal interference, resulting in disorder of the clock signal of the image sensor or the bus signal of the camera module, and thus causing abnormal pixels in the first image.

[0040] For example, such as Figure 2 As shown, the image sensor and main control chip can control the transmission of image data through the MCLCK, PCLCK, VSYNC, and HSYNC clock signals. The MCLCK clock signal is the clock source for the image sensor. The PCLCK clock signal is the clock that controls pixel output, i.e., the pixel sampling clock; one pixel's data is acquired in one PCLCK clock cycle. The VSYNC clock signal is the vertical synchronization signal. The HSYNC clock signal is the horizontal synchronization signal.

[0041] The image sensor receives the MCLCK clock signal from the main control chip and uses it as its clock source. The sensor can calculate the sampling clock PCLCK based on the MCLCK signal. Then, the sensor can use the VSYNC and HSYNC clock signals to output a RAW image. For example, with VSYNC active high, all image data output during the period from high to low constitutes one frame or one image. During the active period of the HSYNC clock signal, the image data output by the sensor is in the same row. The sensor can use a Mobile Industry Processor Interface (MIPI) to output both the VSYNC and HSYNC clock signals. Figure 2 The main control chip can be a baseband chip or a central processing unit.

[0042] Figure 2 This demonstrates the process by which the sensor converts photoelectric signals based on the MCLCK clock signal of the main control chip. During this process, the radio frequency signal can interfere with functions such as... Figure 2 The reception or transmission of at least one of the clock signals MCLCK, PCLCK, VSYNC, and HSYNC causes the sensor output to include a noisy RAW image, which in turn causes the first image finally displayed by the mobile phone to also include noise.

[0043] Noise refers to unnecessary or redundant interference information in the first image. Noise can include stripe noise, dot noise, and mosaic noise. The noise appears randomly at any location in one or more of the first images.

[0044] A first image including stripe noise may include colored stripes or black and white stripes, which may be straight lines or curves. The stripes may be displayed in an array across all or part of the first image, or they may be displayed randomly in a portion of the first image. Dots may be colored dots or black and white dots. A first image including dot noise may include colored, black, or white dots, which may be displayed in an array across all or part of the first image, or they may be displayed randomly in a portion of the first image. In a first image including mosaic noise, mosaics may be displayed across all or part of the first image. For example, as... Figure 3 As shown. Figure 3 Image a shows a normal first image, a first image including stripe noise, a first image including dot noise, a first image including mosaic noise, and another first image including stripe noise. Figure 3 Image b shows the normal first image and the abnormal image with radio frequency signal interference, including stripe noise.

[0045] Typically, before a mobile phone leaves the factory, testers perform hardware and software tests to select phones that meet factory quality requirements. For example, a mobile phone may include multiple hardware and software modules, such as a display module, camera module, Bluetooth module, Wi-Fi module, speaker module, battery module, video module, and radio frequency (RF) system. The screen module, depending on its function, may include a fingerprint module, button module, etc. Before leaving the factory, testers test each hardware and software module. For example, they test the RF system to check for malfunctions. They also test the camera module to check for malfunctions in its various shooting functions. Conventional testing methods involve testing the RF system and camera module separately. For example, the RF system might be tested according to specifications first, followed by the camera module. However, conventional testing methods do not specifically test for interference from the RF system on the camera module that can cause shooting malfunctions, and therefore cannot effectively identify problems caused by this interference.

[0046] Furthermore, during routine testing of camera modules, testers manually identify whether the images captured by the phone are abnormal images containing noise. For example, testers prepare test images in advance, such as... Figure 3 As shown in 'c', each pixel in the test image is a black dot. The tester prints the test image as a physical file and places it at the test location. The phone includes this test image. Then, the tester uses the phone under test to capture the test image. Finally, the tester compares the captured image with the test image on the phone under test, primarily comparing whether the image captured by the phone under test includes noise compared to the test image. For example, the tester checks each black dot in the captured image to see if it matches the corresponding black dot in the test image. If the image captured by the phone under test matches the test image, the phone under test is considered to be fault-free. If the image captured by the phone under test does not match the test image, such as if the image captured by the phone under test includes at least one of the following noise types: stripe noise, dot noise, or mosaic noise, the phone under test is considered faulty and requires further analysis and repair. Figure 3 As shown in c, the black dots at some locations in the captured image are different from the black dots at the same locations in the test image.

[0047] Furthermore, some noise is quite obvious in the image, allowing testers to visually identify it. Other noise is more subtle and may be difficult for testers to detect, potentially leading to overlooked phone malfunctions and posing a risk to users. Additionally, interference from the radio frequency (RF) system with the camera module causing abnormal camera shooting is a random occurrence, and conventional testing methods lack a comprehensive testing mechanism, potentially failing to detect whether the phone is experiencing RF system interference leading to abnormal shooting.

[0048] Therefore, this application provides an image-based device fault testing method. First, M*N sets of first images are acquired by N cameras of a first electronic device when it operates in each of M radio frequency bands. The first electronic device supports M radio frequency bands and includes N cameras. The first electronic device can traverse and activate each of the M radio frequency bands, and after activating each radio frequency band, it uses the N cameras to acquire first images respectively. In this way, M*N sets of first images can be obtained by each of the N cameras when the first electronic device operates sequentially in the M radio frequency bands. Each of the M*N sets of first images includes at least one second image.

[0049] Next, a first neural network model is used to identify the type of at least one second image in each of the M*N groups of first images. Image types include normal images and abnormal images. A second image is considered normal if it does not contain any of the Q preset noise types. A second image is considered abnormal if it contains at least one of the Q preset noise types. This yields M*N groups of identification information, each group including the type of at least one corresponding second image. This M*N group of identification information can be used to generate a test report.

[0050] As can be seen, this solution uses a first neural network model to identify whether the M*N sets of first images acquired by the first electronic device include noisy or abnormal images, thereby identifying whether the first electronic device is a faulty device. Furthermore, the M*N sets of first images acquired by the first electronic device can cover all radio frequency bands and all cameras in which the first electronic device operates. Fault testing based on a large number of images improves the accuracy of abnormal image recognition, thus contributing to the accuracy of faulty device identification.

[0051] The method provided in this application can be applied to electronic devices with data processing capabilities. The aforementioned first electronic device may include servers, mobile phones, tablets, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smartwatches, smart bracelets, etc.), in-vehicle devices, virtual reality devices, etc., and this application does not impose any limitations on these. In this application embodiment, the aforementioned electronic device is an electronic device capable of running an operating system and installing applications. Optionally, the operating system running on the electronic device may be... system, system, Systems, etc.

[0052] For example, please refer to Figure 4 The diagram illustrates the structure of an electronic device 400. The electronic device 400 may include a processor 410, an external memory interface 420, an internal memory 421, an audio module 430, a speaker 430A, a microphone 430B, a display screen 440, a communication module 450, a power module 460, an input device 470, a sensor module 480, a camera 490, etc. The sensor module 480 may include a pressure sensor, a touch sensor, etc.

[0053] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 400. In other embodiments of this application, the electronic device 400 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0054] Processor 410 may include one or more processing units. For example, processor 410 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, electronic device 400 may also include one or more processors 410.

[0055] The processor 410 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. This memory can store instructions or data that the processor 410 has just used or that are used repeatedly. If the processor 410 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 410, and thus improves the efficiency of the system.

[0056] The external storage interface 420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 400. The external memory card communicates with the processor 410 through the external storage interface 420 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0057] Internal memory 421 can be used to store one or more computer programs, which include instructions. Processor 410 can execute the instructions stored in internal memory 421, thereby causing electronic device 400 to perform application running methods, as well as various applications and data management, as provided in some embodiments of this application. In some embodiments, processor 410 can execute instructions stored in internal memory 421 and / or instructions stored in memory disposed in processor 410 to cause electronic device 400 to perform application running methods, as well as other applications and data management, as provided in embodiments of this application.

[0058] Electronic device 400 can implement audio functions through audio module 430, speaker 430A, microphone 430B, and application processor, such as music playback and recording. Speaker 430A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals.

[0059] Microphone 430B, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. Users can speak by bringing their mouth close to microphone 430B to input sound signals into it.

[0060] The communication function of electronic device 400 can be realized through antenna 1, antenna 2 and communication module 450, etc.

[0061] Communication module 450 can provide solutions for wireless communication applications on electronic device 400, including cellular, Wi-Fi, Bluetooth, and wireless data transmission modules (e.g., 433MHz, 868MHz, 915MHz). Communication module 450 can be one or more devices integrating at least one communication processing module. Communication module 450 receives electromagnetic waves via antenna 1 or antenna 2, filters and frequency-modulates the electromagnetic wave signals, and sends the processed signal to processor 410. Communication module 450 can also receive signals to be transmitted from processor 410, frequency-modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 1 or antenna 2.

[0062] Electronic device 400 implements display functions through a GPU, a display screen 440, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 440 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 410 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0063] Display screen 440 is used to display images, videos, etc. Display screen 440 includes a display panel. In some embodiments, electronic device 400 may include one or N displays screens 440, where N is a positive integer greater than 1. In this embodiment, display screen 440 can be used to display a user interface (UI) and receive user operations on the UI.

[0064] The power module 460 can be used to supply power to the various components included in the electronic device 400. In some embodiments, the power module 460 can be a battery, such as a rechargeable battery.

[0065] Input device 470 may include a keyboard, mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc. into electronic device 400, thereby issuing commands to electronic device 400 and inputting data.

[0066] Electronic device 400 can implement shooting functions through an ISP, camera 490, video codec, GPU, display 440, and application processor. The ISP is used to process data fed back by the camera 490. The camera 490 is used to capture still images or videos. In some embodiments, electronic device 400 may include one or N cameras 490, where N is a positive integer greater than 1. The digital signal processor is used to process digital signals, including digital image signals and other digital signals. For example, when electronic device 400 selects a frequency, the digital signal processor performs Fourier transforms on the frequency energy. The video codec is used to compress or decompress digital video.

[0067] Taking the aforementioned electronic device 400 as an example, which is a mobile phone, the software system of the electronic device 400 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 400.

[0068] Figure 5 This is a system architecture diagram of an image-based device fault testing system provided in an embodiment of this application. The system architecture diagram includes a software structure block diagram of the electronic device 400.

[0069] For example, the server includes an image management module, a model training module, a model conversion and integration module, and a transmission module 2.

[0070] The image management module is used to construct training samples for training a first neural network model. The server may include the first neural network model, and the server trains the first neural network model using the training samples. The training samples include pre-configured training samples within the service, and images uploaded by the electronic device 400. These images can be second images or first gradient images uploaded by the electronic device. The images can be normal images or abnormal images. For example, the pre-configured training samples within the service can be images with noise such as stripes, dots, and mosaics, captured by the manufacturer of the electronic device 400 when the electronic device 400 is abnormally subjected to radio frequency interference and bus interference. The image management module can also identify and classify the images uploaded by the electronic device 400, for example, classifying and storing normal images and abnormal images. Furthermore, for abnormal images, the images can be classified and stored according to the noise type they contain.

[0071] The model training module is used to construct and train the first neural network model. Specifically, the model training module can construct the first neural network model, automatically iterate the first neural network model using training samples, and save the first neural network model. Model parameters can include the number of nodes and node weights. The model training module can periodically and automatically train the first neural network model, gradually improving the accuracy of the first neural network model as the number of training samples increases.

[0072] The model conversion and integration module can convert the first neural network into a preset format, which can be an integrated and recognizable format by the electronic device 400. For example, the format of the first neural network model before conversion can be an .H5 file, and the format after conversion can be a .tflite file.

[0073] The layered architecture of the electronic device 400 divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime, the system libraries, and the kernel layer.

[0074] The electronic device 400 may include a fault testing application (APP), an image recognition module, a testing module, a report generation module, a transmission module 1, and a memory. The testing APP is contained in the application layer of the electronic device 400. The image recognition module, the testing module, and the report generation module are contained in the application framework layer of the electronic device 400. The transmission module 1 and the memory are hardware components of the electronic device 400.

[0075] The test app includes a user interface. It can respond to various user actions on the UI, enabling secondary image recognition and test report display.

[0076] The test module is used to activate each of the M radio frequency bands and call N cameras to acquire M*N sets of second images.

[0077] The image recognition module is used to identify the type of the second image in the M*N group of second images. Specifically, the image recognition module can initialize the first neural network model, and the initialization process may include loading model parameters. The image recognition model can also preprocess the second image, including image differentiation. The image recognition module can use the second image or the first gradient image as input to the first neural network model, run the first neural network model, and obtain the recognition result of the second image.

[0078] The report generation module can generate a test report based on the M*N sets of identification information output by the first neural network model.

[0079] like Figure 5 As shown, the application framework layer may also include an input system (not shown), a view system (not shown), a camera service, and a multimedia service.

[0080] In this embodiment, the input system is used to drive the touch screen of the monitoring electronic device 400 through the display screen, and convert the touch parameters generated by the touch operation input by the touch screen into usable events and pass them to the upper-layer test APP.

[0081] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build the display interface of an application. The display interface can consist of one or more views.

[0082] The camera service provides interfaces that can be called by upper-layer applications (such as camera apps), enabling image capture and preview based on these applications' calls. The multimedia service provides interfaces that can be called by upper-layer applications (such as gallery apps), enabling video or audio playback based on these applications' calls.

[0083] The Android Runtime consists of the core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0084] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0085] The kernel layer is the layer between hardware and software. The kernel layer can contain display drivers, audio drivers, sensor drivers, camera drivers, touchscreen drivers, etc.

[0086] The hardware includes a camera, a display screen, a memory, and a transmission module 1. The memory is used to store images and videos taken by the user. The memory can also be used to store the first gradient image of the preprocessed second image and the test report generated by the report generation module.

[0087] Transmission module 1 is used to send data to or receive data sent by transmission module 2 on the server. For example, transmission module 1 can download a first neural network model from the server. Transmission module 1 is also used to send a second image or a first gradient image and / or a test report for this test to the server.

[0088] The following is combined with Figure 6 a and Figure 6Taking the example of a test version mobile phone (hereinafter referred to as a mobile phone) as the first electronic device, b in this application, we introduce an image-based device fault testing method provided by the embodiments of this application.

[0089] S601, the mobile phone acquires M*N sets of first images captured by N cameras when the mobile phone is operating in each of the M radio frequency bands.

[0090] The mobile phone can be fixed in a preset test position. Then, in response to a trigger by the tester, the phone can acquire M*N sets of first images. For example, the phone can be fixed to a stand or to a test device. With the phone fixed in the preset test position, its N cameras can be pointed at a preset object, which can be a preset photographed object or a preset test image. In this way, the captured image and background are uniform in each of the M*N sets of first images, which helps improve the recognition efficiency of the first images.

[0091] The mobile phone's response to the tester's trigger includes the phone responding to a first operation performed by the tester on the phone itself, or a first operation performed by the tester on a second electronic device. Specifically, for example, the phone may include a test app. In response to the tester clicking the test app's icon on the phone, the phone executes S601. As another example, the second electronic device could be a test computer. The phone can establish a connection with the test computer, for example, by establishing a universal serial bus (USB) connection. The first operation could be the tester entering a run command on the test computer. Figure 6 As shown in b, in response to the first operation, the test computer sends a test command to the mobile phone. In response to the test command, the mobile phone executes S601.

[0092] The mobile phone supports M radio frequency bands. A radio frequency band can be the frequency range of electromagnetic waves emitted by the radio frequency module, measured in Hz. The mobile phone may include multiple radio frequency bands for supporting cellular network communication, multiple radio frequency bands for supporting short-range wireless communication, and multiple radio frequency bands for supporting satellite positioning and navigation.

[0093] The various radio frequency bands used to support cellular network communication include those of different network operators. For example, the mobile phone supports one or more of the following radio frequency bands: Global System for Mobile Communications (GSM), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), and 5G New Radio (5GNR). For instance, the mobile phone supports the n41 and n79 radio frequency bands in Operator 1's 5G NR. The n41 radio frequency band has a frequency range of 2515MHz-2675MHz, and the n79 radio frequency band has a frequency range of 4800MHz-4900MHz.

[0094] Multiple radio frequency bands are used to support near-field wireless communication, including those supporting wireless local area network (WLAN) communication, Bluetooth communication, near-field communication (NFC), and ultra-wideband (UWB) communication for precise wireless positioning and sensing. For example, WLAN communication bands can be 2.4 GHz and 5 GHz, Bluetooth communication bands can be 2.4 GHz, and UWB communication bands can range from 3.1 GHz to 10.6 GHz.

[0095] A mobile phone can include N cameras. These cameras can be front-facing, rear-facing, or side-facing cameras. Cameras can also be any of the following: infrared cameras, visible light cameras, ultraviolet cameras, etc. Cameras can also be single or dual cameras.

[0096] The M radio frequency bands can be radio frequency band 1, radio frequency band 2, radio frequency band 3, ..., radio frequency band M, where M is an integer greater than or equal to 1. The N cameras can be camera 1, camera 2, camera 3, ..., camera N, where N is an integer greater than or equal to 1.

[0097] For example, such as Figure 7As shown, a mobile phone can sequentially capture N sets of images using each of N cameras in each of M radio frequency bands, resulting in M*N sets of images. Specifically, in one radio frequency band, one camera captures one set of images, and N cameras capture N sets of images. A total of M*N sets of images can be captured across the M radio frequency bands. For example, in frequency band 1, the mobile phone uses N cameras to capture N sets of first images. These N sets of first images can include the first image captured by camera 1 in radio frequency band 1, the first image captured by camera 2 in radio frequency band 1, the first image captured by camera 3 in radio frequency band 1, ..., the first image captured by camera N in radio frequency band 1. Similarly, the N sets of first images captured by the mobile phone in radio frequency band 2 can include the first image captured by camera 1 in radio frequency band 2, the first image captured by camera 2 in radio frequency band 2, the first image captured by camera 3 in radio frequency band 2, ..., the first image captured by camera N in radio frequency band 2. Similarly, the N sets of first images collected by the mobile phone in the radio frequency band M may include the first image collected by camera 1 in the radio frequency band M, the first image collected by camera 2 in the radio frequency band M, the first image collected by camera 3 in the radio frequency band M, ..., the first image collected by camera N in the radio frequency band M.

[0098] Taking M=2 as an example, the mobile phone can operate on either the n41 or n79 RF bands in 5G NR. The frequency range of the n41 RF band is 2515MHz-2675MHz, and the frequency range of the n79 RF band is 4800MHz-4900MHz.

[0099] The phone first activates one of the two radio frequency bands, such as the n41 radio frequency band, and operates within that band. This operation can be understood as the phone transmitting radio frequency signals in that band. Simultaneously, the phone acquires N sets of initial images captured by N cameras while operating in that band. Then, the phone activates the other of the two radio frequency bands, such as the n79 radio frequency band, and operates within that band. Again, the phone acquires N sets of initial images captured by the N cameras while operating in that band. For example, as shown... Figure 8 As shown, S601 may include S801-S804.

[0100] S801, the mobile phone activates the n41 radio frequency band and transmits a first radio frequency signal, the transmission frequency of the first radio frequency signal being within the n41 radio frequency band.

[0101] After the phone activates the n41 radio frequency band, it can send a first radio frequency signal. The frequency of the first radio frequency signal is within the n41 radio frequency band, specifically between 2515MHz and 2675MHz. The first radio frequency signal can be pre-configured in the phone.

[0102] Mobile phones may include a radio frequency (RF) system. The RF system includes, but is not limited to, baseband circuits and RF circuits. The baseband circuits include baseband chips. The RF circuits include, but are not limited to, RF transceiver chips, power amplifiers, low-noise amplifiers, RF switches, and antennas.

[0103] After the phone activates the n41 radio frequency band and transmits the first radio frequency signal, it sequentially activates N cameras to capture N sets of first images. Specifically, the phone can execute S802.

[0104] S802, the mobile phone operates in the n41 radio frequency band, turns on the first camera to capture the first image a, turns on the second camera to capture the first image b, ..., turns on the Nth camera to capture the first image n.

[0105] Afterwards, the phone can continue to activate the n79 radio frequency band and use N cameras to capture N sets of initial images. The phone can then execute S803.

[0106] S803: The mobile phone activates the n79 radio frequency band and transmits a second radio frequency signal. The transmission power of the second radio frequency signal is within the n79 radio frequency band.

[0107] After the phone activates the n79 radio frequency band, it can transmit a second radio frequency signal. The frequency of the second radio frequency signal is within the n79 radio frequency band, specifically between 4800MHz and 4900MHz. This second radio frequency signal can be pre-configured in the phone. The content of the first radio frequency signal and the second radio frequency signal can be the same or different. Then, the phone sequentially activates N cameras to capture N sets of first images. Specifically, the phone can execute S804.

[0108] S804, the mobile phone operates in the n79 radio frequency band, turns on the first camera to capture the first image A, turns on the second camera to capture the first image B, ..., turns on the Nth camera to capture the first image N.

[0109] In this way, the mobile phone acquires 2N sets of the first image.

[0110] Optionally, when the mobile phone transmits the first radio frequency signal and the second radio frequency signal, it can use the maximum transmission power to transmit both signals. The maximum transmission power of the mobile phone transmitting the first radio frequency signal can be the same as or different from the maximum transmission power of the mobile phone transmitting the second radio frequency signal.

[0111] Each of the M*N groups of first images can be a video (first video), for example, a first video of a preset duration captured by a camera. The first video includes multiple second images, which are image frames from the first video. Each of the M*N groups of first images can also include one or more photographs (second images), for example, a group of first images can be multiple photographs captured by the camera within a preset duration. The preset duration can be 5 seconds or 10 seconds.

[0112] The first set of M*N images is used to identify whether the mobile phone is faulty. Faults include electromagnetic interference, hardware malfunctions, and software malfunctions. A first neural network model identifies whether any of the M*N first images contain noisy abnormal images. If any of the M*N first images contain noisy abnormal images, the phone is faulty. If none of the M*N first images contain noisy abnormal images, the phone is not faulty.

[0113] The first neural network model can be configured in a mobile phone, which can run the model to identify whether there are noisy or abnormal images in the M*N group of first images. Alternatively, the first neural network model can be configured in a test computer. The mobile phone can send the M*N group of first images to the test computer, which can then run the model to identify whether there are noisy or abnormal images in the group.

[0114] Optionally, the first neural network model can also be deployed on a server. If the first neural network is deployed on a server, the mobile phone can send M*N sets of first images to the server. The server then runs the first neural network model to identify whether there are any abnormal images, including noisy ones, in the M*N sets of first images.

[0115] The following section uses a mobile phone equipped with a first neural network model and running that model as an example to introduce this solution.

[0116] S602, the mobile phone sequentially inputs at least one second image from each of the M*N groups of first images into the first neural network model to obtain M*N groups of recognition information.

[0117] For each of the M*N groups of first images, the mobile phone can identify a portion of the second images within each group, or it can identify all the second images within each group. For example, if a group of first images is a single image, the mobile phone can identify all the second images within that group. If a group of first images includes multiple first images, the mobile phone can identify all or a portion of the second images within that group. Furthermore, if a group of first images includes multiple second images, the mobile phone can identify a portion of the second images according to a preset ratio. If a group of first images is a video, the mobile phone can identify all or a portion of the image frames, or all image frames. Further, the mobile phone can identify a portion of the image frames according to a preset ratio. Recognizing only a portion of the second images within a group of first images improves testing efficiency. Recognizing all the second images within a group of first images improves testing accuracy.

[0118] The mobile phone inputs at least one second image from a set of first images into a first neural network model to obtain a set of recognition information. This set of recognition information includes the recognition information of at least one second image from the set of first images. In this way, the mobile phone can obtain M*N sets of recognition information.

[0119] Each set of identification information may include a type corresponding to at least one second image. The type of the second image may be a normal image. Alternatively, the type of the second image may be an abnormal image. For example, if the second image includes at least one of Q preset noise types, then the second image is an abnormal image. Here, Q is an integer greater than or equal to 1. For example, the Q preset noise types may be stripe noise, dot noise, and mosaic noise. For example, if the second image does not include any of the Q preset noise types, the second image is a normal image. If the second image includes at least one of the Q preset noise types, the second image is an abnormal image. The second image may be an abnormal image including stripe noise, an abnormal image including dot noise, or an abnormal image including mosaic noise. Alternatively, the second image may be an abnormal image that includes both stripe noise and dot noise.

[0120] The Q preset noise types can be caused by phone malfunctions appearing in the first image. Phone malfunctions include radio frequency signal interference, hardware failures, and software failures. Hardware failures can be, for example, component failures, such as camera hardware failures. Software failures can be, for example, missing software data in the camera app, malfunctioning code in the camera app, or bugs in the software system.

[0121] The first neural network has the ability to output the type of the second image based on the input second image. For example, the first neural network model can be trained using a normal image without any noise, and an abnormal image containing one or more types of noise such as stripes, dots, and mosaics caused by a mobile phone malfunction. For a second image without any noise, the trained first neural network model has the ability to output that the type of the second image is a normal image. For a second image containing one or more types of noise such as stripes, dots, and mosaics caused by a mobile phone malfunction, the trained first neural network model has the ability to output that the type of the second image is an abnormal image.

[0122] Specifically, the first neural network model can calculate the probability value P that the second image belongs to a normal image. If the probability value P is greater than or equal to a preset threshold, the second image is a normal image; if the probability value P is less than the preset threshold, the second image is an abnormal image. For example, if the first neural network model calculates the probability value P = 0.6 for the second image to be a normal image, and the preset threshold is 0.5, the first neural network model will output that the type of the second image is a normal image.

[0123] Furthermore, if the second image is an abnormal image, the recognition information may also include the noise type of the second image. This noise type is at least one of Q preset noise types caused by a mobile phone malfunction. That is, the first neural network model can not only output that the second image is a normal image or an abnormal image, but it can also output the noise type included in the abnormal image. This noise type is at least one of M preset noise types.

[0124] As one possible implementation, the first neural network model can calculate the probability value of each of the Q preset noises caused by the phone malfunction in the second image, thus obtaining Q probability values. For each of the Q probability values, if the Q probability values ​​include a probability value that meets a preset condition, then the second image includes the noise corresponding to that probability value. The first neural network model outputs the noise type of the second image as the noise type corresponding to the probability value that meets the preset condition. If the Q probability values ​​include a probability value that does not meet the preset condition, then the second image is a normal image. The preset condition can be that the probability value is the largest among the Q probability values ​​and is greater than a preset threshold. Alternatively, the preset condition can be that the probability value is greater than the preset threshold.

[0125] For example, the first neural network model can output probability values ​​for the second image including stripe noise, mosaic noise, and dot noise caused by a mobile phone malfunction. For instance, the probability values ​​might be 0.05, 0.6, and 0.35, respectively. If the probability value of the second image including mosaic noise is the maximum probability value and greater than a preset threshold of 0.5, then the second image is an abnormal image including mosaic noise, and the first neural network model outputs that the noise type of the second image is mosaic noise. For example, if the probability values ​​of both mosaic noise and dot noise in the second image are greater than the preset threshold of 0.3, then the noise type of the second image is mosaic noise and dot noise. If the probability values ​​of the second image belonging to stripe noise, mosaic noise, and dot noise are all less than the preset threshold of 5, then the second image is a normal image. For example, if the probability values ​​of the second image including stripe noise, mosaic noise, and dot noise are 0.35, 0.35, and 0.3, respectively, and the preset threshold can be 0.4, then since these three probability values ​​do not include probability values ​​greater than the preset threshold, the type of the second image can be a normal image.

[0126] As another possible implementation, the first neural network model includes multiple sub-neural network sub-models, with a predefined noise type corresponding to one sub-neural network model. For example, neural network model one is used to identify whether the second image includes stripe noise caused by a mobile phone malfunction; neural network model two is used to identify whether the second image includes mosaic noise caused by a mobile phone malfunction; and neural network model three is used to identify whether the second image includes dot noise caused by a mobile phone malfunction. For instance, the second image is input into neural network model one, which outputs a probability value one indicating that the second image includes stripe noise caused by a mobile phone malfunction. If probability value one is greater than or equal to a predefined threshold one, the second image is considered to include stripe noise caused by radio frequency signal interference, and the second image is considered an abnormal image with stripe noise as its noise type. The second image is then input into neural network model two, yielding a probability value two indicating that the second image includes mosaic noise caused by a mobile phone malfunction. If probability value two is greater than or equal to a predefined threshold two, the second image is considered to include mosaic noise caused by a mobile phone malfunction, and the second image is considered an abnormal image with mosaic noise as its noise type. The second image is input into neural network model three to obtain a probability value three indicating that the second image includes dot matrix noise caused by mobile phone malfunction. If probability value three is greater than or equal to a preset threshold three, the second image is considered to have dot matrix noise caused by mobile phone malfunction, and is therefore considered an abnormal image with dot matrix noise as its noise type. The preset thresholds one, two, and three can be equal, for example, all of which can be 0.5. If probability value one is less than preset threshold one, probability value two is less than preset threshold two, and probability value three is less than preset threshold three, then the second image is considered a normal image, meaning it does not include any of the preset noise types in Q caused by mobile phone malfunction.

[0127] Afterwards, the mobile phone can generate a test report based on the M*N sets of identification information. If the first image in the M*N set does not contain any abnormal images, the test report indicates that the test passed and the phone is not faulty. If the first image in the M*N set does contain an abnormal image, the test report indicates that the test failed and the phone is faulty.

[0128] For example, the mobile phone can generate a test report on the mobile phone side based on M*N sets of identification information. Alternatively, the mobile phone can send the M*N sets of identification information to a test computer, which will then generate a test report, from which the mobile phone can retrieve the test report.

[0129] Optionally, if the test computer executes S602, it generates a test report based on the M*N groups of identification information. The test computer can then send the test report to the mobile phone.

[0130] Optionally, if the server executes S602, it can generate a test report based on M*N sets of identification information. The server can then send the test report to the mobile phone. The following example demonstrates generating a test report on the mobile phone side.

[0131] S603, generate test report on mobile phone.

[0132] The test report may include the test time, mobile phone identification, test results, and the total number of second images used for the test. The test result may be either a pass or a failure. If the test fails, the test report may include information such as the number of abnormal images out of the total number of second images used for the test, and the type of noise in the abnormal images.

[0133] For each abnormal image, the test report may also include information such as the radio frequency band in which the phone's radio frequency system operated when capturing the abnormal image, and the identifier of the camera that captured the abnormal image. As one possible implementation, the first image includes image data and file data, where the file data stores shooting parameter information. When the phone captures the first image in each set of first images, it can add the camera identifier of the camera that captured the first image and the preset radio frequency band information of the phone to the file data of the first image. That is, the first image carries the camera identifier of the camera that captured the first image and the preset radio frequency band information of the phone. If the second image is an abnormal image, the phone can directly read the above information and include it in the test report.

[0134] Optionally, after S603, the mobile phone or test computer can display the test report. When displaying the test report, the mobile phone or test computer can show part of the test report or show all of the test report. For example, the mobile phone or test computer can only display the test results. For example, the mobile phone can execute S604.

[0135] S604, the phone displays the test report.

[0136] The mobile phone can display the test report on the first screen. This first screen can be an interface that automatically pops up on the phone after the test is completed. Optionally, the first screen can be the interface of the testing app. In some embodiments, the mobile phone may not display the test report but instead send it to the testing computer for display.

[0137] If the phone displays only part of the test report, it can also respond to the tester's instructions to display the entire test report. For example, ... Figure 9 As shown, the phone displays the main interface, which includes the application icon for the test app. In response to a tester clicking the test app's icon on the main interface, the phone executes... Figure 6 In the S601-S603 configuration, the phone can display interface 901. Interface 901 can display "Testing in progress." After the test is completed, the phone can display interface 902 or interface 903. Interface 902 includes the test result, which can be "Test passed." Interface 903 includes the test result, which can be "Test failed." Interfaces 902 and / or 903 can include a first button, which triggers the phone to display a test report. This first button could be, for example, a "View Report" button. Figure 9 As shown, in response to the tester clicking the first button on interface 903, the phone displays interface 904. Interface 904 displays information such as the abnormal second image, the noise type of the second image, the radio frequency band in which the radio frequency system was operating when the phone captured the second image, and the camera identification for capturing the second image in a list format.

[0138] Optional, such as Figure 6 As shown in b, after S603, the test computer can display the test report.

[0139] S605, the mobile phone sends a test report to the test computer.

[0140] S606: In response to receiving a test report, the test computer displays the test report.

[0141] The test report is displayed on the mobile phone or test computer, allowing testers to intuitively see the test results. If the test passes, testers can continue to perform other tests on the phone. If the phone test fails, for each abnormal image, testers can further analyze whether there are design or assembly problems in the RF system based on information such as the RF frequency band in which the phone captured the abnormal image and the camera identification of the camera that captured the abnormal image.

[0142] For example, there is a correspondence between the Q radio frequency bands in a mobile phone and the P antennas in the phone. For an abnormal image, based on the radio frequency bands the phone's RF system operated at when capturing the abnormal image, the antennas that operated at that time can be determined. Then, it's possible to detect whether there are insufficient anti-interference issues between the antennas and the camera that captured the abnormal image. Alternatively, testers can further analyze whether there are hardware faults in the camera module that captured the abnormal image and / or whether there are software abnormalities in the phone when capturing the abnormal image.

[0143] As can be seen, in the image-based device fault testing method provided in this application embodiment, the mobile phone can traverse and activate each of the M radio frequency bands, and after activating each radio frequency band, the N cameras in the mobile phone respectively capture a first image. In this way, the mobile phone obtains M*N sets of first images captured by each of the N cameras when the mobile phone operates sequentially in the M radio frequency bands. This allows for the acquisition of a sufficient number of first images, covering all radio frequency bands and all cameras in the mobile phone, creating a test environment that conforms to the entire fault scenario. Then, for some or all of the second images in the M*N sets of first images, a first neural network model is used to identify whether the second images are abnormal images containing noise, improving the identification accuracy. Therefore, this solution can effectively identify mobile phones with shooting faults, improving the quality of the mobile phone.

[0144] Optionally, a first neural network model can be deployed on the server, and the server can train the first neural network model. Before testing, the mobile phone or test computer can obtain the first neural network model from the server. After testing, the mobile phone or test computer can upload all or part of the second images from the M*N sets of first images used for testing to the server as training samples, continuously improving the detection accuracy of the first neural network model. In this way, a more accurate first neural network model can be obtained when testing the next mobile phone. For example, the mobile phone or test computer can upload normal images from the M*N sets of second images to the server, and these normal images can be used as positive samples. The mobile phone or test computer can also upload abnormal images from the M*N sets of first images to the server, and these abnormal images can be used as negative samples. Furthermore, since positive samples are abundant, the mobile phone or test computer can also upload only the second images of the abnormal image type.

[0145] Optionally, when uploading the second image, the mobile phone or test computer can specify the type of the second image. If the second image is an abnormal image, the mobile phone or test computer can also upload the noise type of the second image.

[0146] Furthermore, when uploading a second image (which can be normal or abnormal), the mobile phone or test computer can perform desensitization processing on the sensitive content in the second image. Desensitization processing removes sensitive content from the second image. Sensitive content can include faces, sensitive text, and sensitive patterns. Sensitive text can include factory information and brand information. Desensitization processing includes blurring, mosaicking, and replacing sensitive content in the image. The following section uses uploading a second image via a mobile phone as an example to introduce one desensitization process.

[0147] For example, the mobile phone can first identify sensitive content in the second image. For instance, the phone can identify whether the second image includes a face, preset sensitive text, preset sensitive patterns, etc. For example, the phone can identify sensitive regions in the second image based on a second neural network model. Sensitive regions are areas containing sensitive content. The second neural network model has the ability to output the sensitive regions in the second image. Then, the phone performs one or more of the following processing on the sensitive content in the sensitive regions of the second image: blurring, mosaicking, or replacement. Furthermore, for abnormal images, in order to distinguish between noise and desensitized sensitive regions in the abnormal image, the phone can upload the location of the sensitive regions in the second image when uploading the second image, thus separating the desensitized sensitive regions from the noise.

[0148] However, if the noise in the abnormal image is located in a sensitive region of the abnormal image, the above method may remove the noise from the abnormal image. For example, in this embodiment, the mobile phone can perform image differentiation on the second image and use the first gradient image after differentiation as the image after desensitization processing. Wherein, if the type of the second image is an abnormal image, its corresponding first gradient image includes the noise features in the second image.

[0149] Image differentiation, also known as gradient calculation, refers to differentiating the grayscale value of each pixel in the second image along the x and y directions. Image differentiation measures the edge information of the image based on changes in brightness. For each pixel, the gradient value is obtained by calculating the grayscale changes of its surrounding pixels. For example, image differentiation includes calculating the gradient value of the second image in the x-direction, calculating the gradient value of the second image in the y-direction, and merging the gradient values ​​in the x and y directions to obtain the first gradient image of the second image. For example, the gradient value of pixel one is dx(i,j) + dy(i,j). dx(i,j) = I(i+1,j) - I(i,j), dy(i,j) = I(i,j+1) - I(i,j). Where I is the grayscale value of pixel one, and (i,j) is the coordinate of pixel one in the second image. The first gradient image reflects the changes in pixel values ​​in the second image; the larger the pixel value change, the larger the corresponding gradient value. For example, if the grayscale values ​​in the x-direction of the second image are (100, 90, 90, 90…), then the gradient values ​​in the x-direction are (10, 0, 0…). The first gradient image can be a black and white image, displaying the grayscale gradient values ​​of each pixel in the second image. The more varied the grayscale values ​​of a pixel in the second image, the larger the gradient value at that pixel in the first gradient image; specifically, the brighter (whiter) the pixel in the first gradient image. Conversely, the less varied the grayscale values ​​of a pixel in the second image, the smaller the gradient value at that pixel in the first gradient image; specifically, the darker (blacker) the pixel in the first gradient image. Areas with large variations in grayscale values ​​in the second image are usually at the edges of the second image; therefore, the first gradient image can reflect the edges of the second image.

[0150] Typically, noise in an image manifests as isolated pixels or pixel blocks that cause a strong visual effect. If the second image includes noise, the noise pixels differ significantly from their surrounding pixels. Therefore, the processed first gradient image also includes noise. For example, if the second image includes black dots, the grayscale value of the pixel at the black dot differs greatly from the grayscale values ​​of the pixels surrounding the black dot. Therefore, the first gradient image can also display the black dot.

[0151] like Figure 10 As shown, Figure 10 The second image shown includes irregular black dots. Figure 10The image shown is the first gradient image obtained by differentiating the second image. The first gradient image includes edge images and irregular black dots in the second image. The process of generating the first gradient image from the second image can be called feature extraction, which involves extracting key features from the second image by differentiating it. Key features include the change values ​​of pixels in the first image, reflecting the changes in pixels. Key features include pixels in the first image whose changes in adjacent pixels exceed a preset change threshold.

[0152] Furthermore, before executing S602, the phone can differentiate each of the M*N groups of second images to obtain the first gradient image of each second image. After obtaining the first gradient image of the second image, the phone can input the first gradient image into the first neural network model to identify the type of the second image. Since the first gradient image contains the key features and noise features of the second image compared to the second image, inputting the first gradient image into the first neural network model to identify the type of the second image can improve computational efficiency.

[0153] Optionally, when uploading the second image to the server, the mobile phone can specifically upload the first gradient image. Since the first gradient image only includes key images and ignores areas in the second image where pixel values ​​change relatively smoothly, it can achieve a desensitization effect and protect privacy.

[0154] The preceding text, with reference to the accompanying drawings, describes an image-based device fault testing method provided by an embodiment of this application. The following text, in conjunction with... Figure 11 This section describes the interaction process between the mobile phone and the various modules of the server. The first neural network model is configured on the server, and the mobile phone downloads the first neural network from the server and integrates it before testing. The mobile phone can automatically download the first neural network model from the server or in response to user triggers. For example, in response to a user clicking the application icon of the test app on the mobile phone's main screen, the mobile phone downloads the first neural network model from the server and integrates it.

[0155] The mobile phone can input the second image into the first neural network model to obtain the recognition information of the second image. After testing, if the second image is an abnormal image, the mobile phone can upload the second image and noise type information to the server, and the server can train the first neural network model based on the second image and noise type.

[0156] Or, such as Figure 11 As shown, the mobile phone can input the first gradient image of the second image into the first neural network model to identify its type and obtain the recognition information of the second image. If the second image is an abnormal image, the mobile phone can upload the first gradient image and noise type information to the server, and the server can train the first neural network model based on the first gradient image and noise type.

[0157] Optionally, if the second image is a normal image, the mobile phone can also send the second image and type, or the first gradient image or type, to the server as a positive sample.

[0158] like Figure 11 As shown, the interaction process between the mobile phone and the server includes S1101-S1115.

[0159] S1101, the image management module in the server obtains the first gradient image previously used for testing from other test mobile phones.

[0160] The first gradient image is the derivative of the second image. If the second image is a normal image, the first gradient image can be a positive sample. If the second image is an abnormal image, the first gradient image can be a negative sample. For example, the second image corresponding to the first gradient image might be an abnormal image containing at least one of Q preset noise types caused by a phone malfunction. If the second image is an abnormal image, the server will also obtain the noise types included in the second image from the phone.

[0161] Optionally, the image management module in the server obtains a second image previously used for testing from other test phones. That is, the phone can input the second image into the first neural network model to identify its type; correspondingly, after testing, the phone can upload the second image to the server, and the server uses the second image to train the first neural network model. Or, as... Figure 11 As shown, the mobile phone can input the first gradient image of the second image into the first neural network model to identify its type. Correspondingly, after the test, the mobile phone can upload the first gradient image to the server, and the server uses the first gradient image to train the first neural network model.

[0162] S1102, the image management module classifies and stores the first gradient image.

[0163] S1103, the model training module in the server obtains training samples from the image management module.

[0164] S1104, The model training module in the server constructs the first neural network model and automatically trains the first neural network model using training samples.

[0165] For example, the model training module in the server can periodically retrieve the latest training sample set from the image management module and use the latest training samples to train the first neural network model. The latest training samples include first gradient images retrieved by the image management module from other test phones and / or training samples stored on the server by the developers.

[0166] S1105, after training is complete, the model training module on the server triggers the model conversion and integration module to convert the first neural network model into a preset format. This preset format is one that can be integrated and recognized by the mobile phone.

[0167] S1106, the transmission module 1 in the mobile phone obtains the first neural network model from the server.

[0168] Among them, the transmission module 1 in the mobile phone can obtain the functional class code used for inference operations and the parameters of the first neural network model.

[0169] Optionally, the model conversion and integration module in the server can store the first neural network in a cloud server. The transmission module 1 in the mobile phone can obtain the first neural network model from the cloud server.

[0170] S1107, the transmission module 1 in the mobile phone can store the first neural network model.

[0171] S1108, the test APP on the mobile phone responds to the tester's first operation, triggering the test module to sequentially turn on each of the M radio frequency bands and call N cameras to collect M*N sets of second images.

[0172] The test module sequentially activates M radio frequency bands, ensuring that only one band is active at a time. The test module can trigger the radio frequency system to transmit radio frequency signals after activating a band. Furthermore, the test module can trigger the radio frequency system to maintain maximum transmit power on the activated bands. For example, the phone stores at least one transmit power for each radio frequency band. For instance, radio frequency band 1 includes three transmit powers: Class1 = 31dBm, Class2 = 29dBm, and Class3 = 26dBm. The test module can trigger radio frequency system 1 to transmit radio frequency signals at the maximum transmit power of 31dBm.

[0173] Optionally, during the entire data acquisition process, the test module can activate a interference device that generates mild interference. This interference device can be a compass device, Wi-Fi device, LCD display device, etc. When in operation, the interference device can cause mild interference to the clock signal of the image sensor or the bus signal of the camera module.

[0174] S1109, The image recognition module in the mobile phone loads the first neural network model and initializes the first neural network model.

[0175] S1110, the image recognition module in the mobile phone preprocesses the second image to obtain the first gradient image.

[0176] The image recognition module may include a message queue for storing multiple second images to be processed. The image recognition module in the mobile phone can execute S1110-S1111 on each of the multiple second images in the message queue.

[0177] S1111, the image recognition module in the mobile phone takes the first gradient image as the input of the first neural network model, runs the first neural network model, and obtains the recognition information of the second image.

[0178] S1112, the report generation module in the mobile phone generates a test report based on the recognition information of the M*N groups of second images.

[0179] S1113, the report generation module in the mobile phone stores test reports.

[0180] S1114, The report generation module triggers the test APP on the mobile phone to display the test report.

[0181] S1115, the transmission module 1 in the mobile phone uploads the first gradient image used for this test and the test report generated this time to the server.

[0182] This application provides an electronic device, comprising: a memory, a communication module, a display screen, and one or more processors. The communication module receives and transmits data under the control of the processors to enable communication between the electronic device and other electronic devices. The memory, display screen, and processors are coupled; wherein, the memory stores computer program code, including computer instructions; the computer instructions are stored in the aforementioned electronic device (e.g., ...). Figure 4 When the electronic device 400 is run, it causes the electronic device to perform the various functions or steps in the above method embodiments.

[0183] This application embodiment also provides a computer storage medium, which includes computer instructions, when the computer instructions are executed in the aforementioned electronic device (such as...). Figure 4 When the electronic device 400 shown is run, it causes the electronic device to perform the various functions or steps in the above method embodiments.

[0184] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps described in the above method embodiments.

[0185] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device, the microphone of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0188] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0189] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0190] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image-based device fault testing method, characterized in that, The method includes: When the first electronic device operates in each of the M radio frequency bands, it acquires M*N sets of first images captured by N cameras of the first electronic device; each set of first images in the M*N sets includes at least one second image; where M is an integer greater than or equal to 1; and N is an integer greater than or equal to 1. At least one second image from each of the M*N groups of first images is sequentially input into a first neural network model to obtain M*N sets of recognition information. Each set of recognition information in the M*N sets includes recognition information for at least one second image. The recognition information for at least one second image includes the type of the second image, indicating whether the second image is a normal image or an abnormal image. If the second image does not contain any of the Q preset noise types, it is a normal image. If the second image contains at least one of the Q preset noise types, it is an abnormal image. Where Q is a large number of noise types... The probability of the second image being an integer equal to or greater than 1 is determined by the following conditions: if the probability of the second image containing stripe noise is greater than a first preset threshold, then the second image is an abnormal image containing stripe noise; if the probability of the second image containing mosaic noise is greater than a second preset threshold, then the second image is an abnormal image containing mosaic noise; if the probability of the second image containing dot noise is greater than a third preset threshold, then the second image is an abnormal image containing dot noise; the first neural network model has the ability to output the recognition information of the corresponding image based on the input image; the M*N sets of recognition information corresponding to the M*N sets of first images are used to generate a test report; The method further includes: The test report is obtained based on the M*N sets of identification information; Wherein, if the first image in the M*N group does not include abnormal images, the test report indicates that the test passed and the first electronic device is not faulty; if the first image in the M*N group includes abnormal images, the test report indicates that the test failed and the first electronic device is faulty.

2. The method according to claim 1, characterized in that, The first image in the M*N group is the first video in the M*N group, and the second image is an image frame of the first video.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The test report indicates that the test failed. The test report also includes the radio frequency band in which the radio frequency system in the first electronic device operates and the camera identifier of the camera that captured the abnormal image when the first electronic device captured the abnormal image.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The test report is displayed.

5. The method according to claim 1 or 2, characterized in that, The step of sequentially inputting at least one second image from each of the M*N groups of first images into the first neural network model to obtain M*N groups of recognition information includes: Differentiating the at least one second image yields at least one first gradient image, which corresponds one-to-one with the at least one second image; the first gradient image includes key features of the second image, or the first gradient image includes key features of the second image and noise features corresponding to noise in the second image; wherein, the key features include features of pixels in the first image whose adjacent pixel changes are greater than a preset change threshold; The at least one first gradient image is input into the first neural network model.

6. The method according to claim 1 or 2, characterized in that, The method further includes: The server sends the second image and its identification information to the server; wherein, if the second image is the abnormal image, the second image is used by the server as a negative sample to train the first neural network model; if the second image is the normal image, the second image is used by the server as a positive sample to train the first neural network model.

7. The method according to claim 6, characterized in that, Sending the second image and its identification information to the server includes: The server sends the first gradient image of the second image and the recognition information of the second image.

8. The method according to any one of claims 1, 2, or 7, characterized in that, Before sequentially inputting at least one second image from each of the M*N groups of first images into the first neural network model to obtain M*N groups of recognition information, the method further includes: A download request is sent to the server, the download request being used to request the download of the first neural network model; Based on the instructions from the server, the first neural network model is downloaded and saved.

9. An electronic device, characterized in that, The electronic device includes: a memory, a display screen, one or more processors, and a communication module; the memory, the communication module, the display screen, and the processor are coupled; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-8.

Images