Image-based equipment fault test method and electronic equipment

Abstract

The embodiment of the invention relates to the field of equipment testing, in particular to an image-based equipment fault testing method and electronic equipment. The first electronic equipment comprises N cameras, and the working frequency band of the first electronic equipment comprises M radio frequency bands. The method comprises the following steps: acquiring M * N groups of first images acquired by N cameras of first electronic equipment when the first electronic equipment works in each radio frequency band in M radio frequency bands; each group of first images in the M * N groups of first images comprises at least one second image; inputting at least one second image in each group of first images in the M * N groups of first images into a first neural network model to obtain a group of identification information; wherein the group of identification information comprises the identification information of at least one second image; the identification information comprises the type of the second image, and the type of the second image indicates that the second image is a normal image or an abnormal image; the M * N groups of identification information corresponding to the M * N groups of first images are used for generating a test report.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of equipment testing, and in particular to an image-based equipment fault testing method and electronic equipment. Background Art

[0002] Today's terminal devices generally have photo and video recording functions. Users can use terminal devices to take pictures or videos and record their lives anytime and anywhere.

[0003] When a terminal device fails, such as when the camera module in the terminal device is subject to electromagnetic interference, the terminal device will experience shooting failure. Specifically, the terminal device will capture abnormal images including noise. For example, when the image sensor in the camera module is subject to electromagnetic interference from the radio frequency signal in the terminal device, the image sensor will capture abnormal images including noise.

[0004] Usually, the software and hardware modules of the terminal device can be tested in the factory before the terminal device leaves the factory to avoid the above problems. Conventional testing methods do not conduct special tests for abnormal images captured by terminal devices due to interference of the RF system on the camera module, and cannot effectively identify the problem of shooting failure of terminal devices due to interference of the RF system on the camera module. Moreover, conventional testing methods rely on testers to manually identify abnormal images, and the misidentification rate is high. In short, conventional testing methods cannot effectively identify terminal devices with shooting failures, and there is a problem of low accuracy in identifying faulty devices. Summary of the invention

[0005] The embodiment of the present application provides an image-based device fault testing method and electronic device, which uses a first neural network model to identify whether the test image collected by the device under test includes an abnormal image of noise, and then identifies whether the device under test is a faulty device. In addition, the test image collected by the device under test can cover all radio frequency bands and all cameras of the device under test. Fault testing is performed based on a large number of test images, and the recognition accuracy of abnormal images is improved, which helps to improve the recognition accuracy of faulty devices. In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

[0006] In a first aspect, an image-based device fault testing method is provided, the method comprising: obtaining M*N groups of first images collected by N cameras of a first electronic device when the first electronic device operates in each of M radio frequency bands; each group of first images in the M*N groups of first images includes at least one second image; wherein M is an integer greater than or equal to 1; N is an integer greater than or equal to 1, and inputting at least one second image in each group of first images in the M*N groups of first images into a first neural network model to obtain a group of identification information; wherein a group of identification information includes identification information of at least one second image; the identification information includes a type of the second image, and the type of the second image indicates that the second image is a normal image or an abnormal image; wherein the second image does not include any one of Q preset noise types, and the second image is a normal image; the second image includes at least one of Q preset noise types, and the second image is an abnormal image; wherein Q is an integer greater than or equal to 1; the first neural network model has the ability to output identification information of the corresponding image according to the input image; the M*N groups of identification information corresponding to the M*N groups of first images are used to generate a test report.

[0007] The present application uses a first neural network model to identify the type of at least one second image in each of the M*N groups of first images collected by the first electronic device. In addition, the M*N groups of first images collected by the first electronic device can cover all radio frequency bands and all cameras of the first electronic device, including enough test samples. Fault testing is performed based on a large number of images, and the recognition accuracy of abnormal images is improved, which helps to improve the recognition accuracy of faulty equipment.

[0008] In a possible implementation of the first aspect, the M*N groups of first images are M*N groups of first videos, and the second images are image frames of the first videos. The first electronic device may traverse and enable each of the M radio frequency bands, and after enabling each radio frequency band, use N cameras in the first electronic device to respectively capture the first videos to obtain the M*N groups of first videos. The second image may be a portion or all of the image frames in the first video.

[0009] In a possible implementation of the first aspect, if the second image is an abnormal image, the identification information further includes a noise type of the second image, and the noise type is at least one of the Q preset noise types included in the second image. That is, the first neural network model can output the type of the second image, which can be a normal image or an abnormal image. If the second image is an abnormal image, the first neural network model can also output the noise type of the second image. The noise type is at least one of the Q preset noise types.

[0010] In a possible implementation of the first aspect, after obtaining the M*N group identification information corresponding to the M*N group first images, a test report can also be obtained based on the M*N group identification information corresponding to the M*N group first images. Specifically, the M*N group first images do not include abnormal images, the test report indicates that the test passed, and the first electronic device is not faulty. The M*N group first images include at least one abnormal image, and the test report indicates that the test failed. The first electronic device is faulty.

[0011] In a possible implementation of the first aspect, the test report indicates that the test failed, and the test report also includes a radio frequency band in which a radio frequency system in the first electronic device operates when the first electronic device captures an abnormal image and a camera identifier of a camera that captures a second image of the abnormal image. The radio frequency band and the camera identifier are used to locate the cause of the abnormal shooting of the first electronic device.

[0012] In a possible implementation of the first aspect, the method further includes displaying a test report. For example, the first electronic device displays the test report. Alternatively, the first electronic device establishes a connection with the second electronic device, and the second electronic device displays the test report. The first electronic device or the second electronic device displays the test report to promptly inform the tester of the test status.

[0013] In a possible implementation of the first aspect, the second image may be first differentiated to obtain a first gradient image of the second image; the first gradient image includes key features of the second image, or the first gradient image includes the key features of the second image and noise features corresponding to noise in the second image; wherein the key features include features of pixels in the first image whose adjacent pixel changes are greater than a preset change threshold, and finally the first gradient image is input into a first neural network model. Compared with the second image, the first gradient image includes fewer features, and inputting the second image into the first neural network model helps to improve the recognition effect.

[0014] In a possible implementation of the first aspect, the second image and identification information of the second image may also be sent to the server. Wherein, when the second image is an abnormal image, the second image is used by the server as a negative sample to train the first neural network model; when the second image is a normal image, the second image is used by the server as a positive sample to train the first neural network model.

[0015] In a possible implementation of the first aspect, the first gradient image of the second image and identification information of the second image may be sent to the server. Since the first gradient image only includes the key image, the area where the pixel value changes more smoothly in the second image is ignored, which can achieve a desensitization effect and protect privacy.

[0016] In a possible implementation of the first aspect, the server may continuously train the first neural network model. Before testing, the electronic device may send a download request to the server, where the download request is used to request downloading of the first neural network model; based on the instruction information of the server, the first neural network model is downloaded and saved.

[0017] In a second aspect, an electronic device is provided, comprising: a memory, a display screen, one or more processors, and a communication module; the memory, the communication module, the display screen and the processor are coupled; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the electronic device executes any one of the methods of the first aspect.

[0018] In a third aspect, the present application provides a chip system, which can be applied to an electronic device including a memory. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The interface circuit is used to receive a signal from the above-mentioned memory and send the signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device executes the method as in the first aspect and any possible design thereof.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and / or the second aspect and any possible design thereof.

[0020] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method according to the first aspect and any possible design thereof.

[0021] It can be understood that the beneficial effects that can be achieved by the electronic device of the second aspect to the computer program product of the fifth aspect can be referred to as the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of a camera provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the interaction process between an image sensor and a main control chip provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of multiple types of first images provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 5 A system architecture diagram of an image-based equipment fault detection system provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a flow chart of an image-based device fault detection method provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of a process flow of another image-based device fault detection method provided in an embodiment of the present application;

[0029] Figure 8 A flowchart of another method for detecting device faults based on an image provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of a display interface of a test APP provided in an embodiment of the present application;

[0031] Fig.10 A comparison diagram of images before and after desensitization provided in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of the interaction process between an electronic device and a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] Today's terminal devices generally have photo and video recording functions. Users can use terminal devices to take pictures or videos and record their lives anytime and anywhere.

[0034] A terminal device such as a mobile phone may include at least one camera. Exemplarily, in response to a user clicking a photo button in a camera application (application, APP) photo interface, the camera may capture an original image. The original image may be a RAW image captured by the camera. Afterwards, an image signal processor in the mobile phone may process the RAW image to obtain a processed first image. Finally, the mobile phone may display the first image in a preview interface of the camera APP. In this way, the user may preview the first image captured by the mobile phone.

[0035] The camera may include a lens, a holder, an image sensor, a voice coil motor (VCM) and a flexible printed circuit (FPC). An image sensor, also known as a photosensitive device, is a device that converts an optical image into an electrical signal. For example, the surface of an image sensor contains hundreds of thousands to millions of photosensitive diodes. When the photosensitive diode is exposed to light, a light signal is generated. Afterwards, the image sensor can convert the light signal into an electrical signal and output a RAW image in the Bayer format. Some image sensors can also output RAW images in RGGB, RGBW, CMYK, RYYB, CMY and other formats.

[0036] Figure 1 FIG. 1 shows a schematic diagram of the structure of a camera. Figure 1 As shown, the image sensor 101 can be soldered on a printed circuit board 1 (PCB). The PCB1 can be connected to the PCB2 through a flexible circuit board 102. A bracket 103 is also included around the image sensor, and the bracket can be a structural member. A lens 104 is also included above the image sensor, and the lens is used to converge light. A voice coil motor 105 is used to control the focal length and focus of the lens.

[0037] The electronic device may also include an image signal processor (ISP), where the ISP is a special type of processor for processing image signals. Taking the RAW image in Bayer format output by the image sensor as an example, the ISP can perform image processing on the RAW image in Bayer format to obtain an image in YUV format or RGB format. Image processing includes but is not limited to automatic exposure control, automatic gain control, automatic white balance, color correction, gamma correction, removal of bad pixels, etc. Afterwards, the ISP can transmit the processed image to the central processing unit. The central processing unit can store the image and / or display the image. Optionally, the mobile phone may also include a digital signal processor (DSP), and the ISP can transmit the processed image to the DSP, which then transmits the image to the central processing unit.

[0038] The camera, ISP, and DSP can be referred to as a camera module. When the mobile phone is shooting, the camera module is interfered by the radio frequency system in the mobile phone, and the mobile phone will have shooting abnormalities. For example, the mobile phone will capture a first image including noise. Exemplarily, when the image sensor is subject to electromagnetic interference from the radio frequency module, an abnormal image including noise will be captured. For example, when the mobile phone receives an incoming call while shooting, or when the mobile phone downloads data in the background while shooting, the mobile phone will cause the mobile phone to capture a first image including noise. Among them, the first image including noise is an abnormal image, and the first image not including noise is a normal image.

[0039] The main reason why the RF system generates electromagnetic interference to the camera module is that the operating frequency of the RF system in certain frequency bands is close to the frequency of the clock signal of the image sensor or the bus signal of the camera module, causing signal interference, resulting in disorder of the clock signal of the image sensor or the bus signal of the camera module, and then causing abnormal pixels in the first image.

[0040] For example, Figure 2 As shown, the image sensor and the main control chip can control the transmission of image data through the MCLCK clock signal, PCLCK clock signal, VSYNC clock signal, and HSYNC clock signal. Among them, the MCLCK clock signal is the clock source of the image sensor. The PCLCK clock signal is the clock that controls the pixel output, that is, the pixel sampling clock, and one pixel point data is collected in one PCLCK clock. The VSYNC clock signal is a field synchronization signal. The HSYNC clock signal is a line synchronization signal.

[0041] The image sensor can receive the MCLCK clock signal of the main control chip and use the MCLCK clock signal as the clock source. The sensor can calculate the sampling clock PCLCK based on the MCLCK clock signal. After that, the sensor can use the VSYNC clock signal and the HSYNC clock signal to output the RAW image. Among them, taking the high level effective as an example, VSYNC is pulled from high to low, and all the image data output in this section constitute a frame or an image. During the effective period of the HSYNC clock signal, the image data output by the sensor belongs to the same row. Among them, the sensor can use the mobile industry processor interface (MIPI) to output the VSYNC clock signal and the HSYNC clock signal. Figure 2 The main control chip in the system can be a baseband chip or a central processing unit.

[0042] Figure 2 The sensor realizes the photoelectric signal conversion process based on the MCLCK clock signal of the main control chip. In this process, when the RF system is working, the RF signal will interfere with the following: Figure 2 The reception or transmission of at least one of the MCLCK clock signal, PCLCK clock signal, VSYNC clock signal, and HSYNC clock signal causes the sensor to output a RAW image including noise, which in turn causes the first image finally displayed by the mobile phone to also include noise.

[0043] Noise refers to unnecessary or redundant interference information in the first image. Noise may include stripe noise, point noise, and mosaic noise. The noise appears randomly at any position in one or more first images.

[0044] The first image including stripe noise may include color stripes or black and white stripes, and the stripes may be straight lines or curves. The stripes may be displayed in an array in the entire area or a partial area of ​​the first image, or the stripes may be displayed disorderly in a partial area of ​​the first image. The dots may be color dots or black and white dots. The first image including dot noise may include color, black or white dots, and the dots may be displayed in an array in the entire area or a partial area of ​​the first image, or may be displayed disorderly and chaotically in the entire area or a partial area of ​​the first image. In the first image including mosaic noise, mosaics may be displayed in the entire area or a partial area of ​​the first image. For example, Figure 3 shown. Figure 3 Figures a and b respectively show a normal first image, a first image including stripe noise, a first image including point noise, a first image including mosaic noise, and another first image including stripe noise. Figure 3 FIG. 1 and FIG. 2 show a normal first image and an abnormal image including stripe noise caused by radio frequency signal interference. FIG.

[0045] Usually, before a mobile phone leaves the factory, the testers in the factory will test the software and hardware in the mobile phone to select the mobile phones that meet the factory quality requirements. For example, a mobile phone may include multiple software and hardware functional modules, such as a display module, a camera module, a Bluetooth module, a Wi-Fi module, a speaker module, a battery module, a video module, a radio frequency system, etc. Among them, the screen module may include a fingerprint module, a button module, etc. according to different functions. Before leaving the factory, the testers will test each software and hardware functional module in the mobile phone. For example, the testers will test the radio frequency system to detect whether the operation of the radio frequency system is faulty. The testers will test the camera module to detect whether the various shooting functions of the camera module are faulty. Conventional testing methods will perform separate tests on the radio frequency system and the camera module, that is, test the radio frequency system and the camera module separately. For example, the radio frequency system is tested according to the test specifications first, and then the camera module is tested according to the test specifications. Conventional testing methods do not perform special tests for mobile phone shooting failures caused by interference of the radio frequency system with the camera module, and cannot effectively identify the problem of mobile phone shooting failures caused by interference of the radio frequency system with the camera module.

[0046] In addition, during the conventional camera module test, the tester manually identifies whether the image captured by the mobile phone is an abnormal image containing noise. For example, the tester prepares the test image in advance, and the test image can be, for example, Figure 3 As shown in c in . Each pixel of the test image is a black dot. The tester prints the test image into a physical file and places it at the test location. The mobile phone includes the test image. Afterwards, the tester uses the mobile phone to be tested to shoot the test image. Finally, the tester compares the difference between the captured image and the test image in the mobile phone to be tested, mainly comparing whether the image captured by the mobile phone to be tested includes noise compared to the test image. Exemplarily, the tester checks one by one whether the black dots at each position in the captured image are consistent with the black dots at the corresponding positions in the test image. If the image captured by the mobile phone to be tested is consistent with the test image, it is considered that the mobile phone to be tested has no faults. If the image captured by the mobile phone to be tested is inconsistent with the test image, such as the image captured by the mobile phone to be tested includes at least one of stripe noise, point noise, and mosaic noise, it is considered that the mobile phone to be tested is faulty and requires further analysis and repair. As Figure 3 As shown in c in FIG. 1 , the black dots at some positions in the captured image are different from the black dots at the same positions in the test image.

[0047] Some noise is more obvious in the image, and the tester can intuitively identify the noise in the image. Some noise is more hidden in the image, and the tester may not be able to identify the noise in the image, which may lead to the omission of mobile phone failures and leave hidden dangers for users to use the mobile phone. In addition, the RF system interferes with the camera module and causes abnormal mobile phone shooting. It occurs randomly. Conventional testing methods do not have a complete testing mechanism and may not be able to test whether the mobile phone has the problem of RF system interfering with the camera module and causing abnormal mobile phone shooting.

[0048] To this end, an embodiment of the present application provides an image-based device fault testing method. First, obtain M*N groups of first images captured by N cameras of the first electronic device when the first electronic device works in each of the M radio frequency bands. Among them, the first electronic device supports M radio frequency bands, and the first electronic device includes N cameras. The first electronic device can traverse and enable each of the M radio frequency bands, and after enabling each radio frequency band, use the N cameras in the first electronic device to respectively capture the first image. In this way, it is possible to obtain M*N groups of first images captured by each of the N cameras when the first electronic device works sequentially in the M radio frequency bands. Each group of first images in the M*N groups of first images includes at least one second image.

[0049] Afterwards, the first neural network model is used to identify the type of at least one second image in each of the M*N groups of first images. The types of images include normal images and abnormal images, wherein the second image does not include any of the Q preset noises, and the second image is a normal image. If the second image includes at least one of the Q preset noises, the second image is an abnormal image. In this way, M*N groups of identification information can be obtained, and each group of identification information includes the type of the corresponding at least one second image. The M*N groups of identification information can be used to generate a test report.

[0050] It can be seen that this solution uses the first neural network model to identify whether the M*N groups of first images collected by the first electronic device include abnormal images with noise, and then identify whether the first electronic device is a faulty device. In addition, the M*N groups of first images collected by the first electronic device can cover all radio frequency bands and all cameras in which the first electronic device works. Fault testing is performed based on a large number of images, and the recognition accuracy of abnormal images is improved, which helps to improve the recognition accuracy of faulty devices.

[0051] The method provided in the embodiment of the present application can be applied to electronic devices with data processing capabilities. The above-mentioned first electronic device may include a server, a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (for example, a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (for example, a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiment of the present application does not impose any restrictions on this. In the embodiment of the present application, the above-mentioned electronic device is an electronic device that can run an operating system and install application programs. Optionally, the operating system running on the electronic device may be system, system, System, etc.

[0052] For example, please refer to Figure 4 , which shows a schematic diagram of the structure of an electronic device 400. The electronic device 400 may include a processor 410, an external memory interface 420, an internal memory 421, an audio module 430, a speaker 430A, a microphone 430B, a display screen 440, a communication module 450, a power module 460, an input device 470, a sensor module 480, a camera 490, etc. Among them, the sensor module 480 may include a pressure sensor, a touch sensor, etc.

[0053] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0054] The processor 410 may include one or more processing units. For example, the processor 410 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 400 may also include one or more processors 410.

[0055] The processor 410 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. The memory may store instructions or data that the processor 410 has just used or cyclically used. If the processor 410 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 410, and thus improves the efficiency of the system.

[0056] The external memory interface 420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 400. The external memory card communicates with the processor 410 through the external memory interface 420 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0057] The internal memory 421 may be used to store one or more computer programs, which include instructions. The processor 410 may enable the electronic device 400 to execute the application running method provided in some embodiments of the present application, as well as various applications and data management, by running the above instructions stored in the internal memory 421. In some embodiments, the processor 410 may enable the electronic device 400 to execute the application running method provided in the embodiments of the present application, as well as other applications and data management, by running the instructions stored in the internal memory 421 and / or the instructions stored in the memory provided in the processor 410.

[0058] The electronic device 400 can implement audio functions such as music playing and recording through the audio module 430, the speaker 430A, the microphone 430B, and the application processor. The speaker 430A, also called the "speaker", is used to convert audio electrical signals into sound signals.

[0059] Microphone 430B, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. A user can make a sound by approaching microphone 430B with his mouth, and input the sound signal into microphone 430B.

[0060] The communication function of the electronic device 400 can be realized through the antenna 1, the antenna 2 and the communication module 450.

[0061] The communication module 450 can provide solutions for wireless communications including cellular, Wi-Fi, Bluetooth (BT), wireless data transmission modules (e.g., 433MHz, 868MHz, 915MHz) applied to the electronic device 400. The communication module 450 can be one or more devices integrating at least one communication processing module. The communication module 450 receives electromagnetic waves via antenna 1 or antenna 2, filters and frequency modulates the electromagnetic wave signals, and sends the processed signals to the processor 410. The communication module 450 can also receive the signal to be sent from the processor 410, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through antenna 1 or antenna 2.

[0062] The electronic device 400 implements the display function through a GPU, a display screen 440, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 440 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 410 may include one or more GPUs, which execute program instructions to generate or change display information.

[0063] The display screen 440 is used to display images, videos, etc. The display screen 440 includes a display panel. In some embodiments, the electronic device 400 may include 1 or N display screens 440, where N is a positive integer greater than 1. In the embodiment of the present application, the display screen 440 may be used to display a UI and receive user operations on the UI.

[0064] The power module 460 can be used to supply power to various components included in the electronic device 400. In some embodiments, the power module 460 can be a battery, such as a rechargeable battery.

[0065] The input device 470 may include a keyboard, a mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc. into the electronic device 400, thereby issuing commands to the electronic device 400, inputting data, etc.

[0066] The electronic device 400 can realize the shooting function through an ISP, a camera 490, a video codec, a GPU, a display screen 440, and an application processor. The ISP is used to process the data fed back by the camera 490. The camera 490 is used to capture static images or videos. In some embodiments, the electronic device 400 includes 1 or N cameras 490, where N is a positive integer greater than 1. The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 290 is at a frequency point selection, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc. The video codec is used to compress or decompress digital video.

[0067] Take the above electronic device 400 as a mobile phone as an example. The software system of the electronic device 400 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 400.

[0068] Figure 5 4 is a system architecture diagram of an image-based device fault testing system provided in an embodiment of the present application. The system architecture diagram includes a software structure block diagram of an electronic device 400.

[0069] Exemplarily, the server includes an image management module, a model training module, a model conversion and integration module, and a transmission module 2.

[0070] The image management module is used to construct training samples for training the first neural network model. The server may include a first neural network model, and the server uses training samples to train the first neural network model. The training samples include training samples pre-configured in the service, and images uploaded by the electronic device 400. The image may be a second image or a first gradient image uploaded by the electronic device. Among them. The image may be a normal image or an abnormal image. Exemplarily, the training samples pre-configured in the service may be images with noise such as stripes, dots, mosaics, etc. taken by the manufacturer of the electronic device 400 when the electronic device 400 is subject to radio frequency interference and bus interference abnormalities. The image management module can also identify the classification and storage of images uploaded by the electronic device 400, such as classifying and storing normal images and abnormal images. Furthermore, for abnormal images, the images can also be classified and stored according to the noise type based on the noise type included in the abnormal image.

[0071] The model training module is used to construct a first neural network model and train the first neural network model. Specifically, the model training module can construct the first neural network model, automatically iterate the first neural network model using training samples, and save the first neural network model. Model parameters may include the number of nodes and node weights. The model training module can periodically and automatically train the first neural network model, and as the number of training samples continues to increase, the accuracy of the first neural network model is gradually improved.

[0072] The model conversion and integration module can convert the first neural network into a preset format, which can be a format that can be integrated and recognized by the electronic device 400. Exemplarily, the format of the first neural network model before conversion can be, for example, a .H5 file, and the format after conversion can be, for example, a .tflite file.

[0073] The layered architecture of the electronic device 400 divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime), the system library, and the kernel layer.

[0074] The electronic device 400 may include a fault test APP, an image recognition module, a test module, a report generation module, a transmission module 1 and a memory. The test APP is included in the application layer of the electronic device 400. The image recognition module, the test module and the report generation module are included in the application framework layer of the electronic device 400. The transmission module 1 and the memory are the hardware of the electronic device 400.

[0075] The test APP includes a UI interface, which can respond to various operations of the user on the UI interface to realize second image recognition and test report display.

[0076] The test module is used to turn on each of the M radio frequency bands and call N cameras to collect M*N groups of second images.

[0077] The image recognition module is used to identify the type of the second image in the M*N group of second images. Specifically, the image recognition module can initialize the first neural network model, and the initialization process can include model parameter loading. The image recognition model can also preprocess the second image, and the preprocessing includes image derivation. The image recognition module can use the second image or the first gradient image as the input of the first neural network model, run the first neural network model, and obtain the recognition result of the second image.

[0078] The report generation module can generate a test report based on the M*N groups of identification information output by the first neural network model.

[0079] like Figure 5 As shown, the application framework layer may also include an input system (not shown), a view system (not shown), a camera service, and a multimedia service.

[0080] In the embodiment of the present application, the input system is used to drive the touch screen of the monitoring electronic device 400 through the display screen, and convert the touch parameters generated by the touch operation input by the touch screen into usable events and pass them to the upper-level test APP.

[0081] The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build the display interface of an application. The display interface can be composed of one or more views.

[0082] The camera service provides an interface that can be called by upper-level applications (such as the camera APP) to realize image capture and preview based on the application's call. The multimedia service provides an interface that can be called by upper-level applications (such as the gallery APP) to realize video or audio playback based on the application's call.

[0083] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system. The core library contains two parts: one is the function that the Java language needs to call, and the other is the Android core library.

[0084] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0085] The kernel layer is the layer between hardware and software. The kernel layer can include display driver, audio driver, sensor driver, camera driver, audio driver, touch screen driver, etc.

[0086] The hardware includes a camera, a display screen, a memory, and a transmission module 1. The memory is used to store pictures and videos taken by the user. The memory can also be used to store the first gradient image of the second image after preprocessing and the test report generated by the report generation module.

[0087] The transmission module 1 is used to send data to the transmission module 2 of the server or receive data sent by the transmission module 2. For example, the transmission module 1 can download the first neural network model from the server. The transmission module 1 is also used to send the second image or the first gradient image and / or the test report of this test to the server.

[0088] Combine the following Figure 6 a and Figure 6In b, taking the first electronic device being a test version mobile phone (hereinafter referred to as mobile phone) as an example, an image-based device fault testing method provided in an embodiment of the present application is introduced.

[0089] S601, the mobile phone obtains M*N groups of first images collected by N cameras when the mobile phone works in each of the M radio frequency bands.

[0090] The mobile phone can be fixed in a preset test position, and then the mobile phone can obtain M*N groups of first images in response to the trigger of the tester. For example, the mobile phone can be fixed on a bracket or on a test device. The mobile phone is fixed in a preset test position, and the N cameras in the mobile phone can shoot at a preset object, wherein the preset object can be a preset shooting object, or can be a preset test picture. In this way, the shooting image and shooting background in each first image in the M*N groups of first images are unified, which helps to improve the recognition efficiency of the first image.

[0091] The mobile phone responds to the trigger of the tester, including the mobile phone responding to the first operation of the tester on the mobile phone, or responding to the first operation of the tester on the second electronic device. Specifically, for example, the mobile phone includes a test APP. In response to the tester clicking the application icon of the test APP on the mobile phone, the mobile phone executes S601. For another example, the second electronic device can be a test computer. The mobile phone can establish a connection with the test computer, for example, the mobile phone establishes a universal serial bus (USB) connection with the test computer, and the first operation can be that the tester enters a run command on the test computer. Figure 6 As shown in step b, in response to the first operation, the test computer sends a test instruction to the mobile phone. In response to the test instruction, the mobile phone executes S601.

[0092] The mobile phone supports M radio frequency bands. The radio frequency band can be the frequency range of electromagnetic waves emitted by the radio frequency module, and the unit is Hz. The mobile phone can include multiple radio frequency bands for supporting cellular network communications, multiple radio frequency bands for supporting short-range wireless communications, and multiple radio frequency bands for supporting satellite positioning and navigation.

[0093] Among them, multiple radio frequency bands for supporting cellular network communications include network communication radio frequency bands of different network operators. For example, the mobile phone supports multiple operators' Global System for Mobile Communications (GSM) radio frequency bands, Wideband Code Division Multiple Access (W-CDMA) radio frequency bands, Long Term Evolution (LTE) radio frequency bands, and one or more of the new wireless standards in the fifth generation mobile communication technology (5G), namely the 5G New Radio (5GNR) radio frequency bands. Exemplarily, the mobile phone supports the n41 radio frequency band and the n79 radio frequency band in the 5G NR of operator 1. The frequency range of the n41 radio frequency band is 2515MHz-2675MHz, and the frequency range of the n79 radio frequency band is 4800MHz-4900MHz.

[0094] The multiple radio frequency bands for supporting short-range wireless communications include radio frequency bands supporting wireless LAN communications, radio frequency bands supporting Bluetooth communications, radio frequency bands supporting near field communications (NFC), and radio frequency bands supporting ultra wide band (UWB) communications for wireless precise positioning and perception. For example, the radio frequency bands supporting wireless LAN communications may be 2.4G and 5G radio frequency bands, the radio frequency bands supporting Bluetooth communications may be 2.4G radio frequency bands, and the frequency range of the radio frequency bands supporting UWB communication technology may be 3.1GHz-10.6GHz.

[0095] The mobile phone may include N cameras. The camera may be a front camera, a rear camera, or a side camera of the electronic device. The camera may also be any of the following: an infrared camera, a visible light camera, an ultraviolet camera, etc. The camera may also be a single camera or a dual camera.

[0096] The M RF bands may be RF band 1, RF band 2, RF band 3, ..., RF band M, where M is an integer greater than or equal to 1. The N cameras may be camera 1, camera 2, camera 3, ..., camera N, where N is an integer greater than or equal to 1.

[0097] For example, Figure 7As shown, the mobile phone can use each of the N cameras to collect N groups of images in each of the M radio frequency bands in turn, so that the mobile phone can collect M*N groups of images. Among them, in one radio frequency band, one camera collects one group of images, and N cameras collect N groups of images. In M radio frequency bands, a total of M*N groups of images can be collected. For example, the mobile phone uses N cameras to collect N groups of first images in band 1. The N groups of first images can include the first image collected by camera 1 in radio frequency band 1, the first image collected by camera 2 in radio frequency band 1, the first image collected by camera 3 in radio frequency band 1, ..., the first image collected by camera N in radio frequency band 1. The N groups of first images collected by the mobile phone in radio frequency band 2 can include the first image collected by camera 1 in radio frequency band 2, the first image collected by camera 2 in radio frequency band 2, the first image collected by camera 3 in radio frequency band 2, ..., the first image collected by camera N in radio frequency band 2. By analogy, the N groups of first images captured by the mobile phone in RF band M may include the first image captured by camera 1 in RF band M, the first image captured by camera 2 in RF band M, the first image captured by camera 3 in RF band M,..., the first image captured by camera N in RF band M.

[0098] Taking M=2 as an example, the operating frequency band of the mobile phone can be the n41 RF band and the n79 RF band in 5G NR. The frequency range of the n41 RF band is 2515MHz-2675MHz, and the frequency range of the n79 RF band is 4800MHz-4900MHz.

[0099] The mobile phone first turns on one of the two RF bands, such as the n41 RF band, and works in the RF band. The work can be understood as the mobile phone emitting RF signals outward in the RF band. At the same time, the mobile phone obtains N groups of first images captured by N cameras when the mobile phone works in the RF band. Afterwards, the mobile phone turns on another RF band of the two RF bands, such as the n79 RF band, and works in the RF band. At the same time, the mobile phone obtains N groups of first images captured by N cameras when the mobile phone works in the RF band. For example, Figure 8 As shown, S601 may include S801-S804.

[0100] S801, the mobile phone turns on the n41 radio frequency band and transmits a first radio frequency signal, and the transmission frequency of the first radio frequency signal is within the n41 radio frequency band.

[0101] After the mobile phone turns on the n41 radio frequency band, the mobile phone can send a first radio frequency signal. The frequency of the first radio frequency signal is within the n41 radio frequency band, that is, the frequency of the first radio frequency signal is between 2515 MHz and 2675 MHz. The first radio frequency signal can be pre-configured in the mobile phone.

[0102] A mobile phone may include a radio frequency system. The radio frequency system includes but is not limited to a baseband circuit and a radio frequency circuit. The baseband circuit includes a baseband chip. The radio frequency circuit includes but is not limited to a radio frequency transceiver chip, a power amplifier, a low noise amplifier, a radio frequency switch, an antenna, etc.

[0103] After the mobile phone turns on the n41 radio frequency band and transmits the first radio frequency signal, the mobile phone turns on N cameras in sequence to collect N groups of first images. Specifically, the mobile phone may execute S802.

[0104] S802, the mobile phone operates in the n41 radio frequency band, turns on the first camera to collect the first image a, turns on the second camera to collect the first image b, ..., turns on the Nth camera to collect the first image n.

[0105] Afterwards, the mobile phone may continue to turn on the n79 radio frequency band and use the N cameras to collect N groups of first images. The mobile phone may execute S803.

[0106] S803, the mobile phone turns on the n79 radio frequency band and transmits a second radio frequency signal, and the transmission power of the second radio frequency signal is within the n79 radio frequency band.

[0107] After the mobile phone turns on the n79 radio frequency band, the mobile phone can send a second radio frequency signal, and the frequency of the second radio frequency signal is within the n79 radio frequency band, that is, the frequency of the second radio frequency signal is between 4800MHz and 4900MHz. The second radio frequency signal can be pre-configured in the mobile phone. The content of the first radio frequency signal and the content of the second radio frequency signal can be the same or different. Afterwards, the mobile phone turns on N cameras in sequence to collect N groups of first images. Specifically, the mobile phone can execute S804.

[0108] S804, the mobile phone operates in the n79 radio frequency band, starts the first camera to collect the first image A, starts the second camera to collect the first image B, ..., starts the Nth camera to collect the first image N.

[0109] In this way, the mobile phone obtains 2N groups of first images.

[0110] Optionally, when the mobile phone transmits the first RF signal and the second RF signal, the first RF signal and the second RF signal can both be transmitted at the maximum transmission power. The maximum transmission power of the mobile phone transmitting the first RF signal and the maximum transmission power of the mobile phone transmitting the second RF signal can be the same or different.

[0111] Each of the M*N groups of first images may be a video (first video), for example, the first image may be a first video of a preset duration captured by a camera. The first video includes a plurality of second images, and the second images are image frames in the first video. Each of the M*N groups of first images may also include one or more photos (second images), for example, a group of first images may be a plurality of photos captured by a camera within a preset duration. The preset duration may be 5 seconds or 10 seconds.

[0112] The M*N groups of first images are used to identify whether the mobile phone has a fault, wherein the fault includes electromagnetic wave interference, hardware fault, software fault, etc. The first neural network model is used to identify whether there are abnormal images including noise in the M*N groups of first images. If there are abnormal images including noise in the M*N groups of first images, the mobile phone has a fault. If there are no abnormal images including noise in the M*N groups of first images, the mobile phone has no fault.

[0113] The first neural network model can be configured in a mobile phone, and the mobile phone can run the first neural network model to identify whether there are abnormal images including noise in the M*N groups of first images. Alternatively, the first neural network model can be configured in a test computer. The mobile phone can send the M*N groups of first images to the test computer. The test computer can run the first neural network model to identify whether there are abnormal images including noise in the M*N groups of first images.

[0114] Optionally, the first neural network model can also be deployed on a server. If the first neural network is deployed on a server, the mobile phone can send the M*N groups of first images to the server. The server runs the first neural network model to identify whether there are abnormal images including noise in the M*N groups of first images.

[0115] The following introduces the present solution by taking a mobile phone configured with a first neural network model and running the first neural network model as an example.

[0116] S602, the mobile phone sequentially inputs at least one second image in each of the M*N groups of first images into the first neural network model to obtain M*N groups of identification information.

[0117] For each group of first images in the M*N groups of first images, the mobile phone can identify part of the second images in each group of first images, or can identify all the second images in each group of first images. For example, a group of first images is one image, and the mobile phone identifies all the second images in a group of first images. A group of first images includes multiple first images, and the mobile phone identifies all or part of the second images in this group of first images. Further, a group of first images includes multiple second images, and the mobile phone can identify part of the second images according to a preset ratio. A group of first images is a video, and the mobile phone identifies all or part of the image frames or all the image frames in the video. Further, the mobile phone can identify part of the image frames according to a preset ratio. The mobile phone only identifies part of the second images in a group of first images, which can improve the test efficiency. The mobile phone identifies all the second images in a group of first images, which can improve the test accuracy.

[0118] The mobile phone inputs at least one second image in a set of first images into the first neural network model to obtain a set of identification information. The set of identification information includes the identification information of at least one second image in the set of first images. In this way, the mobile phone can obtain M*N sets of identification information.

[0119] Wherein, each group of identification information may include a type corresponding to at least one second image. The type of the second image may be a normal image. Alternatively, the type of the second image may be an abnormal image. For example, if the second image includes at least one noise of Q preset noise types, the second image is an abnormal image. Wherein, Q is an integer greater than or equal to 1. Exemplarily, the Q preset noises may be stripe noise, point noise, and mosaic noise. For example, if the second image does not include any one of the Q preset noise types, the second image is a normal image. If the second image includes at least one noise of the Q preset noise types, the second image is an abnormal image. The second image may be an abnormal image including stripe noise, an abnormal image including point noise, or an abnormal image including mosaic noise. Alternatively, the second image may be an abnormal image including both stripe noise and point noise.

[0120] The Q preset noise types may appear in the first image due to a mobile phone failure. Mobile phone failures include radio frequency signal interference, hardware failures, software failures, etc. Hardware failures may be, for example, device failures, such as camera hardware failures. Software failures may be, for example, missing software data of a camera APP, disordered code operation of a camera APP, and software system operation bugs.

[0121] The first neural network has the ability to output the type of the second image according to the input second image. Exemplarily, the first neural network model can be trained using a normal image that does not include any noise, and an abnormal image that includes one or more noises such as stripes, dots, and mosaics caused by a mobile phone malfunction. For the second image that does not include any noise, the trained first neural network model has the ability to output the type of the second image as a normal image. For the second image that includes one or more noises such as stripes, dots, and mosaics caused by a mobile phone malfunction, the trained first neural network model has the ability to output the type of the second image as an abnormal image.

[0122] Specifically, the first neural network model can calculate the probability value P that the second image is a normal image. If the probability value P is greater than or equal to a preset threshold, the second image is a normal image, and if the probability value P is less than the preset threshold, the second image is an abnormal image. For example, the first neural network model calculates the probability value P=0.6 that the second image is a normal image, and the preset threshold is 0.5, and the first neural network model outputs that the type of the second image is a normal image.

[0123] Furthermore, if the second image is an abnormal image, the identification information may also include the noise type of the second image. The noise type is at least one of the Q preset noise types caused by a mobile phone failure. That is, the first neural network model can not only output the type of the second image as a normal image or an abnormal image. The first neural network model can also output the noise type included in the abnormal image. Among them, the noise type is a noise type that includes at least one of the M preset noise types.

[0124] As a possible implementation method, the first neural network model can calculate the probability value of each noise in the Q preset noises caused by the mobile phone failure in the second image, so that Q probability values ​​can be obtained. For the Q probability values, if the Q probability values ​​include a probability value that meets the preset conditions, the second image includes the noise corresponding to the probability value that meets the preset conditions. The first neural network model outputs the noise type of the second image as the noise type corresponding to the probability value that meets the preset conditions. If the Q probability values ​​include a probability value that does not meet the preset conditions, the second image is a normal image. Among them, the preset condition can be that the probability value is the maximum probability value among the Q probability values ​​and is greater than a preset threshold. Alternatively, the preset condition can be that the probability value is greater than a preset threshold.

[0125] Exemplarily, the first neural network model can output the probability value that the second image includes stripe noise, mosaic noise, and point noise caused by a mobile phone failure. For example, the probability values ​​are 0.05, 0.6, and 0.35, respectively. If the probability value of the second image including mosaic noise is the maximum probability value and is greater than the preset threshold value of 0.5, then the second image is an abnormal image including mosaic noise, and the first neural network model outputs the noise type of the second image as mosaic noise. For example, if the probability value of the second image including mosaic noise and the probability value of point noise are both greater than the preset threshold value of 0.3, then the noise type of the second image is mosaic noise and point noise. If the probability values ​​of the second image belonging to stripe noise, mosaic noise, and dot noise are all less than the preset threshold value of five, then the second image is a normal image. For example, the probability values ​​of the second image including stripe noise, mosaic noise, and dot noise are 0.35, 0.35, and 0.3, respectively, and the preset threshold value can be 0.4. Since these three probability values ​​do not include a probability value greater than the preset threshold, the type of the second image can be a normal image.

[0126] As another possible implementation, the first neural network model includes a plurality of sub-neural network sub-models, and a preset noise type includes a corresponding sub-neural network model. For example, neural network model one is used to identify whether the second image includes stripe noise caused by a mobile phone failure; neural network model two is used to identify whether the second image includes mosaic noise caused by a mobile phone failure; neural network model three is used to identify whether the second image includes point noise caused by a mobile phone failure. Exemplarily, the second image is input into neural network model one, and neural network model one outputs a probability value one that the second image includes stripe noise caused by a mobile phone failure. If the probability value one is greater than or equal to the preset threshold one, it is considered that the second image includes stripe noise caused by radio frequency signal interference, the second image is an abnormal image, and the noise type of the second image is stripe noise. The second image is input into neural network model two, and a probability value two that the second image includes mosaic noise caused by a mobile phone failure is obtained. If the probability value two is greater than or equal to the preset threshold two, it is considered that the second image includes mosaic noise caused by a mobile phone failure, the second image is an abnormal image, and the noise type of the second image is mosaic noise. The second image is input into neural network model three, and the probability value three that the second image includes dot noise caused by mobile phone failure is obtained. If the probability value three is greater than or equal to the preset threshold value three, it is considered that the second image is dot noise caused by mobile phone failure, the second image is an abnormal image and the noise type of the second image is dot noise. Among them, the preset threshold value one, the preset threshold value two, and the preset threshold value three can be equal, for example, all can be 0.5. Among them, if the probability value one of the second image is less than the preset threshold value one, and the probability value two is less than the preset threshold value two, and the probability value three is less than the preset threshold value three, then the type of the second image is a normal image, that is, the second image is an image that does not include any of the noise types in the preset noise types in Q caused by mobile phone failure.

[0127] Afterwards, the mobile phone can obtain a test report based on the M*N group identification information. If the first image of the M*N group does not include an abnormal image, the test report indicates that the test passed and the mobile phone is not faulty. If the first image of the M*N group includes an abnormal image, the test report indicates that the test failed and the mobile phone is faulty.

[0128] For example, the mobile phone can generate a test report on the mobile phone side based on the M*N group identification information. Alternatively, the mobile phone can send the M*N group identification information to the test computer, and the test computer generates a test report, and the mobile phone can obtain the test report from the test computer.

[0129] Optionally, if the test computer executes S602, the test computer generates a test report based on the M*N group identification information. The test computer can send the test report to the mobile phone.

[0130] Optionally, if the server executes S602, the server may generate a test report based on the M*N group identification information. The server may send the test report to the mobile phone. The following takes the generation of the test report on the mobile phone side as an example.

[0131] S603, the mobile phone generates a test report.

[0132] The test report may include the test time, mobile phone identification, test results, and the total number of second images used for the test. The test result may be a test pass or a test fail. If the test fails, the test report may include information such as the number of abnormal images in the total number of second images used for the test, the noise type of the abnormal images, etc.

[0133] For each abnormal image, the test report may also include information such as the RF frequency band in which the RF system was operating when the mobile phone took the abnormal image, the camera identification of the camera that took the abnormal image, and other information. As a possible implementation method, the first image includes image data and file data, wherein the file data is used to store shooting parameter information. When the mobile phone takes the first image in each group of first images, the camera identification of the camera that took the first image and the preset RF frequency band information of the mobile phone can be added to the file data of the first image. That is, the first image carries the camera identification of the camera that took the first image and the preset RF frequency band information of the mobile phone. If the second image is an abnormal image, the mobile phone can directly read the above information and count it in the test report.

[0134] Optionally, after S603, the mobile phone or the test computer may display the test report. When displaying the test report, the mobile phone or the test computer may display part of the content in the test report or display the entire content in the test report. For example, the mobile phone or the test computer may only display the test result. For example, the mobile phone may execute S604.

[0135] S604, the mobile phone displays a test report.

[0136] The mobile phone may display the test report on the first interface. The first interface may be a display interface that pops up automatically on the mobile phone after the test is completed. Optionally, the first interface may be an interface of the test APP. In some embodiments, the mobile phone may not display the test report, but instead feeds the test report back to the test computer, which is then displayed by the test computer.

[0137] If the mobile phone displays part of the test report, the mobile phone can also respond to the tester's instructions to display the entire test report. Fig. 9 As shown, the mobile phone displays the main interface, which includes the application icon of the test APP. In response to the tester clicking the application icon of the test APP on the main interface, the mobile phone executes Figure 6 In S601-S603, the mobile phone may display interface 901. Interface 901 may indicate that the test is in progress. After the test is completed, the mobile phone may display interface 902 or interface 903. Interface 902 includes the test result, which may be a test pass. Interface 903 includes the test result, which may be a test fail. Interface 902 and / or interface 903 may include a first button, which is used to trigger the mobile phone to display a test report. The first button may be, for example, a view report button. Fig. 9 As shown, in response to the tester clicking the first button on interface 903, the mobile phone displays interface 904. Interface 904 displays the abnormal second image, the noise type of the second image, the radio frequency band of the radio frequency system when the mobile phone took the second image, the identification of the camera that took the second image, and other information in a list format.

[0138] Optional, such as Figure 6 As shown in b, after S603, the test computer can display the test report.

[0139] S605, the mobile phone sends a test report to the test computer.

[0140] S606, in response to receiving the test report, the test computer displays the test report.

[0141] The mobile phone or test computer displays the test report, and the tester can intuitively see the test results. If the test passes, the tester can continue to perform other tests on the mobile phone. If the mobile phone test fails, for each abnormal image, the tester can further analyze whether there are design or assembly problems in the RF system based on the RF frequency band of the RF system when the mobile phone took the abnormal image, the camera logo of the camera that took the abnormal image, and other information.

[0142] For example, the Q RF bands in a mobile phone correspond to the P antennas in the mobile phone. For an abnormal image, based on the RF band in which the RF system works when the mobile phone takes this abnormal image, the antenna that works when the mobile phone takes this abnormal image is obtained. Afterwards, it can be detected whether there is a problem of insufficient anti-interference between the antenna and the camera that took this abnormal image. Alternatively, the tester can further analyze whether there is a hardware failure in the camera module that took this abnormal image and / or whether there is a software anomaly in the mobile phone when taking this abnormal image.

[0143] It can be seen that in the embodiment of the present application, a device fault testing method based on an image is provided, in which the mobile phone can traverse and open each of the M radio frequency bands, and after opening each radio frequency band, use the N cameras in the mobile phone to respectively capture the first image. In this way, the mobile phone obtains M*N groups of first images captured by each of the N cameras when the mobile phone works sequentially in the M radio frequency bands. In this way, enough first images can be collected, and the first images cover all radio frequency bands and all cameras in the mobile phone, and a test environment that meets the entire fault scenario can be created. Afterwards, for part or all of the second images in the M*N groups of first images, the first neural network model is used to identify whether the second image is an abnormal image including noise, thereby improving the recognition accuracy. Furthermore, this solution can effectively identify mobile phones with shooting faults and improve the quality of mobile phones.

[0144] Optionally, a first neural network model can be deployed on the server, and the server can train the first neural network model. Before the test, the mobile phone or the test computer can obtain the first neural network model from the server. After the test, the mobile phone or the test computer can upload all or part of the second images in the M*N group of first images used for the test to the server as training samples, and continuously improve the detection accuracy of the first neural network model. In this way, a first neural network model with better accuracy can be obtained when testing the next mobile phone. For example, the mobile phone or the test computer can upload the normal image in the M*N group of second images to the server, and the normal image can be used as a positive sample. The mobile phone or the test computer can upload the abnormal image in the M*N group of first images to the server, and the abnormal image can be used as a negative sample. Furthermore, since the positive samples are relatively abundant, the mobile phone or the test computer can also only upload the second image of the abnormal image type.

[0145] Optionally, when uploading the second image, the mobile phone or the test computer may upload the type of the second image. If the second image is an abnormal image, the mobile phone or the test computer may also upload the noise type of the second image.

[0146] Furthermore, when the mobile phone or the test computer uploads the second image (which may be a normal image or an abnormal image), the sensitive content in the second image may be desensitized. Desensitization can make the second image no longer contain sensitive content. Among them, sensitive content may include faces, sensitive text, sensitive patterns, etc. Sensitive text may include factory information and brand information. Desensitization includes blurring, mosaicing, replacing, etc. of sensitive content in the image. The following introduces a desensitization process by taking the example of uploading the second image by a mobile phone.

[0147] Exemplarily, the mobile phone can first identify the sensitive content in the second image. For example, the mobile phone can identify whether the second image includes a face, preset sensitive text, preset sensitive pattern, etc. For example, the mobile phone can identify the sensitive area in the second image based on the second neural network model. The sensitive area is an area that includes sensitive content. The second neural network model has the ability to output the sensitive area in the second image. Afterwards, the mobile phone performs one or more of blurring, mosaic processing, and replacement processing on the sensitive content in the sensitive area in the second image. Furthermore, for abnormal images, in order to distinguish the noise in the abnormal image from the sensitive area after desensitization processing, the mobile phone can upload the position of the sensitive area in the second image when uploading the second image, and separate the sensitive area after desensitization processing from the noise.

[0148] However, if the noise of the abnormal image is in the sensitive area of ​​the abnormal image, the above method may remove the noise in the abnormal image. For example, in the embodiment of the present application, the mobile phone can perform image derivation on the second image, and use the first gradient image after the derivation of the second image as the image after desensitization processing. Among them, if the type of the second image is an abnormal image, the corresponding first gradient image includes the noise characteristics in the second image.

[0149] Image derivation, also known as gradient calculation, refers to the derivation of the grayscale value of each pixel in the second image in the x and y directions respectively. Image derivation measures the edge information of the image based on the brightness change of the image. For each pixel, the gradient value of the point is obtained by calculating the change in the grayscale of the surrounding pixels. Exemplarily, image derivation includes calculating the gradient value of the second image in the x direction, calculating the gradient value of the second image in the y direction, and merging the gradient value in the x direction with the gradient value in the y direction to obtain the first gradient image of the second image. For example, the gradient value of pixel one is dx(i,j)+dy(i,j). dx(i,j)=I(i+1,j)-I(i,j), dy(i,j)=I(i,j+1)-I(i,j). Wherein, I is the grayscale value of pixel one, and (i,j) is the coordinate of pixel one in the second image. The first gradient image can reflect the change in pixel value in the second image. The larger the pixel value change, the larger the corresponding gradient value. For example, if the grayscale value in the x direction in the second image is (100.90.90.90…), then the gradient value in the x direction is (10.0.0…). The first gradient image can be a black and white image, and the first gradient image displays the grayscale gradient value of each pixel in the second image. The more the grayscale value of a pixel in the second image changes, the larger the gradient value at the pixel in the first gradient image. Specifically, the brighter (whiter) the pixel is in the first gradient image. The less the grayscale value of a pixel in the second image changes, the smaller the gradient value at the pixel in the first gradient image. Specifically, the darker (blacker) the pixel is in the first gradient image. The places where the grayscale value of a pixel in the second image changes greatly are usually at the edge of the second image, so the first gradient image can reflect the image edge of the second image.

[0150] Generally, noise often appears in an image as isolated pixels or pixel blocks that cause strong visual effects. If the second image includes noise, the difference between the noise pixels and the surrounding pixels is relatively large. Therefore, the processed first gradient image also includes noise. For example, taking the second image including black dots as an example, the grayscale value of the pixel at the black dot varies greatly from the grayscale value of the pixel around the black dot. Therefore, the first gradient image may also display the black dot.

[0151] like Fig.10 As shown, Fig.10 The second image shown in FIG. 1 includes random black dots. Fig.10The first gradient image after the second image is derived is shown, and the first gradient image includes the edge image in the second image and the irregular black dots in the second image. The process of generating the first gradient image from the second image can be called feature extraction, that is, extracting the key features in the second image by deriving the second image. The key features include the change value of the pixel point in the first image, and the key features can reflect the change of the pixel point in the first image. The key features include the features of the pixels in the first image whose adjacent pixel changes are greater than the preset change threshold.

[0152] Furthermore, before executing S602, the mobile phone may derive each second image in the M*N groups of second images to obtain a first gradient image of each second image. After obtaining the first gradient image of the second image, the mobile phone may input the first gradient image into the first neural network model to identify the type of the second image. Since the first gradient image includes the key features and noise features of the second image compared to the second image, inputting the first gradient image into the first neural network model to identify the type of the second image can improve computational efficiency.

[0153] Optionally, when uploading the second image to the server, the mobile phone can specifically upload the first gradient image. Since the first gradient image only includes the key image, the area with relatively stable pixel value changes in the second image is ignored, which can achieve a desensitization effect and protect privacy.

[0154] The foregoing describes an image-based device fault testing method provided by an embodiment of the present application in conjunction with the accompanying drawings. Fig.11 The interaction process between the mobile phone and each module in the server is introduced. Among them, the first neural network model is configured on the server, and the mobile phone downloads the first neural network from the server and integrates it into the mobile phone before testing. The mobile phone can download the first neural network model from the server automatically or in response to a user's trigger. For example, in response to a user clicking on the application icon of the test APP on the main interface of the mobile phone, the mobile phone downloads the first neural network model from the server and integrates it into the mobile phone.

[0155] The mobile phone can input the second image into the first neural network model to obtain identification information of the second image. After the test, if the second image is an abnormal image, the mobile phone can upload the second image and noise type information to the server, and the server can train the first neural network model based on the second image and noise type.

[0156] Or, if Fig.11 As shown, the mobile phone can input the first gradient image of the second image into the first neural network model to identify its type and obtain identification information of the second image. If the second image is an abnormal image, the mobile phone can upload the first gradient image and noise type information to the server, and the server can train the first neural network model based on the first gradient image and noise type.

[0157] Optionally, if the second image is a normal image, the mobile phone may also send the second image and type, or the first gradient image or type to the server as a positive sample.

[0158] like Fig.11 As shown, the interaction process between the mobile phone and the server includes S1101-S1115.

[0159] S1101, an image management module in a server obtains a first gradient image previously used for testing from other tested mobile phones.

[0160] The first gradient image is an image derived from the second image. If the second image is a normal image, the first gradient image may be a positive sample. If the second image is an abnormal image, the first gradient image may be a negative sample. For example, the second image corresponding to the first gradient image is an abnormal image including at least one of Q preset noise types caused by a mobile phone failure. If the second image is an abnormal image, the server will also obtain the noise type included in the second image from the mobile phone.

[0161] Optionally, the image management module in the server obtains the second image from other test mobile phones before being used for testing. That is, the mobile phone can input the second image into the first neural network model to identify its type. Correspondingly, after the test, the mobile phone can upload the second image to the server, and the server uses the second image to train the first neural network model. Or, Fig.11 As shown, the mobile phone can input the first gradient image of the second image into the first neural network model to identify its type. Correspondingly, after testing, the mobile phone can upload the first gradient image to the server, and the server uses the first gradient image to train the first neural network model.

[0162] S1102: The image management module classifies and stores the first gradient image.

[0163] S1103, the model training module in the server obtains training samples from the image management module.

[0164] S1104: The model training module in the server constructs a first neural network model and uses the training samples to automatically train the first neural network model.

[0165] Exemplarily, the model training module in the server can periodically obtain the latest training sample set from the image management module and use the latest training samples to train the first neural network model. The latest training samples include the first gradient image obtained by the image management module from other test mobile phones and / or the training samples stored in the server by the R&D personnel.

[0166] S1105, after the training is completed, the model training module in the server triggers the model conversion and integration module to convert the first neural network model into a preset format. The preset format may be a format that can be integrated and recognized by the mobile phone.

[0167] S1106, the transmission module 1 in the mobile phone obtains the first neural network model from the server.

[0168] Among them, the transmission module 1 in the mobile phone can obtain the functional class code used for reasoning operations in the first neural network model and the parameters of the first neural network model, etc.

[0169] Optionally, the model conversion and integration module in the server can store the first neural network in the cloud server. The transmission module 1 in the mobile phone can obtain the first neural network model from the cloud server.

[0170] S1107, the transmission module 1 in the mobile phone can store the first neural network model.

[0171] S1108, the test APP in the mobile phone responds to the first operation of the tester, triggers the test module to turn on each of the M radio frequency bands in sequence and calls N cameras to capture M*N groups of second images.

[0172] The test module turns on M RF bands in sequence, ensuring that only one RF band is effective at the same time. The test module can trigger the RF system to transmit RF signals after turning on a RF band. Furthermore, the test module can trigger the RF system to maintain the maximum transmission power in the turned-on RF band. Exemplarily, the mobile phone saves at least one transmission power for each RF band. For example, RF band 1 includes 3 transmission powers, namely Class1=31dBm, class2=29dBm, and class3=26dBm. The test module can trigger RF system 1 to transmit RF signals at a maximum transmission power of 31dBm.

[0173] Optionally, during the entire acquisition process, the test module can turn on an interference device that can generate slight interference to enable it to work. The interference device can be a compass device, a Wi-Fi device, an LCD device, etc. When working, the interference device can generate slight interference to the clock signal of the image sensor or the bus signal of the camera module.

[0174] S1109, the image recognition module in the mobile phone loads the first neural network model and initializes the first neural network model.

[0175] S1110, an image recognition module in the mobile phone pre-processes the second image to obtain a first gradient image.

[0176] The image recognition module may include a message queue for storing a plurality of second images to be processed. The image recognition module in the mobile phone may execute S1110-S1111 one by one for the plurality of second images in the message queue.

[0177] S1111, the image recognition module in the mobile phone uses the first gradient image as the input of the first neural network model, runs the first neural network model, and obtains recognition information of the second image.

[0178] S1112: A report generation module in the mobile phone generates a test report based on the identification information of the M*N groups of second images.

[0179] S1113, the report generation module in the mobile phone stores the test report.

[0180] S1114, the report generation module triggers the test APP in the mobile phone to display the test report.

[0181] S1115, the transmission module 1 in the mobile phone uploads the first gradient image used for the test and the test report generated this time to the server.

[0182] The present application provides an electronic device, which includes: a memory, a communication module, a display screen, and one or more processors. The communication module receives and sends data according to the control of the processor to achieve communication between the electronic device and other electronic devices. The memory and the display screen are coupled to the processor; wherein the memory is used to store computer program code, the computer program code includes computer instructions; the computer instructions are in the above electronic device (such as Figure 4 When the method is executed on an electronic device 400 shown in the figure, the electronic device executes each function or step in the above method embodiment.

[0183] The present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are in the above electronic device (such as Figure 4 When the method is executed on the electronic device 400 shown in the figure, the electronic device executes each function or step in the above method embodiment.

[0184] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step in the above method embodiment.

[0185] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device, a microphone of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0186] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0187] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0191] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image-based device fault testing method, characterized in that: The method comprises: Acquire M*N groups of first images captured by N cameras of the first electronic device when the first electronic device works in each of the M radio frequency bands; each group of first images in the M*N groups of first images includes at least one second image; wherein M is an integer greater than or equal to 1; N is an integer greater than or equal to 1, Input at least one second image in each of the M*N groups of first images into the first neural network model to obtain a group of identification information; wherein, the group of identification information includes the identification information of the at least one second image; the identification information includes the type of the second image, and the type of the second image indicates that the second image is a normal image or an abnormal image; wherein, the second image does not include any one of Q preset noise types, and the second image is a normal image; the second image includes at least one of Q preset noise types, and the second image is an abnormal image; wherein, Q is an integer greater than or equal to 1; the first neural network model has the ability to output the identification information of the corresponding image according to the input image; the M*N groups of identification information corresponding to the M*N groups of first images are used to generate a test report.

2. The method according to claim 1, characterized in that The M*N groups of first images are M*N groups of first videos, and the second images are image frames of the first videos.

3. The method according to claim 1 or 2, characterized in that: If the second image is the abnormal image, the identification information further includes a noise type of the second image, and the noise type is at least one noise type among Q preset noise types included in the second image.

4. The method according to claims 1-3, characterized in that: After inputting at least one second image in each of the M*N groups of first images into the first neural network model to obtain a group of recognition information, the method further includes: Based on the M*N group identification information, obtain the test report; The M*N groups of first images do not include abnormal images, the test report indicates that the test is passed, and the first electronic device is not faulty; The M*N groups of first images include abnormal images, the test report indicates that the test failed, and the first electronic device is faulty.

5. The method according to claim 4, characterized in that The method further comprises: The test report indicates that the test failed, and the test report also includes the radio frequency band in which the radio frequency system in the first electronic device operated when the first electronic device captured the abnormal image and the camera identifier of the camera that captured the abnormal image.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The test report is displayed.

7. The method according to any one of claims 1 to 6, characterized in that: The step of inputting at least one second image in each of the M*N groups of first images into the first neural network model to obtain a set of recognition information includes: Deriving the at least one second image to obtain at least one first gradient image, wherein the at least one first gradient image corresponds to the at least one second image one-to-one; the first gradient image includes a key feature of the second image, or the first gradient image includes the key feature of the second image and a noise feature corresponding to noise in the second image; wherein the key feature includes a feature of a pixel in the first image whose adjacent pixel change is greater than a preset change threshold; The at least one first gradient image is input into the first neural network model.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: The second image and identification information of the second image are sent to the server; wherein, when the second image is the abnormal image, the second image is used by the server as a negative sample to train the first neural network model; and when the second image is the normal image, the second image is used by the server as a positive sample to train the first neural network model.

9. The method according to claim 8, characterized in that The sending the second image and the identification information of the second image to the server includes: The first gradient image of the second image and identification information of the second image are sent to the server.

10. The method according to any one of claims 1 to 9, characterized in that Before inputting at least one second image in each of the M*N groups of first images into the first neural network model to obtain a group of recognition information, the method further includes: Sending a download request to a server, where the download request is used to request downloading of the first neural network model; Based on the instruction information of the server, the first neural network model is downloaded and saved.

11. An electronic device, characterized in that: The electronic device comprises: a memory, a display screen, one or more processors, and a communication module; the memory, the communication module, and the display screen are coupled to the processor; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 10.

Images