Playback anomaly detection method, electronic device, and computer-readable storage medium

Abstract

The embodiments of the present application disclose a playback anomaly detection method, an electronic device, and a computer-readable storage medium. The method may include: the electronic device obtains data to be processed containing preset tags, the data to be processed is data obtained by collecting images and / or sounds played by the electronic device to be detected during the process of the electronic device to be detected playing a target image and / or target audio, and the target image and target audio both contain pre-implanted preset tags; the electronic device obtains a tag detection result by identifying the preset tags in the data to be processed; the electronic device obtains a playback anomaly detection result based on the tag detection result. The embodiments of the present application convert complex playback anomaly detection into preset tag detection by implanting preset tags in videos, pictures, and audio, thereby reducing detection difficulty and improving detection efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a playback anomaly detection method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Electronic devices can generally be used to play sound, as well as videos or pictures. When playing videos or pictures, electronic devices with displays (such as mobile phones, tablets, and large-screen devices) may experience display anomalies such as black screens, flickering screens, distorted screens, and freezes due to hardware or software rendering issues. Similarly, electronic devices with sound playback modules may also experience problems such as abnormal audio playback quality during sound playback.

[0003] Currently, methods for detecting playback anomalies in electronic devices suffer from problems such as high detection difficulty and low detection efficiency. Taking display anomalies as an example, methods for detecting display anomalies in electronic devices generally include similarity algorithm detection and defect algorithm detection. The similarity algorithm detection process may include: using the display device to be detected to play various video scenes (for example, TV series, animations, games, etc.); using a high-definition camera to capture the display screen of the display device to be detected to obtain the display image, and using built-in software to take a screenshot; calculating the similarity between the image captured by the camera and the image captured by the built-in software, and determining that a display anomaly has occurred if the similarity is less than a certain threshold. The defect algorithm detection process may include: collecting a large number of images of video display anomalies, annotating the images of display anomalies, and using a training dataset including the annotated images to train a model; after the model training is completed, using a high-definition camera to capture the display image of the electronic device to be detected, and then inputting the captured display image into the trained model to obtain the display anomaly detection result. Due to the complexity of video scenes and other reasons, current display anomaly detection is difficult and inefficient. Summary of the Invention

[0004] The embodiments of the present application provide a playback anomaly detection method, an electronic device, and a computer-readable storage medium to solve the problems of high detection difficulty and low detection efficiency in existing playback anomaly detection methods.

[0005] In the first aspect, an embodiment of the present application provides a playback anomaly detection method, which specifically includes: obtaining data to be processed containing preset marks, the data to be processed is data obtained by collecting images and / or sounds played by the electronic device to be detected during the process of the electronic device to be detected playing a target image and / or target audio, and the target image and target audio both contain pre-implanted preset marks; obtaining a mark detection result by identifying the preset mark in the data to be processed; and obtaining a playback anomaly detection result based on the mark detection result.

[0006] The embodiment of the present application pre-implants a preset mark in the target image or target audio being played, and then collects the image or sound played by the electronic device to be detected during the process of the target image or target audio being played by the electronic device to be detected to obtain data to be processed, and the data to be processed contains the preset mark. Then, by detecting the preset mark in the data to be processed, a playback anomaly detection result is obtained, thereby converting complex playback anomaly detection into preset mark detection, reducing the detection difficulty and improving the detection efficiency.

[0007] As an example, and not a limitation, the target image is a video, and each frame within the video is pre-implanted with a preset marker. In this case, the preset markers include one or more simple icons, such as smiley faces, and numbers. The icons are identical in each frame. Furthermore, a corresponding number is inserted into each frame according to the order of the images in the video. For example, the number inserted into the first frame is 1, the number inserted into the tenth frame is 10, and the number inserted into the twentieth frame is 20.

[0008] After the simple icons and numbers are implanted into the video, the video is played using a large-screen device (e.g., a smart screen, etc.), and while the video is playing on the large-screen device, a high-definition camera is used to shoot the display screen of the large-screen device to collect the image to be processed. The large-screen device is the above-mentioned electronic device to be detected. Since simple icons, numbers and other marks are implanted in each frame of the video in advance, it is possible to determine whether a display abnormality problem occurs by identifying the simple icons and numbers in the collected image to be processed. Specifically, on the one hand, it is possible to determine whether a freeze occurs by identifying the numbers in the image to be processed. For example, after the video is played for 2 minutes, the image to be processed is collected, and the number in the image to be processed is identified as 990, that is, the image to be processed is the 990th frame. It is expected that at 2 minutes, the image frame expected to be played by the large-screen device is the 1000th frame. At this time, since the image frame expected to be played is inconsistent with the image frame actually played, it can be determined that a freeze occurs during the playback process. On the other hand, the integrity of the simple icons in the image to be processed can be used to determine whether display abnormalities such as flower screen, flashing screen and black screen occur. For example, if the smiley face icon in the image being processed is detected to be inconsistent with the pre-implanted smiley face icon, that is, the displayed smiley face icon is inconsistent with the pre-implanted smiley face icon, then it is considered that there are display anomalies such as screen distortion, screen flickering, and black screen during playback. This shows that by pre-implanting preset markers in the video, the complex problem of video display detection (for example, games, movies, and animations) is transformed into the problem of detecting preset markers, which reduces the detection difficulty and improves the detection efficiency.

[0009] In some possible implementations of the first aspect, the preset mark may include a first preset mark and / or a second preset mark.

[0010] In this implementation, the second preset marker can be used to indicate the order of the image or sound to be processed. This order refers to the position in the image sequence or sound sequence. For example, if the second preset marker is a number and the number 1 for a certain image to be processed is 1, it indicates that the image to be processed is the first frame in the image sequence. Specifically, the second preset marker can be a number, a letter, a combination of letters and numbers, a symbol, or an icon combination. An icon combination refers to a combination of at least two icons, and this icon combination can indicate the order of precedence.

[0011] The first preset marker can be used to detect display anomalies of the image to be processed or abnormal audio playback quality of the sound to be processed. For example, a simple icon can be embedded in the image to be processed, and the integrity of the simple icon in the image to be processed (specifically determined by the marker category and / or icon position) can be used to determine whether display anomalies such as a black screen, a distorted screen, or a flickering screen occur.

[0012] In some possible implementations of the first aspect, the data to be processed includes an image to be processed, the target image is a video or a picture, and each frame of the target image is pre-implanted with a preset mark.

[0013] It should be noted that, in some other embodiments, the preset mark may not be embedded in every frame of image. In contrast, embedding the preset mark in every frame of image can improve the detection accuracy.

[0014] In some possible implementations of the first aspect, the first preset mark includes at least one first preset icon, and the second preset mark includes a number or an icon combination, which may include at least two second preset icons.

[0015] The first preset icon may be the same as the second preset icon. For example, the first preset icon is a smiling face pattern.

[0016] In some possible implementations of the first aspect, the preset marker includes a first preset marker. In this case, the process of obtaining a marker detection result by identifying the preset marker in the data to be processed may include: identifying the first preset marker in the image to be processed to obtain a first marker detection result; wherein the marker detection result includes the first marker detection result. The first marker detection result is a detection result corresponding to the first preset marker.

[0017] In some possible implementations of the first aspect, the specific process of identifying the first preset marker in the image to be processed and obtaining the first marker detection result may include:

[0018] Inputting the image to be processed into a pre-trained first detection model to obtain a first output result of the first detection model, and using the first output result as a first label detection result;

[0019] The first output result may include a marker category and / or position information, wherein the position information is used to describe the position of the first preset marker in the image to be processed, and the marker category is used to describe the category of the first preset marker. The marker category may include a normal category and an abnormal category.

[0020] In some possible implementations of the first aspect, the marker detection result includes a first marker detection result, which includes a marker category. In this case, the process of obtaining a playback anomaly detection result based on the marker detection result may include: determining, based on the marker category, whether a first preset marker belonging to the anomaly category exists in the image to be processed; and if the first preset marker belonging to the anomaly category exists in the image to be processed, determining that a first type of display anomaly has occurred in the electronic device to be detected. This first type of display anomaly may include display issues such as a distorted screen, a black screen, or a flickering screen.

[0021] In some possible implementations of the first aspect, if the first mark detection result also includes position information, after determining whether there is a first preset mark belonging to an abnormal category in the image to be processed according to the mark category, it also includes: if there is no first preset mark belonging to an abnormal category in the image to be processed, determining whether the position information of each first preset mark is consistent with the corresponding preset position information; if there is at least one first preset mark whose position information is inconsistent with the corresponding preset position information, determining that a first type of display abnormality occurs in the electronic device to be detected.

[0022] In some possible implementations of the first aspect, the preset marker includes a second preset marker. In this case, the process of obtaining the marker detection result by identifying the preset marker in the data to be processed may include: identifying the second preset marker in the image to be processed to obtain a second marker detection result; wherein the marker detection result includes the second marker detection result. The second marker detection result is a detection result corresponding to the second preset marker.

[0023] In some possible implementations of the first aspect, the above-mentioned process of identifying the second preset marker in the image to be processed and obtaining the second marker detection result may include: inputting the image to be processed into a pre-trained second detection model, obtaining a second output result of the second detection model, and using the second output result as the second marker detection result; wherein the second output result is used to describe the position of the image to be processed in the image sequence of the target image, and the image to be processed is an image captured at a preset time.

[0024] For example, the second preset mark implanted in the image to be processed is an icon combination, and the image to be processed is input into the second detection model to obtain a second output result of the second detection model, which can represent the number corresponding to the icon combination.

[0025] Alternatively, a second predetermined marker in the image to be processed is identified by optical character recognition to obtain a recognition result, which is used as the second marker detection result; wherein the recognition result is used to describe the position of the image to be processed in the image sequence of the target image, and the image to be processed is an image captured at a predetermined time. For example, if the second predetermined marker embedded in the first frame of the image is 1, the number identified in the first frame of the image by optical character recognition is 1.

[0026] In some possible implementations of the first aspect, the marker detection result includes a second marker detection result. In this case, the process of obtaining the playback abnormality detection result based on the marker detection result may include: if the second marker detection result is less than a target value, determining that a second type of display abnormality has occurred in the electronic device to be detected, where the target value is a value obtained by identifying a second preset marker in a target image frame, where the target image frame is an image frame that the electronic device to be detected is expected to play at a preset time, and the target image includes the target image frame. The second type of display abnormality may be a freeze abnormality.

[0027] In this implementation, the image frame expected to be played is the expected playback position, and the image to be processed is the actual playback position. By comparing the second mark detection result and the target value, it is determined whether the expected playback position and the actual playback position are consistent. If the expected playback position and the actual playback position are inconsistent, it is determined that a jamming abnormality has occurred. Conversely, if the expected playback position and the actual playback position are consistent, it is determined that no jamming abnormality has occurred. For example, the image frame expected to be played is the 10000th frame image. At this time, the target value is 10000; the number implanted in the image to be processed (actual playback position) is 9800, that is, the image to be processed is the 9800th frame image. Since 9800 < 10000, it is determined that a jamming abnormality has occurred.

[0028] In some other embodiments, whether a jamming anomaly has occurred can also be determined by comparing the image frame corresponding to the expected playback position with the image frame corresponding to the actual playback position. For example, an icon combination is embedded in the image to be processed. If the icon combination in the expected playback image frame and the image to be processed (the image frame actually played) are consistent, then the expected playback position and the actual playback position are considered to be consistent, and it is determined that no jamming anomaly has occurred. Conversely, if the icon combination in the two images is inconsistent, then it is determined that the expected playback position and the actual playback position are inconsistent, and it is determined that a jamming anomaly has occurred.

[0029] In some possible implementations of the first aspect, the first detection model and the second detection model may be artificial intelligence models. The training process of the first detection model may include obtaining a training dataset comprising videos or images embedded with preset markers. In this case, the preset markers may include markers for normal categories and markers for abnormal categories. The constructed model is trained using the training dataset to obtain a trained model. It is understood that the trained model can identify whether a marker in an image is of a normal or abnormal category, and can also identify the location of the marker in the image.

[0030] In some possible implementations of the first aspect, the image to be processed includes a sound to be processed. In this case, the process of obtaining a marker detection result by identifying a preset marker in the data to be processed may include: separating a preset marker from the sound to be processed, the preset marker including an audio signal in a preset frequency band, which is higher or lower than the frequency band of the target audio; identifying the preset marker to obtain a marker detection result.

[0031] In this implementation, an easily separable sound marker is pre-implanted into the target audio. The sound marker can exist as a high-frequency or low-frequency audio signal. By identifying the sound marker implanted in the audio, the audio playback quality of the electronic device to be tested is detected to see if there is any abnormality.

[0032] In a second aspect, an embodiment of the present application provides a playback anomaly detection device, which may include:

[0033] an acquisition module, configured to acquire data to be processed containing preset markers, the data to be processed being data obtained by capturing images and / or sounds played by the electronic device to be detected during the process of the electronic device to be detected playing a target image and / or target audio, wherein both the target image and target audio contain pre-implanted preset markers;

[0034] A marker detection module is used to obtain a marker detection result by identifying a preset marker in the data to be processed;

[0035] The anomaly detection module is used to obtain a playback anomaly detection result based on the marked detection result.

[0036] In some possible implementations of the second aspect, the preset mark may include a first preset mark and / or a second preset mark.

[0037] In some possible implementations of the second aspect, the data to be processed includes an image to be processed, the target image is a video or a picture, and each frame of the target image is pre-implanted with a preset marker.

[0038] In some possible implementations of the second aspect, the first preset mark includes at least one first preset icon, the second preset mark includes a number or an icon combination, and the icon combination includes at least two second preset icons.

[0039] In some possible implementations of the second aspect, the preset mark includes a first preset mark, and the mark detection module is specifically used to: identify the first preset mark in the image to be processed and obtain a first mark detection result; wherein the mark detection result includes the first mark detection result.

[0040] In some possible implementations of the second aspect, the above-mentioned marker detection module is specifically used to: input the image to be processed into a pre-trained first detection model, obtain a first output result of the first detection model, and use the first output result as the first marker detection result; the first output result includes a marker category and / or position information, the position information is used to describe the position of the first preset marker in the image to be processed, and the marker category is used to describe the category of the first preset marker.

[0041] In some possible implementations of the second aspect, the mark detection result includes a first mark detection result, and the first mark detection result includes a mark category; the above-mentioned abnormality detection module is specifically used to: determine whether there is a first preset mark belonging to the abnormality category in the image to be processed according to the mark category; if there is a first preset mark belonging to the abnormality category in the image to be processed, determine that the electronic device to be detected has a first type of display abnormality.

[0042] In some possible implementations of the second aspect, the first mark detection result also includes position information, and the above-mentioned abnormality detection module is specifically used to: if there is no first preset mark belonging to the abnormal category in the image to be processed, determine whether the position information of each first preset mark is consistent with the corresponding preset position information; if there is at least one first preset mark whose position information is inconsistent with the corresponding preset position information, determine that the electronic device to be detected has a first type of display abnormality.

[0043] In some possible implementations of the second aspect, the preset mark includes a second preset mark, and the above-mentioned mark detection module is specifically used to: identify the second preset mark in the image to be processed and obtain a second mark detection result; wherein the mark detection result includes the second mark detection result.

[0044] In some possible implementations of the second aspect, the marker detection module is specifically configured to: input the image to be processed into a pre-trained second detection model, obtain a second output result of the second detection model, and use the second output result as a second marker detection result; wherein the second output result is used to describe a position of the image to be processed in an image sequence of the target image, where the image to be processed is an image captured at a preset time;

[0045] Alternatively, a second preset mark in the image to be processed is identified by optical character recognition to obtain a recognition result, and the recognition result is used as the second mark detection result; wherein the recognition result is used to describe the position of the image to be processed in the image sequence of the target image, and the image to be processed is an image captured at a preset time.

[0046] In some possible implementations of the second aspect, the mark detection result includes a second mark detection result, and the above-mentioned abnormality detection module is specifically used to: if the second mark detection result is less than the target value, determine that the electronic device to be detected has a second type of display abnormality, the target value is a value obtained by identifying the second preset mark in the target image frame, the target image frame is the image frame expected to be played by the electronic device to be detected at a preset time, and the target image includes the target image frame.

[0047] In some possible implementations of the second aspect, the image to be processed includes a sound to be processed, and the above-mentioned marker detection module is specifically used to: separate a preset marker from the sound to be processed, the preset marker including an audio signal in a preset frequency band, and the preset frequency band is higher or lower than the frequency band of the target audio; identify the preset marker and obtain a marker detection result.

[0048] The above-mentioned playback anomaly detection device has the function of implementing the above-mentioned playback anomaly detection method. This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, and the modules can be software and / or hardware.

[0049] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods of the first aspect described above when executing the computer program.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the above-mentioned first aspects.

[0051] In a fifth aspect, embodiments of the present application provide a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement any of the methods described in the first aspect. The chip system can be a single chip or a chip module consisting of multiple chips.

[0052] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute any of the methods described in the first aspect above.

[0053] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the hardware architecture of the electronic device 100 provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of the software architecture of the electronic device 100 provided in an embodiment of the present application;

[0056] Figure 3 A schematic block diagram of the playback anomaly detection system architecture provided in an embodiment of the present application;

[0057] Figure 4 A schematic flow chart of a method for detecting abnormal playback provided in an embodiment of the present application;

[0058] Figure 5 A simple icon diagram provided for an embodiment of the present application;

[0059] Figure 6 A schematic diagram of an icon combination provided in an embodiment of the present application;

[0060] Figure 7 An image schematic diagram provided for an embodiment of the present application;

[0061] Figure 8 A simple icon diagram provided for an embodiment of the present application;

[0062] Figure 9 A schematic diagram of an image after an icon is implanted according to an embodiment of the present application;

[0063] Figure 10 A schematic diagram of the jamming anomaly detection provided in an embodiment of the present application;

[0064] Figure 11 Another schematic diagram of stuttering anomaly detection provided in an embodiment of the present application;

[0065] Figure 12 A schematic diagram showing an anomaly detection process provided in an embodiment of the present application;

[0066] Figure 13 A schematic diagram of an image after implantation of a preset marker provided in an embodiment of the present application;

[0067] Figure 14 Schematic diagram of the black screen, distorted screen and flickering screen detection process provided in the embodiment of the present application;

[0068] Figure 15 A schematic diagram of the jamming anomaly detection process provided in an embodiment of the present application;

[0069] Figure 16 Schematic diagram of the playback anomaly detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0071] In an embodiment of the present application, after a preset marker is embedded in a video or audio, the video or audio with the embedded preset marker is played using an electronic device to be detected. During the process of the electronic device to be detected playing the video or audio, an acquisition device captures the image or sound played by the electronic device to be detected to obtain data to be processed. Then, an electronic device with data processing capabilities processes the collected data to identify the preset marker in the data to be processed, obtains a marker detection result, and then obtains a playback anomaly detection result based on the marker detection result.

[0072] Wherein, the above-mentioned electronic device to be detected refers to an electronic device that needs to perform playback anomaly detection, and playback anomaly can be divided into picture or video display anomaly, and audio playback anomaly. In general, if it is necessary to perform display anomaly detection, the electronic device to be detected generally needs to have a video (or picture) playback function, and generally has a display screen, for example, a smart phone, a large-screen device (smart screen, etc.) and a tablet computer and other display devices. If it is necessary to perform audio playback anomaly detection, the electronic device to be detected generally needs to have an audio playback function, for example, an audio playback device such as a mobile phone, a smart TV and a tablet computer. Of course, if it is necessary to perform display anomaly detection and audio playback anomaly detection at the same time, that is, the display anomaly and audio playback anomaly of the electronic device to be detected are detected, the electronic device to be detected generally needs to have a video (or picture) playback function and an audio playback function at the same time.

[0073] The aforementioned method for capturing images played by the electronic device to be tested can include, but is not limited to, camera photography or screen recording. Specifically, when the electronic device to be tested plays a video or image, the image played on the display screen of the electronic device to be tested can be captured using a high-definition camera, or the electronic device to be tested can perform screen recording or other operations to capture the image to be processed. Capturing audio played by the electronic device to be tested generally requires the use of an audio capture device, which can be independently provided outside the electronic device to be tested or integrated within the electronic device to be tested.

[0074] That is to say, the above-mentioned acquisition device can be the electronic device to be detected itself. For example, when the electronic device to be detected is a mobile phone, the mobile phone can capture the image to be processed by screen recording, or the mobile phone can capture the sound to be processed by recording. At this time, the mobile phone serves as an acquisition device; it can also be a device independent of the electronic device to be detected. For example, the electronic device to be detected is a large-screen device. When the large-screen device plays a video or picture, the display screen of the large-screen device is photographed by a high-definition camera to capture the image to be processed. At this time, the high-definition camera serves as the acquisition device.

[0075] The electronic device with data processing function can be, but is not limited to, a mobile phone, a personal computer, a tablet or a cloud server. Generally, after the acquisition device acquires the sound to be processed or the image to be processed, the electronic device with data processing function can obtain the image to be processed or the sound to be processed, and process the image to be processed or the sound to be processed to obtain the playback abnormality detection result. Among them, the electronic device to be detected, the acquisition device and the electronic device for processing the data to be processed can be the same device. For example, the electronic device to be detected, the acquisition device and the electronic device for processing the data to be processed are all mobile phones, that is, the mobile phone is used to play video or audio, and the mobile phone can capture the image to be processed or the sound to be processed by recording the screen or recording the sound, and then the mobile phone can process the collected image to be processed or the sound to be processed; or they can not be the same device, in which case, the three are not integrated into the same electronic device. The three can be independent electronic devices. For example, the electronic device to be detected is a large-screen device, the acquisition device is a high-definition camera, and the electronic device for data processing is a personal computer. Of course, two of the three can also be integrated into one electronic device. For example, the electronic device to be detected is a large-screen device, and the acquisition device and the electronic device for data processing are both integrated into a mobile phone, that is, the mobile phone is used to acquire the video or audio played by the large-screen device, and the mobile phone is used to process the acquired data. For another example, the electronic device to be detected and the acquisition device are both integrated into a mobile phone, that is, the mobile phone is used to play video or audio, and the mobile phone is used to acquire images to be processed or sounds to be processed. Then, the mobile phone transmits the acquired images to be processed and sounds to be processed to a personal computer, and the personal computer processes the acquired data. In this case, the electronic device for data processing is a personal computer.

[0076] In specific applications, the types of electronic devices mentioned above can be any, as an example but not a limitation, such as Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0078] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0079] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0080] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0081] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0082] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device 100.

[0083] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170.

[0084] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. Both the I2S interface and the PCM interface can be used for audio communication.

[0085] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality.

[0086] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0087] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0088] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0089] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0090] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0091] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0092] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0093] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0094] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0095] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0096] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0097] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0098] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0099] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0100] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0101] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0102] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0103] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0104] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0105] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0106] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0107] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0108] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0109] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0110] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0111] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0112] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0113] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0114] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0115] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0116] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0117] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0118] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0119] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0120] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0121] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0122] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0123] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the electronic device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0124] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0125] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0126] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0127] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0128] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0129] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0130] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0131] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0132] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0133] The application layer can include a series of application packages.

[0134] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0135] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0136] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0137] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0138] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0139] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0140] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0141] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0142] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0143] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0144] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0145] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0146] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0147] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0148] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0149] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0150] A 2D graphics engine is a drawing engine for 2D drawings.

[0151] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0152] The following describes the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0153] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a still image or video through the camera 193.

[0154] In some embodiments, when the electronic device 100 is an electronic device for data processing, after the electronic device 100 captures an image played on the display screen of the electronic device to be detected through the camera 193, the processor 110 of the electronic device 100 processes the image to be processed to identify a preset mark in the image to be processed and obtain a display abnormality detection result. Alternatively, after the electronic device 100 captures an audio signal played by the electronic device to be detected through the microphone 170C, the processor 110 of the electronic device 100 processes the sound to be processed to identify a preset mark in the sound to be processed and obtain an audio playback quality detection result.

[0155] The following first introduces the system architecture and scenarios that may be involved in the embodiments of the present application.

[0156] See Figure 3 , which is a schematic block diagram of the playback anomaly detection system architecture provided in an embodiment of the present application. The architecture may include a server 31, a transmission link 32, an electronic device to be detected 33, a collection device 34, and a data processing device 35. The server 31 can exchange data with the electronic device to be detected 33 via the transmission link 32. The transmission link 32 may specifically be a communication network, such as the Internet.

[0157] The server 31 is used to store target images and / or target audio files embedded with preset markers. In a specific application, the preset markers are first inserted into videos, images, and audio files using a script, and then the videos, images, and audio files with the preset markers are stored on the server. Typically, the videos and audio files stored on the server cover a wide range of scenes. For example, videos may include scenes from TV series, animations, and games.

[0158] The electronic device to be detected 33 refers to the electronic device that needs to be detected for playback anomalies. It may include but is not limited to mobile phones, tablets, large-screen devices such as smart TVs, personal computers, and other playback devices. The electronic device to be detected 33 is used to play the target image and / or target audio that has been embedded with a preset mark. In a specific application, when the detection electronic device 33 needs to play video or audio, it can obtain the video or audio with the preset mark from the server 31 via the transmission link 32 and then play the obtained video or audio.

[0159] It is understandable that the electronic device 33 to be detected can only play videos or pictures with preset marks implanted, or only play audio with preset marks implanted, or can also play videos with preset marks implanted and audio with preset marks implanted at the same time. For example, a TV series generally includes video and audio, and both the video and audio of the TV series are implanted with preset marks. When the smart TV plays the TV series, it can play videos with preset marks implanted and audio with preset marks implanted at the same time.

[0160] The acquisition device 34 is used to capture the sound and / or image played by the electronic device to be detected 33 and then transmit the captured sound and / or image to the data processing device 35. It is understood that the acquisition device can simultaneously capture both the sound and image played by the electronic device to be detected. For example, if the electronic device to be detected is a smart TV, and the video and audio of the TV series played by the smart TV are embedded with preset tags, the acquisition device can simultaneously capture both the image and the sound to be processed. The acquisition device can be, but is not limited to, a high-definition camera and an audio acquisition device.

[0161] The data processing device 35 is used to obtain the audio and / or image to be processed collected by the acquisition device, and process the obtained audio or image to be processed to obtain a playback anomaly detection result. The data processing device 35 can be, but is not limited to, an electronic device with data processing capabilities, such as a personal computer, a mobile phone, or a server.

[0162] It should be noted that, in some embodiments, Figure 3 The system architecture may not include the server 31. In this case, the target image and / or target audio with the preset marker implanted may be stored on an electronic device (e.g., a personal computer), which may be connected to the electronic device to be detected via a data connection cable (e.g., a USB data cable) to transmit the target image and / or target audio with the preset marker implanted to the electronic device to be detected. Alternatively, the target image and / or target audio with the preset marker implanted may be directly stored on the electronic device to be detected 31, eliminating the need for the server 31 and the transmission link 32.

[0163] In some embodiments, the electronic device to be detected 33, the acquisition device 34 and the data processing device 35 can be specifically manifested as an electronic device, for example, as a mobile phone, that is, the mobile phone not only serves as the electronic device to be detected, but also as an acquisition device and a data processing device. Specifically, the mobile phone plays a video with a preset mark implanted. During the video playback process, the mobile phone collects the image to be processed by recording the screen. Then, the mobile phone processes the collected image to be processed to obtain a playback abnormality detection result.

[0164] Of course, in some other embodiments, any two of the electronic device to be detected 33, the acquisition device 34 and the data processing device 35 can be externally manifested as one electronic device. For example, the electronic device to be detected 33 is a mobile phone. While playing a video, the mobile phone captures the image to be processed by recording the screen and transmits the image to be processed to a personal computer. At this time, the acquisition device 34 is a mobile phone and the data processing device 35 is a personal computer.

[0165] In the embodiments of the present application, playback anomalies may include display anomalies and audio playback quality anomalies. Display anomalies refer to problems such as freezes, black screens, distorted screens, and flickering screens when the display device is playing a video or image. These problems may be caused by the hardware components or software rendering of the display device. Audio playback quality anomalies refer to abnormal audio playback quality (for example, freezes, etc.) during audio playback by the audio playback device.

[0166] In specific applications, playback anomaly detection can be performed in situations such as new product launches, product maintenance inspections, and competitive product comparison tests. For example, before a new mobile phone is released, the playback anomaly detection solution provided in the embodiments of this application can be used to detect display anomalies and audio playback quality anomalies on the new mobile phone. Of course, in addition to scenarios such as new product launches, product maintenance inspections, and competitive product comparison tests, the playback anomaly detection solution provided in the embodiments of this application can also be applied to other scenarios, and the application scenarios are not limited here.

[0167] It should be noted that the system architecture and application scenarios mentioned above are merely examples and do not limit the system architecture and application scenarios of the embodiments of the present application.

[0168] After introducing the system architecture and application scenarios that may be involved in the embodiments of the present application, the playback anomaly detection process will be introduced with reference to the accompanying drawings.

[0169] See Figure 4 , is a schematic flow chart of a playback anomaly detection method provided in an embodiment of the present application. The method can be applied to an electronic device having a data processing function, which can be the aforementioned data processing device 35 or electronic device 100. The method may include the following steps:

[0170] Step S401: The electronic device obtains data to be processed containing preset tags. The data to be processed is data obtained by collecting images and / or sounds played by the electronic device to be detected during the process of the electronic device to be detected playing the target image and / or target audio. The target image and target audio both contain pre-implanted preset tags.

[0171] In a specific application, if the electronic device to be detected plays both the target image and the target audio, the collected data to be processed includes both the sound to be processed and the image to be processed. If the electronic device to be detected plays either the target image or the target audio, the collected data to be processed may include either the image to be processed or the sound to be processed. In other words, the data to be processed may include both the image to be processed and the sound to be processed, or may be either the image to be processed or the sound to be processed.

[0172] The target image can be a video or a picture, which has a pre-marked image embedded in it. Typically, each frame of the target image is embedded with the pre-marked image. However, in practical applications, the pre-marked image can be embedded selectively, for example, at a predetermined interval. Because each frame may exhibit display anomalies, embedding the corresponding pre-marked image in each frame is preferred.

[0173] In some embodiments, the preset marker embedded in the video or picture may include a first preset marker and / or a second preset marker, that is, the image in the target image may contain the first preset marker or the second preset marker, or may contain both the first preset marker and the second preset marker.

[0174] The first preset marker can be used to detect a first type of display anomaly, which may include a black screen, a distorted screen, or a flickering screen. In other words, the electronic device can determine whether the electronic device under inspection has a display issue such as a black screen, a distorted screen, or a flickering screen by detecting the first preset marker contained in the image frame. Generally, the first preset marker embedded in each image frame is the same.

[0175] The first preset mark may specifically include one or more simple icons, and the shape and type of the simple icons may be arbitrary. As an example and not a limitation, see Figure 5 The simple icon diagram provided by the embodiment of the present application is shown. The first preset mark may include icon 51, icon 52 and icon 53. In a specific application, icons 51, 52 and 53 may be embedded in each frame of the video. It is understandable that the icons included in the first preset mark may be more than Figure 5 The icons in the example are more complex or simpler, and the shapes and types of icons are not limited to Figure 5 shown.

[0176] It should be noted that, compared to a first preset marker containing only one icon, including multiple icons in the first preset marker can cover a larger display area, thereby making the display anomaly detection results more accurate. Because every area of the display screen of the electronic device to be detected may experience display issues such as a black screen, a distorted screen, and a flickering screen, using multiple icons can cover as much of the display area as possible.

[0177] In addition, the icon included in the first preset mark is generally preferably a simple icon, which is easier to recognize and can improve the icon recognition rate. Of course, the icon included in the first preset mark can also be a more complex icon.

[0178] The second preset mark can be used to detect the second type of display abnormality, which may refer to a stuttering abnormality. That is, the electronic device can determine whether the electronic device to be detected has a stuttering abnormality by detecting the second preset mark in the image frame. The second preset mark implanted in each frame of the image is used to describe the order of the frame images, and the order of order refers to the order in the image sequence. For example, the image sequence of a video can be divided into the first frame image, the second frame image, the third frame image...the nth frame image according to the order of playback, where n is a positive integer. In this case, the second preset mark is a number, that is, by implanting a corresponding number in each frame of the image, the order of each frame of the image in the image sequence is represented.

[0179] It can be understood that the second preset mark in each frame image is generally different. For example, when the second preset mark is a number, the second preset mark implanted in the first frame image of the video is 1, the second preset mark implanted in the second frame image is 2, the second preset mark implanted in the third frame image is 3, and so on. The second preset mark implanted in the nth frame image is n.

[0180] In specific applications, the second preset mark can be expressed in the digital form mentioned above, as well as in other forms, as long as it can play the role of representing the order. As an example but not limitation, the first preset mark can be expressed as Arabic numerals, that is, 1, 2, 3, 4...n is used to represent the order of each frame image in the image sequence; it can be expressed as Chinese numerals, that is, one, two, three, four... are used to represent the order of each frame image in the image sequence; it can be expressed as a combination of numbers and letters, for example, the second preset mark implanted in the first frame image is A1, the second preset mark implanted in the second frame image is A2, and the second preset mark implanted in the third frame image is A3; it can also be expressed as a combination of icons, that is, a combination of at least two icons is used to represent the order. See. Figure 6 The schematic diagram of the icon combination provided by the embodiment of the present application is shown as follows: Figure 6 As shown, it includes icon combination 61, icon combination 62, and icon combination 63. Icon combination 61 is embedded in the first frame image, icon combination 62 is embedded in the second frame image, and icon combination 63 is embedded in the third frame image. At this time, icon combination 61 represents the number 1, icon combination 62 represents the number 2, and icon combination 63 represents the number 3. Of course, the second preset mark can also be expressed in other forms, for example, using a combination of special symbols and numbers to represent the order of each frame image in the image sequence, which will not be listed here one by one.

[0181] In actual applications, only the corresponding first preset mark or the second preset mark can be embedded in the image, or both the first preset mark and the second preset mark can be embedded at the same time. Figure 7 The image schematic diagram provided by the embodiment of the present application is shown, wherein, Figure 7 The image 71 in (a) is the original image, that is, the image 71 is an image without any preset mark implanted therein. Figure 7 (b) shows an image 72 in which a first preset mark has been implanted. Icons 73, 74 and 75 in the image 72 are identical to those in the image 72. Figure 5 In this case, the first preset mark embedded in the image 72 includes icon 73, icon 74 and icon 75. In specific applications, Figure 5 The three simple icons in are implanted into image 71 to obtain image 72. Figure 7 (c) shows an image 76 in which a second preset mark has been implanted. The second preset mark is a number 77, which is 10,000 for example. Figure 7 (d) shows an image 78 in which a second preset mark has been implanted. Figure 6 Icon combination 63 in . Figure 7 (e) shows an image 79 in which a first preset mark and a second preset mark have been embedded. The first preset mark in the image 79 includes simple icons 711 to 713 , and the second preset mark is a number 710 , which is 10,000 for example.

[0182] It is understandable that the position of the preset mark in the image is arbitrary.

[0183] Similar to the principle and process of embedding preset markers in videos or pictures mentioned above, preset markers can also be embedded in target audio. The preset markers embedded in the target audio are generally easy-to-separate sound markers, which generally exist in the form of signals. For example, a high-frequency signal is embedded in the sound signal at each moment. The frequency of this high-frequency signal is higher than the frequency of the target audio and cannot be heard by the human ear, but can be detected by a machine. The high-frequency signal embedded in the target audio carries corresponding marking information. For example, the marking information carried by the high-frequency information is information that characterizes the sequence of the sound signals.

[0184] Similarly, a first preset marker and / or a second preset marker may be embedded in the target audio. The first preset marker may be used to detect a first type of audio playback anomaly, which generally refers to abnormal audio playback quality. The second preset marker may be used to detect a second type of audio playback anomaly, which may refer to a freeze or freeze.

[0185] The preset markers embedded in the target audio can exist as high-frequency signals or low-frequency signals, where the frequency of the low-frequency signal is lower than that of the target audio. Both high-frequency and low-frequency signals can carry or represent marker information, such as sequence information or playback quality information.

[0186] Based on the methods and principles mentioned above, preset tags can be embedded in videos, images, and audio. Generally, the collected video sources and audio should cover as many scenes as possible. For example, for videos, you can collect video sources for scenes such as TV series, animations, and games, and use scripts to embed corresponding preset tags in the video sources for each scene.

[0187] After obtaining the target image and target audio that have been implanted with preset marks, the electronic device to be detected can first obtain the target image and / or target audio that needs to be played, and then the electronic device to be detected plays the acquired target image and / or target audio. Playing the target image refers to displaying the corresponding image on the display screen of the electronic device to be detected. For example, when the target image is an animated video and the electronic device to be detected is a smart phone, when the smart phone plays the animated video, the corresponding image will be displayed on the display screen of the smart phone. In the process of the electronic device to be detected playing the video or audio, the data to be processed can be acquired by the acquisition device. For example, the display screen of a smart TV that is playing a TV series video can be photographed by a high-definition camera to obtain the image to be processed, and the TV series video has been implanted with preset marks. After acquiring the data to be processed, the electronic device can acquire the data to be processed, and then the electronic device can identify the preset marks in the data to be processed to obtain the corresponding mark detection results.

[0188] Step S402: The electronic device obtains a marker detection result by identifying a preset marker in the data to be processed.

[0189] It should be noted that if the data to be processed includes a first preset marker, the marker detection result may include the marker category and / or marker position. The marker category may include a normal category and an abnormal category, and the marker position refers to the position of the preset marker in the image.

[0190] It is understandable that only the mark category of the first preset mark may be identified, or only the mark position of the first preset mark may be identified. Of course, both the mark category of the first preset mark and the mark position of the first preset mark may be identified.

[0191] For example, the data to be processed includes an image to be processed, and the image to be processed includes Figure 5 The three simple icons shown in the figure are: Figure 5 The electronic device obtains the icon categories and / or icon locations of the simple icons in the image to be processed by identifying the simple icons in the image to be processed.

[0192] If the data to be processed includes a second preset marker, the marker detection result may include information indicating the order of precedence. For example, the data to be processed includes an image to be processed, and the second preset marker in the image to be processed is a number, such as 10000. The electronic device recognizes the second preset marker in the image to be processed and determines that the number in the image to be processed is 10000.

[0193] If the data to be processed includes a first preset marker and a second preset marker, the marker detection result may include marker category and / or marker position, as well as information representing the order.

[0194] In specific applications, the method for identifying the preset markers in the data to be processed can be arbitrary. For example, the electronic device can identify the preset markers through a pre-trained artificial intelligence model. In this case, the data to be processed can be input into the pre-trained model to obtain the output result of the model, which is the above-mentioned marker detection result. For another example, if the second preset marker implanted in the image to be processed is a number or text, the electronic device can use optical character recognition (OCR) to identify the text or number in the image to be processed.

[0195] In practical applications, when the data to be processed includes the sound to be processed, the sound tag in the sound to be processed can be separated first, and then the tag information carried by the sound tag can be determined, and the information specifically represented by the tag information can be determined. For example, when the sound tag represents information of a certain order, it is identified that the order represented by the sound tag in the sound to be processed is 1.

[0196] Step S403: The electronic device obtains a playback abnormality detection result based on the mark detection result.

[0197] It should be noted that the marker detection results may include the detection results of the first preset marker and / or the detection results of the second preset marker. In specific applications, the electronic device can determine whether a playback anomaly has occurred based on the detection results of the first preset marker. For example, if the electronic device to be tested experiences a display issue such as a black screen, a distorted screen, or a flickering screen, the first preset marker in the displayed image may be inconsistent with the first preset marker in the original image. Specifically, this may manifest as inconsistent categories and / or inconsistent positions. For example, if the first preset markers embedded in the original image are all normal markers, but the image to be processed contains a first preset marker of an abnormal category, then the marker categories of the original image and the image to be processed may be inconsistent, and it can be determined that the electronic device to be tested has experienced a display issue such as a black screen, a distorted screen, or a flickering screen. For another example, if the first preset marker in the original image is located at position A, but the same first preset marker in the image to be processed is located at position B, then the marker positions in the original image and the processed image are inconsistent, and it can be determined that the electronic device to be tested has experienced a display issue such as a black screen, a distorted screen, or a flickering screen. The original image here refers to the image that has not been displayed by the electronic device to be tested.

[0198] That is to say, if the mark detection result includes the mark category and / or mark position of the first preset mark, the electronic device can determine the playback abnormality detection result based on the mark category and / or mark position. The playback abnormality detection result may refer to whether the electronic device to be detected has display problems such as black screen, flowery screen and flashing screen.

[0199] The electronic device can also determine whether a jamming anomaly occurs based on the detection result of the second preset mark in the mark detection result. For display anomaly detection, the image to be processed at the preset moment is first collected, and then the electronic device identifies the second preset mark in the image to be processed at the preset moment. The second preset mark in the image to be processed can indicate which frame the image to be processed is; the electronic device then compares whether the image to be processed is consistent with the image frame expected to be played. The image frame expected to be played refers to the image frame expected to be played by the electronic device to be detected at the above-mentioned preset moment. If the image to be processed is inconsistent with the image frame expected to be played, it can be considered that a jamming anomaly has occurred.

[0200] For example, a smart TV plays a video source with embedded digital content. After 10 minutes of playback, a high-definition camera captures the image to be processed by photographing the smart TV's display. The personal computer then recognizes the number 9800 in the image to be processed, indicating that the image to be processed is the 9800th frame. The expected number of image frames to be played in 10 minutes is 10000. After 10 minutes of playback, the actual playback position is the 9800th frame, while the expected playback position is the 10000th frame. The actual playback position is smaller than the expected playback position, indicating that a pause occurred during playback and the actual playback position does not match the expected playback position.

[0201] Of course, the electronic device can determine whether a first type of display abnormality, such as a black screen, a distorted screen, or a flickering screen, occurs during video or image playback, and whether a second type of display abnormality, such as a freeze, occurs, based on the detection results of the first preset marker and the detection results of the second preset marker. For display abnormality detection, the playback abnormality detection results may include the first type of display abnormality detection results and / or the second type of display abnormality results.

[0202] Audio playback anomaly detection is similar to display anomaly detection. For audio playback anomaly detection, the electronic device can determine whether there is a jam during playback by implanting a second preset mark in the audio. Specifically, the electronic device can first separate the second preset mark from the collected sound to be processed, and then identify the sequence represented by the second preset mark, and then determine whether there is a jam based on the actual playback position and the expected playback position. Similarly, the electronic device can first separate the first preset mark from the collected sound to be processed, and then identify whether the first preset mark is consistent with the originally implanted second preset mark. If they are consistent, it is considered that there is no playback anomaly. If they are inconsistent, it can be considered that there is an audio playback quality anomaly.

[0203] As can be seen from the above, the embodiment of the present application implants a preset mark in the video, picture or audio, and collects the sound to be processed or the image to be processed during the process of the electronic device to be detected playing the video, picture or audio, and then determines whether a playback abnormality occurs by identifying the integrity of the image to be processed or the sound to be processed. For example, for display abnormality detection, it is possible to determine whether display problems such as a flowery screen, a black screen and a flashing screen occur by detecting the integrity of the first preset mark in the image to be processed. For another example, for audio playback abnormality detection, it is possible to determine whether the audio playback quality is abnormal by detecting the integrity of the first preset mark in the sound to be processed.

[0204] The integrity of the first preset marker can be determined by comparing the first preset marker in the image to be processed or the audio to be processed with the originally embedded first preset marker. The originally embedded first preset marker refers to a marker in the video source or audio that has not been played. For example, the integrity of the first preset marker in the image to be processed can be determined based on the marker category and / or marker position of the first preset marker in the image to be processed. If the marker category is different and / or the marker position is changed, it is considered that a display abnormality has occurred.

[0205] The embodiment of the present application can also obtain the actual playback position by identifying the sequence of preset mark representations of the image to be processed or the sound to be processed, and then compare the actual playback position with the expected playback position to determine whether a jamming abnormality occurs.

[0206] It can be seen that compared with the existing playback anomaly detection method, the embodiment of the present application transforms the complex playback anomaly detection problem into the problem of detecting preset marks, thereby greatly reducing the detection difficulty and improving the detection efficiency.

[0207] In the embodiment of the present application, the preset mark embedded in the video, picture or audio may include only the first preset mark, only the second preset mark, or both the first preset mark and the second preset mark. The above three situations will be introduced respectively below.

[0208] When the preset tags include a first preset tag, the electronic device can obtain a first tag detection result by identifying the first preset tag in the data to be processed. The first tag detection result is the detection result corresponding to the first preset tag. The specific identification method can be arbitrary. For example, the electronic device can apply a pre-trained model to determine whether the first preset tag is present. In this case, the electronic device can input the data to be processed into the pre-trained model, obtain the output result of the model, and use the output result as the first tag detection result.

[0209] The following will take display anomaly detection as an example to introduce the process of using a pre-trained model to identify the first preset marker.

[0210] Before using a pre-trained model to identify the first preset marker in the image to be processed, it is necessary to first build a model, then use a training data set to train the built model, and then use the trained model to identify the first preset marker in the image to be processed.

[0211] First, the specific representation of the first preset mark is determined, that is, the icon format and the number of icon groups used for the first preset mark are determined. Then, corresponding abnormal icons are created or generated. Finally, model training is performed based on the normal and abnormal icons.

[0212] By way of example and not limitation, see Figure 8 The following is a simple icon diagram provided by an embodiment of the present application. Figure 8 As shown in (a) of FIG, it includes 3 groups of simple icons, which can be marked as PIC1, PIC2 and PIC3 respectively. Figure 8 The three groups of simple icons shown in (a) are used to make or generate corresponding abnormal icons, wherein each group of icons generates two abnormal icons. The generated abnormal icons can be as follows: Figure 8 As shown in (b), each abnormal icon can be labeled as PICErr, PICErr2, PICErr3... in sequence, or all can be labeled as PICErr. Figure 8 In (b), all are marked as PICErr. Figure 8 The abnormal icon in (b) is shown in the figure. Figure 8 The simple icon shown in (a) in FIG. 1 is called a normal icon.

[0213] It is understandable that the number of abnormal icons generated can be arbitrary, and the number of abnormal icons generated is not limited here.

[0214] After generating abnormal icons, both normal icons and abnormal icons can be embedded into various video sources. Figure 8 The three simple icons in (a) and Figure 8 The 6 abnormal icons in (b) are embedded into various video sources. Figure 8 The three simple icons in (a) and Figure 8 The 6 abnormal icons in (b) are shown in Figure 2. Normal icons refer to icons belonging to the normal category, and abnormal icons refer to icons belonging to the abnormal category.

[0215] By way of example and not limitation, see Figure 9 The image diagram after the icon is implanted provided in the embodiment of the present application is shown as follows: Figure 9 As shown, 6 icons are embedded in the image, 3 of which are normal icons and the other 3 are abnormal icons.

[0216] After normal icons and abnormal icons are implanted into each frame of the video, the video source with the implanted icons is used as training data, and the constructed model is trained using the training data.

[0217] It should be noted that the model in the embodiments of the present application can be any type of artificial intelligence model, for example, the model can be a machine learning model or a neural network model.

[0218] The trained model can not only identify the category of the icons embedded in the image, but also the location of the icons in the image. Icon categories can be divided into normal icons and abnormal icons. Furthermore, normal icons can be divided into first-category icons, second-category icons, etc. For example, Figure 9 Taking the image shown in as an example, the icon categories include PIC1, PIC2, PIC3 and PICErr, wherein PIC1, PIC2 and PIC3 are normal icons, and PICErr is an abnormal icon. In some other embodiments, Figure 9 PIC1, PIC2, and PIC3 are classified as normal icons.

[0219] based on Figure 9In the image shown, taking the icon in the upper left corner (i.e., PIC1) as an example, the trained model not only identifies the icon category as PIC1 (or as a normal icon category), but also recognizes the icon's image coordinates as (0.2, 0.1, 0.2, 0.3). It is understood that depending on the recognition needs, it is possible to determine whether to only recognize the icon category, the icon category and icon position, or only the icon position.

[0220] After model training is complete, the trained model can be deployed to an electronic device used to detect playback anomalies, such as a mobile phone or personal computer. Of course, the model training device and the actual detection device can be the same device; for example, a personal computer can be used for both model training and playback anomaly detection.

[0221] Then, you can collect video sources of various scenes and embed normal icons in the video sources. For example, you can embed normal icons in each frame of the video source. Figure 8 Specifically, the icons can be embedded into each frame of each video using a script or other method. In addition, the position of the icons in each frame can be the same or different.

[0222] Next, use the electronic device to be tested to play the video with the normal icon implanted in it. At the same time, use a high-definition camera to shoot the display screen of the electronic device to be tested, or capture the image to be processed by screen recording, which includes the pre-implanted normal icon.

[0223] After capturing the image to be processed, the electronic device can obtain the image to be processed and input the image to be processed into a pre-deployed first detection model to obtain a first output result of the first detection model. The first output result of the first detection model can be an icon category representing a marker category, location information representing an icon location, or both a marker category and location information. This output result is the first marker detection result described above. The first detection model is a pre-trained model.

[0224] The electronic device can obtain the corresponding display anomaly detection result based on the output of the model. Specifically, if the first output result only includes the tag category, the electronic device can determine whether there is an abnormal icon in the image to be processed based on the tag category. If an abnormal icon is present, it can be determined that a first-category display anomaly has occurred. The first-category display anomaly refers to display anomalies such as a black screen, a distorted screen, and a flickering screen. If no abnormal icon is present, it is considered that the first-category display anomaly does not exist.

[0225] If the first output result only includes position information, the electronic device can determine whether the icon position has changed based on the position information and pre-set icon position information. If the icon position has changed, it can be determined that the first type of display abnormality has occurred.

[0226] If the first output result includes the tag category and location information, the electronic device may first determine whether there is an abnormal icon in the image to be processed based on the tag category. If an abnormal icon is present, it may be determined that a first type of display abnormality has occurred. If no abnormal icon is present in the image to be processed, the electronic device may further determine whether the icon position has changed based on the location information. If the icon position has changed, it may be determined that a first type of display abnormality has occurred. If the icon position has not changed, it may be determined that no first type of display abnormality has occurred.

[0227] In a specific application, the electronic device may determine whether the icon position has changed by comparing the preset icon position with the identified icon position.

[0228] It should be noted that when the electronic device to be detected plays the target audio and the collected data to be processed includes the sound to be processed, the process of identifying the first preset marker in the sound to be processed is similar to the process of identifying the first preset marker in the image to be processed. At this time, the electronic device can input the sound to be processed into a pre-trained model to obtain the output result of the model. The output result may include the tag category and / or location information of the first preset marker in the sound to be processed. Finally, the electronic device can determine whether there is an abnormality in the audio playback quality based on the tag category and / or location information. The process is similar to the display abnormality detection process and will not be repeated here.

[0229] When the preset mark includes a second preset mark, the electronic device can determine the actual playback position according to the second preset mark, and then compare the actual playback position with the expected playback position to determine whether a freeze occurs.

[0230] It should be noted that the specific form of the second preset mark may vary, and the recognition method may also vary. For example, when the second preset mark is a number, the electronic device uses OCR to recognize the number in the image to be processed. When the second preset mark is a combination of icons, the electronic device can use a pre-trained model to recognize the combination of icons in the image to be processed.

[0231] When the second preset mark is specifically expressed in the form of numbers, letters, or letters and numbers, the electronic device can identify the second preset mark through OCR. At this time, first collect videos of various scenes, and then implant the corresponding numbers in each frame of the video. For example, the number implanted in the first frame is 1, the number implanted in the second frame is 2, the number implanted in the 10,000th frame is 10,000, and so on. Then, the electronic device to be detected plays the video with the second preset mark implanted, and collects the image to be processed by a high-definition camera or screen recording. Finally, after the electronic device obtains the image to be processed, it can identify the second preset mark in the image to be processed by OCR, and then compare the actual playback position with the expected playback position to determine whether the second type of display abnormality occurs. The second type of display abnormality is a freeze abnormality.

[0232] For example, see Figure 10 The schematic diagram of the jam abnormality detection provided by the embodiment of the present application is shown as follows: Figure 10 As shown, the image on the left is the initial playback position, and the second preset mark in the image is 1, that is, the image frame at the initial playback position is the first frame. After playing for a period of time (for example, 10 minutes), the expected playback position is the 18000th frame. The image frame at the expected playback position is as follows: Figure 10 As shown in the image in the upper right corner, the second preset mark in the image is 18000. However, the actual playback position is the 17800th frame. The image frame of the actual playback position is as follows Figure 10 As shown in the image in the lower right corner, the second preset mark in the image is 178000. In actual application, the collected image to be processed is Figure 10 In the image in the lower right corner, the electronic device recognizes the second preset mark in the image as 17800 through OCR, and then compares 17800 with 18000. Since the actual playback position is closer to the expected playback position, that is, 17800 < 18000, it can be determined that a freeze occurs during playback.

[0233] When the second preset marker is represented by an icon combination, videos of various scenes can be collected first, and then the icon combination can be embedded in each frame of the video, with each frame having a different icon combination, and each icon combination in each frame representing a different number. Then, the electronic device to be tested plays the video with the second preset marker embedded. After the electronic device obtains the collected image to be processed, it can identify whether the icon combination in the image to be processed is consistent with the icon combination in the image frame to be played. If they are inconsistent, it can be determined that a freeze has occurred. If they are consistent, it can be determined that no freeze has occurred.

[0234] For example, see Figure 11 Another schematic diagram of abnormal detection of jamming provided by an embodiment of the present application is shown as follows: Figure 11As shown, the image on the left is the initial playback position, and the number represented by the icon combination in the image is 1, that is, the image is the first frame. After playing for a period of time (for example, 10 minutes), the expected playback position is the image in the upper right corner. The actual playback position is the image in the lower right corner. At this time, after the electronic device obtains the image to be processed at the actual playback position, it can compare the image to be processed with the image frame expected to be played, that is, Figure 10 The image in the upper right corner and the image in the lower right corner are compared to determine whether the icon combination in the two images is consistent. If the icon combination in the two images is consistent, it is considered that the actual playback position is the same as the expected playback position and there is no lag. If the icon combination in the two images is inconsistent, it is considered that the actual playback position is different from the expected playback position and there is a lag.

[0235] In specific applications, a pre-trained model can be used to identify icon combinations in an image to be processed. Specifically, the image to be processed is input into the pre-trained model to obtain a second output result of the model. This second output result can be specifically represented by a number, i.e., a number representing the icon combination in the image to be processed; it can also be represented by an icon combination. The electronic device then determines whether the actual playback position is consistent with the expected playback position based on the second output result.

[0236] It should be noted that audio playback anomaly detection is similar to display anomaly detection. Specifically, when the sound to be processed contains a second preset mark, the electronic device can first separate the second preset mark from the sound to be processed, and then identify the number corresponding to the second preset mark, and then determine whether the expected playback position and the actual playback position are consistent based on the identified number.

[0237] When the preset mark includes a first preset mark and a second preset mark, at this time, the detection method of the first preset mark mentioned above can be used to identify and detect the first preset mark, and the detection method of the second preset mark mentioned above can be used to detect the second preset mark. The specific process can be found in the above content and will not be repeated here.

[0238] It is understandable that the above exemplary embodiments provide a first preset mark detection method and a second preset mark detection method, but the preset mark detection method is not limited to those mentioned above.

[0239] The following is an illustrative introduction to the display anomaly detection process.

[0240] See also Figure 12 The schematic diagram of the abnormality detection process provided by the embodiment of the present application is shown as follows: Figure 12As shown, first, a video source is collected; then, a dynamic tag is embedded in the video source to obtain a tagged video source: the electronic device to be detected obtains the tagged video source through a transmission link and plays the video source, wherein, Figure 12 The example in the figure shows that the electronic devices to be tested may include, but are not limited to, mobile phone model 1, mobile phone model 2, mobile phone model 1, tablet model X, large-screen device model X, or other display terminals. While the electronic devices to be tested are playing the marked video, a high-definition camera captures the screen of each display terminal to obtain an image to be processed. The processed images are then detected and recognized, and a detection result is output, which is a display anomaly detection result.

[0241] In the above process, the dynamic tag may include a first preset tag and / or a second preset tag. The transmission link may be the Internet or other channels, such as a USB data transmission channel.

[0242] In the process of embedding preset marks in the video, the position of the embedded marks can be fixed, that is, the position of the preset marks in each frame image is consistent; or it can be dynamically random, that is, the position of the preset marks in each frame image is not fixed, but random.

[0243] By way of example and not limitation, see Figure 13 The image diagram after implantation of the preset mark provided in the embodiment of the present application is shown as follows: Figure 13 As shown in (a), for black, distorted, and flickering screen detection, three simple icons, PIC1, PIC2, and PIC3, are embedded in image 131 of the video source. The positions of the three simple icons are consistent in each frame of the video. For example, the position of PIC1 in the first frame is the same as the position of PIC1 in the second frame, and the position of PIC2 in the first frame is the same as the position of PIC2 in the second frame.

[0244] based on Figure 13 In (a) of Figure 1, during the model training phase, four categories of icons are embedded in the image: PIC1, PIC2, PIC3, and PICErr. Each normal icon is considered a class, and all abnormal icons are grouped together. At this point, labeling all icons yields four classes, indicating that the trained model is a four-class model, or in other words, a four-class training task.

[0245] like Figure 13As shown in (b), for black screen flashing detection, that is, when detecting display anomalies such as black screen, flower screen and flashing screen, three types of simple icons can be implanted in the image 132 of the video source. These three icons are PIC1, PIC2 and PIC3. Each type of icon in the image has multiple patterns to indicate that the position of the icon in each frame is random. For example, the icon position of PIC1 in the first frame is A, the icon position of PIC1 in the second frame is B, and the icon position of PIC1 in the third frame is C. Figure 13 Taking PIC3 in (b) as an example, icon 133 and icon 134 are located at different positions in the figure. PIC3 in the first frame image is located at the position of icon 133, and PIC3 in the second frame image is located at the position of icon 134. That is to say, the same type of icons in multiple different positions indicate that the positions of the icons in each frame image are random.

[0246] based on Figure 13 In (b), during the model training stage, four types of icons can be implanted in the image, namely PIC1, PIC2, PIC3 and PICErr, that is, each type of normal icon is regarded as a class, and all abnormal icons are classified into one class. At this time, the model training task is a four-classification task. Of course, in some other embodiments, each type of icon can have two positive and negative samples. At this time, there are three normal icons and three abnormal icons, namely PIC1, PIC2, PIC3, PICErr corresponding to PIC1, PICErr corresponding to PIC2, and PICErr corresponding to PIC3. By labeling these six icons separately, six types of labels can be obtained. At this time, the model training task is a six-classification task.

[0247] That is to say, in actual applications, N types of icons are inserted into the image, and the model training task in the model stage can be an N+1 classification task or a 2N classification task.

[0248] See also Figure 14 The schematic diagram of the black screen, flower screen and flash screen detection process provided by the embodiment of the present application is shown as follows Figure 14As shown, first, an icon is inserted into the video source, and then the electronic device to be detected plays the video to be displayed. The image to be processed is obtained by shooting or recording the screen, and then the icon in the image to be processed is identified by the icon detection and recognition algorithm to obtain the icon category and icon position. Determine whether PICErr is detected. If PICErr is detected, it is determined that display abnormalities such as black screen, flowery screen and flashing screen occur, where PICErr refers to an abnormal icon or an icon of an abnormal category. If PICErr is not detected, it is determined whether the icon position has changed. If the icon position has changed, it is determined that display abnormalities such as black screen, flowery screen and flashing screen occur. Conversely, if the icon position has not changed, it is determined that the video is normal, that is, there are no display abnormalities such as black screen, flowery screen and flashing screen.

[0249] The icon detection and recognition algorithm in the above process may be specifically expressed as a model or in other forms.

[0250] During the jamming anomaly detection process, whether an anomaly occurs can be determined by determining whether the actual playback position is consistent with the expected playback position. Figure 15 The schematic diagram of the jamming anomaly detection process provided by the embodiment of the present application is shown. At this time, assuming that the frame rate of the video source being played is 30fps and the video length is 10 minutes, there are a total of 30x10x60=18,000 frames, and numbers are inserted in each frame of the image, with a total of 18,000 numbers from the first frame to the last frame.

[0251] like Figure 15 As shown in (a), first, a script is used to insert continuous numbers into each frame of the video source, and then the electronic device to be detected plays the video, and at time T1, the image to be processed is captured by shooting with a high-definition camera or recording the screen. Then, the electronic device uses OCR to identify the numbers in the image to be detected, and the number is obtained as T1'. At time T2, the image to be processed is captured by shooting or recording the screen, and then the electronic device uses OCR to identify the numbers in the detection image, and the number is obtained as T2'. Next, calculate T2-T1. At this time, assume that T2-T1 is equal to 10 minutes and the frame rate is 30fps. Then determine whether T2'-T1' is approximately equal to 18000. If so, the video is normal and there is no abnormal freeze. If not, there is an abnormal freeze. In Figure 15 In (a), the first frame image and the image after 10 minutes of playback are captured, and the difference between the numbers in the two images is determined to be approximately equal to 18,000 to determine whether a freeze occurs.

[0252] like Figure 15As shown in (b), at this time, what is inserted into the image is an icon combination, not a number. First, an icon is inserted into the video source, and what is inserted is an icon combination. Then, the electronic device to be detected plays the video, and at time T1, the image to be processed is captured by shooting with a high-definition camera or recording the screen, and then the electronic device determines the frame corresponding to time T1 based on the icon combination. The image to be processed is captured at time T2, and then the electronic device determines the frame corresponding to time T2 based on the icon combination. Then, T2-T1 is calculated. At this time, T2-T1=10 minutes, and the frame rate is 30fps. Determine whether the expected frame at time T2 and the detection frame corresponding to time T2 are consistent. If they are inconsistent, a stuttering abnormality is detected. If they are consistent, the video is normal, that is, there is no stuttering abnormality.

[0253] It should be noted that the display abnormality detection process exemplarily introduced above can be referred to the relevant introduction of the above embodiment, and will not be repeated here.

[0254] In addition, a video source can be embedded with icons multiple times and then detected multiple times to increase the detection coverage.

[0255] As can be seen from the above, by implanting preset marks in the image, which can be icons, numbers or icon combinations, complex video display detection problems (for example, complex and changeable video scenes such as movies, games, animations, etc.) are converted into mark detection problems, or icon combination detection problems, which greatly reduces the detection difficulty and improves the detection efficiency.

[0256] See also Figure 16 A schematic diagram of a playback anomaly detection device provided in an embodiment of the present application is shown, and the device may include:

[0257] An acquisition module 161 is configured to acquire data to be processed that includes a preset marker. The data to be processed is data obtained by capturing images and / or sounds played by the electronic device to be detected during the process of the electronic device to be detected playing a target image and / or target audio. Both the target image and target audio include pre-implanted preset markers.

[0258] A marker detection module 162 is configured to obtain a marker detection result by identifying a preset marker in the data to be processed;

[0259] The anomaly detection module 163 is used to obtain a playback anomaly detection result based on the mark detection result.

[0260] In some possible implementations, the above-mentioned preset mark may include a first preset mark and / or a second preset mark.

[0261] In some possible implementations, the data to be processed includes an image to be processed, the target image is a video or a picture, and each frame of the target image is pre-implanted with a preset mark.

[0262] In some possible implementations, the first preset mark includes at least one first preset icon, the second preset mark includes a number or an icon combination, and the icon combination includes at least two second preset icons.

[0263] In some possible implementations, the preset mark includes a first preset mark, and the mark detection module is specifically used to: identify the first preset mark in the image to be processed and obtain a first mark detection result; wherein the mark detection result includes the first mark detection result.

[0264] In some possible implementations, the above-mentioned marker detection module is specifically used to: input the image to be processed into a pre-trained first detection model, obtain a first output result of the first detection model, and use the first output result as the first marker detection result; the first output result includes a marker category and / or position information, the position information is used to describe the position of the first preset marker in the image to be processed, and the marker category is used to describe the category of the first preset marker.

[0265] In some possible implementations, the mark detection result includes a first mark detection result, and the first mark detection result includes a mark category; the above-mentioned abnormality detection module is specifically used to: determine whether there is a first preset mark belonging to the abnormality category in the image to be processed according to the mark category; if there is a first preset mark belonging to the abnormality category in the image to be processed, determine that the electronic device to be detected has a first type of display abnormality.

[0266] In some possible implementations, the first mark detection result also includes position information, and the above-mentioned abnormality detection module is specifically used to: if there is no first preset mark belonging to the abnormal category in the image to be processed, determine whether the position information of each first preset mark is consistent with the corresponding preset position information; if there is at least one first preset mark whose position information is inconsistent with the corresponding preset position information, determine that the electronic device to be detected has a first type of display abnormality.

[0267] In some possible implementations, the preset mark includes a second preset mark, and the mark detection module is specifically used to: identify the second preset mark in the image to be processed and obtain a second mark detection result; wherein the mark detection result includes the second mark detection result.

[0268] In some possible implementations, the marker detection module is specifically configured to: input the image to be processed into a pre-trained second detection model, obtain a second output result of the second detection model, and use the second output result as a second marker detection result; wherein the second output result is used to describe the position of the image to be processed in the image sequence of the target image, and the image to be processed is an image captured at a preset time;

[0269] Alternatively, a second preset mark in the image to be processed is identified by optical character recognition to obtain a recognition result, and the recognition result is used as the second mark detection result; wherein the recognition result is used to describe the position of the image to be processed in the image sequence of the target image, and the image to be processed is an image captured at a preset time.

[0270] In some possible implementations, the mark detection result includes a second mark detection result, and the above-mentioned abnormality detection module is specifically used to: if the second mark detection result is less than the target value, determine that the electronic device to be detected has a second type of display abnormality, the target value is a value obtained by identifying the second preset mark in the target image frame, the target image frame is the image frame expected to be played by the electronic device to be detected at a preset time, and the target image includes the target image frame.

[0271] In some possible implementations, the image to be processed includes a sound to be processed, and the above-mentioned marker detection module is specifically used to: separate a preset marker from the sound to be processed, the preset marker including an audio signal in a preset frequency band, and the preset frequency band is higher or lower than the frequency band of the target audio; identify the preset marker and obtain a marker detection result.

[0272] The above-mentioned playback anomaly detection device has the function of implementing the above-mentioned playback anomaly detection method. This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, and the modules can be software and / or hardware.

[0273] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0274] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0275] The present application also provides a chip system comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the method described in the above-mentioned embodiment of the playback anomaly detection method. The chip system can be a single chip or a chip module composed of multiple chips.

[0276] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0277] It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance. The reference to "one embodiment" or "some embodiments" described in the present application specification means that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in the different places in this specification are not necessarily all references to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized.

[0278] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for detecting abnormal playback, characterized in that: include: Acquiring an image to be processed containing preset markers, wherein the image to be processed is data obtained by capturing an image played by the electronic device to be detected during the process of the electronic device to be detected playing a target image, wherein the target image contains pre-implanted preset markers; wherein the preset markers include a first preset marker and a second preset marker; Obtaining a first marker detection result by identifying the first preset marker in the image to be processed; Obtaining, based on the first mark detection result, a detection result of whether the electronic device to be detected has a first type of display abnormality; the first type of display abnormality includes at least one of a black screen, a distorted screen, or a flickering screen; Obtaining a second marker detection result by identifying the second preset marker in the image to be processed; If the detection result of the second mark is less than the target value, it is determined that the electronic device to be detected has a stuck abnormality, and the target value is obtained by identifying the second preset mark in the target image frame. The target image frame is the image frame that the electronic device to be detected is expected to play at the preset moment, and the target image includes the target image frame.

2. The method according to claim 1, characterized in that The first preset mark includes one or more first preset icons.

3. The method according to claim 1, characterized in that Each frame of the target image includes the first preset mark.

4. The method according to any one of claims 1 to 3, characterized in that Obtaining a first marker detection result by identifying the first preset marker in the data to be processed includes: Identifying a first preset marker in the image to be processed and obtaining a first marker detection result; Wherein, the data to be processed includes the image to be processed.

5. The method according to claim 4, characterized in that The identifying the first preset mark in the image to be processed to obtain a first mark detection result includes: Inputting the image to be processed into a pre-trained first detection model to obtain a first output result of the first detection model, and using the first output result as the first mark detection result; The first output result includes a mark category and / or position information, the position information is used to describe the position of the first preset mark in the image to be processed, and the mark category is used to describe the category of the first preset mark.

6. The method according to claim 5, characterized in that The first marker detection result includes the marker category; Obtaining a detection result of whether the electronic device to be detected has a first type of display abnormality according to the first mark detection result includes: Determining, based on the mark category, whether there is a first preset mark belonging to an abnormal category in the image to be processed; If a first preset mark belonging to an abnormal category exists in the image to be processed, it is determined that a first type of display abnormality occurs in the electronic device to be detected.

7. The method according to claim 6, characterized in that The first mark detection result also includes the position information. After determining whether there is a first preset mark belonging to an abnormal category in the image to be processed according to the mark category, the method further includes: If there is no first preset mark belonging to the abnormal category in the image to be processed, determining whether the position information of each first preset mark is consistent with the preset position information; If the position information of at least one of the first preset marks is inconsistent with the preset position information, it is determined that the electronic device to be detected has the first type of display abnormality.

8. The method according to claim 1, characterized in that Obtaining a second marker detection result by identifying the second preset marker in the image to be processed includes: Inputting the image to be processed into a pre-trained second detection model to obtain a second output result of the second detection model, and using the second output result as the second mark detection result; The second output result is used to describe the position of the image to be processed in the image sequence of the target image, and the image to be processed is an image acquired at a preset time; or, Recognize the second preset mark in the image to be processed by optical character recognition to obtain a recognition result, and use the recognition result as the second mark detection result; The recognition result is used to describe the position of the image to be processed in the image sequence of the target image, and the image to be processed is an image collected at a preset time.

9. The method according to any one of claims 1 to 3, characterized in that The second preset mark includes a number or an icon combination, and the icon combination includes at least two second preset icons.

10. A method for detecting abnormal playback, characterized in that: include: Acquiring a sound to be processed containing a preset tag, wherein the sound to be processed is data obtained by collecting the sound played by the electronic device to be detected during the process of the electronic device to be detected playing the target audio, and the target audio contains a pre-implanted preset tag; wherein the preset tag includes a first preset tag; Separating the first preset marker from the sound to be processed, where the first preset marker includes an audio signal in a preset frequency band, where the preset frequency band is higher or lower than a frequency band of the target audio; Identifying the first preset mark and obtaining a detection result of the first mark; A detection result of whether the electronic device to be detected has a first type of audio playback abnormality is obtained according to the first mark detection result.

11. The method according to claim 10, characterized in that The preset mark further includes a second preset mark, and after obtaining the sound to be processed including the preset mark, the method further includes: Obtaining a second marker detection result by identifying the second preset marker in the sound to be processed; According to the second flag detection result, a detection result of whether the electronic device to be detected has a second type of audio playback abnormality is obtained.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images