A method for real-time detection of black screen and fixed screen of smart phone

Abstract

The present invention discloses a framework method for real-time detection of black screen fixation of smart phones, comprising the following steps: Step 1, constructing a mobile phone-side experimental platform with a built-in experimental data collection program, wherein the mobile phone is a common Android mobile phone model, and the experimental data collection program supports triggering the black screen fixation of the mobile phone and collecting the user's behavior of operating the mobile phone, and sets the usage model type to a classification model, that is, task_name is set to classification. The framework method for real-time detection of black screen fixation of smart phones includes a complete set of ergonomic experiments to collect a series of panic behavior data when users face the black screen fixation phenomenon, performs a series of preprocessing operations on the data, and uses an advanced time series classification model to complete training based on the data, and finally completes deployment testing on the mobile phone side. It includes a complete framework and implementation for real-time detection of black screen fixation on the mobile phone side, and provides a new solution framework from the intersection of ergonomics and computers.

Description

Technical Field

[0001] The present invention relates to the field of ergonomics and deep learning technology, and specifically to a method for detecting a black screen or a fixed screen of a smartphone in real time. Background Art

[0002] The black screen and frozen screen phenomenon of smartphones refers to the screen suddenly turning black or freezing while the device is in use and becoming unresponsive to user operations. Such problems not only seriously affect the user experience and satisfaction, but also have a negative impact on the brand image and market reputation of mobile phone manufacturers. Although existing black screen and frozen screen detection solutions (such as Theia) can detect these problems to a certain extent, their detection accuracy and reliability still have much room for improvement. In traditional black screen and frozen screen detection methods, they mainly rely on system-level monitoring and log analysis, and judge black screen and frozen screen phenomena by recording system crash logs and error reports. Although this method can capture some obvious faults, due to the lack of real-time monitoring of user behavior, it cannot effectively distinguish between normal operations and faulty behaviors, resulting in serious false positives and missed reports. The coverage of system detection points is limited, and it is impossible to fully capture all abnormal situations when black screen and frozen screen phenomena occur, resulting in limited detection accuracy. In recent years, with the development of big data and machine learning technologies, black screen and frozen screen detection has gradually developed in the direction of intelligence and dynamism. By collecting real-time user data in black screen and frozen screen scenarios, the system can accurately detect the black screen and frozen screen phenomena. Analyzing and modeling operational behavior data using machine learning and artificial intelligence can significantly improve detection accuracy and reliability. For example, users often exhibit specific behavioral patterns when encountering black screen or frozen screen issues, such as frequently tapping the screen and pressing the back or power buttons. These behavioral patterns can serve as important reference data. By combining system detection and user behavior analysis, black screen and frozen screen issues can be more comprehensively and accurately identified. Furthermore, leading mobile phone manufacturers such as Huawei have adopted a method that combines system detection and user behavior analysis for black screen and frozen screen detection. Specifically, Huawei has deployed 32 detection points in system detection (including applications, frameworks, native, and kernel), and four types of detection points in user behavior detection (including rapid screen taps, pressing the back button, pressing the home button, and pressing the power button). By flexibly combining and correlating system detection points, black screen and frozen screen issues can be more accurately identified. For example, through such optimization, Huawei has increased the detection rate of freezes (application freezes and system freezes) from 34% to 80%.

[0003] Although existing technologies have solved the problems of black screen and fixed screen detection to a certain extent, their limitations still exist. Existing detection schemes are prone to false positives and missed positives in complex usage scenarios, affecting user experience and device trust. At the same time, system detection methods mainly rely on preset detection points, and the setting and debugging of these detection points require a lot of manpower and time costs. Therefore, existing detection schemes are difficult to fully capture all abnormal situations, resulting in limited detection accuracy and reliability. Therefore, we propose a method for real-time detection of black screen and fixed screen on smartphones to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for real-time detection of black screen and fixed screen of smart phones, so as to solve the problem proposed in the above background technology that the existing detection schemes on the market are difficult to fully capture all abnormal situations, resulting in limited detection accuracy and reliability.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a method for detecting the black screen and fixed screen of a smart phone in real time, comprising the following steps: step 1, constructing a mobile phone experimental platform, which has a built-in experimental data collection program, the mobile phone is an Android mobile phone model, the experimental data collection program supports triggering the black screen and fixed screen of the mobile phone and collecting the user's operation of the mobile phone, and sets the model type to be a classification model, that is, task_name is set to classification, and is set to enter training mode, that is, is_training=1, and sets the model training parameters network layer number to 3, window size to 200, batch size to 8, and model training rounds to 20; step 2, on the mobile phone experimental platform On the top, design user experiments and related tasks, data feeding and processing: first feed the data, then standardize the data and encode the data, the data encoding is divided into time encoding and numerical encoding; main model: then enter the 3-layer TimesNet network, the structure of the single-layer TimesNet network is fast Fourier transform FFT to convert the data into the frequency domain, and use AMP to calculate the average amplitude and obtain the highest k, and pad the length. At the same time, one-dimensional data is transferred to two-dimensional data to obtain two-dimensional features, two-dimensional data is converted to one-dimensional data, and then the length is truncated and the result is obtained by channel weighting, and finally after a residual layer, it enters the next TimesNet block or fully connected layer; the output result Result: The final output result is output through a fully connected layer to output the prediction result; Step 3, based on the designed user experiment, recruit test users to complete the task; Step 4, export the data collected by the experimental data collection program, and design the corresponding data preprocessing and screening methods; Step 5, use the deep learning time series model to train the preprocessed data to obtain a time series classification model, use the TimesNet time series classification model, complete the model parameter setting and definition, and use the model for training to obtain a model that completes the black screen and fixed screen classification based on the time series model input, test the test set data, and obtain the final model accuracy; Step 6, serialize the trained time series classification model using torchscript ptl file, and test the model on an Android device. Use torchscript to serialize the pth file model and convert it into a ptl file. The specific process is to load the pth model file in a new python script, perform a single test output, use the torchscript corresponding API to track the calculation process of the input in the model, and then convert it into a ptl file. Deploy the Android test environment and configure dependencies. Create a new Android project based on gradle configuration, and then import two pytorch-related dependencies in the configuration file, namely (org.pytorch:pytorch_android_lite) and (org.pytorch:pytorch_android_torchvision), set up a single input test output result, import the ptl model file and place it in the project's assets resource path, then import a 200-length data point used in the training data, use the model for classification, and print the classification results directly to the console to verify that the model can run normally and complete the classification.

[0006] Preferably, the step 1 specifically includes the following steps: (1) installing a built-in experimental data collection program on the experimental device Android smartphone; (2) using the experimental data collection program to collect user behavior at a fixed frequency of 200 times per second in different scenarios.

[0007] Preferably, step one includes the following steps: A1, designing the first group of built-in experimental data collection programs, camera: black screen and fixed screen when switching between horizontal and vertical screens; Douyin: black screen and fixed screen during application startup or after switching back to the desktop; WeChat: black screen and fixed screen during application startup or after switching back to the desktop; Honor of Kings: black screen and fixed screen after calling out the WeChat floating window; the power button is invalid under the black screen and fixed screen; A2, designing the second group of built-in experimental data collection programs, camera: black screen and fixed screen during application startup; Douyin: black screen and fixed screen during application use; WeChat: black screen and fixed screen during application use; Honor of Kings: black screen and fixed screen during application use or after switching back to the desktop; the power button is invalid under the black screen and fixed screen; A3, designing the third group of built-in experimental data collection programs, camera: black screen and fixed screen for built-in camera preview; Douyin: black screen and fixed screen for video area, and black screen and fixed screen for other areas Normal; WeChat: Black screen and fixed screen during use of mini program; Honor of Kings: Black screen and fixed screen when calling out WeChat floating window; The power button is valid under black screen and fixed screen, but the black screen and fixed screen phenomenon cannot be terminated; A4, design the fourth group of built-in experimental data collection programs, Camera: Black screen and fixed screen when the built-in camera preview is built-in; Douyin: Black screen and fixed screen when the video area is built-in, and other areas are normal; WeChat: Black screen and fixed screen when browsing Moments; Honor of Kings: Black screen and fixed screen when the main interface of the game is built-in, and other functions are normal; The power button is valid under black screen and fixed screen, but the black screen and fixed screen phenomenon cannot be terminated; A5, design the fifth group of built-in experimental data collection programs, Camera: Black screen and fixed screen when switching between horizontal and vertical screens; Douyin: Black screen and fixed screen when shooting videos; WeChat: Black screen and fixed screen during video calls; Honor of Kings: Black screen and fixed screen when the main interface of the game is built-in, and other functions are normal; The power button is valid under black screen and fixed screen, but the black screen and fixed screen phenomenon cannot be terminated.

[0008] Preferably, the step 2 specifically includes the following steps: B1, collecting the user's touch data on the touch screen (x, y, t): taking the upper left corner of the screen as the origin of the coordinate axis, the x-axis direction is from left to right, and the y-axis direction is from top to bottom; B2, collecting the user's pressing data on the physical buttons on the screen (press, t1, t2): press is a string that represents the following buttons: the up volume button, the down volume button and the power button, t1 and t2 represent the time of pressing and releasing the button respectively; B3, collecting the scene and time data (scene, event, t1, t2) when the black screen or fixed screen occurs: scene is divided into two categories, namely application level and global level, among which the whole machine black screen fixed screen There are differences between screen-level and application-level black screen freezing as follows: A black screen freeze of the entire device cannot be ended by pressing the phone's home button. In application-level scenarios, pressing the home button returns to the phone's desktop. The application level is divided into photo album scenarios, Honor of Kings game scenarios, WeChat and TikTok scenarios, and global scenarios are divided into desktop and multitasking center. Events are divided into two categories: black screen and frozen screen. t1 and t2 indicate the start and end time of the black screen or frozen screen, respectively. B4. Collect IMU sensor data (sensor, x, y, z, t) on the phone: sensor indicates the type of sensor, including accelerometers and gyroscopes. x, y, z indicate the corresponding data, representing data on the x, y, and z axes, respectively.

[0009] Preferably, the step 2 also includes the following steps: determining the research problem: identifying the panic behavior of users in the black screen fixed screen scenario through the study of user behavior, conducting behavioral cluster analysis, and then defining quantitative behavioral patterns through behavioral experiments, outputting user panic behavior information, and assisting the business team to more accurately identify the black screen fixed screen problem; proposing a research hypothesis: the panic behavior generated by users after different black screen fixed screen events has certain differences. By analyzing the indicators and regularities of these panic behaviors, the occurrence of black screen fixed screen events can be reversely inferred; setting the research dependent variables as follows: physical button behavior: the user's panic behavior in the black screen fixed screen event After the screen is locked, the user will try to recover by locking the screen and waking up through physical buttons such as the lock screen button. The data collection indicators (t, Δt, scene, event, button) are designed. Among them, t represents the time when each physical button is pressed, Δt is the time difference with the black screen fixed screen burying time, scene is the specific scene where the burying occurs, event is black screen or fixed screen, and button is the physical button pressed; virtual button behavior: after the black screen fixed screen occurs, the user will directly return to the desktop or open the process to close the related application tasks through the virtual button of the system interaction layer. The data collection indicators (t, x, y, Δt, scene, event, key), where t represents the time when each touch screen occurs, x and y are coordinates, Δt is the time difference from the black screen fixed screen burying time, scene is the specific scene where the burying occurs, event is black screen or fixed screen, key is the system-level button pressed; IMU behavior: the user's IMU data may be obviously abnormal after the black screen fixed screen occurs. When playing games in horizontal mode, once the screen goes black, the user will immediately adjust it back to vertical mode. Design data collection indicators (t, x1, y1, z1, x2, y2, z2, scene, event), where t is the time of each trigger In the experiment, x1, y1, and z1 are the values ​​of the corresponding axes of the gyroscope, x2, y2, and z2 are the values ​​of the corresponding directional acceleration, scene is the scene where the specific embedding occurs, and event is a black screen or a fixed screen. The experimental steps are set as follows: each subject needs to experience four types of app scenes and trigger the specific black screen and fixed screen problem in each scene. The subjects are divided into 5 groups and assigned black screen and fixed screen scenes. The black screen and fixed screen of the whole device must be triggered once in the experiment of each subject, at most twice, and the scene is random. The black screen and fixed screen in the multi-tasking scene must appear in the 2nd and 4th groups, and appear randomly in the 1st, 3rd, and 5th groups. The specific experimental steps are divided into six steps (see Figure 2 ), namely: the subjects fill out the informed consent form and are then assigned to groups; then the users check the instructions; the users become familiar with the phone and use it normally for a period of time; after confirming familiarity, a formal experiment is conducted. The formal experiment is completed in four application scenarios: camera, TikTok, WeChat, and Honor of Kings. Corresponding tasks are performed in each scenario. Then, the data collection program in step one will trigger the black screen fixed screen scenario and collect user behavior indicators. After completing the formal experiment, the subjects fill out the relevant questionnaires, and finally the data is exported and verified.

[0010] Preferably, the step three specifically includes the following steps: recruiting 183 subjects to complete the experimental content according to the experimental process described in claim 6; ensuring that the subject composition includes different genders, age ranges, occupations, educational backgrounds, monthly incomes and personal mobile phone models.

[0011] Preferably, the step four specifically includes the following steps: exporting the txt data collected from the experimental data and directly merging it into csv data to convert the data into structured data; manually marking and proofreading the data and annotating it in Excel to solve a series of problems; observing the data and finding quantitative indicators by comparing it with the experimental video; using Python language combined with Excel data to format the csv data into ts data to facilitate subsequent batch processing and model training.

[0012] Preferably, the step 2 also includes the following steps: A1. Loss of black screen fixed screen event record: This problem is very likely to occur when the entire device is fixed to a black screen and the mobile phone needs to be restarted to continue the experiment. The solution is to complete it in Excel for easy subsequent processing; A2. Continuous triggering of black screen fixed screen and recent tasks: After applying the black screen fixed screen, the recent task deletion process will be entered. The recent task is also a preset black screen fixed screen scene. Therefore, two types of black screen fixed screen scenes will be triggered continuously, causing the data records to be overwritten. The solution is to complete it in Excel; A3. Data recording error: including scene recording errors discovered through video proofreading, and inconsistent triggering of the original task of the group. The solution is to correct it in Excel; A4. Data delay: There will be a discrepancy of 1-5s between the manually proofread event and the script export time. The solution is to record the correct time obtained by manual proofreading in Excel.

[0013] Preferably, the ts data format in step 4 specifically includes the following input format and output format requirements: Input format: a matrix of seq_len length and 10 width, each row is a complete time series data, from top to bottom are the features of each dimension, and each column can be regarded as a time downsampling data; (1) seq_x: represents the input data. In the original model, the input is a tensor with a shape of [batch_size, seq_len, feature_dim]; (2) batch_size: the batch of data processed by the model at one time. This value is 1 during testing or actual prediction of the model; (3) seq_len: the time series input of the model at one time Data length, this value is temporarily set to the data that can be obtained within 5s. When sampling the mobile phone, a fixed interval needs to be specified, and sampling is performed continuously according to this interval. The value of seq_len is set to 5s / sampling interval. However, this value needs to be determined through subsequent ablation experiments to find the optimal value under the balance between algorithm overhead and accuracy; (4) feature_dim: the feature dimension of the data, giving the final feature dimension length, the total length is 10, representing the data obtained at each timestamp, each dimension represents: touch x, touch y, button, IMU x, IMU y, IMU z, hour, minute, second, millisecond; output format: the probability of a black screen or fixed screen in the corresponding time window (the collected data only contains 0 and 1).

[0014] Preferably, the step four also includes the following specific steps: B1. Read the corresponding CSV data according to the name of the first column in Excel; B2. Read the required data of the corresponding time period in CSV according to the corresponding black screen fixed screen time marked in Excel, fill it into a vector, and mark it, the black screen fixed screen is 1, and the normal is 0; B3. The data required for the vector is in the ts data format, the window size is 200, and it is the data that can be sampled within 1s to improve the immediacy of detection feedback; B4. Divide the data into training set and test set in an 8:2 ratio, and finally write the ts data header metadata.

[0015] Compared with the existing technology, the beneficial effects of the present invention are: the method for real-time detection of black screen and fixed screen of smartphones adopts a new structural design, and its specific contents are as follows: it includes a complete set of ergonomic experiments to collect a series of panic behavior data of users when facing the black screen and fixed screen phenomenon, performs a series of preprocessing operations on the data, and adopts an advanced time series classification model to complete training based on the data, and finally completes deployment testing on the mobile phone side. It includes a complete framework and implementation for real-time detection of black screen and fixed screen on the mobile phone side, and provides a new solution framework from the intersection of ergonomics and computers. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1Schematic diagram of four types of app scenarios and corresponding black screen fixed screen occurrence timing designed for the experiment of this invention;

[0017] Figure 2 Schematic diagram of the experimental process designed for the present invention;

[0018] Figure 3 A schematic diagram of manual data labeling in the present invention;

[0019] Figure 4 Schematic diagram of training data and metadata of the present invention;

[0020] Figure 5 A schematic diagram of the parameters required for model training of the present invention;

[0021] Figure 6 Schematic diagram of a method for real-time detection of a black screen and a fixed screen framework on a mobile phone of the present invention;

[0022] Figure 7 Schematic diagram of setting research independent variables for the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] See also Figure 1-Figure 7 The present invention provides a technical solution: a method for real-time detection of black screen and fixed screen of a smart phone, comprising the following steps: step 1, constructing a mobile phone experimental platform, which has a built-in experimental data collection program, the mobile phone is an Android mobile phone model, and the experimental data collection program supports making the mobile phone screen black and fixed and collecting the user's operation behavior of the mobile phone; step 2, designing user experiments and related tasks on the mobile phone experimental platform; step 3, based on the user experiment designed in step 2, recruiting test users, completing the tasks, and using the experimental platform described in step 1 to complete the collection of user behavior; step 4, exporting the data collected by the experimental data collection program, and designing corresponding data preprocessing and screening methods; step 5, using the deep learning time series model to train the data preprocessed in step 4 to obtain a time series classification model.

[0025] : The detailed introduction is as follows: Design and development of experimental data collection program on mobile phone. According to the independent variables in the user experiment designed in step 2 above, five experimental program scenarios are designed as follows: The first group, camera: black screen and fixed screen when switching between horizontal and vertical screens; Douyin: black screen and fixed screen during application startup or after switching back to the desktop; WeChat: black screen and fixed screen during application startup or after switching back to the desktop; Honor of Kings: black screen and fixed screen after calling out WeChat floating window; the power button is invalid under black screen and fixed screen; the second group, camera: black screen and fixed screen during application startup; Douyin: black screen and fixed screen during application use; WeChat: black screen and fixed screen during application use; Honor of Kings: black screen and fixed screen during application use or after switching back to the desktop; the power button is invalid under black screen and fixed screen; the third group, camera: black screen and fixed screen for built-in camera preview; Douyin: video The video area is black and fixed, while other areas are normal; WeChat: the screen is black and fixed during the use of the mini program; Honor of Kings: the WeChat floating window is black and fixed; the power button is valid under the black screen fixation, but the black screen fixation phenomenon cannot be terminated; the fourth group, camera: the built-in camera preview is black and fixed; Douyin: the video area is black and fixed, while other areas are normal; WeChat: the browsing circle of friends is black and fixed; Honor of Kings: the main interface of the game is black and fixed, and other functions are normal; the power button is valid under the black screen fixation, but the black screen fixation phenomenon cannot be terminated; the fifth group, camera: the screen is black and fixed when switching between horizontal and vertical; Douyin: the screen is black and fixed during video shooting; WeChat: the screen is black and fixed during video calls; Honor of Kings: the main interface of the game is black and fixed, and its text function is normal; the power button is valid under the black screen fixation, but the black screen fixation phenomenon cannot be terminated.

[0026] The experimental data collection program is used to collect user behaviors at a fixed frequency of 200 times per second in different scenarios. The collected user behaviors specifically include the following: user touch data on the touch screen (x, y, t): the upper left corner of the screen is the origin of the coordinate axis, the x-axis direction is from left to right, and the y-axis direction is from top to bottom; user pressing data on the physical buttons on the screen (press, t1, t2): press is a string representing the following buttons: up volume button, down volume button and power button, t1 and t2 represent the time of pressing and releasing the button respectively; scene and time data of black screen or fixed screen (scene, event, t1, t2): scene is divided into two categories, namely application level and the global level. Among them, there are differences between black screen fixation of the entire device and black screen fixation at the application level as follows: the black screen fixation of the entire device cannot be ended by pressing the home button of the mobile phone. In the application-level scenario, pressing the home button can return to the mobile phone desktop. The application level is divided into album scene, Honor of Kings game scene, WeChat and Douyin scene, and the global scene is divided into desktop and multitasking center; events are divided into two categories: black screen and fixed screen; t1 and t2 respectively indicate the start and end time of the black screen or fixed screen; mobile phone sensor data (sensor, x, y, z, t): sensor indicates the type of sensor, including accelerometer and gyroscope; x, y, z indicate the corresponding data, indicating the data on the x, y and z axes respectively.

[0027] An experimental data collection program with the above functions was installed on an Android phone, and a user experiment was designed to determine the research question: Through the study of user behavior, the panic behavior of users in the black screen fixed screen scenario was identified, and behavioral cluster analysis was performed. Then, quantitative behavioral patterns were defined through behavioral experiments, and user panic behavior information was output to assist the business team in more accurately identifying the black screen fixed screen problem; a research hypothesis was proposed: the panic behavior generated by users after different black screen fixed screen events has certain different characteristics. By analyzing the indicators and regularities of these panic behaviors, the occurrence of black screen fixed screen events can be reversely inferred.

[0028] The research dependent variables are determined as follows: Physical button behavior: after the black screen is fixed, the user will try to lock the screen and wake up by using physical buttons such as the lock screen button to try to recover. The data collection indicators (t, Δt, scene, event, button) are designed, where t represents the time when each physical button is pressed, Δt is the time difference with the black screen fixed point burying time, scene is the specific scene where the burying occurs, event is black screen or fixed screen, and button is the physical button pressed; Virtual button behavior: after the black screen is fixed, the user will directly return to the desktop or open the process to close the related application tasks through the virtual button of the system interaction layer. The data collection indicators (t, x, y, Δt, scene, event, key) are designed. In the figure, t represents the time when each screen touch occurs, x and y are coordinates, Δt is the time difference from the black screen and fixed screen tracking time, scene is the specific scene in which the tracking occurs, event is a black screen or fixed screen, and key is the system-level button pressed; IMU behavior: The IMU data of the user may be obviously abnormal after the black screen and fixed screen occur. When playing games in landscape mode, once the screen goes black, the user will immediately adjust it back to portrait mode. Design data collection indicators (t, x1, y1, z1, x2, y2, z2, scene, event), where t is the time of each trigger, x1, y1, z1 are the values ​​of the corresponding axes of the gyroscope, x2, y2, z2 are the values ​​of the corresponding directional acceleration, scene is the specific scene in which the tracking occurs, and event is a black screen or fixed screen.

[0029] The experimental steps are as follows: each subject needs to experience four types of app scenarios and trigger the specific black screen fixation problem in each scenario. The subjects are divided into 5 groups and assigned black screen fixation scenarios. The whole device black screen fixation must be triggered once in each subject's experiment, at most twice, and the scenarios are random. The multi-tasking scenario black screen fixation must appear in the 2nd and 4th groups, and randomly appears in the 1st, 3rd, and 5th groups; the specific experimental steps are divided into six steps (see Figure 2 ), which are: the subjects fill out the informed consent form and are then assigned to groups; then the users check the instructions; the users become familiar with the phone and use it normally for a period of time; after confirming that they are familiar with it, a formal experiment is conducted. The formal experiment is completed in four application scenarios: camera, TikTok, WeChat and King of Glory. Corresponding tasks are performed in each scenario, and then the data collection program in step (1) triggers the black screen fixed screen scenario and collects user behavior indicators; after completing the formal experiment, the subjects fill out the relevant questionnaire; finally, the data is exported and verified.

[0030] Recruit subjects to complete the experimental content according to the above experimental steps, and recruit 183 subjects to complete the experimental content according to the experimental process described in step (2); preferably, the subjects are divided into: by gender, males account for 102 people, females account for 81 people; by age, 96 people under 25 years old, 51 people aged 26-35 years old, and 36 people over 35 years old; by occupation, 86 people are students and 97 people are workers; by education level, 51 people are college graduates and below, and 132 people are undergraduate graduates and above; by monthly income, 84 people are below 6,000 yuan, 88 people are between 6,000 and 19,999 yuan, and 11 people are above 20,000 yuan; by mobile phone model, 115 people are Android users and 68 people are iOS users. A series of preprocessing operations are completed for the data collected during the experiment.

[0031] Manually label the data to solve the following problems: Loss of black screen fixed screen event records: This problem is very likely to occur when the entire device is fixed to a black screen and the phone needs to be restarted to continue the experiment. The solution is to complete it in Excel for easy subsequent processing; Apply black screen fixed screen and continuous triggering of recent tasks: After applying black screen fixed screen, the subjects will enter the recent task deletion process. The recent task is also a preset black screen fixed screen scene, so two types of black screen fixed screen scenes will be triggered continuously, causing the data records to be overwritten. The solution is to complete it in Excel; Data recording errors: This includes discovering scene recording errors through video proofreading and inconsistent triggering of the original task of the group. The solution is to correct it in Excel; Data delay: There will be a discrepancy of 1-5s between the manually proofread event and the script export time. The solution is to record the correct time obtained by manual proofreading in Excel.

[0032] The final ts data format used for training is set, including the input format and output format as follows: Input format: a matrix of seq_len length and 10 width, each row is a complete time series data, and from top to bottom are the features of each dimension. Each column can be regarded as a time downsampling data: (1) seq_x: represents the input data. In the original model, the input is a tensor with a shape of [batch_size, seq_len, feature_dim]: (2) batch_size: the batch of data processed by the model at one time. This value is 1 during testing or actual prediction of the model; (3) seq_len: the length of the time series data input by the model at one time. This value is temporarily set to the data that can be obtained within 5s. When sampling the mobile phone, a fixed interval needs to be specified, and sampling is continuously performed according to this interval. The value of seq_len is set to 5s / sampling interval; however, this value needs to be determined through subsequent ablation experiments to find the optimal value under the balance between algorithm overhead and accuracy; (4) feature_dim: the feature dimension of the data, giving the final feature dimension length, with a total length of 10. Represents the data obtained at each timestamp, and each dimension represents: touch x, touch y, button, IMU x, IMU y, IMU z, hours, minutes, seconds, and milliseconds; output format: the probability of a black screen fixed screen within the corresponding time window (collected only contains 0 and 1); the steps for using Python combined with the labeled Excel data to process the structured CSV data merged from the txt data directly collected from the experiment are as follows: read the corresponding CSV data according to the subject name in the first column of Excel; read the required data of the corresponding time period in the CSV according to the corresponding black screen fixed screen time marked in Excel, fill it into a vector, and mark it, black screen fixed screen is 1, normal is 0; the data required for the vector is the TS data format, with a window size of 200, which is the data that can be sampled within 1s to improve the immediacy of detection feedback; divide the data into training set and test set in an 8:2 ratio, and finally write the TS data header metadata; use the deep learning time series model to train the preprocessed data to obtain a time series classification model; use the TimesNet time series classification model to complete the model parameter setting and definition.

[0033] Model parameter settings are as follows Figure 5First, set the model to be used as a classification model, that is, task_name is set to classification; then enter the training mode, that is, is_training=1; then set the model training parameters such as the number of network layers to 3, the window size to 200, the batch size to 8, and the model training rounds to 20; use this model for training to obtain a model that completes the black screen and fixed screen classification based on the time series model input; the model process is as follows: first feed the data, then standardize the data and encode the data, the data encoding is divided into time encoding and value encoding; then feed the 3-layer TimesNet network, and finally output the result through a fully connected layer to output the prediction result; single-layer TimesNet The network structure is as follows: Fast Fourier Transform (FFT) converts data to the frequency domain; AMP is used to calculate the average amplitude and obtain the highest k; padding is performed on the length; one-dimensional data is converted to two-dimensional data to obtain two-dimensional features; the two-dimensional data is converted to one-dimensional data, and then the length is truncated and the result is obtained through channel weighting. Finally, after passing through a residual layer, it enters the next TimesNet block or fully connected layer. During the actual training process, the above model parameters were modified several times. For example, the window (that is, the length of the classification sequence) was found to be a relatively suitable value of 200. The model training rounds were set to 20. The model loss and accuracy have converged with the amount of data collected in the current experiment. Test the test set data to get the final model accuracy; after the model training is completed, test it on the test set and get a final model accuracy of 90.27%; serialize the trained time series classification model into a ptl file using torchscript, and test the model on an Android device; use torchscript to serialize the pth file model and convert it into a ptl file. The specific process is to load the pth model file in a new python script, perform a single test output, use the torchscript corresponding api to track the calculation process of the input in the model, and then convert it into a ptl file; deploy the Android test environment and configure dependencies, and then Create an Android project based on Gradle configuration, and then import two PyTorch-related dependencies in the configuration file: 'org.pytorch:pytorch_android_lite' and 'org.pytorch:pytorch_android_torchvision'; set up a single input test output result, import the PTL model file and place it in the project's assets resource path, and then import a 200-length data used in the training data. Use the model for classification, and print the classification results directly to the console. The model can be loaded normally without errors, and the classification results are output correctly. The test verification is complete.

[0034] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for real-time detection of black screen and fixed screen of smart phone, characterized in that: The steps include: Step 1: Build a mobile phone experimental platform with a built-in experimental data collection program. The mobile phone is an Android phone model. The experimental data collection program supports triggering the phone's black screen and freezing the screen as well as collecting the user's mobile phone operation behavior. The model type is set to classification model, that is, task_name is set to classification, and the training mode is set to 1. The model training parameters are set to 3 network layers, 200 window size, 8 batch size, and 20 model training rounds. Step 2: On the mobile phone experimental platform, design user experiments and related tasks, data feeding and processing: first feed the data, then standardize the data and encode the data, the data encoding is divided into time encoding and numerical encoding; main model: then enter the 3-layer TimesNet network, the structure of the single-layer TimesNet network is fast Fourier transform FFT to convert the data to the frequency domain, and use AMP to calculate the average amplitude and obtain the highest k, and pad the length. At the same time, one-dimensional data is transferred to two-dimensional data to obtain two-dimensional features, and two-dimensional data is converted to one-dimensional data. Then the length is truncated and the result is obtained by channel weighting, and finally after a residual layer, it enters the next TimesNet block or fully connected layer; output result: the final output result is output through a fully connected layer to output the prediction result; The step 2 specifically includes the following steps: B1, collecting the user's touch data on the touch screen (x, y, t): taking the upper left corner of the screen as the origin of the coordinate axis, the x-axis direction is from left to right, and the y-axis direction is from top to bottom; B2, collecting the user's pressing data on the physical buttons on the screen (press, t1, t2): press is a string representing the following buttons: the up volume button, the down volume button and the power button, t1 and t2 represent the time of pressing and releasing the button respectively; B3, collecting the scene and time data (scene, event, t1, t2) when the black screen or fixed screen occurs: scene is divided into two categories, namely application level and global level, among which the whole machine black screen fixed screen and Application-level black screen and fixed screen scenarios differ as follows: A device-wide black screen fix cannot be ended by pressing the phone's home button. In application-level scenarios, pressing the home button returns you to the phone's desktop. Application-level scenarios are categorized into photo album, Honor of Kings, WeChat, and TikTok. Global scenarios are divided into desktop and multitasking center. Events are categorized into two types: black screen and fixed screen. t1 and t2 indicate the start and end times of a black screen or fixed screen, respectively. B4. Collecting IMU sensor data (sensor, x, y, z, t): sensor indicates the sensor type, including accelerometers and gyroscopes. x, y, and z indicate the corresponding data, representing data on the x, y, and z axes, respectively. Step 3: Based on the designed user experiment, recruit test users to complete the task; Step 4: Export the data collected by the experimental data collection program and design corresponding data preprocessing and screening methods; Step 5: Use a deep learning time series model to train the preprocessed data to obtain a time series classification model. Use the TimesNet time series classification model to complete the model parameter setting and definition, and use this model for training to obtain a model that completes the black screen and fixed screen classification based on the time series model input. Test the test set data to obtain the final model accuracy; Step 6. Use torchscript to serialize the trained time series classification model into a ptl file, and test the model on an Android device. Use torchscript to serialize the pth file model and convert it into a ptl file. The specific process is to load the pth model file in a new python script, perform a single test output, use the torchscript corresponding api to track the calculation process of the input in the model, and then convert it into a ptl file for Android test environment deployment and dependency configuration. Create a new Android project based on gradle configuration, and then import two pytorch-related dependencies in the configuration file, set the single input test output result, import the ptl model file and place it in the assets resource path of the project, and then import a 200-length data used in the training data, use the model for classification, and print the classification results directly to the console to verify that the model can run normally to complete the classification.

2. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 1, characterized in that: The step 1 specifically includes the following steps: (1) installing a built-in experimental data collection program on the experimental device Android smartphone; (2) using the experimental data collection program to collect user behavior at a fixed frequency of 200 times per second in different scenarios.

3. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 1, characterized in that: The step one includes the following steps: A1, designing a first group of built-in experimental data collection programs, camera: black screen and fixed screen when switching between horizontal and vertical screens; Douyin: black screen and fixed screen during application startup or after switching back to the desktop; WeChat: black screen and fixed screen during application startup or after switching back to the desktop; Honor of Kings: black screen and fixed screen after calling out the WeChat floating window; the power button is invalid when the screen is black and fixed; A2, designing a second group of built-in experimental data collection programs, camera: black screen and fixed screen during application startup; Douyin: black screen and fixed screen during application use; WeChat: black screen and fixed screen during application use; Honor of Kings: black screen and fixed screen during application use or after switching back to the desktop; the power button is invalid when the screen is black and fixed; A3. Design the third set of built-in experimental data collection programs. Camera: The built-in camera preview is black and fixed; Tik Tok: The video area is black and fixed, and other areas are normal. WeChat: The screen freezes while the mini-program is in use; Honor of Kings: The WeChat floating window freezes while in use; the power button remains active while in use, but the freeze is unavailable; A4, design the fourth group of built-in experimental data collection programs, Camera: The built-in camera preview freezes while in use; TikTok: The video area freezes while in use, but other areas remain normal; WeChat: Browsing Moments results in a black and frozen screen; Honor of Kings: The main interface of the game appears black and frozen, but other functions are normal. The power button is effective in the black screen freeze state, but it cannot stop the black screen freeze phenomenon; A5, design the fifth group of built-in experimental data collection programs, camera: black screen freeze when switching between horizontal and vertical screens; Tik Tok: black screen freeze when shooting videos; WeChat: The screen freezes and goes black during a video call; Honor of Kings: The main interface of the game freezes and goes black, but other functions are normal. The power button is effective when the screen is stuck in black, but it cannot stop the black screen.

4. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 1, characterized in that: The step 2 also includes the following steps: determining the research problem: identifying the panic behavior of users in the black screen fixed screen scenario through the study of user behavior, conducting behavioral cluster analysis, and then defining quantitative behavioral patterns through behavioral experiments, outputting user panic behavior information, and assisting the business team in more accurately identifying the black screen fixed screen problem; proposing a research hypothesis: the panic behavior generated by users after different black screen fixed screen events has certain different characteristics. By analyzing the indicator rules of these panic behaviors, the occurrence of black screen fixed screen events can be reversely inferred; setting the research dependent variables as follows: physical button behavior: after the black screen fixed screen occurs, the user will try to lock the screen and wake up by using a physical button such as the lock screen button to try to recover, and designing data collection indicators (t, Δt, scene, event, button), where t represents the time when each physical button occurs, Δt is the time difference with the black screen fixed screen burying time, scene is the specific burying scene, event is black screen or fixed screen, and button is the physical button pressed; virtual button behavior: after the black screen fixed screen occurs, the user will directly return to the desktop or open the process and close the related application tasks through the virtual button of the system interaction layer, and designing data collection indicators (t, x, y, Δt, scene, event, key), where t represents the time when each touch occurs, x and y are coordinates, Δt is the time difference from the black screen fixed screen burying time, scene is the specific scene where the burying occurs, event is black screen or fixed screen, and key is the system-level button pressed; IMU behavior: The user's IMU data may be obviously abnormal after the black screen fixed screen occurs. When playing games in landscape mode, once the screen goes black, the user will immediately adjust it back to portrait mode. Design data collection indicators (t, x1, y1, z1, x2, y2, z2, scene, event), where t is the time of each trigger, x1, y1, and z1 are the values ​​of the corresponding axes of the gyroscope, x2, y2, and z2 are the values ​​of the corresponding directional acceleration, scene is the scene in which the specific tracking point occurs, and event is a black screen or fixed screen. The experimental steps are as follows: each subject needs to experience four types of app scenarios and trigger the specific black screen and fixed screen problem in each scenario. The subjects are divided into five groups and assigned black screen and fixed screen scenarios. The whole device black screen and fixed screen problem will be triggered once in the experiment for each subject, and at most twice, in random scenarios. The black screen and fixed screen problem in the multitasking scenario will occur in groups 2 and 4, and randomly in groups 1, 3, and 5. The specific experimental steps are divided into six steps, namely: the subjects fill out the informed consent form and are then assigned to groups; then the users check the instructions; the users become familiar with the mobile phone and use it normally for a period of time; after confirming familiarity, a formal experiment is conducted. The formal experiment is completed in four application scenarios: camera, TikTok, WeChat and Honor of Kings, and corresponding tasks are performed in each scenario. Then, the data collection program in step one will trigger the black screen and fixed screen scenario and collect user behavior indicators. After completing the formal experiment, the subjects fill out the relevant questionnaire, and finally the data is exported and verified.

5. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 1, characterized in that: The step three specifically includes the following steps: recruiting 183 subjects to complete the experimental content according to the experimental process, ensuring that the subject composition includes different genders, age ranges, occupations, educational backgrounds, monthly incomes, and personal mobile phone models.

6. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 1, characterized in that: The step 4 specifically includes the following steps: exporting the txt data collected from the experimental data and directly merging it into csv data, converting the data into structured data; manually marking and proofreading the data and annotating it in excel to solve a series of problems; Observe data and find quantitative indicators by comparing experimental videos; use Python language combined with Excel data to format CSV data into TS data for subsequent batch processing and model training.

7. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 1, characterized in that: The step 2 also includes the following steps: A1, black screen fixed screen event record loss: This problem is very likely to occur when the entire device is fixed to a black screen and the phone needs to be restarted to continue the experiment. The solution is to complete it in Excel to facilitate subsequent processing; A2, applying black screen fixed screen and recent tasks to trigger continuously: After applying black screen fixed screen, the recent task deletion process will be entered. The recent task is also a preset black screen fixed screen scenario, so two types of black screen fixed screen scenarios will be continuously triggered, causing the data records to be overwritten. The solution is to complete it in Excel; A3. Data recording errors: This includes errors in scene recording and inconsistencies in the group's original task triggering discovered through video proofreading. The solution is to correct them in Excel. A4. Data delay: There may be a 1-5 second discrepancy between the manually proofread event and the script export time. The solution is to record the correct time obtained by manual proofreading in Excel.

8. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 6, characterized in that: The ts data format in step 4 specifically includes the following input format and output format requirements: Input format: a matrix with a length of seq_len and a width of 10, each row is a complete time series data, and from top to bottom are the features of each dimension, which can be regarded as data obtained by time downsampling; (1) seq_x: represents the input data. In the original model, the input is a tensor with a shape of [batch_size, seq_len, feature_dim]; (2) batch_size: the batch of data processed by the model at one time. This value is 1 during testing or actual prediction of the model; (3) seq _len: the length of the time series data input to the model at one time. This value is tentatively set to the data that can be obtained within 5 seconds. When sampling the mobile phone, a fixed interval needs to be specified, and sampling is performed continuously according to this interval. The value of seq_len is set to 5 seconds / sampling interval. However, this value needs to be determined through subsequent ablation experiments to find the optimal value under the balance between algorithm overhead and accuracy; (4) feature_dim: the feature dimension of the data, giving the final feature dimension length. The total length is 10, representing the data obtained at each timestamp. Each dimension represents: touch x, touch y, button, IMU x, IMU y, IMU z, hours, minutes, seconds, and milliseconds. Output format: The probability of a black screen within the corresponding time window.

9. The method for real-time detection of a black screen or a fixed screen of a smart phone according to claim 8, characterized in that: The step four also includes the following specific steps: B1. Read the corresponding CSV data according to the name of the first column in Excel; B2. Read the required data of the corresponding time period in CSV according to the corresponding black screen fixed screen time marked in Excel, fill it into a vector, and mark it, with the black screen fixed screen as 1 and the normal as 0; B3. The data required for the vector is in the TS data format, with a window size of 200, which is the data that can be sampled within 1 second to improve the immediacy of detection feedback; B4. Divide the data into training set and test set in an 8:2 ratio, and finally write the TS data header metadata.

Images