Framework method for detecting black screen and fixed screen of smart phone in real time

Abstract

The invention discloses a frame method for detecting black screen and fixed screen of a smart phone in real time, which comprises the following steps of: 1, constructing a mobile phone end experiment platform in which an experiment data collection program is built, the experimental data collection program supports triggering of a black screen and a fixed screen of a mobile phone and collection of behaviors of operating the mobile phone by a user, and a use model type is set as a classification model, namely, taskname is set as classification. The frame method for detecting the black screen and the fixed screen of the smart phone in real time comprises the steps that a set of complete ergonomics experiments is used for collecting a series of panic behavior data when a user faces a black screen and fixed screen phenomenon, a series of preprocessing operation is carried out on the data, and training is completed according to the data by adopting an advanced time sequence classification model; finally, deployment testing is completed on the mobile phone end side, a complete frame for detecting blank screen and screen fixing on the mobile phone end side in real time and implementation are included, and a new solution frame is provided from the crossing field of ergonomics and computers.

Description

Technical Field

[0001] The present invention relates to the field of human factors engineering and deep learning technology, and in particular to a framework method for real-time detection of a black screen and a fixed screen of a smart phone. Background Art

[0002] The black screen and frozen screen phenomenon of smartphones refers to the screen suddenly turning black or freezing during the use of the device and failing to respond to user operations. Such problems not only seriously affect the user experience and satisfaction, but also have a negative impact on the brand image and market reputation of mobile phone manufacturers. Although existing black screen and frozen screen detection solutions (such as Theia) can detect these problems to a certain extent, there is still a lot of room for improvement in the accuracy and reliability of their detection. In traditional black screen and frozen screen detection methods, they mainly rely on system-level monitoring and log analysis to judge black screen and frozen screen phenomena by recording system crash logs and error reports. Although this method can capture some obvious faults, due to the lack of real-time monitoring of user behavior, it is impossible to effectively distinguish between normal operations and faulty behaviors, resulting in serious false positives and missed reports. The coverage of system detection points is limited, and it is impossible to fully capture all abnormal situations when black screen and frozen screen phenomena occur, resulting in limited detection accuracy. In recent years, with the development of big data and machine learning technologies, black screen and frozen screen detection has gradually developed in the direction of intelligence and dynamism. By collecting users' black screen and frozen screen scenarios in real time, the detection of black screen and frozen screen is carried out in real time. Operational behavior data, and the use of machine learning and artificial intelligence technology for analysis and modeling can significantly improve the accuracy and reliability of detection. For example, when users encounter black screen and frozen screen problems, they often show specific behavior patterns, such as frequently clicking the screen, pressing the back key or the power key, etc. These behavior patterns can be used as important reference data, and by combining system detection and user behavior analysis, black screen and frozen screen phenomena can be identified more comprehensively and accurately. In addition, leading mobile phone manufacturers in the industry, such as Huawei, have adopted a method that combines system detection and user behavior analysis in black screen and frozen screen detection. Specifically, Huawei has arranged 32 detection points in system detection (including applications, Framework, Native, Kernel, etc.), and arranged 4 types of detection points in user behavior detection (including quickly clicking the screen, pressing the back key, pressing the Home key, pressing the power key, etc.). By flexibly combining and associating system detection points and user behavior detection points, black screen and frozen screen phenomena can be judged more accurately. For example, through such optimization, Huawei has increased the detection rate of freezes (application freezes and system freezes) from 34% to 80%.

[0003] Although the existing technology has solved the problem of black screen and fixed screen detection to a certain extent, its limitations still exist; the existing detection scheme is prone to false positives and missed positives in complex usage scenarios, affecting user experience and device trust; at the same time, the system detection method mainly relies on preset detection points, and the setting and debugging of these detection points require a lot of manpower and time costs; therefore, the existing detection scheme is difficult to fully capture all abnormal situations, resulting in limited detection accuracy and reliability, so we propose a framework method for real-time detection of black screen and fixed screen of smartphones to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide a framework method for real-time detection of black screen and fixed screen of smart phones, so as to solve the problem proposed by the above background technology that the existing detection schemes on the market are difficult to fully capture all abnormal situations, resulting in limited detection accuracy and reliability.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a framework method for real-time detection of black screen and fixed screen of smart phone, comprising the following steps: step one, constructing a mobile phone experimental platform, which has a built-in experimental data collection program, the mobile phone is a common Android mobile phone model, the experimental data collection program supports triggering the black screen and fixed screen of the mobile phone and collecting the user's operation of the mobile phone, and sets the model type to be a classification model, that is, task_name is set to classification, and sets it to enter training mode, that is, is_training=1, sets the model training parameters such as the number of network layers to 3, the window size to 200, the batch size to 8, and sets the model training rounds to 20; step two, on the mobile phone On the experimental platform, user experiments and related tasks are designed. Data feeding and processing: first feed the data, then standardize and encode the data. The data encoding is divided into time encoding and numerical encoding. Main model: Then enter the 3-layer TimesNet network. The structure of a single-layer TimesNet network is that the fast Fourier transform FFT converts the data to the frequency domain, and uses AMP to calculate the average amplitude and obtain the highest k, and pads the length. At the same time, one-dimensional data is transferred to two-dimensional data to obtain two-dimensional features, and two-dimensional data is converted to one-dimensional data. Then the length is truncated and the result is obtained through channel weighting, and finally after a residual layer, it enters the next TimesNet block or fully connected layer. Output result: The final output result is output through a fully connected layer to output the prediction result; Step 3: Based on the designed user experiment, recruit test users to complete the task; Step 4: Export the data collected by the experimental data collection program, and design the corresponding data preprocessing and screening methods; Step 5: Use the deep learning time series model to train the preprocessed data to obtain a time series classification model, use the TimesNet time series classification model, complete the model parameter setting and definition, and use the model for training to obtain a model that completes the black screen and fixed screen classification according to the time series model input, and test the test set data to obtain the final model accuracy; Step 6: Use the torchscript sequence to train the trained time series classification model ptl file, and test the model on an Android device. Use torchscript to serialize the pth file model and convert it into a ptl file. The specific process is to load the pth model file in a new python script, perform a single test output, use the torchscript corresponding api to track the calculation process of the input in the model, and then convert it into a ptl file, deploy the Android test environment and configure dependencies. Create a new Android project based on gradle configuration, and then import two pytorch-related dependencies in the configuration file, namely (org.pytorch:pytorch_android_lite) and (org.pytorch:pytorch_android_torchvision), set the single input test output result, import the ptl model file and put it in the assets resource path of the project, then import a data with a length of 200 used in the training data, use the model for classification, and print the classification results directly to the console to verify that the model can run normally to complete the classification. .

[0006] Preferably, the step one specifically includes the following steps: (1) installing a built-in experimental data collection program on the experimental device Android smartphone; (2) using the experimental data collection program to collect user behavior at a fixed frequency of 200 times per second in different scenarios.

[0007] Preferably, step one includes the following steps: A1, designing a first group of built-in experimental data collection programs, camera: black screen and fixed screen when switching between horizontal and vertical screens; Douyin: black screen and fixed screen during application startup or after switching back to the desktop; WeChat: black screen and fixed screen during application startup or after switching back to the desktop; Honor of Kings: black screen and fixed screen after calling out the WeChat floating window; the power button is invalid under the black screen and fixed screen; A2, designing a second group of built-in experimental data collection programs, camera: black screen and fixed screen during application startup; Douyin: black screen and fixed screen during application use; WeChat: black screen and fixed screen during application use; Honor of Kings: black screen and fixed screen during application use or after switching back to the desktop; the power button is invalid under the black screen and fixed screen; A3, designing a third group of built-in experimental data collection programs, camera: black screen and fixed screen for built-in camera preview; Douyin: black screen and fixed screen in the video area, and normal in other areas; WeChat: Black screen and fixed screen during the use of mini-programs; Honor of Kings: Black screen and fixed screen when calling out the WeChat floating window; the power button is valid under the black screen and fixed screen, but the black screen and fixed screen phenomenon cannot be terminated; A4, design the fourth group of built-in experimental data collection programs, Camera: Black screen and fixed screen when the built-in camera preview is built-in; Tik Tok: Black screen and fixed screen in the video area, and other areas are normal; WeChat: Black screen and fixed screen when browsing Moments; Honor of Kings: Black screen and fixed screen when the main interface of the game is built-in, and other functions such as typing are normal; the power button is valid under the black screen and fixed screen, but the black screen and fixed screen phenomenon cannot be terminated; A5, design the fifth group of built-in experimental data collection programs, Camera: Black screen and fixed screen when switching between horizontal and vertical screens; Tik Tok: Black screen and fixed screen when shooting videos; WeChat: Black screen and fixed screen during video calls; Honor of Kings: Black screen and fixed screen when the main interface of the game is built-in, and other functions such as typing are normal; the power button is valid under the black screen and fixed screen, but the black screen and fixed screen phenomenon cannot be terminated.

[0008] Preferably, the step 2 specifically includes the following steps: B1, collecting the user's touch data (x, y, t) on the touch screen: taking the upper left corner of the screen as the origin of the coordinate axis, from left to right is the x-axis direction, and from top to bottom is the y-axis direction; B2, collecting the user's pressing data on the physical buttons on the screen (press, t1, t2): press is a string that represents the following buttons: the up volume button, the down volume button and the power button, t1 and t2 represent the time of pressing and releasing the button respectively; B3, collecting the scene and time data (scene, event, t1, t2) when a black screen or a fixed screen occurs: scene is divided into two categories, namely application level and global level, among which the whole machine black screen fixed screen There are differences between this and application-level black screen and fixed screen as follows: the whole device black screen and fixed screen cannot be ended by pressing the home button of the mobile phone. In the application-level scenario, pressing the home button can return to the mobile phone desktop. The application level is divided into album scene, King of Glory game scene, WeChat and Douyin scene, and the global scene is divided into desktop and multi-task center. Events are divided into two categories: black screen and fixed screen. t1 and t2 also represent the start and end time of black screen or fixed screen respectively. B4. Collect mobile phone IMU sensor data (sensor, x, y, z, t): sensor represents the type of sensor, including accelerometer and gyroscope. x, y, z represent the corresponding data, which represent the data on the x, y and z axes respectively.

[0009] Preferably, the step 2 specifically includes the following steps: determine the research problem: identify the panic behavior of users in the black screen and fixed screen scenario through the study of user behavior, conduct behavioral cluster analysis, and then define quantitative behavior patterns through behavioral experiments, output user panic behavior information, and assist the business team to more accurately identify the black screen and fixed screen problem; propose a research hypothesis: the panic behavior of users after different black screen and fixed screen events have certain difference characteristics, and through the analysis of the indicator rules of these panic behaviors, the occurrence of black screen and fixed screen events can be reversely inferred; set the research dependent variables as follows: physical button behavior: after the black screen and fixed screen occur, the user will try to lock the screen and wake up through physical buttons such as the lock screen button to try to recover, design data collection indicators (t, Δt, scene, event, button), where t represents the time when each physical button occurs, Δt is the time difference with the black screen / fixed screen burying time (which can be calculated later), scene is the scene where the specific burying occurs, event is a black screen or fixed screen, and button is the physical button pressed; virtual button behavior: after the black screen and fixed screen occur, the user will directly interact with the system to wake up The virtual keys of the layer return to the desktop or open the process to close the related application tasks, and design data collection indicators (t, x, y, Δt, scene, event, key), where t represents the time when each touch screen occurs, x and y are coordinates, Δt is the time difference with the black screen / fixed screen tracking time (which can be calculated later), scene is the scene where the specific tracking occurs, event is black screen or fixed screen, and key is the system-level button pressed; IMU behavior: After the black screen or fixed screen occurs, the IMU data (accelerometer and gyroscope) of the user may be obviously abnormal, such as When playing games in horizontal mode, once the screen goes black, the user will immediately adjust back to the vertical mode. The data collection indicators (t, x1, y1, z1, x2, y2, z2, scene, event) are designed, where t is the time of each trigger, x1, y1, z1 are the values ​​of the corresponding axes of the gyroscope, x2, y2, z2 are the values ​​of the corresponding direction acceleration, scene is the scene where the specific embedding occurs, and event is a black screen or a fixed screen. The experimental steps are set as follows: Each subject needs to experience four types of app scenes and trigger specific black screen and fixed screen problems in each scene. The subjects are divided into 5 groups, according to Figure 1 The black screen fixed screen scenario is assigned in the table. The whole machine black screen fixed screen must be triggered once in each subject's experiment, at most twice, and the scenario is random. The multi-task scenario black screen fixed screen must appear in the 2nd and 4th groups, and randomly appears in the 1st, 3rd, and 5th groups; the specific experimental steps are divided into six steps (see Figure 2), respectively: the subjects fill in the informed consent form and are then assigned to groups; then the users check the instructions; the users become familiar with the phone and use it normally for a period of time; after confirming that they are familiar with it, a formal experiment is conducted. The formal experiment is completed in four application scenarios (camera, TikTok, WeChat, and Honor of Kings), and corresponding tasks are performed in each scenario. Then, the data collection program in step one triggers the black screen fixed screen scenario and collects user behavior indicators. After completing the formal experiment, the subjects fill in the relevant questionnaires, and finally the data is exported and verified.

[0010] Preferably, the step three specifically includes the following steps: recruiting 183 subjects to complete the experimental content according to the experimental process described in claim 6; ensuring that the subject composition includes different genders, age ranges, occupations, educational backgrounds, monthly income situations, and self-used mobile phone models.

[0011] Preferably, the step four specifically includes the following steps: exporting the txt data collected from the experimental data and directly merging it into csv data to convert the data into structured data; manually marking and proofreading the data and annotating it in excel to solve a series of problems; observing the data and finding quantitative indicators by comparing the experimental video; using python language combined with excel data to format the csv data into ts data for subsequent batch processing and model training.

[0012] Preferably, the step two specifically includes the following steps: A1. Loss of black screen fixed screen event record: This problem is very likely to occur when the entire machine is fixed to a black screen and the mobile phone needs to be restarted to continue the experiment. The solution is to complete it in Excel for easy subsequent processing; A2. Continuous triggering of black screen fixed screen and recent tasks: After applying the black screen fixed screen, the recent task deletion process will be entered. The recent task is also a preset black screen fixed screen scene. Therefore, two types of black screen fixed screen scenes will be continuously triggered, causing the data record to be overwritten. The solution is to complete it in Excel; A3. Data recording error: This includes discovering scene recording errors and inconsistent triggering of original tasks of the group through video proofreading. The solution is to correct it in Excel; A4. Data delay: There is sometimes a discrepancy of 1-5s between the manually proofread event and the script export time. The solution is to record the correct time obtained by manual proofreading in Excel.

[0013] Preferably, the ts data format in step 4 specifically includes the following input format and output format requirements: Input format: a matrix with a length of seq_len and a width of 10, each row is a complete time series data, and from top to bottom are the features of each dimension, and each column can be regarded as data obtained by time downsampling; (1) seq_x: represents the input data. In the original model, the input is a tensor with a shape of [batch_size, seq_len, feature_dim], and each dimension of the tensor represents; (2) batch_size: the data batch processed by the model at one time. This value is 1 during testing or actual prediction of the model; (3) seq_len: the model once The length of the input time series data. This value is temporarily set to the data that can be obtained within 5 seconds. Please note that a fixed interval needs to be specified when sampling the mobile phone in this part, and sampling is performed continuously according to this interval. The value of seq_len is set to 5 seconds / sampling interval. However, this value needs to be determined through subsequent ablation experiments to find the optimal value under the balance between algorithm overhead and accuracy; (4) feature_dim: the feature dimension of the data. The final feature dimension length is given here. The total length is 10, representing the data obtained at each timestamp. Each dimension represents: touch x, touch y, button, IMU x, IMU y, IMU z, hour, minute, second, millisecond; Output format: the probability of a black screen in the corresponding time window (the collected data only contains 0 and 1).

[0014] Preferably, the step four includes the following specific steps: B1, read the corresponding csv data according to the name of the first column in excel; B2, read the required data of the corresponding time period in csv according to the corresponding black screen fixed screen time marked in excel, fill it into a vector, and mark it, the black screen fixed screen is 1, and the normal is 0; B3, the data required by the vector is in the ts data format as above, the window size is 200, and it is the data that can be sampled within 1s to improve the immediacy of the detection feedback; B4, divide the data into a training set and a test set in an 8:2 ratio, and finally as Figure 4 Write ts data header metadata.

[0015] Compared with the prior art, the beneficial effect of the present invention is that the framework method for real-time detection of black screen and fixed screen of smart phones adopts a new structural design, and its specific contents are as follows: it includes a complete set of ergonomic experiments to collect a series of panic behavior data of users when facing the black screen and fixed screen phenomenon, performs a series of preprocessing operations on the data, and adopts an advanced time series classification model to complete training according to the data, and finally completes the deployment test on the mobile phone side, which includes a complete framework and implementation for real-time detection of black screen and fixed screen on the mobile phone side, and provides a new solution framework from the intersection of ergonomics and computers. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention designs four types of app scenarios for experiments and the corresponding schematic diagram of the black screen and fixed screen occurrence timing; Figure 2 A schematic diagram of the experimental flow chart for designing the present invention; Figure 3 A schematic diagram of manual data labeling for the present invention; Figure 4 Schematic diagram of training data and metadata of the present invention; Figure 5 A schematic diagram of the parameters required for training the model of the present invention; Figure 6 A schematic diagram of a method for real-time detection of a black screen and a fixed screen framework on a mobile phone of the present invention; Figure 7 Schematic diagram of setting research independent variables for the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1-Figure 7 The present invention provides a technical solution: a framework method for real-time detection of black screen and fixed screen of smart phones, comprising the following steps: step one, constructing a mobile phone experimental platform, which has a built-in experimental data collection program, the mobile phone is a common Android mobile phone model, and the experimental data collection program supports making the mobile phone black and fixed screen and collecting the user's operation behavior of the mobile phone; step two, on the mobile phone experimental platform, designing user experiments and related tasks; step three, based on the user experiment designed in step two, recruiting test users, completing the tasks, and using the experimental platform described in step one to complete the collection of user behaviors; step four, exporting the data collected by the experimental data collection program, and designing corresponding data preprocessing and screening methods; step five, using the deep learning time series model to train the data preprocessed in step four to obtain a time series classification model.

[0019] The detailed introduction is as follows: Design and development of experimental data collection program on mobile phone. Five experimental program scenarios are designed according to the independent variables in the user experiment designed in step 2 above: The first group, camera: black screen and fixed screen when switching between horizontal and vertical screens; Douyin: black screen and fixed screen during application startup or after switching back to the desktop; WeChat: black screen and fixed screen during application startup or after switching back to the desktop; Honor of Kings: black screen and fixed screen after calling out the WeChat floating window; the power button is invalid under the black screen and fixed screen; the second group, camera: black screen and fixed screen during application startup; Douyin: black screen and fixed screen during application use; WeChat: black screen and fixed screen during application use; Honor of Kings: black screen and fixed screen during application use or after switching back to the desktop; the power button is invalid under the black screen and fixed screen; the third group, camera: black screen and fixed screen for built-in camera preview; Douyin: video area Black screen fixed, other areas are normal; WeChat: black screen fixed during use of mini program; Honor of Kings: black screen fixed when calling out WeChat floating window; the power button is valid under black screen fixed, but the black screen fixed phenomenon cannot be terminated; the fourth group, camera: black screen fixed when built-in camera preview; Tik Tok: black screen fixed in video area, other areas are normal; WeChat: black screen fixed when browsing Moments; Honor of Kings: black screen fixed on the main interface of the game, other functions such as typing are normal; the power button is valid under black screen fixed, but the black screen fixed phenomenon cannot be terminated; the fifth group, camera: black screen fixed when switching between horizontal and vertical screens; Tik Tok: black screen fixed during video shooting; WeChat: black screen fixed during video calls; Honor of Kings: black screen fixed on the main interface of the game, other functions such as typing are normal; the power button is valid under black screen fixed, but the black screen fixed phenomenon cannot be terminated.

[0020] The experimental data collection program is used to collect user behaviors at a fixed frequency of 200 times per second in different scenarios. The collected user behaviors specifically include the following: user touch data on the touch screen (x, y, t): the upper left corner of the screen is the origin of the coordinate axis, the x-axis direction is from left to right, and the y-axis direction is from top to bottom; user pressing data on the physical buttons on the screen (press, t1, t2): press is a string representing the following buttons: the up volume button, the down volume button, and the power button, t1 and t2 represent the time of pressing and releasing the button respectively; scene and time data of black screen or fixed screen (scene, event, t1, t2): scene is divided into two categories, namely application level and event level. At the global level, there are differences between the black screen fixation of the entire device and the black screen fixation at the application level as follows: the black screen fixation of the entire device cannot be ended by pressing the home button of the mobile phone, but in the application-level scenario, pressing the home button can return to the mobile phone desktop. The application level is divided into album scenes, King of Glory game scenes, WeChat and Douyin scenes, and the global scenes are divided into desktop and multi-task center; events are divided into two categories: black screen and fixed screen; t1 and t2 also represent the start and end time of the black screen or fixed screen respectively; mobile phone sensor data (sensor, x, y, z, t): sensor represents the type of sensor, including accelerometer and gyroscope; x, y, z represent the corresponding data, which represent the data on the x, y and z axes respectively.

[0021] An experimental data collection program with the above functions is installed on an Android phone, and a user experiment is designed to determine the research problem: Through the study of user behavior, the panic behavior of users in the black screen fixed screen scenario is identified, and behavioral clustering analysis is performed. Then, quantitative behavioral patterns are defined through behavioral experiments, and user panic behavior information is output to assist the business team in more accurately identifying the black screen fixed screen problem; a research hypothesis is proposed: the panic behavior of users after different black screen fixed screen events has certain differences. By analyzing the indicators and rules of these panic behaviors, the occurrence of black screen fixed screen events can be reversely inferred.

[0022] The research dependent variables are determined as follows: Physical button behavior: after a black screen or fixed screen occurs, the user will try to restore by locking the screen and waking up through physical buttons such as the lock screen button. The data collection indicators (t, Δt, scene, event, button) are designed, where t represents the time when each physical button occurs, Δt is the time difference with the black screen / fixed screen burying time (which can be calculated later), scene is the specific scene where the burying occurs, event is a black screen or fixed screen, and button is the physical button pressed; Virtual button behavior: after a black screen or fixed screen occurs, the user will directly return to the desktop or open the process to close the related application tasks through the virtual buttons of the system interaction layer. The data collection indicators (t, x, y, Δt, scene, event, key) are designed, where t represents each The time when the screen touch occurs, x and y are coordinates, Δt is the time difference with the black screen / fixed screen tracking time (which can be calculated later), scene is the specific scene where the tracking occurs, event is black screen or fixed screen, and key is the system-level button pressed; IMU behavior: the IMU data (accelerometer and gyroscope) of the user may be obviously abnormal after the black screen or fixed screen occurs. For example, when playing games in horizontal mode, once the screen goes black, the user will immediately adjust it back to vertical mode. Design data collection indicators (t, x1, y1, z1, x2, y2, z2, scene, event), where t is the time of each trigger, x1, y1, z1 are the values ​​of the corresponding axes of the gyroscope, x2, y2, z2 are the values ​​of the corresponding direction acceleration, scene is the specific scene where the tracking occurs, and event is black screen or fixed screen.

[0023] The experimental steps are as follows: Each subject needs to experience four types of app scenarios and trigger a specific black screen problem in each scenario. The subjects are divided into 5 groups according to Figure 1 The black screen fixed screen scenario is assigned in the table. The whole machine black screen fixed screen must be triggered once in each subject's experiment, at most twice, and the scenario is random. The multi-task scenario black screen fixed screen must appear in the 2nd and 4th groups, and randomly appears in the 1st, 3rd, and 5th groups; the specific experimental steps are divided into six steps (see Figure 2 ), which are: the subjects fill in the informed consent form and are then assigned to groups; then the users check the instructions; the users become familiar with the phone and use it normally for a period of time; after confirming that they are familiar with it, a formal experiment is conducted. The formal experiment is completed in four application scenarios (camera, TikTok, WeChat and King of Glory), and corresponding tasks are performed in each scenario. Then, the data collection program in step (1) triggers the black screen freeze scenario and collects user behavior indicators; after completing the formal experiment, the subjects fill in the relevant questionnaire; finally, the data is exported and verified.

[0024] Recruit subjects to complete the experiment according to the above experimental steps, and recruit 183 subjects to complete the experiment according to the experimental process described in step (2); preferably, the subjects are divided into: by gender, 102 males and 81 females; by age, 96 people under 25 years old, 51 people between 26 and 35 years old, and 36 people over 35 years old; by occupation, 86 students and 97 workers; by education level, 51 people with college degree or below, and 132 people with bachelor degree or above; by monthly income, 84 people with less than 6,000 yuan, 88 people with 6,000-19,999 yuan, and 11 people with more than 20,000 yuan; by self-used mobile phone model, 115 people with Android and 68 people with iOS, and complete a series of preprocessing operations on the data collected for the experiment.

[0025] Manually label the data to solve the following problems: Loss of black screen fixed screen event records: This problem is very likely to occur when the entire device is black screen fixed and the phone needs to be restarted to continue the experiment. The solution is to complete it in Excel for subsequent processing; Apply black screen fixed screen and continuous triggering of recent tasks: After applying black screen fixed screen, the subjects will enter the recent task deletion process. The recent task is also a preset black screen fixed screen scene, so two types of black screen fixed screen scenes will be triggered continuously, causing the data records to be overwritten. The solution is to complete it in Excel; Data recording errors: including scene recording errors discovered through video proofreading, and inconsistent triggering of the original task of the group. The solution is to correct it in Excel; Data delay: There is sometimes a discrepancy of 1-5s between the manually proofread event and the script export time. The solution is to record the correct time obtained by manual proofreading in Excel.

[0026] The final ts data format used for training is set, including the input format and output format as follows: Input format: a matrix with a length of seq_len and a width of 10. Each row is a complete time series data. From top to bottom, they are the features of each dimension. Each column can be regarded as the data obtained by time downsampling: (1) seq_x: represents the input data. In the original model, the input is a tensor with a shape of [batch_size, seq_len, feature_dim]. The dimensions of this tensor represent: (2) batch_size: the batch of data processed by the model at one time. This value is 1 during testing or actual prediction of the model; (3) seq_len: the length of the time series data input by the model at one time. This value is temporarily set to the data that can be obtained within 5 seconds. Please note that a fixed interval needs to be specified when sampling the mobile phone in this part, and sampling is performed continuously according to this interval. The value of seq_len is set to 5 seconds / sampling interval; however, this value needs to be determined through subsequent ablation experiments to achieve the best balance between algorithm overhead and accuracy. Merit value; (4) feature_dim: feature dimension of data. Here, the final feature dimension length is given. The total length is 10, representing the data obtained at each timestamp. Each dimension represents: touch x, touch y, button, IMU x, IMU y, IMU z, hour, minute, second, millisecond; output format: the probability of a black screen in the corresponding time window (the collected data only contains 0 and 1); the steps of using python combined with the excel data marked to process the structured csv data merged from the txt data directly collected from the experiment are as follows: read the corresponding csv data according to the subject name in the first column of excel; read the required data of the corresponding time period in csv according to the corresponding black screen fixed time marked in excel, fill it into a vector, and mark it. The black screen fixed time is 1 and the normal is 0; the required data of the vector is in the ts data format as above, with a window size of 200, which is the data that can be sampled within 1s to improve the immediacy of detection feedback; divide the data into training set and test set in 8:2, and finally as Figure 4 Write ts data header metadata; use the deep learning time series model to train the preprocessed data to obtain a time series classification model; use the TimesNet time series classification model to complete the model parameter setting and definition.

[0027] The model parameters are set as Figure 5First, set the model to be used as a classification model, that is, task_name is set to classification; then enter the training mode, that is, is_training=1; then set the model training parameters such as the number of network layers to 3, the window size to 200, the batch size to 8, and the model training rounds to 20; use this model for training to obtain a model that completes the black screen and fixed screen classification based on the time series model input; the model process is as follows: first feed the data, then standardize and encode the data, the data encoding is divided into time encoding and value encoding, etc.; then feed the 3-layer TimesNet network, and finally output the result through a fully connected layer to output the prediction result; a single-layer TimesNet The structure of the t network is: fast Fourier transform FFT converts data to frequency domain; AMP is used to calculate the average amplitude and obtain the highest k; padding is performed on the length; one-dimensional data is converted to two-dimensional data to obtain two-dimensional features; two-dimensional data is converted to one-dimensional data, and then the length is truncated and the result is obtained through channel weighting, and finally enters the next TimesNet block or fully connected layer after passing through a residual layer; in the actual training process, the above model parameters were modified many times, such as the window (that is, the sequence length of the classification), and it was found that 200 was a more appropriate value. The model training rounds were set to 20. The loss loss and accuracy of the model have converged under the amount of data collected in the current experiment. ; Test the test set data to get the final model accuracy; After the model training is completed, test it on the test set and get a final model accuracy of 90.27%; Use torchscript to serialize the trained time series classification model into a ptl file and test the model on an Android device; Use torchscript to serialize the pth file model and convert it into a ptl file. The specific process is to load the pth model file in a new python script, perform a single test output, use the torchscript corresponding api to track the calculation process of the input in the model, and then convert it into a ptl file; Deploy the Android test environment and configure dependencies, and then Build an Android project based on gradle configuration, and then import two pytorch-related dependencies in the configuration file, namely: 'org.pytorch:pytorch_android_lite' and 'org.pytorch:pytorch_android_torchvision'; set up a single input test output result, import the ptl model file and put it in the assets resource path of the project, and then import a data of 200 length used in the training data, use the model for classification, and print the classification results directly to the console. The model can be loaded normally without error, and the classification results are output correctly, and the test verification is completed.

[0028] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A framework method for real-time detection of black screen and fixed screen of smart phone, characterized in that: The steps include: Step 1: Build a mobile phone experimental platform with a built-in experimental data collection program. The mobile phone is a common Android mobile phone model. The experimental data collection program supports triggering the black screen of the mobile phone and collecting the user's operation behavior of the mobile phone. The model type is set to the classification model, that is, task_name is set to classification, and the training mode is set to 1. The model training parameters are set, such as the number of network layers is 3, the window size is 200, the batch size is 8, and the model training rounds are set to 20; Step 2: Design user experiments and related tasks on the mobile phone experimental platform, data feeding and processing: first feed the data, then standardize and encode the data, the data encoding is divided into time encoding and numerical encoding, etc.; main model: then enter the 3-layer TimesNet network, the structure of the single-layer TimesNet network is that the fast Fourier transform FFT converts the data to the frequency domain, and uses AMP to calculate the average amplitude and obtain the highest k, and pads the length. At the same time, the one-dimensional data is transferred to the two-dimensional data to obtain the two-dimensional features, the two-dimensional data is converted to the one-dimensional data, and then the length is truncated and the result is obtained by channel weighting, and finally after a residual layer, it enters the next TimesNet block or fully connected layer; output result: the final output result is output through a fully connected layer to output the prediction result; Step 3: Based on the designed user experiment, recruit test users to complete the task; Step 4: Export the data collected by the experimental data collection program and design corresponding data preprocessing and screening methods; Step 5: Use the deep learning time series model to train the preprocessed data to obtain a time series classification model. Use the TimesNet time series classification model to complete the model parameter setting and definition, and use the model for training to obtain a model that completes the black screen and fixed screen classification based on the time series model input. Test the test set data to obtain the final model accuracy; Step 6. Use torchscript to serialize the trained time series classification model into a ptl file, and test the model on an Android device. Use torchscript to serialize the pth file model and convert it into a ptl file. The specific process is to load the pth model file in a new python script, perform a single test output, use the torchscript corresponding api to track the calculation process of the input in the model, and then convert it into a ptl file, deploy the Android test environment and configure dependencies, create a new Android project based on gradle configuration, and then import two pytorch-related dependencies in the configuration file, namely (org.pytorch:pytorch_android_lite) and (org.pytorch:pytorch_android_torchvision), set the single input test output result, import the ptl model file and place it in the assets resource path of the project, and then import a 200-length data used in the training data, use the model for classification, and print the classification results directly to the console to verify that the model can run normally to complete the classification.

2. A framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step one specifically includes the following steps: (1) installing a built-in experimental data collection program on the experimental device Android smartphone; (2) using the experimental data collection program to collect user behavior at a fixed frequency of 200 times per second in different scenarios.

3. A framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step one includes the following steps: A1, designing a first group of built-in experimental data collection programs, camera: black screen and fixed screen when switching between horizontal and vertical screens; Douyin: black screen and fixed screen during application startup or after switching back to the desktop; WeChat: black screen and fixed screen during application startup or after switching back to the desktop; Honor of Kings: black screen and fixed screen after calling out the WeChat floating window; the power button is invalid under the black screen and fixed screen; A2, designing a second group of built-in experimental data collection programs, camera: black screen and fixed screen during application startup; Douyin: black screen and fixed screen during application use; WeChat: black screen and fixed screen during application use; Honor of Kings: black screen and fixed screen during application use or after switching back to the desktop; the power button is invalid under the black screen and fixed screen; A3. Design the third group of built-in experimental data collection programs. Camera: built-in camera preview black screen fixed screen; Tik Tok: video area black screen fixed screen, other areas normal; WeChat: The screen is black and fixed when the mini program is in use; Honor of Kings: The WeChat floating window is black and fixed; The power button is valid under the black screen and fixed screen, but it cannot stop the black screen and fixed screen phenomenon; A4, design the fourth group of built-in experimental data collection programs, camera: the built-in camera preview is black and fixed; Douyin: The video area is black and fixed, and other areas are normal; WeChat: Browsing Moments will result in a black screen and freeze. Honor of Kings: The main interface of the game will be black and frozen, but other functions such as typing are normal. The power button is effective under the black screen and fixed screen condition, but it cannot stop the black screen and fixed screen condition; A5, design the fifth group of built-in experimental data collection programs, camera: black screen and fixed screen when switching between horizontal and vertical screens; Tik Tok: black screen and fixed screen when shooting videos; WeChat: The screen is black and frozen during a video call; Honor of Kings: The main interface of the game is black and frozen, but other functions such as typing are normal; The power button is effective when the screen is stuck in black, but it cannot stop the black screen.

4. A framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step 2 specifically includes the following steps: B1, collecting the user's touch data on the touch screen (x, y, t): taking the upper left corner of the screen as the origin of the coordinate axis, the x-axis direction is from left to right, and the y-axis direction is from top to bottom; B2, collecting the user's pressing data on the physical buttons on the screen (press, t1, t2): press is a string that represents the following buttons: the up volume button, the down volume button and the power button, t1 and t2 represent the time of pressing and releasing the button respectively; B3, collecting the scene and time data (scene, event, t1, t2) when the black screen or fixed screen occurs: scene is divided into two categories, namely application level and global level, among which the whole machine black screen fixed screen and application There are differences in user-level black screen and fixed screen as follows: the whole device cannot be ended by pressing the home button of the mobile phone when the screen is black. In the application-level scenario, pressing the home button can return to the mobile phone desktop. The application level is divided into album scene, King of Glory game scene, WeChat and Douyin scene, and the global scene is divided into desktop and multi-task center; event is divided into two categories: black screen and fixed screen; t1 and t2 also represent the start and end time of black screen or fixed screen respectively; B4. Collect mobile phone IMU sensor data (sensor, x, y, z, t): sensor represents the type of sensor, including accelerometer and gyroscope, x, y, z represent the corresponding data, which represent the data on the x, y and z axes respectively.

5. The framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step 2 specifically includes the following steps: determine the research problem: identify the panic behavior of users in the black screen fixed screen scenario through the study of user behavior, conduct behavioral cluster analysis, and then define quantitative behavior patterns through behavioral experiments, output user panic behavior information, and assist the business team to more accurately identify the black screen fixed screen problem; propose a research hypothesis: the panic behavior of users after different black screen fixed screen events have certain differences. By analyzing the indicators and rules of these panic behaviors, the occurrence of black screen fixed screen events can be reversely inferred; set the research dependent variables as follows: physical button behavior: after the black screen fixed screen occurs, the user will try to Physical buttons such as the lock screen button lock the screen and wake up to try to recover. Design data collection indicators (t, Δt, scene, event, button), where t represents the time when each physical button occurs, Δt is the time difference with the black screen / fixed screen tracking time (which can be calculated later), scene is the scene where the specific tracking occurs, event is a black screen or fixed screen, and button is the physical button pressed; Virtual button behavior: After the black screen or fixed screen occurs, the user will directly return to the desktop or open the process to close the related application tasks through the virtual buttons of the system interaction layer. Design data collection indicators (t, x, y, Δt, scene, event, key), where t represents the time when each touch occurs, x and y are coordinates, Δt is the time difference with the black screen / fixed screen tracking time (which can be calculated later), scene is the scene where the specific tracking occurs, event is black screen or fixed screen, and key is the system-level button pressed; IMU behavior: The IMU data (accelerometer and gyroscope) of the user may be obviously abnormal after the black screen or fixed screen occurs. For example, when playing a game in horizontal mode, once the screen goes black, the user will immediately adjust it back to vertical mode. Design data collection indicators (t, x1, y1, z1, x2, y2, z2, scene, event ), where t is the time of each trigger, x1, y1, z1 are the values ​​of the corresponding axes of the gyroscope, x2, y2, z2 are the values ​​of the corresponding direction acceleration, scene is the scene where the specific embedding occurs, and event is a black screen or a fixed screen; the experimental steps are set as follows: each subject needs to experience four types of app scenes and trigger a specific black screen and fixed screen problem in each scene. The subjects are divided into 5 groups, and the black screen and fixed screen scenes are assigned according to the table in Figure 1. The whole machine black screen and fixed screen must be triggered once in the experiment of each subject, at most twice, and the scene is random. The black screen and fixed screen in the multi-task scene must appear in the 2nd and 4th groups, and appear randomly in the 1st, 3rd, and 5th groups; The specific experimental steps are divided into six steps, namely: the subjects fill out the informed consent form and are then assigned to groups; then the users check the instructions; the users become familiar with the mobile phone and use it normally for a period of time; after confirming that they are familiar with it, a formal experiment is conducted. The formal experiment is completed in four application scenarios (camera, TikTok, WeChat and Honor of Kings), and corresponding tasks are performed in each scenario. Then, the data collection program in step one will trigger the black screen fixed screen scenario and collect user behavior indicators. After completing the formal experiment, the subjects fill out the relevant questionnaires, and finally the data is exported and verified.

6. A framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step three specifically includes the following steps: recruiting 183 subjects to complete the experimental content according to the experimental process, ensuring that the subjects include different genders, age ranges, occupations, educational backgrounds, monthly incomes, and self-used mobile phone models.

7. A framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step 4 specifically includes the following steps: exporting the txt data collected from the experimental data and directly merging it into csv data, converting the data into structured data; manually marking and proofreading the data and annotating it in excel to solve a series of problems; Observe data and find quantitative indicators by comparing the experimental video; use Python language combined with Excel data to format CSV data into TS data for subsequent batch processing and model training.

8. The framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step 2 specifically includes the following steps: A1. Loss of black screen fixed screen event records: This problem is very likely to occur when the entire device is black screen fixed and the mobile phone needs to be restarted to continue the experiment. The solution is to complete it in Excel for subsequent processing; A2. Apply black screen fixed screen and recent tasks to trigger continuously: After applying black screen fixed screen, the recent task deletion process will be entered. The recent task is also a preset black screen fixed screen scene, so two types of black screen fixed screen scenes will be continuously triggered, causing the data records to be overwritten. The solution is to complete it in Excel; A3. Data recording errors: This includes scene recording errors and inconsistent triggering of the original tasks of the groups found through video proofreading. The solution is to correct them in Excel; A4. Data delay: There may be a 1-5 second discrepancy between the manually proofread event and the script export time. The solution is to record the correct time obtained by manual proofreading in Excel.

9. A framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The ts data format in step 4 specifically includes the following input format and output format requirements: Input format: a matrix with a length of seq_len and a width of 10. Each row is a complete time series data. From top to bottom, they are the features of each dimension. Each column can be regarded as data obtained by time downsampling; (1) seq_x: represents the input data. In the original model, the input is a tensor with a shape of [batch_size, seq_len, feature_dim], and each dimension of the tensor represents; (2) batch_size: the batch of data processed by the model at one time. This value is 1 during testing or actual prediction of the model; (3) ) seq_len: the length of the time series data input to the model at one time. This value is temporarily set to the data that can be obtained within 5 seconds. Please note that a fixed interval needs to be specified when sampling the mobile phone in this part, and sampling is performed continuously according to this interval. The value of seq_len is set to 5 seconds / sampling interval. However, this value needs to be determined through subsequent ablation experiments to find the optimal value under the balance between algorithm overhead and accuracy; (4) feature_dim: the feature dimension of the data. Here, the final feature dimension length is given. The total length is 10, representing the data obtained at each timestamp. Each dimension represents: touch x, touch y, button, IMU x, IMU y, IMU z, hour, minute, second, millisecond; Output format: the probability of a black screen in the corresponding time window (only 0 and 1 are collected).

10. The framework method for real-time detection of black screen and fixed screen of smart phone according to claim 1, characterized in that: The step 4 includes the following specific steps: B1. Read the corresponding csv data according to the name of the first column in excel; B2. Read the required data of the corresponding time period in csv according to the corresponding black screen fixed screen time marked in excel, fill it into a vector, and mark it. The black screen fixed screen is 1 and the normal is 0; B3. The data required by the vector is in the ts data format as above, and the window size is 200, which is the data that can be sampled within 1s to improve the immediacy of detection feedback; B4. Divide the data into training set and test set in 8:2, and finally write the ts data header metadata as shown in Figure 4.

Images