Intelligent machine description fault identification auxiliary processing method and system for elderly population
By establishing a fault association network graph and using multimodal feature fusion technology, the problem of low identification accuracy in auxiliary methods for fault identification of smartphones for the elderly has been solved, and accurate fault type diagnosis and personalized solutions have been achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2025-05-21
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, the methods for assisting the identification of faults in smartphones for the elderly rely on keyword matching and simple classification. These methods cannot understand ambiguous expressions, make it difficult to distinguish between real faults and operational errors, resulting in low identification accuracy and poor suitability for the elderly.
By establishing a fault association network graph and combining multimodal feature fusion and knowledge graph technologies, we analyze video and audio data provided by the elderly to generate personalized solutions, including multimodal models and interactive clarification tasks, thereby improving recognition accuracy.
It achieves accurate identification of fault descriptions in the elderly, reduces misjudgments, improves the accuracy of fault type diagnosis, and reduces computing load through edge computing, providing personalized age-friendly solutions.
Smart Images
Figure CN120524429B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, specifically to a method and system for assisting in the identification and processing of descriptive faults in smartphones for the elderly. It can be used for identifying and assisting in the processing of descriptive faults in smartphones for the elderly. Background Technology
[0002] Smartphone fault identification and assistance for the elderly refers to automatically identifying device fault types by analyzing video and audio data describing faults provided by the elderly, combining multimodal feature fusion (visual + voice) and knowledge graph technology, and generating personalized solutions (such as voice guidance, graphic guidance, or remote assistance) that are easy for the elderly to understand.
[0003] Smartphones are currently the most widely used fault identification and assistance system service target among the elderly. However, their primary target audience is a younger user group. This results in complex hardware design and operating methods of the built-in system, as well as numerous functions. Consequently, the learning curve for elderly people to use smartphones is high, making it difficult for them to independently resolve both malfunctions and non-malfunctions.
[0004] Existing methods for assisting the identification of faults in smartphones for the elderly mainly rely on keyword matching and simple classification. They extract keywords solely from voice and text, which cannot understand ambiguous expressions. Furthermore, single-modal comprehensive analysis makes it difficult to distinguish between real faults and operational errors. This results in methods that are highly versatile but poorly suited to the elderly. In addition, the complex technical terms output lead to low accuracy in identifying unstructured descriptions of problems in the elderly. Summary of the Invention
[0005] To address the aforementioned technical issues, this paper provides a method and system for assisting in the identification of smartphone malfunctions for the elderly. This solution overcomes the shortcomings of current smartphone malfunction identification methods for the elderly, which primarily rely on keyword matching and simple classification. These methods suffer from significant deficiencies, relying solely on extracting keywords from voice and text, failing to understand ambiguous expressions. Furthermore, single-modal comprehensive analysis struggles to distinguish between genuine malfunctions and operational errors, resulting in strong versatility but poor suitability for the elderly. Additionally, the complex technical terminology output leads to low accuracy in identifying unstructured descriptions of problems affecting the elderly.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A smartphone-based fault identification and assistance method for the elderly includes:
[0008] Based on the official debug logs and user feedback communities of known smartphone operating systems, calculate the prior probability of historical fault types for each smartphone and establish a fault association network graph of known smartphones.
[0009] Based on edge devices, acquire video data describing the faults of the target elderly person's smartphone, and divide it into video stream and audio stream to obtain video-audio stream data describing the faults of the target elderly person's smartphone.
[0010] Verify the similarity score between the video vector and audio stream vector describing the fault in the smartphone, and determine whether the similarity score is within the consistency threshold range. If it is, the fault description is considered normal; otherwise, the fault description is considered abnormal.
[0011] If the fault description is abnormal, locate the smartphone fault description video vector-audio stream vector whose similarity score is not in the consistency threshold range, generate an interactive clarification task, and update the smartphone fault description video-audio stream data of the target elderly person.
[0012] Based on the video-audio stream data describing smartphone malfunctions of the target elderly, a multimodal model for smartphone malfunction identification is established, and a smartphone malfunction localization-confidence matrix is generated for the target elderly.
[0013] By matching the smartphone fault location-confidence matrix of the target elderly with the fault association network map of known smartphones, the fault type of the target elderly's smartphone is determined and an auxiliary processing plan is generated.
[0014] Preferably, based on the official debug logs of the operating system of known smartphones and user feedback communities, historical fault type data of each smartphone is obtained and data preprocessed.
[0015] Based on the kernel version of each smartphone's system, the historical fault type data is divided. The operating time of the smartphone's system kernel version is used as the observation window, and the absolute frequency of each fault type in the historical fault type data of each smartphone is used as the observation object to obtain the time series data of the frequency characteristics of historical fault types of each smartphone per unit time.
[0016] Collect time-series data on the frequency characteristics of historical fault types for each smartphone per unit time, determine the total number of historical fault types for each smartphone, and calculate the prior probability of each fault type for each smartphone per unit time.
[0017] Preferably, ARMA time series analysis is trained based on historical fault type data of smartphones, with the prior probability of fault type of each smartphone per unit time as input and the fault probability of fault type of each smartphone per unit time as output.
[0018] Based on the fault probability of each smartphone per unit time for each fault type, calculate the fault dependency relationship of each smartphone per unit time for each fault type, and determine the fault co-occurrence probability of each smartphone per unit time for each fault type.
[0019] By utilizing the co-occurrence probability of fault types for each smartphone per unit time, the fault influence strength value of fault types for each smartphone per unit time is calculated, and the causal association probability of fault types for each smartphone per unit time is determined.
[0020] Based on the causal association probability of fault types for each smartphone per unit time, calculate the fault interaction intensity of fault types for each smartphone per unit time, and establish the fault type association matrix for each smartphone per unit time.
[0021] Based on the fault type association matrix of each smartphone per unit time, a fault association network graph of known smartphones is established, with fault type as the central node, fault dependency relationship of fault type of each smartphone per unit time as the edge, and fault interaction strength of fault type as the edge weight.
[0022] Preferably, based on the smartphone fault description video-audio stream data of the target elderly person, frame extraction is performed according to the sampling rate of the smartphone fault description video stream data to obtain smartphone fault description video frame data;
[0023] The size of the video frame data describing the fault of the smartphone is pre-adjusted, and the pixels of each video frame describing the fault of the smartphone are standardized. The data are then fed into a CNN convolutional neural network to extract the frame-level feature vector of the smartphone fault description of the target elderly person, and a frame-level feature matrix of the smartphone fault description of the target elderly person is constructed.
[0024] Based on the frame-level feature matrix of the smartphone fault description video of the target elderly, the mean of the corresponding smartphone fault description video frame-level feature vector is extracted according to the time attribute of each frame and average pooling is performed to obtain the global feature vector of the smartphone fault description video of the target elderly.
[0025] Preferably, based on the smartphone fault description video-audio stream data of the target elderly person, the smartphone fault description audio stream data is divided into segments according to unit time to obtain smartphone fault description audio segment data;
[0026] The audio segment data describing the fault based on the smartphone will be preprocessed, and the time attribute of the audio segment data describing the fault based on the smartphone will be used as a time window. Frame segmentation-frame shifting-windowing will be performed on the audio segment data describing the fault based on the smartphone within the time window to obtain the short-time signal of the audio describing the fault based on the smartphone.
[0027] Using short-time Fourier transform, the spectrum of each frame in the short-time signal of the fault description audio frame of the smartphone is calculated to obtain the linear spectrum of the fault description audio of the smartphone.
[0028] Based on the Mel filter, the linear spectrogram of the fault description audio of the smartphone is converted into Mel frequency, and the Mel spectrogram of the fault description audio of the smartphone is generated.
[0029] The size of the Mel spectrogram of the smartphone fault description audio is adjusted and substituted into VGGish deep learning. Mel frequency feature vectors of the smartphone fault description audio frames are extracted according to each time window to construct the time-feature matrix of the smartphone fault description audio for the target elderly.
[0030] Based on the self-attention mechanism of VGGish deep learning, a weight is assigned to the feature vector of each frame in the time-feature matrix of the smartphone fault description audio of the target elderly person. The global feature vector of the smartphone fault description audio of the target elderly person is calculated by weighted summation.
[0031] Preferably, based on cosine similarity, a similarity score is calculated between the global feature vector of the smartphone describing the fault in the target elderly person's video and the global feature vector of the smartphone describing the fault in the target elderly person's audio.
[0032] Determine whether the similarity score between the global feature vector of the smartphone describing the fault in the target elderly person's video and the global feature vector of the smartphone describing the fault in the target elderly person's audio is within the consistency threshold range. If yes, the fault description is considered normal; otherwise, the fault description is considered abnormal.
[0033] Preferably, the global feature vectors of the target elderly person’s smartphone description video and audio stream corresponding to the smartphone description video vector-audio stream vector whose similarity score is not in the consistency threshold range are denoted as the target elderly person’s smartphone description video-audio global feature anomaly vector.
[0034] Based on SVM (Support Vector Machine), the input is the global feature anomaly vector of the smartphone fault description video-audio of the target elderly person, and the output is the fault description category of the smartphone of the target elderly person.
[0035] The NLP (Natural Language Processing) model inputs the smartphone description fault classification categories of the target elderly into the dictionary mapping, determines the corresponding question templates for the smartphone description fault classification categories of the target elderly, and generates an interactive clarification task.
[0036] Preferably, based on the Transformer architecture, the global feature vector of the smartphone describing the fault video and the global feature vector of the smartphone describing the fault audio of the target elderly are used as outputs. Linear projection is performed on the global feature vector of the smartphone describing the fault video and the global feature vector of the smartphone describing the fault audio to obtain the global feature space vector of the smartphone describing the fault video and the global feature space vector of the smartphone describing the fault audio.
[0037] The cross-modal attention weights of the global feature space vector of the video describing the fault of the intelligent machine and the global feature space vector of the audio describing the fault of the intelligent machine are used to fuse the features to obtain the global fused feature vector of the video-audio describing the fault of the intelligent machine.
[0038] Based on the global fusion feature vector of video-audio describing faults in smartphones, the input is into the MLP multilayer perceptron classifier, and the output is the probability distribution of smartphone description fault categories of the target elderly person.
[0039] Based on the video frame-level feature matrix describing the smartphone fault of the target elderly and the audio time-feature matrix describing the smartphone fault of the target elderly, the data is input into the spatiotemporal localization classifier. Using a temporal convolutional network and attention mechanism, the temporal attention weights corresponding to each video frame-level feature describing the smartphone fault and the audio time-feature describing the smartphone fault are determined, and the smartphone fault localization heatmap of the target elderly is calculated.
[0040] Based on the probability distribution of smartphone fault description categories and the smartphone fault location heatmap of the target elderly, the confidence value corresponding to each smartphone fault description category per unit time is calculated, and a smartphone fault location-confidence matrix of the target elderly is established.
[0041] Preferably, based on the smartphone fault location-confidence matrix of the target elderly, the global confidence score of each fault type is calculated;
[0042] Based on the global confidence score of each fault type, the positively correlated sub-fault association network graphs in the known fault association network graphs of smartphones are selected and denoted as the positively correlated sub-fault association network graphs of smartphone faults for the target elderly.
[0043] Based on the positive correlation sub-fault association network graph of smartphone faults among the target elderly, the fault type category corresponding to the correlation sub-fault association network graph is determined, and then substituted into the NLP natural language processing model to generate an auxiliary processing solution.
[0044] Furthermore, a fault identification and assistance system for smartphones used by the elderly is described.
[0045] Knowledge graph construction module, data acquisition module, data rationality verification module, data update module, fault task identification module, and fault task processing module.
[0046] The knowledge graph construction module is used to calculate the prior probability of historical fault type data for each smartphone based on the official debug logs of the operating system of known smartphones and user feedback communities, and to establish a fault association network graph of known smartphones.
[0047] The data acquisition module is used to acquire video data describing the fault of the target elderly person's smartphone based on the edge device, and divide it according to the video stream and audio stream to obtain the video-audio stream data describing the fault of the target elderly person's smartphone.
[0048] The data rationality verification module is wirelessly connected to the data acquisition module. The data rationality verification module is used to verify the similarity score between the video vector and audio stream vector describing the fault by the smartphone, and to determine whether the similarity score is within the consistency threshold range. If it is, the fault description is determined to be normal; otherwise, the fault description is determined to be abnormal.
[0049] The data update module is electrically connected to the data rationality verification module and the data acquisition module. The data update module is used to locate the smartphone description fault video vector-audio stream vector that is not in the consistency threshold range if the fault description is abnormal, generate an interactive clarification task, and update the smartphone description fault video-audio stream data of the target elderly person.
[0050] The fault task identification module is electrically connected to the data update module. The fault task identification module is used to establish a multimodal model for identifying faults in the smartphone based on the video-audio stream data describing the faults in the target elderly person's smartphone, and generate a smartphone fault location-confidence matrix for the target elderly person.
[0051] The fault task processing module is electrically connected to the fault task identification module. The fault task processing module is used to perform correlation matching between the fault location-confidence matrix of the target elderly person's smart phone and the fault association network map of known smart phones to determine the fault type of the target elderly person's smart phone and generate an auxiliary processing plan.
[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0053] This invention improves recognition accuracy through multimodal fusion, aiding in the understanding of ambiguous expressions, reducing misjudgments, and enhancing the accuracy of fault type diagnosis. Simultaneously, it can be implemented using edge computing, reducing computational load and improving applicability. By combining correlation and causality in fault analysis and incorporating fault propagation prediction, it ultimately provides precise decision support for fault types, representing a personalized, age-friendly solution. Attached Figure Description
[0054] Figure 1 A flowchart describing a fault identification and auxiliary processing method for smartphones used by the elderly;
[0055] Figure 2 A framework diagram of a fault identification and auxiliary processing system for smartphones used by the elderly is provided. Detailed Implementation
[0056] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.
[0057] Reference Figure 1 As shown, a fault identification and assistance method for smartphones used by the elderly includes:
[0058] Step 1: Based on the official debug logs and user feedback communities of the known smartphone operating systems, calculate the prior probability of historical fault types for each smartphone and establish a fault association network graph for the known smartphones.
[0059] Step one includes the following:
[0060] Step 101: Based on the official debug logs of the operating system of known smartphones and user feedback communities, obtain historical fault type data for each smartphone and perform data preprocessing.
[0061] Based on the kernel version of each smartphone's system, the historical fault type data is divided. The operating time of the smartphone's system kernel version is used as the observation window, and the absolute frequency of each fault type in the historical fault type data of each smartphone is used as the observation object to obtain the time series data of the frequency characteristics of historical fault types of each smartphone per unit time.
[0062] Collect time-series data on the frequency characteristics of historical fault types for each smartphone per unit time, determine the total number of historical fault types for each smartphone, and calculate the prior probability of each fault type for each smartphone per unit time.
[0063] Step 102: Based on the historical fault type data of smartphones, train ARMA time series analysis, using the prior probability of fault type of each smartphone per unit time as input and the fault probability of fault type of each smartphone per unit time as output.
[0064] Based on the fault probability of each smartphone per unit time for each fault type, the fault dependency relationship of each smartphone per unit time for each fault type is calculated, and the co-occurrence probability of each smartphone per unit time for each fault type is determined, as follows:
[0065]
[0066] in, For the i-th smartphone, the j-th fault type per unit time is compared with the j-th fault type. The co-occurrence probability of each fault type. For the i-th smartphone, the j-th fault type per unit time is compared with the j-th fault type. The number of times each fault type occurs simultaneously Let be the total number of times the j-th fault type occurs per unit time for the i-th smartphone. For the i-th smartphone, the unit time is the... The total number of times each fault type occurred;
[0067] By utilizing the co-occurrence probability of fault types for each smartphone per unit time, the fault influence strength value of fault types for each smartphone per unit time is calculated, and the causal association probability of fault types for each smartphone per unit time is determined, as follows:
[0068]
[0069] Among them, For the i-th smartphone, the j-th fault type per unit time is... The causal correlation probability of each fault type For i smartphones, the unit time of the first Under the condition of fault type, the first The conditional probability of each fault type occurring For i smartphones, the number of times the i-th event does not occur within a unit of time is... Under the condition of fault type, the first The conditional probability of each fault type occurring;
[0070] Based on the causal association probability of fault types for each smartphone per unit time, the fault interaction intensity of fault types for each smartphone per unit time is calculated, and a fault type association matrix for each smartphone per unit time is established, as follows:
[0071]
[0072] in, For the i-th smartphone, the j-th fault type is randomly selected per unit time and the... Interaction strength values for each fault type, Let be the marginal probability of the j-th fault type randomly occurring per unit time for the i-th smartphone. Let be the marginal probability of the j-th fault type randomly occurring per unit time for the i-th smartphone. For the i-th smartphone, the j-th fault type is randomly selected per unit time and the... The joint probability of multiple fault types occurring simultaneously A value of 0 indicates that a fault has occurred, and a value of 1 indicates that a fault has not occurred.
[0073] Based on the fault type association matrix of each smartphone per unit time, a fault association network graph of known smartphones is established, with fault type as the central node, fault dependency relationship of fault type of each smartphone per unit time as the edge, and fault interaction strength of fault type as the edge weight.
[0074] When using it, refer to steps 101 to 102:
[0075] As a further development, by integrating debug logs from the smartphone operating system and user feedback data, and dividing time windows based on kernel versions, this study utilizes LSTM networks to model the temporal dynamics of fault probability. Combined with co-occurrence analysis and causal inference to calculate fault interaction strength, a dynamically weighted fault association network graph is ultimately constructed. The technical principle lies in combining temporal modeling, probabilistic graphical models, and complex network theory. Through hierarchical data preprocessing → probability calculation → association analysis → network construction, multi-dimensional quantification of fault relationships is achieved. The beneficial effects are: 1. The kernel version-aligned time window division improves the spatiotemporal consistency of fault data; 2. The LSTM network enhances the temporal generalization ability of fault probability prediction; 3. The fusion of causal association and co-occurrence analysis simultaneously captures statistical correlation and potential causality; 4. The dynamically weighted network representation supports accurate root cause analysis and fault propagation prediction, providing decision support for smartphone fault types.
[0076] Step 2: Based on edge devices, acquire video data describing the smartphone malfunction of the target elderly person, and divide it into video stream and audio stream to obtain video-audio stream data describing the smartphone malfunction of the target elderly person.
[0077] Step 3: Verify the similarity score between the video vector and audio stream vector describing the fault in the smartphone. Determine whether the similarity score is within the consistency threshold range. If yes, the fault description is considered normal; otherwise, the fault description is considered abnormal.
[0078] Step three includes the following:
[0079] Step 301: Based on the smartphone fault description video-audio stream data of the target elderly person, extract frames according to the sampling rate of the smartphone fault description video stream data to obtain smartphone fault description video frame data;
[0080] The size of the video frame data describing the fault of the smartphone is pre-adjusted, and the pixels of each video frame describing the fault of the smartphone are standardized. The data are then fed into a CNN convolutional neural network to extract the frame-level feature vector of the smartphone fault description of the target elderly person, and a frame-level feature matrix of the smartphone fault description of the target elderly person is constructed.
[0081] Based on the frame-level feature matrix of the smartphone fault description video of the target elderly, the mean of the corresponding smartphone fault description video frame-level feature vector is extracted according to the time attribute of each frame and average pooling is performed to obtain the global feature vector of the smartphone fault description video of the target elderly.
[0082] Step 302: Based on the video-audio stream data describing the fault of the target elderly person's smartphone, divide the audio stream data describing the fault of the smartphone into segments according to the unit time to obtain the audio segment data describing the fault of the smartphone.
[0083] The audio segment data describing the fault based on the smartphone will be preprocessed, and the time attribute of the audio segment data describing the fault based on the smartphone will be used as a time window. Frame segmentation-frame shifting-windowing will be performed on the audio segment data describing the fault based on the smartphone within the time window to obtain the short-time signal of the audio describing the fault based on the smartphone.
[0084] Using short-time Fourier transform, the spectrum of each frame in the short-time signal of the fault description audio frame of the smartphone is calculated to obtain the linear spectrum of the fault description audio of the smartphone.
[0085] Based on the Mel filter, the linear spectrogram of the fault description audio of the smartphone is converted into Mel frequency, and the Mel spectrogram of the fault description audio of the smartphone is generated.
[0086] The size of the Mel spectrogram of the smartphone fault description audio is adjusted and substituted into VGGish deep learning. Mel frequency feature vectors of the smartphone fault description audio frames are extracted according to each time window to construct the time-feature matrix of the smartphone fault description audio for the target elderly.
[0087] Based on the self-attention mechanism of VGGish deep learning, a weight is assigned to the feature vector of each frame in the time-feature matrix of the smartphone fault description audio of the target elderly person. The global feature vector of the smartphone fault description audio of the target elderly person is calculated by weighted summation.
[0088] Step 303: Based on cosine similarity, calculate the similarity score between the global feature vector of the smartphone describing the fault in the target elderly person's video and the global feature vector of the smartphone describing the fault in the target elderly person's audio, as follows:
[0089]
[0090] in, A similarity score is calculated between the global feature vector of the video describing the fault in the smartphone of the target elderly person and the global feature vector of the audio describing the fault in the smartphone of the target elderly person. Describe the global feature vector of faulty videos on smartphones targeting elderly users. Describe the global feature vector of fault audio in smartphones targeting elderly users;
[0091] Determine whether the similarity score between the global feature vector of the smartphone describing the fault video and the global feature vector of the smartphone describing the fault audio of the target elderly person is within the consistency threshold range. If yes, the fault description is considered normal; otherwise, the fault description is considered abnormal.
[0092] When using this method, refer to steps 301 to 303:
[0093] As a further step, fault description video data from elderly users' smartphones was collected via edge devices, and the video and audio streams were separated for multimodal analysis. A CNN convolutional neural network was used to extract frame-level visual features, and average pooling was used to generate a global feature vector. For the audio stream, short-time Fourier transform and Mel-spectrum analysis were used to extract acoustic features, and a VGGish deep learning model combined with a self-attention mechanism was used to generate a global audio feature vector. The cosine similarity between the video and audio global feature vectors was calculated to determine their consistency, thereby assessing the reliability of the fault description. The benefits are: 1. Improved accuracy in fault description by reducing misjudgments through multimodal fusion; 2. Optimized edge computing efficiency by reducing computational load through frame segmentation and pooling; 3. Enabled automated real-time detection, improving the response speed of smartphone assistance services.
[0094] Step 4: If the fault description is abnormal, locate the smartphone fault description video vector-audio stream vector whose similarity score is not in the consistency threshold range, generate an interactive clarification task, and update the smartphone fault description video-audio stream data of the target elderly person.
[0095] Step four includes the following:
[0096] Step 401: Mark the global feature vector of the smartphone description video and the global feature vector of the smartphone description audio of the target elderly person whose similarity score is not in the consistency threshold range. This is denoted as the abnormal global feature vector of the smartphone description video-audio of the target elderly person.
[0097] Based on SVM (Support Vector Machine), the input is the global feature anomaly vector of the smartphone fault description video-audio of the target elderly person, and the output is the fault description category of the smartphone of the target elderly person.
[0098] 402. Substitute the smartphone description fault classification categories of the target elderly into the NLP natural language processing model, perform dictionary mapping according to the smartphone description fault classification categories, determine the question template classification corresponding to the smartphone description fault classification categories of the target elderly, and generate an interactive clarification task.
[0099] When using this, refer to sections 401 to 402:
[0100] As a further development, SVM and NLP in the above scheme are both lightweight model sub-processes embedded in the Transformer architecture. Since the global feature anomaly vector data ontology of the smartphone description fault video-audio of the target elderly people corresponding to the classification and recognition task is processed by the model training in the previous step and then used as the output, the fine granularity of the data can meet the classification task of the model in this step.
[0101] Secondly, for clarification tasks, in the case of instances, anomaly vectors are first analyzed using an SVM classifier to identify fault categories (such as lip-syncing mismatch, background noise, etc.), and then targeted clarification questions are generated using NLP template mapping. For example, if the classification is "audio anomaly - speaking too fast", the system automatically calls the template "please slow down and re-describe the problem".
[0102] Step 5: Based on the video-audio stream data describing the smartphone malfunctions of the target elderly, establish a multimodal model for smartphone malfunction identification and generate a smartphone malfunction localization-confidence matrix for the target elderly.
[0103] Step five includes the following:
[0104] Step 501: Based on the Transformer architecture, take the global feature vector of the smartphone describing the fault video and the global feature vector of the smartphone describing the fault audio of the target elderly as the output, and perform linear projection on the global feature vector of the smartphone describing the fault video and the global feature vector of the smartphone describing the fault audio to obtain the global feature space vector of the smartphone describing the fault video and the global feature space vector of the smartphone describing the fault audio.
[0105] The cross-modal attention weights of the global feature space vector of the video describing the fault of the intelligent machine and the global feature space vector of the audio describing the fault of the intelligent machine are used to fuse the features to obtain the global fused feature vector of the video-audio describing the fault of the intelligent machine.
[0106] Step 502: Based on the global fusion feature vector of video-audio describing the fault of the smartphone, input it into the MLP multilayer perceptron classifier, and output the probability distribution of the smartphone description fault category of the target elderly person;
[0107] Based on the video frame-level feature matrix describing the smartphone fault of the target elderly and the audio time-feature matrix describing the smartphone fault of the target elderly, the data is input into the spatiotemporal localization classifier. Using a temporal convolutional network and attention mechanism, the temporal attention weights corresponding to each video frame-level feature describing the smartphone fault and the audio time-feature describing the smartphone fault are determined, and the smartphone fault localization heatmap of the target elderly is calculated.
[0108] Based on the probability distribution of smartphone fault description categories and the smartphone fault location heatmap of the target elderly, the confidence value corresponding to each smartphone fault description category per unit time is calculated, and a smartphone fault location-confidence matrix of the target elderly is established, as follows:
[0109]
[0110] in, Confidence matrix for smartphone fault localization targeting elderly users. Given a smartphone, describe the probability distribution of the c-th fault type when describing the global fusion feature vector of fault video-audio. For the first Heatmap of fault location for smartphones per unit time. The weight of the c-th fault type;
[0111] Step 6: Based on the smartphone fault location-confidence matrix of the target elderly person and the fault association network map of known smartphones, perform association matching to determine the smartphone fault type of the target elderly person and generate an auxiliary processing plan;
[0112] Step six includes the following:
[0113] Step 601: Based on the smartphone fault location-confidence matrix of the target elderly, calculate the global confidence score for each fault type;
[0114] Based on the global confidence score of each fault type, the positively correlated sub-fault association network graphs in the known fault association network graphs of smartphones are selected and denoted as the positively correlated sub-fault association network graphs of smartphone faults for the target elderly.
[0115] Based on the positive correlation sub-fault association network graph of smartphone faults among the target elderly, the fault type category corresponding to the correlation sub-fault association network graph is determined, and then substituted into the NLP natural language processing model to generate an auxiliary processing solution.
[0116] Reference Figure 2 As shown, a fault identification and assistance system for smartphones used by the elderly is described.
[0117] Knowledge graph construction module, data acquisition module, data rationality verification module, data update module, fault task identification module, and fault task processing module.
[0118] The knowledge graph construction module is used to calculate the prior probability of historical fault type data for each smartphone based on the official debug logs of the operating system of known smartphones and user feedback communities, and to establish a fault association network graph of known smartphones.
[0119] The data acquisition module is used to acquire video data describing the fault of the target elderly person's smartphone based on the edge device, and divide it according to the video stream and audio stream to obtain the video-audio stream data describing the fault of the target elderly person's smartphone.
[0120] The data rationality verification module is wirelessly connected to the data acquisition module. The data rationality verification module is used to verify the similarity score between the video vector and audio stream vector describing the fault by the smartphone, and to determine whether the similarity score is within the consistency threshold range. If it is, the fault description is determined to be normal; otherwise, the fault description is determined to be abnormal.
[0121] The data update module is electrically connected to the data rationality verification module and the data acquisition module. The data update module is used to locate the smartphone description fault video vector-audio stream vector that is not in the consistency threshold range if the fault description is abnormal, generate an interactive clarification task, and update the smartphone description fault video-audio stream data of the target elderly person.
[0122] The fault task identification module is electrically connected to the data update module. The fault task identification module is used to establish a multimodal model for identifying faults in the smartphone based on the video-audio stream data describing the faults in the target elderly person's smartphone, and generate a smartphone fault location-confidence matrix for the target elderly person.
[0123] The fault task processing module is electrically connected to the fault task identification module. The fault task processing module is used to perform correlation matching between the fault location-confidence matrix of the target elderly person's smart phone and the fault association network map of known smart phones to determine the fault type of the target elderly person's smart phone and generate an auxiliary processing plan.
[0124] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. A method for assisting in fault identification and processing in smartphones for the elderly, characterized in that, include: S1. Based on the official debug logs and user feedback communities of the known smartphone operating system, calculate the prior probability of historical fault type data for each smartphone and establish a fault association network graph of the known smartphones. S2. Based on edge devices, acquire video data describing the fault of the target elderly person's smartphone, and divide it into video stream and audio stream to obtain video-audio stream data describing the fault of the target elderly person's smartphone; S3. Verify the similarity score between the video vector and audio stream vector describing the fault in the smartphone. Determine whether the similarity score is within the consistency threshold range. If yes, the fault description is normal; otherwise, the fault description is abnormal. S4. If the fault description is abnormal, locate the smartphone fault description video vector-audio stream vector whose similarity score is not in the consistency threshold range, generate an interactive clarification task, and update the smartphone fault description video-audio stream data of the target elderly person. S5. Based on the video-audio stream data describing the smartphone malfunctions of the target elderly, establish a multimodal model for smartphone malfunction identification and generate a smartphone malfunction localization-confidence matrix for the target elderly. S6. Based on the smartphone fault location-confidence matrix of the target elderly and the fault association network map of known smartphones, perform association matching to determine the smartphone fault type of the target elderly and generate an auxiliary processing plan. Wherein, S5 includes: Based on the Transformer architecture, the global feature vectors of the video describing the fault of the target elderly person's smartphone and the global feature vectors of the audio describing the fault of the target elderly person's smartphone are used as outputs. Linear projection is performed on the global feature vectors of the video describing the fault of the smartphone and the global feature vectors of the audio describing the fault of the smartphone to obtain the global feature space vectors of the video describing the fault of the smartphone and the global feature space vectors of the audio describing the fault of the smartphone. The cross-modal attention weights of the global feature space vector of the video describing the fault of the intelligent machine and the global feature space vector of the audio describing the fault of the intelligent machine are used to fuse the features to obtain the global fused feature vector of the video-audio describing the fault of the intelligent machine. Based on the global fusion feature vector of video-audio describing faults in smartphones, the input is into the MLP multilayer perceptron classifier, and the output is the probability distribution of smartphone description fault categories of the target elderly person. Based on the video frame-level feature matrix describing the smartphone fault of the target elderly and the audio time-feature matrix describing the smartphone fault of the target elderly, the data is input into the spatiotemporal localization classifier. Using a temporal convolutional network and attention mechanism, the temporal attention weights corresponding to each video frame-level feature describing the smartphone fault and the audio time-feature describing the smartphone fault are determined, and the smartphone fault localization heatmap of the target elderly is calculated. Based on the probability distribution of smartphone fault description categories and the smartphone fault location heatmap of the target elderly, the confidence value corresponding to each smartphone fault description category per unit time is calculated, and a smartphone fault location-confidence matrix of the target elderly is established.
2. The method for assisting in fault identification and processing of smartphones for the elderly according to claim 1, characterized in that, S1 includes: Based on the official debug logs of the operating system of known smartphones and user feedback communities, historical fault type data of each smartphone was obtained and data preprocessed. Based on the kernel version of each smartphone's system, the historical fault type data is divided. The operating time of the smartphone's system kernel version is used as the observation window, and the absolute frequency of each fault type in the historical fault type data of each smartphone is used as the observation object to obtain the time series data of the frequency characteristics of historical fault types of each smartphone per unit time. Collect time-series data on the frequency characteristics of historical fault types for each smartphone per unit time, determine the total number of historical fault types for each smartphone, and calculate the prior probability of each fault type for each smartphone per unit time.
3. The method for assisting in fault identification and processing of smartphones for the elderly according to claim 2, characterized in that, S1 further includes: Based on historical fault type data of smartphones, ARMA time series analysis is trained, with the prior probability of fault type of each smartphone per unit time as input and the fault probability of fault type of each smartphone per unit time as output. Based on the fault probability of each smartphone per unit time for each fault type, calculate the fault dependency relationship of each smartphone per unit time for each fault type, and determine the fault co-occurrence probability of each smartphone per unit time for each fault type. By utilizing the co-occurrence probability of fault types for each smartphone per unit time, the fault influence strength value of fault types for each smartphone per unit time is calculated, and the causal association probability of fault types for each smartphone per unit time is determined. Based on the causal association probability of fault types for each smartphone per unit time, calculate the fault interaction intensity of fault types for each smartphone per unit time, and establish the fault type association matrix for each smartphone per unit time. Based on the fault type association matrix of each smartphone per unit time, a fault association network graph of known smartphones is established, with fault type as the central node, fault dependency relationship of fault type of each smartphone per unit time as the edge, and fault interaction strength of fault type as the edge weight.
4. The method for assisting in fault identification and processing of smartphones for the elderly according to claim 1, characterized in that, S3 includes: Based on the smartphone fault description video-audio stream data of the target elderly, frame extraction is performed according to the sampling rate of the smartphone fault description video stream data to obtain smartphone fault description video frame data. The size of the video frame data describing the fault of the smartphone is pre-adjusted, and the pixels of each video frame describing the fault of the smartphone are standardized. The data are then fed into a CNN convolutional neural network to extract the frame-level feature vector of the smartphone fault description of the target elderly person, and a frame-level feature matrix of the smartphone fault description of the target elderly person is constructed. Based on the frame-level feature matrix of the smartphone fault description video of the target elderly, the mean of the corresponding smartphone fault description video frame-level feature vector is extracted according to the time attribute of each frame and average pooling is performed to obtain the global feature vector of the smartphone fault description video of the target elderly.
5. The method for assisting in fault identification and processing of smartphones for the elderly according to claim 4, characterized in that, S3 further includes: Based on the video-audio stream data describing the faults of the target elderly's smartphones, the audio stream data describing the faults of the smartphones is divided into segments according to unit time to obtain the audio segment data describing the faults of the smartphones; The audio segment data describing the fault based on the smartphone will be preprocessed, and the time attribute of the audio segment data describing the fault based on the smartphone will be used as a time window. Frame segmentation-frame shifting-windowing will be performed on the audio segment data describing the fault based on the smartphone within the time window to obtain the short-time signal of the audio describing the fault based on the smartphone. Using short-time Fourier transform, the spectrum of each frame in the short-time signal of the fault description audio frame of the smartphone is calculated to obtain the linear spectrum of the fault description audio of the smartphone. Based on the Mel filter, the linear spectrogram of the fault description audio of the smartphone is converted into Mel frequency, and the Mel spectrogram of the fault description audio of the smartphone is generated. The size of the Mel spectrogram of the smartphone fault description audio is adjusted and substituted into VGGish deep learning. Mel frequency feature vectors of the smartphone fault description audio frames are extracted according to each time window to construct the time-feature matrix of the smartphone fault description audio for the target elderly. Based on the self-attention mechanism of VGGish deep learning, a weight is assigned to the feature vector of each frame in the time-feature matrix of the smartphone fault description audio of the target elderly person. The global feature vector of the smartphone fault description audio of the target elderly person is calculated by weighted summation.
6. The method for assisting in fault identification and processing of smartphones for the elderly according to claim 5, characterized in that, S3 further includes: Based on cosine similarity, a similarity score is calculated between the global feature vector of the smartphone describing the fault in the target elderly person's video and the global feature vector of the smartphone describing the fault in the target elderly person's audio. Determine whether the similarity score between the global feature vector of the smartphone describing the fault in the target elderly person's video and the global feature vector of the smartphone describing the fault in the target elderly person's audio is within the consistency threshold range. If yes, the fault description is considered normal; otherwise, the fault description is considered abnormal.
7. The method for assisting in fault identification and processing of smartphones for the elderly according to claim 6, characterized in that, S4 includes: The global feature vectors of the smartphone description video and audio stream corresponding to the smartphone description video vectors and audio stream vectors of the target elderly person whose similarity scores are not in the consistency threshold range are denoted as the global feature vectors of the smartphone description video and audio stream vectors of the target elderly person. Based on SVM (Support Vector Machine), the input is the global feature anomaly vector of the smartphone fault description video-audio of the target elderly person, and the output is the fault description category of the smartphone of the target elderly person. The NLP (Natural Language Processing) model inputs the smartphone description fault classification categories of the target elderly into the dictionary mapping, determines the corresponding question templates for the smartphone description fault classification categories of the target elderly, and generates an interactive clarification task.
8. The method for assisting in fault identification of smartphones for the elderly according to claim 1, characterized in that, S6 includes: Based on the smartphone fault location-confidence matrix for the target elderly, calculate the global confidence score for each fault type; Based on the global confidence score of each fault type, the positively correlated sub-fault association network graphs in the known fault association network graphs of smartphones are selected and denoted as the positively correlated sub-fault association network graphs of smartphone faults for the target elderly. Based on the positive correlation sub-fault association network graph of smartphone faults among the target elderly, the fault type category corresponding to the correlation sub-fault association network graph is determined, and then substituted into the NLP natural language processing model to generate an auxiliary processing solution.
9. A smartphone fault identification and auxiliary processing system for the elderly, characterized in that, include: The module includes a knowledge graph construction module, a data acquisition module, a data rationality verification module, a data update module, a fault task identification module, and a fault task processing module. The knowledge graph construction module is used to calculate the prior probability of historical fault type data for each smartphone based on the official debug logs of the operating system of known smartphones and user feedback communities, and to establish a fault association network graph of known smartphones. The data acquisition module is used to acquire video data describing the fault of the target elderly person's smartphone based on the edge device, and divide it according to the video stream and audio stream to obtain the video-audio stream data describing the fault of the target elderly person's smartphone. The data rationality verification module is wirelessly connected to the data acquisition module. The data rationality verification module is used to verify the similarity score between the video vector and audio stream vector describing the fault by the smartphone, and to determine whether the similarity score is within the consistency threshold range. If it is, the fault description is determined to be normal; otherwise, the fault description is determined to be abnormal. The data update module is electrically connected to the data rationality verification module and the data acquisition module. The data update module is used to locate the smartphone description fault video vector-audio stream vector that is not in the consistency threshold range if the fault description is abnormal, generate an interactive clarification task, and update the smartphone description fault video-audio stream data of the target elderly person. The fault task identification module is electrically connected to the data update module. The fault task identification module is used to establish a multimodal model for identifying faults in the smartphone based on the video-audio stream data describing the faults in the target elderly person's smartphone, and generate a smartphone fault location-confidence matrix for the target elderly person. The fault task processing module is electrically connected to the fault task identification module. The fault task processing module is used to perform correlation matching between the fault location-confidence matrix of the target elderly person's smart phone and the fault association network map of known smart phones to determine the fault type of the target elderly person's smart phone and generate an auxiliary processing plan. The fault task identification module performs the following steps: Based on the Transformer architecture, the global feature vectors of the video describing the fault of the target elderly person's smartphone and the global feature vectors of the audio describing the fault of the target elderly person's smartphone are used as outputs. Linear projection is performed on the global feature vectors of the video describing the fault of the smartphone and the global feature vectors of the audio describing the fault of the smartphone to obtain the global feature space vectors of the video describing the fault of the smartphone and the global feature space vectors of the audio describing the fault of the smartphone. The cross-modal attention weights of the global feature space vector of the video describing the fault of the intelligent machine and the global feature space vector of the audio describing the fault of the intelligent machine are used to fuse the features to obtain the global fused feature vector of the video-audio describing the fault of the intelligent machine. Based on the global fusion feature vector of video-audio describing faults in smartphones, the input is into the MLP multilayer perceptron classifier, and the output is the probability distribution of smartphone description fault categories of the target elderly person. Based on the video frame-level feature matrix describing the smartphone fault of the target elderly and the audio time-feature matrix describing the smartphone fault of the target elderly, the data is input into the spatiotemporal localization classifier. Using a temporal convolutional network and attention mechanism, the temporal attention weights corresponding to each video frame-level feature describing the smartphone fault and the audio time-feature describing the smartphone fault are determined, and the smartphone fault localization heatmap of the target elderly is calculated. Based on the probability distribution of smartphone fault description categories and the smartphone fault location heatmap of the target elderly, the confidence value corresponding to each smartphone fault description category per unit time is calculated, and a smartphone fault location-confidence matrix of the target elderly is established.