Mental disease data processing method and device based on visual tracking

Abstract

The invention provides a mental disease data processing method and device based on visual tracking, and relates to the field of data processing. The method comprises the following steps: acquiring visual tracking data when a target patient executes a target task; according to the visual tracking data, in combination with audio data and physiological data when the target patient executes the target task, a target feature group is determined, and the target feature group comprises visual tracking features, audio features and physiological features; and inputting the target feature group into a preset analysis model to generate an analysis result which is used for indicating the mental disease condition of the target patient. By implementing the technical scheme provided by the invention, the accuracy of data processing is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a method and device for processing mental illness data based on visual tracking. Background Art

[0002] In the current medical field, significant progress has been made in the auxiliary processing methods for mental illnesses. However, these methods often rely on single-modal data processing techniques when dealing with relevant data. This limitation greatly restricts the accuracy and comprehensiveness of data processing, posing challenges to doctors in the process of diagnosing and treating mental illnesses.

[0003] Traditional data processing methods usually focus on data from a single source. For example, in the data processing process, only single-dimensional information such as the patient's clinical description, symptom self-rating scale, or physiological indicators is used for analysis. However, it is difficult to comprehensively reflect the complex condition of the patient only through data in one aspect, and the symptoms and manifestations of mental illnesses often change over time. Single-modal data processing is difficult to achieve real-time tracking and dynamic analysis of the patient's condition, resulting in relatively low accuracy of data processing.

[0004] Therefore, there is an urgent need for a method and device for processing mental illness data based on visual tracking to improve the accuracy of data processing. Summary of the Invention

[0005] This application provides a method and device for processing mental illness data based on visual tracking, which is convenient for improving the accuracy of data processing.

[0006] In the first aspect of this application, a method for processing mental illness data based on visual tracking is provided. The method includes: obtaining visual tracking data of a target patient when performing a target task; determining a target feature group according to the visual tracking data, in combination with audio data and physiological data of the target patient when performing the target task, where the target feature group includes visual tracking features, audio features, and physiological features; inputting the target feature group into a preset analysis model to generate an analysis result, where the analysis result is used to indicate the mental illness condition of the target patient.

[0007] By adopting the above technical solution, by simultaneously collecting the visual tracking data, audio data, and physiological data of the target patient when performing the target task, this process can obtain information on various aspects such as the patient's behavior, emotion, and cognitive state. The fusion of such multi-dimensional data can reflect the true situation of the patient more comprehensively compared to the analysis of single-modal data, thereby improving the accuracy of the assessment. Target feature groups such as eye movement features, expression and behavior recognition features, audio features, and physiological features are extracted from the multi-modal data. These features are independent of each other but interrelated, jointly constituting a complex map of the patient's state. Careful feature extraction helps to capture the subtle changes that may be overlooked in single-modal data, thereby enhancing the sensitivity and accuracy of the assessment. The target feature group is input into a preset analysis model to generate a personalized analysis result for the target patient. This customized analysis method can take into account the individual differences between patients and provide tailored diagnosis and treatment suggestions for each patient. Through the preset analysis model to automatically process and analyze the multi-modal data, the need for manual intervention is reduced, and the efficiency of medical services is improved. At the same time, the intelligent analysis model can continuously optimize its own performance based on big data and machine learning algorithms, further enhancing the accuracy and reliability of the analysis.

[0008] Optionally, the obtaining of the visual tracking data of the target patient when performing the target task specifically includes: when the target patient performs the target task, using an image acquisition device to perform real-time image acquisition on the target patient to obtain facial image data and limb image data; setting a tracking task for the facial image data and the limb image data to obtain eye movement image data and expression and behavior image data; performing preprocessing on the eye movement image data and the expression and behavior image data to obtain the visual tracking data, and the preprocessing includes tracking information calculation, tracking index selection, and tracking index comparison.

[0009] By adopting the above technical solutions, real-time image acquisition of the target patient is carried out through an image acquisition device, which can capture the immediate reactions of the patient during the task execution, including subtle facial expression changes and limb movements. This real-time nature ensures the timeliness and accuracy of the data. At the same time, facial image data and limb image data are collected, not only paying attention to the patient's facial expressions, but also taking into account limb behaviors, which provides more comprehensive data support for subsequent analysis. The fusion of multi-source data helps to understand the patient's psychological and behavioral states more comprehensively. Tracking task settings are set for the collected facial and limb image data, clearly focusing on the changes in eye movements and expressions and behaviors. This targeted tracking can extract key information and provide key data for subsequent analysis. Preprocessing is performed on the eye movement image data and the expression and behavior image data, including tracking information calculation, tracking index selection, and tracking index comparison. This step ensures the accuracy and analyzability of the data. Through a refined processing process, features valuable for evaluating the mental illness condition are extracted. The tracking information calculation, tracking index selection, and tracking index comparison in the preprocessing step are processes of cleaning, screening, and standardizing the original image data. This helps to eliminate noise, reduce errors, and improve the accuracy and reliability of subsequent analysis.

[0010] Optionally, determining the target feature group by combining the visual tracking data with the audio data and physiological data of the target patient during the execution of the target task specifically includes: receiving the voice data of the target patient during the execution of the target task sent by a microphone array device; receiving the biosignal data of the target patient during the execution of the target task sent by a wearable physiological monitoring device; performing data processing on the voice data and the biosignal data to obtain the audio data and the physiological data, and the data processing includes denoising, filtering, and normalization processing.

[0011] By adopting the above technical solutions, through the combination of visual tracking data, audio data, and physiological data, this process realizes the comprehensive fusion of multi-modal data. This fusion can capture information presented by the patient from multiple dimensions during task execution, such as visual attention, language communication, emotional state, and physiological reactions, thus providing a more comprehensive and in-depth portrait of the patient's state. Performing denoising, filtering, and normalization processing on the received voice data and biosignal data is a key step to ensure data quality. Denoising and filtering can reduce the influence of environmental noise and device interference on the data, while normalization processing enables data from different sources to be compared and analyzed on the same scale, improving the consistency and comparability of the data. Operations such as denoising, filtering, and normalization in the data processing process help to eliminate noise and errors in the data and improve the accuracy and reliability of subsequent analysis. This is particularly important for the assessment of mental illnesses because any minor error may affect the accuracy of diagnosis and the effectiveness of treatment.

[0012] Optionally, based on the visual tracking data, in combination with the audio data and physiological data of the target patient when performing the target task, determining a target feature group specifically further includes: extracting features from the visual tracking data to obtain the visual tracking features, where the visual tracking features include eye movement trajectories, fixation point distributions, facial expression changes, and whole-body behavior patterns; extracting features from the audio data to obtain the audio features, where the audio features include intonation, speech rate, and volume; extracting features from the physiological data to obtain the physiological features, where the physiological features include heart rate variability and skin conductance; fusing the visual tracking features, the audio features, and the physiological features to obtain the target feature group.

[0013] By adopting the above technical solution, by separately extracting features from visual tracking data, audio data, and physiological data, multiple key features of the patient when performing tasks can be obtained. These features include eye movement trajectories, fixation point distributions, facial expression changes, whole-body behavior patterns; intonation, speech rate, volume; and heart rate variability and skin conductance. These features together constitute a comprehensive map of the patient's mental state and behavioral performance, and the specific and detailed features provide a rich information basis for subsequent analysis and diagnosis. Fusing the visual tracking features, audio features, and physiological features to form a target feature group, this process realizes the comprehensive fusion of multi-modal data. Different modal data provides information in different aspects, is complementary to each other, and together constitutes a comprehensive description of the patient's state. The fusion of multi-modal data can reduce the information one-sidedness and errors that may be brought by single-modal data, thereby improving the accuracy and reliability of the evaluation.

[0014] Optionally, inputting the target feature group into a preset analysis model to generate an analysis result specifically includes: generating a comprehensive feature vector according to the target feature group; using the preset analysis model to process the comprehensive feature vector to obtain the analysis result, where the preset analysis model is a combined model of a convolutional neural network and a recurrent neural network, and the analysis result includes the type of potential mental illness and the probability corresponding to the type of potential mental illness.

[0015] By adopting the above technical solution, through generating a comprehensive feature vector, the features from different modalities are effectively integrated. This integration method not only retains the unique information of each modality feature, but also realizes the unified representation of information in the form of a vector, providing convenience for subsequent analysis. The generation of the comprehensive feature vector is usually accompanied by dimensionality reduction processing of the features, which helps to reduce the computational complexity and improve the processing efficiency. At the same time, the dimensionality-reduced feature vector can better highlight the key information, contributing to improving the accuracy of the analysis. Using a combined model of a convolutional neural network and a recurrent neural network as the preset analysis model is based on the powerful capabilities of these two networks in image processing and sequence analysis. The convolutional neural network is good at capturing spatial features, such as facial expressions and eye movement trajectories; while the recurrent neural network is good at processing time series data, such as speech and biological signals. This combination can make full use of the advantages of different networks to improve the accuracy and generalization ability of the analysis model. The model has advantages such as automatically extracting features, non-linear modeling, and powerful data processing capabilities. By training a large amount of data, the model can learn complex feature representations and decision boundaries, so as to more accurately identify potential mental illness types.

[0016] Optionally, before inputting the target feature group into the preset analysis model to generate an analysis result, train the preset analysis model; the training of the preset analysis model specifically includes: obtaining a plurality of original example features and the original example analysis result features corresponding to each of the original example features; fusing at least two of the original example features to obtain a fused example feature, and determining the fused example analysis result feature corresponding to the fused example feature; training a pre-established initial model based on the original example features, the original example analysis result features, the fused example feature, and the fused example analysis result feature to obtain the preset analysis model.

[0017] By adopting the above technical solution, by obtaining multiple original example features and fusing at least two original example features to obtain a fused example feature, this process greatly enriches the diversity and complexity of the training data. The fusion of such multi-source data enables the model to learn more comprehensive and detailed feature representations, thereby improving the generalization ability and accuracy of the model. The diverse training data helps the model learn the feature variation rules in different situations, reduces the risk of overfitting, and thus enhances the robustness and stability of the model. The fused example feature is obtained by fusing multiple original example features, and this fusion can capture the complementary information between different features. Compared with a single feature, the fused example feature usually has higher recognition accuracy and reliability. This is because the fused feature can comprehensively consider information from multiple aspects and reduce the errors and uncertainties that may be brought by a single feature. By providing corresponding analysis result features for each original example feature and fused example feature, supervised learning of the model is achieved. This learning method can ensure that the model continuously approaches the mapping relationship in the real world during training, thereby improving the prediction accuracy and reliability of the model.

[0018] Optionally, the method further includes: determining the mental illness onset probability corresponding to the target patient according to the analysis result; if it is determined that the mental illness onset probability is greater than a preset threshold, generating an analysis report.

[0019] By adopting the above technical solution, determining the mental illness onset probability corresponding to the target patient through the analysis result provides a quantitative basis for risk assessment. Such an analysis report of quantitative assessment is more objective and accurate than traditional qualitative assessment, which helps doctors, general practitioners, and family members better understand the mental health status of the patient. When the mental illness onset probability is greater than the preset threshold, it means that the patient may face a relatively high onset risk.

[0020] In a second aspect of the present application, there is provided a mental illness data processing device based on visual tracking. The mental illness data processing device includes an acquisition module and a processing module. Among them, the acquisition module is used to acquire visual tracking data of a target patient when performing a target task; the processing module is used to determine a target feature group according to the visual tracking data in combination with the audio data and physiological data of the target patient when performing the target task. The target feature group includes visual tracking features, audio features, and physiological features; the processing module is further used to input the target feature group into a preset analysis model to generate an analysis result, and the analysis result is used to indicate the mental illness status of the target patient.

[0021] In a third aspect of the present application, an electronic device is provided. The electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. Both the user interface and the network interface are used for communicating with other devices. The processor is used to execute the instructions stored in the memory, so that the electronic device executes the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions, and when the instructions are executed, the method described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By simultaneously collecting visual tracking data, audio data, and physiological data of a target patient when performing a target task, this process can obtain information on various aspects such as the patient's behavior, emotion, and cognitive state. The fusion of such multi-dimensional data can reflect the patient's true situation more comprehensively compared to single-modal data analysis, thereby improving the accuracy of assessment. Target feature groups such as eye movement features, expression and behavior recognition features, audio features, and physiological features are extracted from the multi-modal data. These features are independent of each other but interrelated, jointly constituting a complex map of the patient's state. Careful feature extraction helps capture subtle changes that may be overlooked in single-modal data, thereby enhancing the sensitivity and accuracy of assessment. The target feature group is input into a preset analysis model to generate a personalized analysis result for the target patient. This customized analysis method can take into account individual differences among patients and provide tailored diagnosis and treatment suggestions for each patient. Through the preset analysis model to automatically process and analyze multi-modal data, the need for manual intervention is reduced, and the efficiency of medical services is improved. At the same time, the intelligent analysis model can continuously optimize its own performance based on big data and machine learning algorithms, further enhancing the accuracy and reliability of the analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic flowchart of a method for processing mental illness data based on visual tracking provided by an embodiment of the present application.

[0025] Figure 2 It is another schematic flowchart of a method for processing mental illness data based on visual tracking provided by an embodiment of the present application.

[0026] Figure 3 It is a schematic block diagram of a device for processing mental illness data based on visual tracking provided by an embodiment of the present application.

[0027] Figure 4A schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0028] Explanation of reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. Detailed implementation manners

[0029] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0030] In the description of the embodiments of the present application, words such as "for example" or "for illustration" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "for example" or "for illustration" in the embodiments of the present application should not be construed as being more preferred or more advantageous than other embodiments or design solutions. Rather, the use of words such as "for example" or "for illustration" is intended to present relevant concepts in a specific manner.

[0031] In the description of the embodiments of the present application, the meaning of the term "a plurality" refers to two or more. For example, a plurality of systems refers to two or more systems, and a plurality of screen terminals refers to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly indicating the technical features indicated. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0032] At the current forefront of medical technology, although significant progress has been made in the technology for the assisted treatment of mental illnesses, its data processing strategies are often limited by the framework of a single modality. This current situation significantly restricts the accuracy and integrity of data analysis, and further poses obstacles to the accurate diagnosis of mental illnesses and the formulation of personalized treatment strategies.

[0033] Traditionally, data processing mechanisms often focus on isolated data sources, such as separately analyzing single-dimensional information such as a patient's clinical narrative, the results of a symptom self-assessment questionnaire, or physiological monitoring data. However, the complexity of the mental health field cannot be comprehensively captured by a single dimension. A patient's mental state, symptom manifestations, and physiological responses often change intertwined and exhibit dynamics over time. Therefore, relying solely on single-modality data processing methods is difficult to achieve an in-depth understanding of a patient's overall condition, resulting in relatively low accuracy of data processing.

[0034] To solve the above technical problems, the present application provides a method for processing mental illness data based on visual tracking. Refer to Figure 1 , Figure 1 FIG. is a schematic flowchart of a method for processing mental illness data based on visual tracking provided by an embodiment of the present application. This data processing method is applied to a server and includes steps S110 to S130. The above steps are as follows: S110. Obtain visual tracking data of a target patient when performing a target task.

[0035] Specifically, in the embodiment of the present application, a server refers to one or more computer systems that can process requests, manage resources, and provide services to clients (such as computers, mobile phones, or other devices). In the medical or scientific research field, servers are often used to store, process, and analyze large amounts of data. A target patient refers to an individual selected as the object of research or treatment. These patients may suffer from a certain disease, participate in a clinical trial, or require specific rehabilitation training. The selection of target patients is usually based on certain criteria, such as age, gender, and disease severity. Performing a target task refers to a series of specific actions or tasks that a patient is required to complete. These tasks may be used to evaluate a patient's certain abilities (such as cognitive ability, motor ability), observe the patient's response to treatment, or be part of rehabilitation training. For example, the tasks may include tracking a moving target on the screen, reading and answering questions, performing specific limb movements, etc. When a patient is performing a target task, a series of data will be generated by their eye movements. These data can be captured and recorded by an eye movement tracking device (such as an eye tracker). Visual tracking data includes, but is not limited to, the position of the fixation point, fixation time, amplitude and direction of saccades (rapid eye movements), etc. These data are of great significance for understanding aspects such as the patient's attention allocation, cognitive processing process, and the impact of the disease on the visual system.

[0036] In a possible implementation manner, obtaining visual tracking data of a target patient when performing a target task specifically includes: when the target patient is performing the target task, using an image acquisition device to perform real-time image acquisition on the target patient to obtain facial image data and limb image data; setting a tracking task for the facial image data and the limb image data to obtain eye movement image data and expression and behavior image data; performing preprocessing on the eye movement image data and the expression and behavior image data to obtain visual tracking data, and the preprocessing includes tracking information calculation, tracking index selection, and tracking index comparison.

[0037] Specifically, a target patient is recorded in real time using an image acquisition device (such as a high-definition camera, an eye tracker, etc.). These devices can capture the patient's facial and limb movements, generating a continuous video stream or image sequence. The facial image data is mainly used to analyze eye movements, facial expressions, etc.; the limb image data is used to analyze the patient's limb movements and postures. In image processing software, specific task settings are made for the collected facial image data and limb image data. These settings are based on research or treatment goals. For example, a tracking algorithm is set to identify the pupil of the eye and track its movement, or to identify specific changes in facial expressions. Through these settings, the software can distinguish between eye movement image data (such as pupil position, gaze direction, etc.) and expression and behavior image data (such as smiling, frowning, gestures, etc.). Tracking information calculation: The eye movement image data and expression and behavior image data are deeply processed to calculate relevant tracking information, such as the speed and direction of eye movement, changes in the fixation point, etc.; as well as the intensity and frequency of expression changes, and the patterns of limb movements. Tracking index selection: According to the needs of research or treatment, the most relevant tracking indexes are selected for analysis. For example, when evaluating attention, the stability of eye fixation and the switching speed of the fixation point may be concerned; when evaluating emotional responses, the intensity and duration of facial expressions may be concerned. Tracking index comparison: The calculated tracking indexes are compared with preset standard or baseline data to evaluate whether the patient's performance is normal or has improved. This helps doctors or researchers understand the specific situation of the patient and formulate personalized treatment or rehabilitation plans.

[0038] In the embodiment of the present application, before actual data collection, calibration needs to be carried out first. The calibration method includes 5-point calibration or 9-point calibration. The 5-point calibration is to set fixation points at the upper left, lower left, upper right, lower right boundaries and the center of the screen. The user fixates on these 5 points respectively to calibrate whether the eye tracking is accurate, so as to obtain eye movement data; the 9-point calibration is to set fixation points at the upper left, middle, and right, middle left, middle, and right, and lower left, middle, and right of the screen. The user fixates on these 9 points respectively to calibrate whether the eye tracking is accurate, so as to obtain eye movement data.

[0039] Among them, the above-mentioned target tasks include smooth pursuit, free viewing, and fixation gazing. Smooth pursuit is an eye movement in which the eyes follow a smoothly moving object at a slow speed, which can reflect people's abilities in perceiving and utilizing motion information, predicting the motion direction of an object, etc., and is also an important part of measuring the visual attention mechanism. The patient performs fixation point tracking by moving a fixation point at different speeds (settable) and different trajectories (settable). Free viewing is the free viewing of given materials, which is presented for a relatively long time. The patient can saccade back and forth to different contents, which can reflect people's attention preferences and visual search habits. Fixation gazing requires the patient to maintain a stable gaze on a given target and ignore other interferences, which can reflect people's ability to control fixation and the ability to inhibit saccades.

[0040] S120. According to the visual tracking data, combined with the audio data and physiological data of the target patient when performing the target task, determine the target feature group, where the target feature group includes visual tracking features, audio features, and physiological features.

[0041] Specifically, as mentioned before, these data are obtained through image acquisition and tracking technologies, reflecting the visual behaviors of the patient's eye movement trajectories, fixation point changes, etc. during the task execution. Audio data: may include the patient's speech, intonation, speech rate, and conversation content with the environment or others. These data can be collected through devices such as microphones and are used to analyze the patient's language communication ability, emotional expression, etc. Physiological data: This type of data is usually collected through wearable devices (such as heart rate monitors, skin conductance meters, etc.), reflecting the patient's physiological responses, such as heart rate, blood pressure, skin conductivity, etc. These indicators can indirectly reflect the patient's emotional state, stress level, or physical fatigue degree. The server will use advanced algorithms and technologies to extract key features from the above three types of data. For example, extract visual tracking features such as the speed of eye movement and the stability of the fixation point from the visual tracking data; extract audio features such as the pitch, speech rate, and emotional color of the speech from the audio data; extract physiological features such as heart rate variability and skin conductance response from the physiological data. After extracting various features, the server will select a group of the most representative features as the target feature group according to the research or treatment objectives. These features should be able to comprehensively and accurately reflect the patient's comprehensive performance during the task execution, including but not limited to aspects such as attention level, emotional state, cognitive ability, and communication ability.

[0042] In a possible implementation, according to the visual tracking data, combining the audio data and physiological data of the target patient when performing the target task, the target feature group is determined, specifically including: receiving the speech data of the target patient when performing the target task sent by the microphone array device; receiving the biological signal data of the target patient when performing the target task sent by the wearable physiological monitoring device; performing data processing on the speech data and biological signal data to obtain audio data and physiological data, and the data processing includes denoising, filtering, and normalization processing.

[0043] Specifically, the speech data of the target patient when performing the target task is received through the microphone array device. The microphone array usually contains multiple microphones, which can capture sounds from different directions and improve the clarity and directivity of speech through signal processing techniques. This is particularly important for accurately capturing the patient's speech in a noisy environment. The biological signal data of the target patient when performing the target task is received through the wearable physiological monitoring device. These devices can monitor various physiological parameters such as heart rate, blood pressure, skin conductivity, etc., and these data are of great significance for evaluating the patient's emotional state, stress level, etc. Since there are various noise sources in the actual environment (such as background noise, device self-noise, etc.), the received raw data often contains noise components. The denoising process aims to reduce or eliminate these noises to improve the signal-to-noise ratio of the data and make subsequent analysis more accurate. Filtering is a signal processing technique used to remove unwanted frequency components from the signal or enhance specific frequency components. In the processing of audio data and physiological data, filtering can help remove high-frequency noise or low-frequency interference and retain the useful signal part for analysis. Since the data from different devices, different patients, or different time points may have differences in dimension or magnitude, the normalization process is to convert these data to the same scale or range for comparison and analysis. Normalization can ensure that data from different sources have the same weight in subsequent feature extraction and model training.

[0044] In a possible implementation, according to the visual tracking data, combining the audio data and physiological data of the target patient when performing the target task, the target feature group is determined, specifically further including: extracting features from the visual tracking data to obtain visual tracking features, and the visual tracking features include eye movement trajectories, fixation point distributions, facial expression changes, and whole-body behavior patterns; extracting features from the audio data to obtain audio features, and the audio features include intonation, speech rate, and volume; extracting features from the physiological data to obtain physiological features, and the physiological features include heart rate variability and skin conductance; fusing the visual tracking features, audio features, and physiological features to obtain the target feature group.

[0045] Specifically, by analyzing the position changes of the eyeballs in the image sequence, the movement trajectory of the eyeballs can be traced. This helps to understand the patient's attention distribution and visual search pattern. Calculating the dwell time and frequency of the eye fixation points on different regions or objects can reveal the patient's points of interest or attention focus. By identifying the movements and morphological changes of the facial muscles, the patient's emotional state (such as happy, sad, angry, etc.) and cognitive load can be analyzed. Although mainly focusing on visual tracking, the whole-body behavior patterns (such as gestures, body postures, etc.) can also be captured through the image data and used as supplementary features. The change in intonation can reflect the speaker's emotional state and intention. For example, a rising intonation may indicate a question or surprise, while a falling intonation may indicate a statement or affirmation. The speed of speech can reflect the speaker's level of tension, thinking activity, or emotional state. The change in volume can reveal the speaker's emotional intensity or communication willingness. Heart rate variability refers to the minute changes in the time intervals between heartbeats, which reflects the activity level of the autonomic nervous system and is closely related to emotional state, stress response, etc. Skin conductance (also known as skin electrical response or GSR) is the reciprocal of skin resistance, which reflects the changes in sweat gland activity and is commonly used to evaluate the level of emotional arousal or stress response. Feature fusion is the process of combining features from different data sources into a comprehensive feature set. In this example, the visual tracking features, audio features, and physiological features are fused into a target feature group. This fusion can be based on simple feature concatenation or more complex fusion methods (such as weighted fusion, feature selection, etc.) to make full use of the complementary information of various data sources.

[0046] S130. Input the target feature group into a preset analysis model to generate an analysis result, where the analysis result is used to indicate the mental illness condition of the target patient.

[0047] Specifically, the target feature group is a set of features extracted from the visual tracking data, audio data, and physiological data of the target patient, which jointly reflect the comprehensive performance of the patient when performing a specific task or in a specific state. The preset analysis model is a model that has been trained with a large amount of training data and can understand and identify the association between the patterns in the target feature group and specific mental illness conditions. This model is constructed based on a neural network combination. When the server inputs the target feature group into the preset analysis model, the model will comprehensively analyze these features according to the knowledge and rules it has learned and generate one or more analysis results. These results may be presented in the form of probabilities (such as the probability of suffering from a certain mental illness is XX%) or in a categorical form (such as diagnosed with XX mental illness). The main purpose of the analysis result is to indicate or prompt the mental illness condition of the target patient. This helps doctors or therapists to more accurately understand the patient's condition, thereby formulating more appropriate treatment plans or intervention measures.

[0048] For example, assume that a screening study on depression is underway, and the target patients are a group of individuals who may have depression. Visual tracking data, audio data, and physiological data of the target patients are collected through a visual tracking device, a microphone, and a wearable physiological monitoring device. Then, these data are processed to extract a target feature group, including features such as the fixation stability in the eye movement trajectory, the degree of low intonation in the voice data, and the decrease in heart rate variability in the physiological data. Secondly, the server has a preset analysis model trained with a large amount of data of depression patients. This model has learned how to identify signs of depression from similar feature groups. The server inputs the target feature group into the preset analysis model. After the model comprehensively analyzes these features, an analysis result is generated, such as "the probability that the patient has depression is 85%". This result indicates that the target patient is likely to have depression. Doctors or therapists can further evaluate the patient's condition based on this indication and conduct necessary diagnoses and treatments.

[0049] In a possible implementation manner, inputting the target feature group into the preset analysis model to generate an analysis result specifically includes: generating a comprehensive feature vector according to the target feature group; using the preset analysis model to process the comprehensive feature vector to obtain the analysis result, where the preset analysis model is a combined model of a convolutional neural network and a recurrent neural network, and the analysis result includes the type of potential mental illness and the probability corresponding to the type of potential mental illness.

[0050] Specifically, in this step, it is necessary to convert the target feature group extracted from the visual tracking data, audio data, and physiological data into a comprehensive feature vector. This vector is a high-dimensional numerical representation that contains information on all important features and can be directly processed by a machine learning model. The methods for generating the comprehensive feature vector may include steps such as feature concatenation, feature selection, and feature dimensionality reduction. Feature concatenation is the most straightforward method, that is, simply concatenating all features into a long vector; feature selection is to select the most useful features from the original features; feature dimensionality reduction is to map the high-dimensional features to a low-dimensional space through a certain transformation while trying to retain the original information as much as possible. Here, the preset analysis model is a composite model composed of a convolutional neural network (CNN) and a recurrent neural network (RNN). This combined model can capture both the spatial features of the data (such as textures and shapes in images) and the temporal features (such as temporal relationships in sequence data) simultaneously. CNN is usually used to process images or data with spatial structures, and it extracts local features of the data through structures such as convolutional layers and pooling layers.

[0051] In this scenario, CNN is used to process image features in visual tracking data (such as eye movement trajectories, facial expressions). RNN, on the other hand, is good at processing sequential data, such as audio signals or time-series physiological data. It can capture the temporal dependencies in the data, which is very useful for understanding continuously changing emotional states or physiological responses. Combining CNN and RNN can make full use of the advantages of both to comprehensively analyze multi-source and multi-dimensional data. After being processed by a preset analysis model, an analysis result will be generated. This result usually includes the types of potential mental illnesses and their corresponding probabilities. The probability value represents the confidence level of the model that the patient has the mental illness. It should be noted that the analysis result may not be single, but a list of multiple potential disease types and their probabilities. This is because the manifestations of mental illnesses are often complex and diverse, and there is a phenomenon of comorbidity (that is, suffering from multiple mental illnesses at the same time).

[0052] For example, assume that a screening study on the comorbidity of depression and anxiety is being conducted. Data of target patients are collected through visual tracking devices, microphones, and wearable physiological monitoring devices, and target feature groups are extracted. These features may include the fixation stability in the eye movement trajectory, the degree of sadness in facial expressions, the emotional state in speech, and heart rate variability, etc. All the extracted features are concatenated into a comprehensive feature vector. For example, if each feature is quantified as a specific numerical value, then these numerical values can be concatenated into a long vector. The comprehensive feature vector is input into a preset analysis model composed of CNN and RNN. The CNN part may be used to process facial expression images and extract features related to the emotional state; the RNN part is used to process audio signals and physiological data sequences to capture the temporal dependencies therein. After being processed by the model, an analysis result containing the types of potential mental illnesses and their probabilities is generated. For example, the model may output "Depression: 70%, Anxiety: 50%", which means that the model believes the probability of the patient having depression is 70% and the probability of having anxiety is 50%. This result helps doctors or therapists to more comprehensively understand the mental health status of the patient and formulate corresponding treatment plans.

[0053] In a possible implementation manner, before inputting the target feature group into the preset analysis model to generate an analysis result, the preset analysis model is trained; training the preset analysis model specifically includes: obtaining a plurality of original example features and the original example analysis result features corresponding to each of the original example features; fusing at least two original example features to obtain a fused example feature, and determining the fused example analysis result feature corresponding to the fused example feature; training a pre-established initial model based on the original example features, the original example analysis result features, the fused example features, and the fused example analysis result features to obtain the preset analysis model.

[0054] Specifically, first, a large amount of raw data needs to be collected, which contains various information about different mental illness patients (such as visual tracking data, audio data, physiological data, etc.). Then, features are extracted from these raw data to form raw example features. Each example feature corresponds to a specific patient or situation. For each raw example feature, there also needs to be a corresponding analysis result feature, which is usually given by professional doctors or researchers based on the patient's clinical manifestations, diagnosis results, etc. These analysis result features serve as the labels or target values for training the model. In some cases, to improve the generalization ability and accuracy of the model, multiple raw example features may be fused to obtain fused example features. The fusion method can be simple concatenation or more complex feature transformation. The fused example features aim to capture the potential associations and interactions between the original features. At the same time, it is also necessary to determine the fused example analysis result features corresponding to the fused example features. This is usually obtained by merging or transforming the raw example analysis result features through a certain rule or algorithm. Using the raw example features, raw example analysis result features, fused example features, and fused example analysis result features as training data, the pre-established initial model is trained. The initial model may be a simple neural network structure or a more complex model framework. During the training process, the model learns how to map the input features (whether raw example features or fused example features) to the corresponding analysis result features. By continuously adjusting the model parameters and optimizing the loss function, the performance of the model on the training set gets better and better. Finally, when the model reaches a certain performance index (such as accuracy, recall rate, etc.) on the training set, the training can be stopped, and the model at this time can be saved as the preset analysis model.

[0055] In one possible implementation, with reference to Figure 2 , Figure 2 FIG.

[0056] Specifically, after inputting the target feature group into the preset analysis model and generating the analysis result, the result usually includes the potential mental illness types and the corresponding onset probabilities. These probabilities are obtained based on the model's evaluation of the input features and reflect the confidence level of the model that the patient has a certain mental illness. Next, the server will determine the mental illness onset probability corresponding to the target patient according to these probability values, combined with the preset rules or logic. This step may involve comprehensively evaluating the probabilities of multiple potential disease types or only focusing on the probability of a specific disease. The server will compare the determined mental illness onset probability with a preset threshold. This threshold is set according to the actual situation (such as the allocation of medical resources, the severity of the disease, the personal situation of the patient, etc.) and is used to judge whether the patient's mental illness condition has reached the level that requires an alarm. If the determined mental illness onset probability is greater than the preset threshold, it indicates that the patient's mental illness condition may be relatively serious or requires urgent attention. At this time, the server will generate an analysis report containing alarm data. Once the analysis report is generated, the server can control the printing device to print out a paper report on-site.

[0057] The present application also provides a mental illness data processing device based on visual tracking. Refer to Figure 3 , Figure 3 which is a schematic diagram of the modules of a mental illness data processing device based on visual tracking provided by an embodiment of the present application. The mental illness data processing device is a server, and the server includes an acquisition module 31 and a processing module 32. Among them, the acquisition module 31 acquires the visual tracking data of the target patient when performing the target task; the processing module 32 determines the target feature group according to the visual tracking data, combined with the audio data and physiological data of the target patient when performing the target task. The target feature group includes visual tracking features, audio features, and physiological features; the processing module 32 inputs the target feature group into the preset analysis model to generate an analysis result, and the analysis result is used to indicate the mental illness condition of the target patient.

[0058] In a possible implementation manner, the acquisition module 31 acquires the visual tracking data of the target patient when performing the target task, specifically including: when the target patient performs the target task, the processing module 32 uses an image acquisition device to perform real-time image acquisition on the target patient to obtain facial image data and limb image data; the processing module 32 sets the tracking task for the facial image data and the limb image data to obtain eye movement image data and expression and behavior image data; the processing module 32 preprocesses the eye movement image data and the expression and behavior image data to obtain visual tracking data, and the preprocessing includes tracking information calculation, tracking index selection, and tracking index comparison.

[0059] In a possible implementation, the processing module 32 determines a target feature group according to the visual tracking data, in combination with the audio data and physiological data of the target patient when performing the target task. Specifically, it includes: the acquisition module 31 receives the speech data of the target patient when performing the target task sent by the microphone array device; the acquisition module 31 receives the biological signal data of the target patient when performing the target task sent by the wearable physiological monitoring device; the processing module 32 performs data processing on the speech data and biological signal data to obtain audio data and physiological data, and the data processing includes denoising, filtering, and normalization processing.

[0060] In a possible implementation, the processing module 32 determines a target feature group according to the visual tracking data, in combination with the audio data and physiological data of the target patient when performing the target task. Specifically, it further includes: the processing module 32 extracts features from the visual tracking data to obtain visual tracking features, and the visual tracking features include eye movement trajectories, fixation point distributions, facial expression changes, and whole-body behavior patterns; the processing module 32 extracts features from the audio data to obtain audio features, and the audio features include intonation, speech rate, and volume; the processing module 32 extracts features from the physiological data to obtain physiological features, and the physiological features include heart rate variability and skin conductance; the processing module 32 fuses the visual tracking features, audio features, and physiological features to obtain a target feature group.

[0061] In a possible implementation, the processing module 32 inputs the target feature group into a preset analysis model to generate an analysis result. Specifically, it includes: the processing module 32 generates a comprehensive feature vector according to the target feature group; the processing module 32 uses the preset analysis model to process the comprehensive feature vector to obtain an analysis result, and the preset analysis model is a combined model of a convolutional neural network and a recurrent neural network, and the analysis result includes the type of potential mental illness and the probability corresponding to the type of potential mental illness.

[0062] In a possible implementation, before the processing module 32 inputs the target feature group into the preset analysis model to generate an analysis result, it trains the preset analysis model; training the preset analysis model specifically includes: the acquisition module 31 acquires a plurality of original example features and the original example analysis result features corresponding to each original example feature; the processing module 32 fuses at least two original example features to obtain a fused example feature, and determines the fused example analysis result feature corresponding to the fused example feature; the processing module 32 trains a pre-established initial model based on the original example features, original example analysis result features, fused example features, and fused example analysis result features to obtain the preset analysis model.

[0063] In a possible implementation, the processing module 32 determines the onset probability of mental illness corresponding to the target patient according to the analysis result; if the processing module 32 determines that the onset probability of mental illness is greater than a preset threshold, it generates an analysis report.

[0064] It should be noted that when the device provided in the above embodiment realizes its functions, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be elaborated here.

[0065] This application also provides an electronic device. Refer to Figure 4 , Figure 4 which is a schematic structural diagram of an electronic device provided by an embodiment of this application. The electronic device may include: at least one processor 41, at least one network interface 44, a user interface 43, a memory 45, and at least one communication bus 42.

[0066] Among them, the communication bus 42 is used to realize the connection and communication between these components.

[0067] Among them, the user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may further include a standard wired interface and a wireless interface.

[0068] Among them, the network interface 44 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0069] Among them, the processor 41 may include one or more processing cores. The processor 41 connects various parts within the entire server using various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 45, and by calling the data stored in the memory 45, it performs various functions of the server and processes data. Optionally, the processor 41 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 41 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 41 and may be implemented separately by a single chip.

[0070] Among them, the memory 45 may include random access memory (RAM) and may also include read-only memory. Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 45 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store the data involved in the above-mentioned method embodiments. Optionally, the memory 45 may also be at least one storage device located far from the aforementioned processor 41. As Figure 4 shown, the memory 45, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a method of processing mental illness data based on visual tracking.

[0071] In Figure 4In the electronic device shown, the user interface 43 is mainly used to provide an interface for the user to input and obtain the data input by the user; while the processor 41 can be used to call an application program stored in the memory 45 for a method of processing mental illness data based on visual tracking. When executed by one or more processors, the electronic device is caused to execute the method as described in one or more of the above embodiments.

[0072] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0073] This application also provides a computer-readable storage medium storing instructions. When executed by one or more processors, the electronic device is caused to execute the method as described in one or more of the above embodiments.

[0074] In the above embodiments, the descriptions of the various embodiments each have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0075] In several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some service interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0076] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0077] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0078] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned memory includes: various media such as USB flash drives, mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0079] The above are only exemplary embodiments of the present disclosure, and the scope of the present disclosure cannot be limited thereby. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. After considering the specification and the disclosure of the practical truth, those skilled in the art will easily think of other implementation schemes of the present disclosure. This application aims to cover any variations, uses, or adaptive changes of the present disclosure, and these variations, uses, or adaptive changes follow the general principles of the present disclosure and include the common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for processing mental illness data based on visual tracking, characterized in that: The method comprises: Obtain visual tracking data of the target patient while performing the target task; Determine a target feature group according to the visual tracking data in combination with audio data and physiological data of the target patient when performing the target task, wherein the target feature group includes visual tracking features, audio features, and physiological features; The target feature group is input into a preset analysis model to generate an analysis result, wherein the analysis result is used to indicate the mental illness condition of the target patient.

2. The method for processing mental illness data based on visual tracking according to claim 1, characterized in that: The obtaining of visual tracking data of the target patient when performing the target task specifically includes: When the target patient performs the target task, an image acquisition device is used to acquire real-time images of the target patient to obtain facial image data and limb image data; Setting tracking tasks for the facial image data and the limb image data to obtain eye movement image data and expression and behavior image data; The eye movement image data and the expression and behavior image data are preprocessed to obtain the visual tracking data, wherein the preprocessing includes tracking information calculation, tracking index selection, and tracking index comparison.

3. The method for processing mental illness data based on visual tracking according to claim 1, characterized in that: Determining a target feature group based on the visual tracking data in combination with the audio data and physiological data of the target patient when performing the target task specifically includes: receiving voice data of the target patient when performing the target task sent by the microphone array device; Receiving biological signal data of the target patient when performing the target task sent by a wearable physiological monitoring device; The voice data and the biological signal data are processed to obtain the audio data and the physiological data, wherein the data processing includes denoising, filtering and normalization processing.

4. The method for processing mental illness data based on visual tracking according to claim 3 is characterized in that: Determining a target feature group based on the visual tracking data in combination with audio data and physiological data of the target patient when performing the target task specifically includes: Extracting features from the visual tracking data to obtain the visual tracking features, wherein the visual tracking features include eye movement trajectories, gaze point distribution, facial expression changes, and whole body behavior patterns; Extracting features from the audio data to obtain the audio features, where the audio features include intonation, speech speed, and volume; Extracting features from the physiological data to obtain the physiological features, wherein the physiological features include heart rate variability and skin conductance; The visual tracking feature, the audio feature and the physiological feature are fused to obtain the target feature group.

5. The method for processing mental illness data based on visual tracking according to claim 1, characterized in that: The step of inputting the target feature group into a preset analysis model to generate an analysis result specifically includes: generating a comprehensive feature vector according to the target feature group; The preset analysis model is used to process the comprehensive feature vector to obtain the analysis result, wherein the preset analysis model is a combination model of a convolutional neural network and a recurrent neural network, and the analysis result includes the probability of a potential mental illness type corresponding to the potential mental illness type.

6. The method for processing mental illness data based on visual tracking according to claim 1, characterized in that: Before inputting the target feature group into the preset analysis model to generate the analysis result, training the preset analysis model; the training of the preset analysis model specifically includes: Acquire a plurality of original example features and an original example analysis result feature corresponding to each of the original example features; Fusing at least two of the original example features to obtain a fused example feature, and determining a fused example analysis result feature corresponding to the fused example feature; The pre-established initial model is trained based on the original example features, the original example analysis result features, the fused example features and the fused example analysis result features to obtain the preset analysis model.

7. The method for processing mental illness data based on visual tracking according to claim 1, characterized in that: The method further comprises: Determine the probability of mental illness corresponding to the target patient according to the analysis results; If it is determined that the probability of the mental illness is greater than a preset threshold, an analysis report is generated.

8. A device for processing mental illness data based on visual tracking, characterized in that: The mental illness data processing device comprises an acquisition module (31) and a processing module (32), wherein: The acquisition module (31) is used to acquire visual tracking data of the target patient when performing the target task; The processing module (32) is used to determine a target feature group based on the visual tracking data in combination with audio data and physiological data of the target patient when performing the target task, wherein the target feature group includes visual tracking features, audio features and physiological features; The processing module (32) is further used to input the target feature group into a preset analysis model to generate an analysis result, wherein the analysis result is used to indicate the mental illness condition of the target patient.

9. An electronic device, characterized in that: The electronic device comprises a processor (41), a memory (45), a user interface (43) and a network interface (44), wherein the memory (45) is used to store instructions, the user interface (43) and the network interface (44) are both used to communicate with other devices, and the processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images