Psychological abnormality detection method and system based on deep learning

Abstract

The application discloses a kind of psychological abnormality detection method and system based on deep learning, it is related to psychological abnormality detection field, first to original audio-visual stream is user and situation separation, and user behavior feature and environmental situation feature are extracted respectively.Subsequently, these features are input into a personalized baseline model that is trained.The model can learn and understand the typical behavior pattern of a specific user in a specific situation, and then reconstruct the expected user behavior in the situation.By calculating the personalized consistency deviation between the actual user behavior and the reconstructed user behavior, the method can accurately quantify the deviation between individual behavior and the expected behavior in the current situation.This deviation analysis based on individual baseline and situation awareness effectively avoids misjudging situation-adaptive behavior as abnormal, thereby enabling more accurate capture of early and weak psychological abnormality signals, significantly improving the accuracy and practicality of detection.

Description

Technical Field

[0001] This application relates to the field of psychological anomaly detection, and more specifically, to a method and system for psychological anomaly detection based on deep learning. Background Technology

[0002] Early identification of mental health issues is crucial for individual health and social stability. With the rapid development of artificial intelligence, especially breakthroughs in deep learning for pattern recognition, new possibilities have emerged for automated, non-invasive detection of mental health problems. Deep learning-based methods can process complex audiovisual data, extracting subtle patterns that are difficult for humans to perceive, thus potentially enabling the accurate capture of weak early signals of mental health issues.

[0003] However, existing deep learning-based psychological anomaly detection schemes generally suffer from significant shortcomings, primarily in their insufficient understanding of context-behavior mismatches. Traditional deep learning models are often decontextualized in their design; they are trained as simple input-output mappers, for example, taking an image of a face as input and outputting its emotional state. These models lack a parallel information pathway to effectively process and integrate environmental information, resulting in their inability to fully consider the specific context in which the behavior occurs when understanding human behavior. Furthermore, most publicly available datasets used to train these models are inherently flawed, often consisting of cropped faces or isolated audio clips, stripping away all contextual information. This prevents the models from learning how to integrate context into their understanding of behavior from the outset. This one-sided feature extraction, focusing only on the user's own behavioral characteristics (such as facial expressions and tone of voice) while ignoring equally important environmental contextual features (such as the surrounding environment and background noise), makes it difficult for the models to distinguish between normal contextual adaptive behavior and genuine abnormal behavior. For example, a person who laughs loudly while watching a comedy might be misjudged as having abnormally high emotions by a decontextualized model. This greatly limits the accuracy and reliability of existing technologies in practical applications, making it difficult to bring products to market.

[0004] To overcome the limitations of existing technologies, an optimized deep learning-based scheme for detecting psychological anomalies is proposed. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this application provides a deep learning-based method for detecting psychological anomalies, comprising: User / context separation is performed on the acquired raw video and audio streams to obtain user behavior segments and environmental context segments; Two-stream context-behavior feature extraction is performed on user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors; The user behavior vector and context embedding vector are input into the trained personalized baseline model to obtain the reconstructed user behavior vector; Calculate the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector; Anomaly accumulation judgment and graded early warning are performed on the time series of personalized consistency deviation scores to obtain early warning signals.

[0006] This application also provides a deep learning-based psychological anomaly detection system, which includes: The user / context separation module is used to separate the acquired raw video stream and raw audio stream into user / context segments to obtain user behavior segments and environmental context segments; The dual-stream feature extraction module is used to extract dual-stream context-behavior features from user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors. The user behavior reconstruction module is used to input user behavior vectors and context embedding vectors into the trained personalized baseline model to obtain reconstructed user behavior vectors. The consistency deviation score calculation module is used to calculate the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector. The early warning signal generation module is used to perform anomaly accumulation judgment and graded early warning on the time series of personalized consistency deviation scores to obtain early warning signals.

[0007] Compared with existing technologies, this application provides a deep learning-based method and system for detecting psychological anomalies. By introducing context awareness and personalized baseline modeling, it solves the problems of context-behavior mismatch and the relativity of anomaly definitions in existing deep learning methods for detecting psychological anomalies. First, it separates the user from the context in the original audiovisual stream, extracting user behavior features and environmental context features respectively. Then, these features are input into a trained personalized baseline model. This model can learn and understand the typical behavioral patterns of a specific user in a specific context, and thus reconstruct the expected user behavior in that context. By calculating the personalized consistency deviation score between the actual user behavior and the reconstructed user behavior, this method can accurately quantify the degree of deviation between an individual's behavior and their expected behavior in the current context. This deviation analysis based on individual baselines and context awareness effectively avoids misjudging context-adaptive behaviors as anomalies, thereby more accurately capturing early, weak psychological anomaly signals and significantly improving the accuracy and practicality of detection. Attached Figure Description

[0008] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings.

[0009] Figure 1This is a flowchart of a deep learning-based psychological anomaly detection method according to an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of the data flow of a deep learning-based psychological anomaly detection method according to an embodiment of this application.

[0011] Figure 3 This is a flowchart of step S1 in the deep learning-based psychological anomaly detection method according to an embodiment of this application.

[0012] Figure 4 This is a flowchart of step S3 in the deep learning-based psychological anomaly detection method according to an embodiment of this application.

[0013] Figure 5 This is a block diagram of a deep learning-based psychological anomaly detection system according to an embodiment of this application. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. It should be understood that the drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] In view of the shortcomings in the above-mentioned technical fields, this application proposes a psychological anomaly detection method based on deep learning. Figure 1 This is a flowchart of a deep learning-based psychological anomaly detection method according to an embodiment of this application. Figure 2 This is a schematic diagram of the data flow in a deep learning-based psychological anomaly detection method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the deep learning-based psychological anomaly detection method according to an embodiment of this application includes: S1, performing user / context separation on the acquired original video stream and original audio stream to obtain user behavior segments and environmental context segments; S2, performing dual-stream context-behavior feature extraction on the user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors; S3, inputting the user behavior vectors and context embedding vectors into a trained personalized baseline model to obtain reconstructed user behavior vectors; S4, calculating the personalized consistency deviation score between the reconstructed user behavior vector and the original user behavior vector; S5, performing anomaly accumulation judgment and graded early warning on the time series of personalized consistency deviation scores to obtain early warning signals.

[0016] In step S1, the acquired raw video and audio streams are separated into user / context segments to obtain user behavior fragments and environmental context fragments. It should be understood that existing deep learning-based psychological anomaly detection methods, due to limitations in their model design, often treat human behavior as isolated events, lacking a deep understanding of the context in which the behavior occurs. Traditional models are typically trained as simple input-output mappers, for example, judging emotional states solely based on facial images, ignoring the environmental background of the behavior. This decontextualized approach causes the model to be unable to distinguish between normal adaptive behavior exhibited by an individual in a specific context and genuine psychological anomaly signals. For example, if a person laughs heartily while watching a comedy show, and the model fails to perceive the comedy show context, it may misjudge this as abnormal emotional elevation, resulting in a large number of false positives and severely impacting the accuracy and practicality of the detection. Therefore, in order to overcome this situation-behavior mismatch and achieve accurate capture of early, weak signals of psychological abnormalities, this application performs user / situation separation on the acquired raw video stream and raw audio stream. The aim is to effectively decouple the user's own behavioral information from the external environmental situational information from the source, laying the foundation for subsequent situation-behavior dual-stream feature extraction and personalized baseline modeling, so that the model can more comprehensively and accurately understand the deeper meaning of behavior.

[0017] Figure 3 This is a flowchart of step S1 in the deep learning-based psychological anomaly detection method according to an embodiment of this application. Figure 3 As shown, an exemplary implementation of step S1 of this application involves performing user / context separation on the acquired raw video stream and raw audio stream to obtain user behavior segments and environmental context segments, including: S11, inputting the raw video stream into a face detection module to obtain a face image sequence and an environmental background image sequence; S12, inputting the raw audio stream into a speech activity detection module to obtain user speech segments and environmental sound segments; S13, performing data alignment and encapsulation on the face image sequence and user speech segments to obtain user behavior segments; and S14, performing data alignment and encapsulation on the environmental background image sequence and environmental sound segments to obtain environmental context segments.

[0018] Specifically, step S1 of this application is performed as follows: The raw video stream refers to a sequence of continuous image frames acquired in real time or offline by a camera (e.g., a smart home camera, a computer built-in camera, etc.), transmitted at a rate of 25 or 30 frames per second, and each frame has a specific resolution, such as 1920×1080 pixels.

[0019] In step S11: The face detection module is a pre-trained deep learning model, such as a detector based on a convolutional neural network (CNN). This model contains multiple convolutional layers, pooling layers, and fully connected layers. Its architecture is capable of learning and extracting multi-scale features from images to identify the presence of faces. For example, the model might first extract low-level features (such as edges and textures) and high-level features (such as combinations of eyes, nose, and mouth) from the image through a series of convolutional and pooling layers. Then, it predicts regions in the image that may contain faces using a Region Proposal Network (RPN) or direct regression, outputting the coordinates of a bounding box for each region (e.g., x, y coordinates of the top-left corner and width and height) and a confidence score indicating the probability that the region is a face. Specifically, the parameters (i.e., weights and biases) of this face detection model are trained on a large-scale face dataset, such as the WIDER FACE dataset or the CelebA dataset. During training, the model continuously adjusts its internal parameters by minimizing the error between the predicted bounding boxes and the ground truth face bounding boxes (e.g., using Intersection over Union (IoU) loss) and the classification error (face / non-face), enabling it to accurately locate faces in images. When each frame is input into the face detection module, the model outputs the bounding boxes of all detected faces in that frame and their corresponding confidence scores. To ensure detection accuracy, a confidence threshold, such as 0.9, is set. This threshold is determined through performance evaluation on a validation set, combined with a trade-off between recall and precision requirements in the actual application scenario. Only detections with confidence scores higher than this threshold are considered valid faces. For each frame, once valid face bounding boxes are determined, image separation can begin. Specifically, the generation of face image sequences is achieved by cropping the regions corresponding to the face bounding boxes from the original frames. For example, if the face detection module identifies a face with a bounding box of (x=500, y=300, width=300, height=400) and a confidence score of 0.95 in a 1920×1080 pixel video frame, then this 300×400 pixel region will be cropped out as the face image for the current frame. If multiple faces are detected in a frame, multiple face images will be cropped out. Meanwhile, the generation of the ambient background image sequence is achieved by masking or removing all detected face regions in the original frame, and then using the remaining image regions as the ambient background. For example, in the aforementioned 1920×1080 pixel frame, all pixels except for the face region (500, 300, 300, 400) will be considered as ambient background. If no valid faces are detected in a frame, then the entire original frame will be considered as the ambient background image.In this way, the original video stream is effectively decomposed into two independent image sequences: one is a sequence of face images focusing on the user's facial expressions and head posture, and the other is a sequence of environmental background images reflecting the user's physical environment. Both sequences are synchronized with the original video stream in time.

[0020] In step S12: The raw audio stream refers to the continuous digital audio signal acquired synchronously with the raw video stream, encoded at a specific sampling rate (e.g., 16 kHz or 44.1 kHz) and bit depth (e.g., 16 bits), which may contain a mixture of various acoustic components such as human voice, ambient noise, music, and silence. The speech activity detection module is a pre-trained deep learning model whose function is to determine whether human speech exists in an audio frame. The implementation of this module includes the following stages: First, audio preprocessing and feature extraction. The raw audio stream is segmented into short, overlapping audio frames. For example, for an audio stream with a sampling rate of 16 kHz, it can be segmented into frames that are 25 milliseconds long and have a frame shift of 10 milliseconds. This means that 100 frames of audio are generated per second. For each frame, a window function (such as a Hamming window) is applied to reduce spectral leakage, and then a series of acoustic features are extracted. Commonly used acoustic features include: Mel-frequency cepstral coefficients (MFCCs), which effectively characterize the spectral envelope of speech, extracting 13 to 39 dimensions; frame energy, reflecting the loudness of the sound; zero-crossing rate, used to distinguish between unvoiced and voiced sounds; and features describing the shape of the spectrum, such as spectral centroid and spectral spread. For example, each frame can extract 39 dimensions of MFCCs, 1 dimension of energy, and 1 dimension of zero-crossing rate, forming a 41-dimensional feature vector. These acoustic feature sequences are then fed into a deep learning model for speech activity detection. This model employs a recurrent neural network (RNN) architecture, such as a gated recurrent unit (GRU) or a long short-term memory (LSTM) network. A typical model architecture may contain two to three GRU or LSTM layers, each with 128 to 256 units. These recurrent layers capture the temporal dependencies in the audio sequence, which is crucial for distinguishing between speech and non-speech patterns. One or two fully connected (or dense) layers, for example, 64 to 128 units per layer, are then connected after the recurrent layers for further feature transformation. The final layer of the model is a fully connected layer that outputs a single neuron using the sigmoid activation function. Its output value, ranging from 0 to 1, represents the probability that the current audio frame contains speech. The model's weights and biases are trained on a large-scale, manually annotated speech activity detection dataset. This dataset contains a large amount of audio mixed with various background noises, and each audio frame is precisely labeled as either speech or non-speech. The model learns to accurately identify speech activity by iteratively updating its parameters by minimizing the binary cross-entropy loss function between the predicted probability and the true label. During the model inference phase, the sequence of acoustic feature vectors for each frame after feature extraction is input into the trained speech activity detection module, and the model outputs a speech presence probability for each frame. To convert these probabilities into discrete speech or non-speech labels, a probability threshold, such as 0.5 or 0.7, is set. Frames above this threshold are labeled as speech, and frames below this threshold are labeled as non-speech.To improve detection robustness and avoid short-term false positives, post-processing is performed. For example, a median filter can be used to smooth consecutive speech / non-speech tags, eliminating speech segments or non-speech gaps that are too short in duration. Furthermore, minimum speech durations (e.g., 100 milliseconds) and minimum non-speech durations can be set, merging or correcting segments below these thresholds. Finally, based on the speech / non-speech tag sequence obtained after thresholding and post-processing, the original audio stream is precisely segmented. All consecutive audio segments labeled as speech are extracted to form user speech segments. For example, if in a 10-second original audio stream, the speech activity detection module identifies speech from seconds 2.5 to 4.8 and from 6.1 to 8.3, then the audio data corresponding to these two time periods will be extracted as user speech segments. All consecutive audio segments labeled as non-speech are extracted to form ambient sound segments. For example, in the above example, audio data from 0 seconds to 2.5 seconds, from 4.8 seconds to 6.1 seconds, and from 8.3 seconds to 10 seconds will be extracted as environmental sound segments. In this way, the original audio stream is clearly decoupled from the user's own vocal behavior and the acoustic environment in which they are located, providing a clean and semantically clear input for subsequent feature extraction and psychological anomaly detection.

[0021] In step S13: The face image sequence consists of a series of consecutive image frames, each containing the user's face region and bearing precise timestamp information. For example, in a video stream of 25 frames per second, the timestamp interval between each frame is 40 milliseconds. The user's voice segment consists of a series of discrete audio segments, each containing the user's voice and also bearing start and end timestamps. For example, a user's voice segment might start at 2.5 seconds into the video stream and last until 4.8 seconds. Since the original video stream and the original audio stream are acquired synchronously, their time bases are unified. Therefore, the alignment process primarily relies on timestamp information. Specifically, for each face image frame in the video stream, its timestamp is compared with the timestamp of the user's voice segment. If the timestamp of a face image frame falls within the start and end time range of a user's voice segment, then the face image frame and the voice segment are considered to correspond temporally. For example, if frame 63 in a face image sequence (corresponding to timestamp 2.52 seconds) overlaps with a user's voice segment (from 2.5 seconds to 4.8 seconds), then that face image frame is considered part of the voice segment. In practice, to construct a consistent user behavior segment, a fixed time window length is set, such as 5 seconds. The system iterates through the aligned face image sequence and user voice segments. When user voice activity is detected, a fixed-length time window is extracted forward or backward from the start time of that activity. Within this time window, all aligned face image frames and user voice data are collected. For example, if the user starts speaking at 2.5 seconds and the voice activity continues until 4.8 seconds, a user behavior segment starting at 2.5 seconds and lasting 5 seconds can be constructed. This segment will contain all face image frames from 2.5 seconds to 7.5 seconds, such as 25 frames / second * 5 seconds = 125 face images, as well as all user voice data within that time period. If the user's speech segment is discontinuous within this 5-second window, only the actual speech portion is included. The encapsulation process integrates these aligned visual and auditory data into a single logical unit. A user behavior segment can be represented as a data structure containing: a time range identifier, such as a start time of 2.5 seconds and an end time of 7.5 seconds; a list containing all cropped face images within that time range; and an audio file or audio feature sequence containing all user speech data within that time range. This encapsulation ensures that the visual and auditory information of the user behavior is synchronous and complete in time. For example, a user behavior segment might contain 125 frames of face images (each frame with a resolution of, for example, 200×200 pixels) and a 5-second mono user speech data segment with a sampling rate of 16kHz.

[0022] In step S14: Similar to S13, this step aims to precisely time-align the environmental background image sequence extracted from the original video stream (from S11) with the environmental sound segments separated from the original audio stream (from S12), and encapsulate them into a unified environmental context segment. The environmental background image sequence consists of a series of consecutive image frames, each excluding the user's face region and bearing precise timestamp information. The environmental sound segments consist of a series of discrete audio segments, each containing non-user-generated environmental sounds and also bearing start and end timestamps. For example, an environmental sound segment might start at second 0 of the video stream and last until second 2.5, or from second 4.8 to second 6.1. Data alignment also relies on timestamp information. For each frame of the environmental background image in the video stream, its timestamp is compared with the timestamp of the environmental sound segment. If the timestamp of an environmental background image frame falls within the start and end time range of an environmental sound segment, then the environmental background image frame and the environmental sound segment are considered to correspond temporally. For example, if the 10th frame in the environmental background image sequence (corresponding to a timestamp of 0.4 seconds) overlaps with an ambient sound segment (0 to 2.5 seconds), then that frame of environmental background image is considered part of that ambient sound segment. Similar to the construction of user behavior segments, the system also sets a fixed time window length, such as 5 seconds, to construct uniform environmental context segments. The system iterates through the aligned environmental background image sequence and ambient sound segments. When ambient sound activity is detected, or to maintain consistency with the time window of the user behavior segment, a fixed-length time window is extracted based on a certain point in time. Within this time window, all aligned environmental background image frames and ambient sound data are collected. For example, if the user behavior segment is from 2.5 seconds to 7.5 seconds, then the corresponding environmental context segment should also cover the same time range, i.e., from 2.5 seconds to 7.5 seconds. This segment will contain all environmental background image frames from 2.5 seconds to 7.5 seconds, such as 125 frames of environmental background images, as well as all ambient sound data within that time period. If the ambient sound segments are intermittent within this 5-second window, only the portions where ambient sound actually occurs are included. The encapsulation process integrates these aligned visual and auditory data into a single logical unit. An environmental context segment can be represented as a data structure containing: a time range identifier, such as a start time of 2.5 seconds and an end time of 7.5 seconds; a list containing all ambient background images within that time range; and an audio file or audio feature sequence containing all ambient sound data within that time range. This encapsulation ensures that the visual and auditory information of the environmental context is temporally synchronized and complete.For example, an environmental scene segment might contain 125 frames of environmental background images (each frame with a resolution of 1920×1080 pixels, minus the face area) and a 5-second mono ambient sound data with a sampling rate of 16kHz.

[0023] In step S2, dual-stream context-behavior feature extraction is performed on user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors. Correspondingly, in psychological anomaly detection, although the aforementioned steps have successfully separated the original audiovisual data into user behavior segments and environmental context segments, solving the context-behavior decoupling problem at the data level, these segments are still at the stage of primitive or low-level feature representation. For example, user behavior segments contain a series of facial images and raw speech waveforms, while environmental context segments contain environmental background images and environmental sound waveforms. These raw data are high-dimensional, informationally redundant, and lack high-level semantic information, making them difficult to directly use for accurate judgment of psychological states in deep learning models. When processing such complex multimodal data, traditional deep learning models are limited in their learning efficiency and feature representation capabilities if the raw data is directly input, failing to effectively capture subtle patterns related to psychological anomalies. Therefore, in order to transform these separated raw segments into model-understandable, discriminative, and compact feature representations, and to ensure that user behavior and environmental context can be analyzed independently and in depth, dual-stream context-behavior feature extraction is further performed on user behavior segments and environmental context segments.

[0024] An exemplary implementation of step S2 of this application involves extracting dual-stream context-behavior features from user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors, including: S21, inputting a sequence of face images from the user behavior segments into a pre-trained 3D-CNN model to obtain spatiotemporal visual features of the face; S22, inputting a user speech segment from the user behavior segments into a pre-trained speech emotion model to obtain user behavior acoustic features; and S23, fusing the spatiotemporal visual features of the face and the acoustic features of the user behavior to obtain user behavior vectors.

[0025] An exemplary implementation of step S2 of this application, which involves extracting dual-stream context-behavior features from user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors, further includes: S24, inputting the environmental background image sequence in the environmental context segment into a scene recognition model to obtain scene visual features; S25, inputting the environmental sound segment in the environmental context segment into an audio scene classification model to obtain scene auditory features; and S26, performing feature fusion on the scene auditory features and scene visual features to obtain a context embedding vector.

[0026] Specifically, step S2 of this application is performed as follows: In step S21: First, the input face image sequence is preprocessed. Since the 3D-CNN model has specific requirements for the length and image size of the input sequence, the face image sequence needs to be sampled and scaled. For example, if the pre-trained 3D-CNN model expects a fixed number of frames (e.g., 16 or 32 frames) as input, a specified number of frames need to be uniformly sampled from the 125-frame face image sequence. Simultaneously, each frame is scaled to the standard size required by the model, such as 112×112 pixels. Furthermore, the pixel values ​​of the image are normalized, for example, scaled to the range of [0,1] or [-1,1] to adapt to the model's input requirements. Next, these preprocessed face image sequences are input into the pre-trained 3D-CNN model. The 3D-CNN model is a deep learning architecture specifically designed for processing video data. It extracts features simultaneously in the spatial dimension (width and height of the image) and the temporal dimension (video frame sequence) using three-dimensional convolutional kernels (rather than traditional two-dimensional convolutional kernels). A 3D-CNN model comprises multiple 3D convolutional layers, 3D pooling layers, batch normalization layers, and activation functions (such as ReLU). For example, the model might start with an input sequence and progressively extract different levels of spatiotemporal features through a series of 3D max-pooling layers with 3×3×3 kernels, 64 different ReLU activation functions, and 1×2×2 pooling windows. These layers capture local facial texture and shape features, as well as the temporal patterns of these features, such as the raising of eyebrows and the twitching of the corners of the mouth. After multiple layers of 3D convolution and pooling, the model's output passes through a global average pooling layer or a flattening layer, and is then connected to one or more fully connected layers. Specifically, the weights and biases of this 3D-CNN model are pre-trained on large-scale video datasets. These datasets are used for action recognition or facial expression recognition in videos. During pre-training, the model learns how to identify different action or expression categories from videos by minimizing the loss function of the classification task (e.g., cross-entropy loss). In step S21, the final classification result of the model is not needed; instead, the high-level feature representations it has learned are required. Therefore, the output of the last fully connected layer in the pre-trained 3D-CNN model (before the classification head) is extracted as the spatiotemporal visual features of the face. For example, if the last fully connected layer of the model outputs a 256-dimensional vector, then this 256-dimensional vector represents the compact and semantically rich spatiotemporal visual features of the face image sequence in that user behavior segment. This feature vector can effectively encode information such as the dynamic changes in the user's facial expressions and subtle adjustments in head pose.For example, after processing a 5-second segment of user behavior with a 3D-CNN, a 256-dimensional spatiotemporal visual feature vector of the face can be obtained. This vector comprehensively reflects the overall dynamic pattern of the user's facial expressions and movements within those 5 seconds.

[0027] In step S22: First, the input user speech segments are preprocessed. This includes resampling to the sampling rate desired by the speech emotion model (e.g., 16 kHz) and normalizing the audio to ensure the volume is within a reasonable range, avoiding excessively loud or soft input signals from affecting model performance. Next, these preprocessed user speech segments are input into a pre-trained speech emotion model. The speech emotion model is a deep learning model specifically designed to identify emotional states in speech. Its architecture includes the following components: a feature extraction layer, which transforms the raw audio waveform into acoustic features more suitable for model processing. Common features include Mel frequency cepstral coefficients, Mel spectrograms, or directly using the raw waveform as input (for end-to-end models). For example, for a 5-second speech segment, its Mel spectrogram can be extracted, with dimensions possibly being (number of time frames × number of Mel filter banks), such as (500 frames × 80 Mel filter banks). A sequence modeling layer, which captures the dynamic changes of acoustic features over time and contextual information, crucial for emotion recognition. Common architectures include recurrent neural networks (RNNs), particularly long short-term memory networks (LSTMs) or gated recurrent units (GRUs), or one-dimensional convolutional neural networks (1D CNNs) combined with pooling layers. For example, a model might contain two to three layers of bidirectional long short-term memory networks (Bi-LSTMs), each with 256 units, capable of processing both past and future information of the speech sequence simultaneously. An aggregation layer aggregates the frame-level features output from the sequence modeling layer into a fixed-length segment-level feature vector. This can be achieved through global average pooling, attention mechanisms, or statistical pooling (such as averaging and variance of features across all frames). A classification head (used during pre-training) follows the aggregation layer, connecting one or more fully connected layers, ultimately outputting to a softmax activation function to predict the sentiment category of the speech (e.g., happy, sad, angry, neutral, etc.). The weights and biases of this speech sentiment model are trained on a large-scale, manually labeled speech sentiment dataset. In step S22, the output of the last fully connected layer (before the classification head) in the pre-trained speech emotion model is extracted as the user behavior acoustic feature. For example, if the last fully connected layer of the model outputs a 512-dimensional vector, then this 512-dimensional vector represents the compact and semantically rich emotional acoustic features of the user's speech segment. This feature vector can effectively encode the emotional intensity, polarity, and other acoustic cues related to psychological state in the user's speech, providing crucial auditory input for subsequent user behavior analysis. For example, for a 5-second user speech segment, after processing by the speech emotion model, a 512-dimensional user behavior acoustic feature vector can be obtained, which comprehensively reflects the emotional state of the user's speech within those 5 seconds.

[0028] In step S23: Feature fusion is performed. The most direct and effective method for feature fusion is feature concatenation. Specifically, the spatiotemporal visual feature vector of the face and the acoustic feature vector of user behavior are connected along their feature dimensions. For example, the spatiotemporal visual feature vector of the face is a 256-dimensional vector, and the acoustic feature vector of user behavior is a 512-dimensional vector. The fused user behavior vector will be the concatenation of these two vectors. Through this concatenation operation, the resulting user behavior vector will have a higher dimension, for example, 256 + 512 = 768 dimensions. This fused vector contains both visual and auditory information of user behavior, and can more comprehensively represent the user's psychological state. Visual and auditory information are often complementary; for example, when visual information is unclear (such as in insufficient light), auditory information can provide supplementation, and vice versa. Through fusion, the robustness and accuracy of understanding user behavior can be improved.

[0029] In step S24: First, the input environmental background image sequence is preprocessed. Since most pre-trained scene recognition models expect fixed-size input images, each frame of the environmental background image needs to be scaled or cropped to the model's standard input size, such as 224×224 pixels. If the original image resolution is high, it can be center-cropped or randomly cropped first, and then scaled. Second, the pixel values ​​of the image need to be normalized. This involves converting the pixel values ​​from an integer range of 0-255 to a floating-point range of 0-1, and then standardizing them according to the mean and standard deviation used by the pre-trained model. For example, for an RGB image, the mean of its three color channels might be [0.485, 0.456, 0.406], and the standard deviations might be [0.229, 0.224, 0.225]. These preprocessing steps ensure the consistency of the input data format and help the model generalize better. Next, these preprocessed environmental background image sequences are input into the pre-trained scene recognition model. The scene recognition model is a deep convolutional neural network (CNN) specifically designed to identify and classify different scene categories from images. A typical scene recognition model can be built on the ResNet (residual network) architecture, for example, using ResNet-50 as the backbone. ResNet-50 contains 50 convolutional layers and solves the gradient vanishing problem in deep network training by introducing residual connections, enabling it to build very deep networks and learn rich image features. Its basic architecture includes an initial convolutional layer and a max-pooling layer, followed by four stages of residual blocks, each stage containing multiple convolutional layers and skip connections. Each convolutional layer uses a small-sized convolutional kernel (e.g., 3×3) and is accompanied by batch normalization and an activation function (e.g., ReLU). At the end of the network, there is a global average pooling layer that aggregates features in the spatial dimension into a fixed-length vector, which is then connected to one or more fully connected layers. The weights and biases of this scene recognition model are pre-trained on a large-scale scene classification dataset. In step S24, the learned high-level feature representations are needed. Therefore, for each preprocessed environmental background image in the environmental context segment, it is input into a pre-trained scene recognition model. The model processes the image layer by layer, extracting features at different levels of abstraction. The output of the last global average pooling layer or the last fully connected layer (before the final classification layer) is extracted as the scene visual feature of that frame. For example, if using the ResNet-50 model, the 2048-dimensional feature vector after the global average pooling layer can be extracted as the scene visual feature of a single frame. Since the environmental context segment is an image sequence, these single-frame scene visual features need to be aggregated into a single scene visual feature representing the entire sequence. A commonly used aggregation method is temporal average pooling.Specifically, the scene visual feature vectors extracted from all frames in the sequence are averaged element-wise. For example, if a 5-second environmental context segment contains 125 frames, and each frame extracts a 2048-dimensional scene visual feature vector, then these 125 2048-dimensional vectors are summed element-wise and then divided by 125 to obtain a final 2048-dimensional scene visual feature vector. This average pooling operation effectively captures the overall visual information of the scene over the entire time period while smoothing out noise or transient changes that may exist in a single frame. This results in a fixed-dimensional scene visual feature vector. For example, a 2048-dimensional vector compactly encodes the semantic information of the visual scene in the environmental context segment. This vector can characterize the type of environment the user is in (e.g., indoor, outdoor, office, living room, etc.), the complexity of the environment, and the key visual elements that may be present. For example, for a 5-second environmental context segment, the final output scene visual features may be a 2048-dimensional floating-point vector, such as [0.12, -0.05, 0.87, ..., 0.33], representing the visual semantics of the environment within that time period.

[0030] In step S25: First, the input ambient sound clip is preprocessed. This includes resampling to the sampling rate desired by the audio scene classification model (e.g., 16 kHz or 44.1 kHz) and normalizing the audio to ensure the volume is within a reasonable range. Then, the audio waveform is converted into a time-frequency representation, most commonly a Mel spectrogram. The Mel spectrogram converts the time-domain signal to the frequency domain using a Fourier transform, and then uses a Mel filter bank to simulate the human ear's frequency perception characteristics, ultimately resulting in a two-dimensional image where the horizontal axis represents time, the vertical axis represents Mel frequency, and the pixel value represents energy. For example, for a 5-second audio clip with a sampling rate of 16 kHz, a Mel spectrogram with dimensions of (500 frames × 128 Mel filter banks) can be generated. Next, these preprocessed Mel spectrograms are input into a pre-trained audio scene classification model. The audio scene classification model is a deep learning model specifically designed to identify the environmental scene category represented by the audio clip. Its architecture includes the following parts: Feature extraction layer: The Mel spectrogram itself is a high-level feature, which the model learns directly from. Convolutional Layers: The model employs a two-dimensional convolutional neural network (2D CNN) to process the Mel spectrogram. For example, it may contain multiple 2D convolutional layers using kernels of different sizes (e.g., 3×3 or 5×5) to capture local patterns in the temporal and frequency dimensions of the spectrogram. These convolutional layers are followed by batch normalization layers and activation functions (e.g., ReLU), as well as max-pooling layers to reduce dimensionality and increase feature robustness. For example, the model might contain three to four convolutional blocks, each consisting of a 2D convolutional layer, a batch normalization layer, a ReLU activation function, and a 2D max-pooling layer. Sequence Modeling Layers: After the convolutional layers, recurrent neural networks (e.g., LSTM or GRU) or attention mechanisms are sometimes introduced to further capture long-range dependencies in the temporal dimension of the Mel spectrogram, which is very useful for understanding dynamically changing acoustic scenes. Aggregation Layers: This layer aggregates the feature maps output by the convolutional or sequence modeling layers into a fixed-length fragment-level feature vector. This can be achieved using global average pooling or global max pooling, compressing the spatial dimensions (time and frequency) of the feature maps into a single vector. The classification head (used during pre-training): In the pre-training phase, one or more fully connected layers are connected after the aggregation layer, and the final output is fed into the Softmax activation function to predict the category of the audio scene, such as indoor, outdoor, traffic, music, nature, etc. The weights and biases of this audio scene classification model are trained on a large-scale, manually labeled audio scene classification dataset. In step S25, the learned high-level feature representations are needed. Therefore, the Mel spectrogram in the environmental context segment is input into the pre-trained audio scene classification model. The model processes the spectrogram layer by layer, extracting features at different levels of abstraction.The output of the last global average pooling layer or the last fully connected layer (before the final classification layer) in the model is extracted as the scene auditory feature of the audio segment. For example, if the aggregation layer of the model outputs a 512-dimensional vector, then this 512-dimensional vector represents the compact and semantically rich scene auditory features of the environmental sound segment. This feature vector can effectively encode information such as the type, intensity, and rhythm of the environmental sound. For example, for a 5-second environmental sound segment, after processing by the audio scene classification model, a 512-dimensional scene auditory feature vector can be obtained, which comprehensively reflects the acoustic semantics of the environment within these 5 seconds, such as [0.8, 0.1, -0.3, ..., 0.5].

[0031] In step S26: Feature fusion is performed. This fusion method is the same as that in step S23, namely feature concatenation. Specifically, the scene visual feature vector and the scene auditory feature vector are concatenated along their feature dimensions. For example, the scene visual feature is a 2048-dimensional vector, and the scene auditory feature is a 512-dimensional vector. Then, the fused context embedding vector will have a higher dimension, namely 2048 + 512 = 2560 dimensions. This fused vector contains both visual and auditory information of the environment, and can more comprehensively represent the physical and acoustic environment in which the user is located. The advantage of this multimodal fusion is that visual and auditory information are often complementary and crucial for scene understanding. For example, the visual features of a park scene may include trees, grass, paths, etc., while the auditory features may include birdsong, wind sounds, children playing, etc. Visual or auditory information alone may not be sufficient to fully identify the scene, but combining the two can form a more accurate and richer scene understanding. Through fusion, the robustness and accuracy of environmental context understanding can be improved.

[0032] In step S3, the user behavior vector and context embedding vector are input into the trained personalized baseline model to obtain the reconstructed user behavior vector. It is understandable that in psychological anomaly detection, the background technology clearly states that psychological anomalies are not absolute, but rather inconsistencies relative to the individual baseline and the environment. Although the aforementioned steps (S1 and S2) have successfully decoupled the original audiovisual data into user behavior vectors and context embedding vectors, and extracted high-level semantic features from the multimodal data, these features themselves cannot directly determine whether the behavior is abnormal. A model that only compares behavioral characteristics with the average behavior of the general population will fail to capture the uniqueness and context-dependent nature of individual behavior. For example, a normally quiet person remaining quiet in a quiet environment is normal, but if the model lacks an understanding of this normal state, it may misjudge this as an abnormal low mood. This depersonalized and decontextualized judgment method is the key issue leading to the high false positive rate and poor practicality of existing technologies. Therefore, in order to overcome this limitation and achieve accurate and personalized identification of individual psychological abnormalities, in step S3, the personalized baseline model trained by inputting user behavior vectors and context embedding vectors aims to generate a representation of expected behavior by learning typical behavioral patterns of individuals in specific contexts.

[0033] Figure 4 This is a flowchart of step S3 in the deep learning-based psychological anomaly detection method according to an embodiment of this application. Figure 4 As shown, an exemplary implementation of step S3 of this application involves inputting the user behavior vector and context embedding vector into a trained personalized baseline model to obtain a reconstructed user behavior vector, including: S31, concatenating the user behavior vector and context embedding vector along the feature dimension to obtain a fused input vector; S32, inputting the fused input vector into the encoder network of the trained personalized baseline model to obtain a Gaussian distribution conditioned on the current context-behavior pair; S33, sampling the latent vector from the Gaussian distribution conditioned on the current context-behavior pair to obtain a sampled latent vector; and S34, concatenating the sampled latent vector with the context embedding vector and inputting it into the decoder network of the trained personalized baseline model to obtain a reconstructed user behavior vector.

[0034] Specifically, step S3 of this application is performed as follows: In step S31: the user behavior vector is the output from step S23, for example, a 768-dimensional vector that fuses the user's spatiotemporal visual facial features and user behavior acoustic features. The context embedding vector is the output from step S26, for example, a 2560-dimensional vector that fuses the scene's visual features and scene's auditory features. The concatenation operation is a direct and efficient feature fusion method. Specifically, the user behavior vector and the context embedding vector are concatenated along their feature dimensions. Thus, the dimension of the fused input vector will be the sum of the dimensions of the user behavior vector and the context embedding vector. For example, concatenating the 768-dimensional user behavior vector with the 2560-dimensional context embedding vector will result in a 768 + 2560 = 3328-dimensional fused input vector. This fused input vector contains all the key information about the user's behavior at the current moment, as well as the environmental context information at the time the behavior occurred, providing comprehensive context for the personalized baseline model.

[0035] In step S32: The core of the personalized baseline model is a Conditional Variational Autoencoder (CVAE). A CVAE is a generative model that learns the latent representation of data and generates new data based on given conditions. Here, the user behavior vector is the data to be learned and reconstructed, while the contextual embedding vector is the condition. The encoder network of the personalized baseline model is a deep neural network whose architecture consists of multiple fully connected (or dense) layers. For example, the encoder network might contain the following structure: Input layer: Receives the fused input vector from step S31, with a dimension of 3328. Hidden layers: Multiple fully connected layers, for example, the first layer has 1024 neurons using the ReLU activation function; the second layer has 512 neurons using the ReLU activation function. These hidden layers progressively perform nonlinear transformations and dimensionality reduction on the input features, extracting more abstract and compact feature representations. Output layer: The last layer of the encoder is divided into two parallel fully connected layers. One outputs the latent space mean vector, and the other outputs the latent space log-variance vector. These two vectors have the same dimension, i.e., the dimension of the latent space, for example, 128 dimensions. The mean vector represents the center of the latent distribution, while the log-variance vector (using log-variance instead of direct variance to ensure positive variance and numerical stability) represents the diffusion of the latent distribution. The parameters of this personalized baseline model (including its encoder and decoder), such as weights and biases, are trained on “normal” behavior data for a specific user. The training dataset contains normal user behavior vectors and corresponding contextual embedding vectors for the user in various contexts. The goal of the training process is to enable the model to learn the inherent patterns and variability of the user's behavior across different contexts. CVAE's training objective function consists of two parts: reconstruction loss, which measures the difference between the user behavior vector reconstructed by the decoder and the original user behavior vector, for example, using mean squared error. This prompts the model to learn how to accurately reconstruct user behavior. KL divergence loss, which measures the difference between the conditional Gaussian distribution of the encoder output and the standard normal distribution. This helps to regularize the latent space, giving it a good structure and preventing the encoder from learning overly sparse or discontinuous latent representations. By minimizing the weighted sum of these two losses, the model parameters (including the weights and biases of the encoder network) are iteratively updated through backpropagation and optimization algorithms such as the Adam optimizer. This training enables the encoder to map the input context-behavior pairs to a meaningful latent space and capture their core features in the form of a Gaussian distribution. In step S32, when a fused input vector (e.g., 3328-dimensional) is fed into the trained encoder network, it passes through the encoder layers sequentially. Ultimately, the encoder's output layer generates two 128-dimensional vectors: a latent space mean vector and a latent space log-variance vector. These two vectors together define a conditional Gaussian distribution of the current context-behavior pair in the latent space.For example, for a specific fused input vector, the encoder might output a latent space mean vector of [0.1, -0.5, ..., 0.3] and a latent space log-variance vector of [-1.2, -0.8, ..., -0.1]. This Gaussian distribution represents a probabilistic description of the model's expected behavior for the user in the current context and forms the basis for subsequent latent vector sampling.

[0036] An exemplary embodiment of step S33 of this application, sampling the latent vector of a Gaussian distribution conditioned on the current context-behavior pair to obtain a sampled latent vector, includes: sampling the latent vector of the Gaussian distribution conditioned on the current context-behavior pair using the following formula:

[0037] in, Let the latent space mean vector be a Gaussian distribution conditioned on the current context-behavior pairs. Let the latent space log-variance vector be a Gaussian distribution conditioned on the current context-behavior pairs. It is a random noise vector. This indicates element-wise multiplication. This is the sampling potential vector. It is the mean vector of the latent space from step S32, for example, a 128-dimensional floating-point vector that represents the central location of the current context-behavior pair in the latent space. It is the latent space log-variance vector from step S32, such as a 128-dimensional floating-point vector, which encodes the extent of diffusion of the latent distribution. Log-variance is used to ensure that the variance is always positive and to improve numerical stability. It is a random noise vector with the same dimensions as the latent space (e.g., 128 dimensions). Each element of this vector is independently sampled from a standard normal distribution (mean 0, variance 1). For example, One possible value is [0.23, -1.05, 0.78, ..., -0.12]. Introducing... This is to introduce randomness into the sampling process, enabling the model to explore different possibilities around the mean in the latent space, thereby generating diverse reconstructions of normal behavior. This term calculates the standard deviation of the latent distribution. This is equivalent to log(sqrt(variance)), where variance is the variance of each dimension in the latent space. Taking the exponent exp() on this variance yields the standard deviation for each latent dimension. For example, if... If one of the elements is -1.386, then 0.5 × (-1.386) = -0.693, and exp(-0.693) is approximately equal to 0.5. This means that the standard deviation of this latent dimension is 0.5. ⊙ represents element-wise multiplication (Hadamard product). This implies random noise. Each element is multiplied by the standard deviation of its corresponding latent dimension. If the standard deviation of a latent dimension is large, then the random perturbation along that dimension will be larger, reflecting the model's learning of the variability in behavior along that dimension. The final result... It is a 128-dimensional vector representing a specific sample drawn from a conditional Gaussian distribution. This sample contains the core information represented by the mean, and also reflects the natural variability of user behavior in this context through random noise and variance.

[0038] In step S34: First, feature concatenation is performed. The sampled latent vector (e.g., 128-dimensional) and contextual embedding vector (e.g., 2560-dimensional) are concatenated along the feature dimensions. The resulting concatenated vector has a dimension of 128 + 2560 = 2688. The contextual embedding vector serves as the conditional input to the decoder, ensuring that the decoder fully considers the environmental context in which the behavior occurs when reconstructing user behavior. Next, this concatenated vector is input into the decoder network of the trained personalized baseline model. The decoder network is a deep neural network whose architecture consists of multiple fully connected layers, designed to decode low-dimensional latent representations and conditional information back into high-dimensional user behavior vectors. For example, the decoder network might contain the following structure: Input layer: receives the concatenated 2688-dimensional vector. Hidden layers: multiple fully connected layers, for example, the first layer has 1024 neurons using the ReLU activation function; the second layer has 768 neurons using the ReLU activation function. These hidden layers progressively perform nonlinear transformations and dimensionality increases on the input features, recovering the complex patterns of user behavior from the latent representation. Output Layer: The last layer of the decoder is a fully connected layer whose output dimension is the same as the original user behavior vector (e.g., 768-dimensional). This output vector is the reconstructed user behavior vector. The output layer does not use an activation function, or uses a linear activation function, to output floating-point numbers in any range, because the feature values ​​of the user behavior vector may not have specific upper and lower bounds. The weights and biases of the decoder network in the personalized baseline model are obtained in the same training process as the encoder network. During the training of CVAE, the decoder aims to minimize the reconstruction loss, even if its output reconstructed user behavior vector is as close as possible to the original user behavior vector. Through this training, the decoder learns how to generate a user behavior representation that conforms to the normal pattern of the user in a specific context, based on samples in the latent space and given contextual information.

[0039] The resulting reconstructed user behavior vector is a 768-dimensional vector, for example, [0.08, -0.15, 0.67, ..., 0.22]. This vector represents the expected user behavior generated by the model based on the current context (context embedding vector) and latent behavioral patterns sampled from the individual baseline distribution. It reflects the behavioral characteristics that the user would exhibit in similar contexts, including their visual and acoustic components.

[0040] In step S4, the personalized consistency deviation score between the reconstructed user behavior vector and the actual user behavior vector is calculated. It should be understood that in psychological anomaly detection, the background technique emphasizes the relativity of the definition of anomaly; that is, psychological anomalies are not absolute, but rather inconsistencies relative to the individual baseline and the environment. Although the aforementioned step (S3) has successfully utilized the personalized baseline model to generate an expected user behavior vector, i.e., the reconstructed user behavior vector, based on the current situation, this vector represents the user's normal behavior pattern in the current situation. However, simply obtaining this expected behavior vector cannot directly determine whether the actual behavior is abnormal. Without a quantitative indicator to measure the degree of deviation between actual behavior and expected behavior, it is impossible to objectively and accurately identify those weak abnormal signals that deviate from the individual's normal state. For example, a slight change in facial expression or fluctuation in tone may not be noticeable in the general population, but for a specific individual, if it differs significantly from the personalized baseline behavior, it may indicate an abnormal psychological state. Therefore, in order to transform this inconsistency into a quantifiable indicator and provide a basis for subsequent abnormal accumulation judgment and graded early warning, the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector is calculated. The aim is to accurately capture psychological abnormalities by precisely quantifying the difference between actual behavior and personalized expected behavior.

[0041] An exemplary implementation of step S4 of this application, calculating the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector, includes: calculating the mean square error between the reconstructed user behavior vector and the user behavior vector as the personalized consistency deviation score.

[0042] Specifically, step S4 of this application is performed as follows: The mean squared error (MSE), as the personalized consistency bias score, aims to quantify the difference between the actually observed user behavior and the normal user behavior predicted by the personalized baseline model. MSE is a commonly used metric to measure the difference between two vectors; it calculates the average of the squared differences of corresponding elements. The specific implementation process is as follows: First, obtain the current user behavior vector (actual observed value) and the reconstructed user behavior vector (expected value). Next, subtract each corresponding element of these two vectors to obtain a difference vector. Then, square each element of the difference vector to obtain a squared difference vector. Finally, sum all elements of the squared difference vector and divide by the dimension of the vector (i.e., 768) to obtain the final MSE value, which is the personalized consistency bias score.

[0043] For example, if the value of a user behavior vector in one dimension is 0.5, and the value of the reconstructed user behavior vector in the same dimension is 0.4, then the difference in that dimension is 0.5 - 0.4 = 0.1, and the squared difference is 0.1^2 = 0.01. If, in another dimension, the user behavior vector is -0.2 and the reconstructed user behavior vector is 0.1, then the difference is -0.2 - 0.1 = -0.3, and the squared difference is (-0.3)^2 = 0.09. By summing and averaging the squared differences across all 768 dimensions, the final personalized consistency deviation score is obtained. The larger this mean squared error value, the greater the difference between the actual user behavior and the normal behavior expected by the personalized baseline model, i.e., the higher the degree to which the current behavior deviates from the individual's normal state. Conversely, the smaller the mean squared error value, the more consistent the actual behavior is with the expected behavior, indicating that the user behavior is within the normal range of its personalized base state. For example, a personalized consistency deviation score might be 0.005, indicating a high degree of consistency between behavior and baseline; while another deviation score might be 0.15, indicating a significant deviation from baseline. This quantified deviation score directly reflects the degree of inconsistency mentioned in the background technique, providing a precise numerical basis for subsequent anomaly assessment. In this way, this method can effectively capture those weak anomalous signals relative to the individual, rather than simply comparing them with the average behavior of the general population, thus achieving more accurate and personalized detection of psychological anomalies.

[0044] In step S5, anomaly accumulation judgment and graded early warning are performed on the time series of personalized consistency deviation scores to obtain early warning signals. That is, personalized consistency deviation scores quantify the degree of deviation between actual behavior and an individual's baseline expected behavior, but a deviation score at a single moment is often insufficient as a reliable basis for judging psychological abnormalities. Human psychological states and behavioral performances have a certain degree of fluctuation; occasional deviations may be normal emotional fluctuations or brief stress responses, not true abnormal signals. Issuing early warnings based solely on instantaneous high deviation scores may lead to a large number of false alarms, reducing the reliability of early warnings and user trust. True psychological abnormalities often manifest as persistent, cumulative, or gradual changes in behavioral patterns, rather than isolated instantaneous events. Therefore, to improve the accuracy and robustness of early warnings and to effectively identify early and persistent signals of psychological abnormalities, anomaly accumulation judgment and graded early warning are ultimately performed on the time series of personalized consistency deviation scores to obtain early warning signals. This aims to more accurately and promptly identify potential psychological abnormality risks by analyzing the evolution trend and cumulative effect of deviation scores over time and issuing multi-level early warnings based on the degree of deviation.

[0045] Specifically, an exemplary implementation of step S5 of this application is as follows: Anomaly accumulation judgment involves the following aspects: First, to capture persistent changes in behavior, a sliding time window is set, for example, deviation scores from the last 5, 10, or 30 minutes. The system continuously collects and stores these deviation scores. Next, one or more anomaly thresholds are set. These thresholds can be set by analyzing the distribution of deviation scores from the user's historical normal behavior (e.g., calculating the mean and standard deviation, and setting the mean plus 2 or 3 times the standard deviation as the threshold), or through expert experience. For example, a low threshold such as 0.08 and a high threshold such as 0.15 can be set. Then, within the sliding time window, one or more cumulative indicators are calculated to assess the persistence and intensity of the anomaly. Common indicators include: Duration: The length of time the deviation score continuously exceeds a certain threshold. Frequency: The number or proportion of times the deviation score exceeds the threshold within a certain time window. Cumulative Value: The sum of deviation scores exceeding the threshold, or the sum of the differences between the deviation score and the threshold. For example, it can calculate the sum of the differences between all deviation scores exceeding the lower threshold and the higher threshold in the past 10 minutes.

[0046] The tiered early warning mechanism generates warning signals of varying severity based on these accumulated judgments. For example, three warning levels can be set: No warning: No warning signal is issued when the deviation score remains below all thresholds, or when the accumulated indicators do not meet any warning conditions. Level 1 Warning (Mild Abnormality): A mild warning signal is issued when the deviation score frequently (e.g., more than 3 times) exceeds the low threshold within a short period (e.g., within the past 5 minutes), or exceeds the low threshold for a certain duration (e.g., consecutive 2 minutes), but does not meet the conditions for a higher level. For example, the warning signal might be "User behavior slightly deviates from the baseline; attention is recommended." Level 2 Warning (Moderate Abnormality): A moderate warning signal is issued when the deviation score continuously (e.g., for 5 consecutive minutes) exceeds the low threshold over a longer period (e.g., within the past 15 minutes), and some deviation scores exceed the high threshold, or when the accumulated indicators reach a higher level. For example, the warning signal might be "User behavior moderately deviates from the baseline; close observation is recommended." Level 3 Warning (Severe Anomaly): A severe warning signal is issued when the deviation score consistently and significantly exceeds the high threshold over a prolonged period (e.g., within the past 30 minutes), or when the cumulative indicator reaches a very high level. For example, the warning signal might be "User behavior has severely deviated from the baseline; timely intervention is recommended."

[0047] In a specific example, a low threshold is set to 0.08 and a high threshold to 0.15. If a user has 10 consecutive deviation scores of 0.03 within a certain time window (e.g., calculated every 5 seconds), the cumulative judgment result is no warning. If, within the past 5 minutes, a user has 4 deviation scores of 0.09, 0.11, 0.07, and 0.10, with the rest at 0.03, and 3 of these deviation scores exceed the low threshold but none exceed the high threshold, and the duration is short, a Level 1 warning may be triggered. If, within the past 15 minutes, a user has 8 consecutive deviation scores exceeding the low threshold, with 2 of these reaching 0.16 and 0.18 (exceeding the high threshold), a Level 2 warning may be triggered. If, within the past 30 minutes, a user has 20 consecutive deviation scores exceeding the high threshold, and the average deviation score reaches 0.20, a Level 3 warning may be triggered. This abnormal accumulation judgment and graded early warning mechanism can effectively distinguish between instantaneous fluctuations and persistent abnormalities, avoiding overly sensitive or sluggish early warnings, thereby improving the accuracy and practicality of psychological abnormality detection and providing timely and reliable signals for early intervention.

[0048] In particular, in psychological anomaly detection, measuring anomalies solely through reconstruction error has inherent limitations. Standard reconstruction error, i.e., the individualized consistency deviation score, is primarily sensitive to novel behavioral patterns never seen by the model before (i.e., global novelty anomalies). However, its detection capability is insufficient for another, more subtle type of anomaly: situational incongruity anomalies. Situational incongruity anomalies refer to user behavior that the model is familiar with and understands (e.g., a sad expression), but this behavior occurs in a completely inappropriate context (e.g., a celebratory party). In this case, because the model can easily encode and decode the sad expression, its reconstruction error is very low, causing this significant psychological incongruity to be overlooked. Therefore, to overcome this technical bottleneck and simultaneously capture two different types of anomalous signals, this application proposes an anomaly scoring method based on prior-posterior path fusion. This method aims to construct a more comprehensive and robust final anomaly score by introducing a new index specifically quantifying the degree of situational incongruity and combining it with traditional reconstruction error.

[0049] An exemplary preferred embodiment of step S4 of this application includes: Step S4, calculating the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector, comprising: Step S41, calculating the initial personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector; Step S42, performing anomaly scoring on the user behavior vector based on prior-posterior path fusion to obtain a behavior deviation score; Step S43, calculating the weighted sum of the initial personalized consistency deviation score and the behavior deviation score to obtain the personalized consistency deviation score. Specifically, the personalized consistency deviation score calculated in step S41 is the same as that calculated in the embodiment of step 4 above, i.e., the initial personalized consistency deviation score in step S41 is the personalized consistency deviation score calculated in the embodiment of step 4 above, and therefore will not be specifically described.

[0050] In step S42: The difference between the currently observed user behavior and the expected behavior that the model considers most normal and conventional in this context is quantified, thereby specifically scoring context-inconsistent anomalies. Within the framework of Conditional Variational Autoencoders (CVAEs), standard anomaly scoring relies on the posterior path, i.e., inferring the posterior distribution from the current user behavior vector and context embedding vector, and sampling the latent code from it for reconstruction. However, to evaluate context consistency, it is desirable to bypass the influence of potentially anomalous user behavior on the latent code and instead use the prior path. Specifically, the latent code that best represents the normal or average state is selected directly from the model's conditional prior distribution, denoted as the prior latent code zprior. To simplify the model, it is usually assumed that this prior distribution is independent of the specific context, i.e., a standard normal distribution N(0,I). Under this setting, the latent code that best represents the normal state is the mean of this distribution, which is a zero vector. In practice, a zero vector with the same dimension as the latent space (e.g., 128 dimensions) is first generated as the prior latent code zprior. Then, the zprior, representing absolute normality, is concatenated with the current contextual embedding vector (e.g., 2560-dimensional) to form a decoder input vector. This vector is then fed into the decoder network of the trained personalized baseline model to generate a priori expected behavior vector. This vector (e.g., 768-dimensional) represents the most likely and typical behavior pattern that the model believes the user will exhibit in that specific context. Finally, the mean squared error between the currently observed user behavior vector and this generated priori expected behavior vector is calculated; this error value is the behavior deviation score. By decoupling expectation from observation in this way, the influence of the current observed behavior on the expected behavior benchmark is eliminated, thus giving the score a clear indication of anomalies specific to a particular context. Furthermore, for some behaviors that are anomalous but have simple feature patterns (easy to self-encode), the model is more sensitive to such deviations because the benchmark is a fixed priori expected behavior. For example, if the value of a certain dimension of the actual user behavior vector is 0.9 (representing a certain high emotional characteristic), while the value of the corresponding dimension of the prior expected behavior vector in the current context (such as being alone late at night) is 0.1, then the difference between the two is significant, and the calculated behavior deviation score will be correspondingly higher, thus effectively capturing this situational inconsistency.

[0051] In step S43: A single scoring metric is one-sided. By fusing two complementary anomaly scores, a more comprehensive final anomaly judgment criterion can be constructed. The resulting final score is sensitive to both novel behavioral patterns never seen by the model before (contributed by the initial personalized consistency deviation score) and patterns of behavior that do not match the context (contributed by the behavior deviation score), thereby significantly improving detection accuracy and tolerance to noise and random events. In implementation, this step is calculated using the following formula: Final Personalized Consistency Deviation Score = a1 * Initial Personalized Consistency Deviation Score + a2 * Behavior Deviation Score. In this formula, the initial personalized consistency deviation score is the reconstruction error calculated using the standard CVAE posterior path, which measures global novelty anomalies. The behavior deviation score is calculated in the previous step using the prior path, and it measures contextual inconsistency anomalies. a1 and a2 are preset weighting coefficients used to balance the importance of the two scores. These two weights are hyperparameters whose values ​​can be tuned based on performance on the validation dataset, or, in the absence of specific prior knowledge, can be set to equal values, such as a1=0.5 and a2=0.5, indicating that the two anomaly types are equally important. Specifically, the final score can simultaneously capture two core anomaly types: the first is the context inconsistency anomaly, as mentioned earlier, the behavioral deviation score can quantify this mismatch by comparing the actual behavior with the most normal expected behavior in the current context. The second is the global novelty anomaly, which refers to a new (behavior, context) combination exhibited by the user that is a pattern the model has never seen before. This pattern is so unique that the encoder cannot effectively compress it into the known latent space, and the decoder therefore cannot accurately reconstruct it. The initial personalization consistency deviation score (i.e., reconstruction error) measures the model's ability to encode the current input data points. Therefore, when extreme anomalies or entirely new events occur (e.g., a user makes an unconventional, painful, but not typical facial expression due to physical discomfort), the entire (behavior, context) data point deviates from the distribution of the model's training data. The encoder cannot find a suitable latent encoding, resulting in a significant difference between the reconstructed behavior and the original behavior, leading to a high score. For example, in a festive party context, a user displays a slight expression of sadness. Since the model recognizes sadness, its initial personalization consistency deviation score (reconstruction error) might be low, such as 0.03. However, because sadness is inconsistent with the festive context, its behavior deviation score might be high, such as 0.20. If we set a1=0.5 and a2=0.5, the final personalization consistency deviation score is 0.5*0.03+0.5*0.20=0.115. This final score is significantly higher than the individual reconstruction error, successfully amplifying the signal of situational inconsistency, so that it can be effectively captured by the subsequent anomaly accumulation judgment module, avoiding missed detections.Conversely, a small, transient anomaly may only slightly increase one of the scores. Only when the anomaly signal is reflected in both measures, or is persistent and significant in a certain dimension, will the final score increase significantly, thus effectively reducing the false alarm rate.

[0052] Therefore, the final personalized consistency deviation score obtained through this optimized scheme, due to its higher tolerance for noise and random events, provides a higher-quality and more reliable input signal for the subsequent abnormal accumulation judgment and graded early warning in step S5. Minor and transient anomalies may only slightly increase one of the scores; only when the abnormal signal is reflected in both metrics, or is persistent and significant in one metric, will the final score increase significantly. This allows the subsequent accumulation judgment module to analyze based on a more stable and indicative time series, thereby effectively reducing the false alarm rate and achieving accurate capture of early signals of psychological abnormalities.

[0053] In summary, a deep learning-based psychological anomaly detection method based on embodiments of this application is presented. By introducing context awareness and personalized baseline modeling, it solves the problems of context-behavior mismatch and the relativity of anomaly definitions in existing deep learning methods for psychological anomaly detection. First, the user and context are separated from the original audiovisual stream, and user behavior features and environmental context features are extracted separately. Then, these features are input into a trained personalized baseline model. This model can learn and understand the typical behavioral patterns of a specific user in a specific context, and thus reconstruct the expected user behavior in that context. By calculating the personalized consistency deviation score between the actual user behavior and the reconstructed user behavior, this method can accurately quantify the degree of deviation between an individual's behavior and their expected behavior in the current context. This deviation analysis based on individual baselines and context awareness effectively avoids misjudging context-adaptive behaviors as anomalies, thereby more accurately capturing early, weak psychological anomaly signals and significantly improving the accuracy and practicality of detection.

[0054] Figure 5 This is a block diagram of a deep learning-based psychological anomaly detection system according to an embodiment of this application. Figure 5As shown, the deep learning-based psychological anomaly detection system 100 according to an embodiment of this application includes: a user / context separation module 110, used to perform user / context separation on the acquired original video stream and original audio stream to obtain user behavior segments and environmental context segments; a dual-stream feature extraction module 120, used to perform dual-stream context-behavior feature extraction on the user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors; a user behavior reconstruction module 130, used to input the user behavior vectors and context embedding vectors into a trained personalized baseline model to obtain reconstructed user behavior vectors; a consistency deviation score calculation module 140, used to calculate the personalized consistency deviation score between the reconstructed user behavior vectors and the user behavior vectors; and a warning signal generation module 150, used to perform anomaly accumulation judgment and graded warning on the time series of personalized consistency deviation scores to obtain warning signals.

[0055] Here, those skilled in the art will understand that the specific operations of each step in the aforementioned deep learning-based psychological anomaly detection system have been referenced above. Figures 1 to 4 The description of the deep learning-based psychological anomaly detection method is detailed here, and therefore, its repeated description will be omitted.

Claims

1. A method for detecting psychological anomalies based on deep learning, characterized in that, include: User / context separation is performed on the acquired raw video and audio streams to obtain user behavior segments and environmental context segments; Two-stream context-behavior feature extraction is performed on user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors; The user behavior vector and context embedding vector are input into the trained personalized baseline model to obtain the reconstructed user behavior vector; Calculate the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector; Anomaly accumulation judgment and graded early warning are performed on the time series of personalized consistency deviation scores to obtain early warning signals.

2. The deep learning-based psychological anomaly detection method according to claim 1, characterized in that, User / context separation is performed on the acquired raw video and audio streams to obtain user behavior segments and environmental context segments, including: The original video stream is input into the face detection module to obtain a sequence of face images and a sequence of environmental background images; The raw audio stream is input into the speech activity detection module to obtain user speech segments and ambient sound segments; Data alignment and encapsulation are performed on facial image sequences and user voice segments to obtain user behavior segments; Data alignment and encapsulation are performed on environmental background image sequences and environmental sound segments to obtain environmental context segments.

3. The deep learning-based psychological anomaly detection method according to claim 2, characterized in that, Two-stream context-behavior feature extraction is performed on user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors, including: Input the sequence of facial images from user behavior segments into a pre-trained 3D-CNN model to obtain the spatiotemporal visual features of the face; User speech segments from user behavior segments are input into a pre-trained speech emotion model to obtain user behavior acoustic features; Feature fusion is performed on the spatiotemporal visual features of the face and the acoustic features of user behavior to obtain the user behavior vector.

4. The deep learning-based psychological anomaly detection method according to claim 2, characterized in that, The process involves two-stream context-behavior feature extraction from user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors, and also includes: Input the sequence of environmental background images from environmental context segments into the scene recognition model to obtain the scene visual features; Environmental sound segments from environmental context segments are input into an audio scene classification model to obtain scene auditory features; Feature fusion is performed on the auditory and visual features of the scene to obtain the context embedding vector.

5. The deep learning-based psychological anomaly detection method according to claim 1, characterized in that, The personalized baseline model, trained by inputting user behavior vectors and contextual embedding vectors, is used to obtain reconstructed user behavior vectors, including: The user behavior vector and context embedding vector are concatenated along the feature dimension to obtain the fused input vector; The encoder network of the trained personalized baseline model is fed with the fused input vectors to obtain a Gaussian distribution conditioned on the current context-behavior pairs; Sample the latent vectors from a Gaussian distribution conditioned on the current context-behavior pair to obtain sampled latent vectors; The sampled latent vector is concatenated with the context embedding vector and then input into the decoder network of the trained personalized baseline model to obtain the reconstructed user behavior vector.

6. The deep learning-based psychological anomaly detection method according to claim 5, characterized in that, Sampling the latent vector of a Gaussian distribution conditioned on the current context-behavior pair to obtain a sampled latent vector includes: sampling the latent vector of a Gaussian distribution conditioned on the current context-behavior pair using the following formula: ； in, Let the latent space mean vector be a Gaussian distribution conditioned on the current context-behavior pairs. Let the latent space log-variance vector be a Gaussian distribution conditioned on the current context-behavior pairs. It is a random noise vector. This indicates element-wise multiplication. This is the sampling potential vector.

7. The deep learning-based psychological anomaly detection method according to claim 1, characterized in that, Calculating the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector includes: calculating the mean square error between the reconstructed user behavior vector and the user behavior vector as the personalized consistency deviation score.

8. A deep learning-based psychological anomaly detection system, characterized in that, include: The user / context separation module is used to separate the acquired raw video stream and raw audio stream into user / context segments to obtain user behavior segments and environmental context segments; The dual-stream feature extraction module is used to extract dual-stream context-behavior features from user behavior segments and environmental context segments to obtain user behavior vectors and context embedding vectors. The user behavior reconstruction module is used to input user behavior vectors and context embedding vectors into the trained personalized baseline model to obtain reconstructed user behavior vectors. The consistency deviation score calculation module is used to calculate the personalized consistency deviation score between the reconstructed user behavior vector and the user behavior vector. The early warning signal generation module is used to perform anomaly accumulation judgment and graded early warning on the time series of personalized consistency deviation scores to obtain early warning signals.

Images