Depression risk assessment method and device based on multi-modal gait feature fusion

Abstract

The application relates to the technical field of artificial intelligence, and discloses a depression risk assessment method and device based on multi-modal gait feature fusion. The method constructs a depression risk assessment basic model to extract global features of 3D skeleton coordinate data in the spatial dimension by using a multi-head self-attention mechanism and extract multi-scale time sequence features by using a multi-scale time convolution. Unique features and shared features of multiple isomorphic modes are extracted according to the depression risk assessment basic model, and a comparative distillation learning method is used to maximize the mutual information of the unique features and the shared features, and the parameter quantity is greatly reduced. A space-time feature map space-time attention map is constructed according to the shared features, and RGB image features are extracted accordingly; a comparative learning method is used to align the RGB image features and the shared features. Finally, a depression prediction result is output by using the RGB image features and the shared features, reliable depression risk assessment is realized, and the method can be used to objectively assist medical personnel in diagnosing depression.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and device for depression risk assessment based on multimodal gait feature fusion. Background Technology

[0002] Depression is insidious, and genetics, gender, living environment, and physical illness are all complex factors that influence its occurrence. Its diagnostic and assessment methods rely almost entirely on the severity of depressive symptoms reported by patients or the clinical judgment of the severity of symptoms. This has the disadvantages of lacking objectivity, poor timeliness, and requiring a lot of manpower and resources, making it unsuitable for large-scale promotion.

[0003] Currently, artificial intelligence methods for depression identification include those based on video, physiological signals, and social media data. While these methods can objectively assist medical personnel in diagnosing depression to some extent, they have limitations, primarily in the data collection process. For example, when collecting audio and video data, the experimental paradigm typically involves subjects communicating with AI assistants or researchers in a restricted environment, and the process is lengthy, making large-scale depression screening impractical. When collecting physiological signal data, such as electroencephalography (EEG) data, subjects need to wear EEG devices, such as specially designed helmets, and in some cases, view video stimuli. Regarding social media data, although data acquisition is relatively convenient, it inevitably infringes on patient privacy. Depressed patients are often highly sensitive and prone to overreacting to external stimuli; existing methods are not "user-friendly" enough and are difficult to scale up. Summary of the Invention

[0004] This application provides a depression risk assessment method based on multimodal gait feature fusion to address the problems in existing depression diagnosis and assessment methods, such as lack of objectivity, poor timeliness, high manpower and material resources required, and the ease with which data acquisition can cause overreactions in patients with depression, making large-scale promotion difficult.

[0005] Accordingly, embodiments of this application also provide a depression risk assessment device based on multimodal gait feature fusion, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0006] To address the aforementioned technical problems, this application discloses a depression risk assessment method based on multimodal gait feature fusion, the method comprising:

[0007] A basic model for depression risk assessment is constructed to extract global features of 3D skeleton coordinate data in the spatial dimension using a multi-head self-attention mechanism, and to extract multi-scale time series features using multi-scale temporal convolution.

[0008] Based on the depression risk assessment base model, unique features of multiple isomorphic modalities and shared features that integrate features of multiple isomorphic modalities are extracted, and the mutual information of multiple unique features and shared features is maximized by the contrastive distillation learning method.

[0009] Spatiotemporal feature maps and spatiotemporal attention maps are constructed based on RGB image data and shared features, respectively, and RGB image features are extracted based on the spatiotemporal feature maps and spatiotemporal attention maps;

[0010] A contrastive learning method is used to align RGB image features and shared features, and depression prediction results are obtained based on the output of RGB image features and shared features.

[0011] This application also discloses a depression risk assessment device based on multimodal gait feature fusion, the device comprising:

[0012] The model building module is used to build a basic model for depression risk assessment. It is used to extract global features of 3D skeleton coordinate data in the spatial dimension using a multi-head self-attention mechanism, and to extract multi-scale time series features using multi-scale temporal convolution.

[0013] The feature fusion module is used to extract unique features from multiple isomorphic modalities and shared features that fuse features from multiple isomorphic modalities based on the depression risk assessment base model, and to maximize the mutual information between multiple unique features and shared features through a contrastive distillation learning method.

[0014] The feature fusion and semantic alignment module is used to construct spatiotemporal feature maps and spatiotemporal attention maps based on RGB image data and shared features, respectively, and to extract RGB image features based on the spatiotemporal feature maps and spatiotemporal attention maps;

[0015] The feature fusion and semantic alignment module is also used to align RGB image features and shared features using a contrastive learning method, and output depression prediction results based on RGB image features and shared features.

[0016] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements one or more of the methods described in this application.

[0017] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements one or more of the methods described in this application.

[0018] In this embodiment, a basic model for depression risk assessment is constructed. This model utilizes a multi-head self-attention mechanism to extract global features from 3D skeleton coordinate data in the spatial dimension and employs multi-scale temporal convolution to extract multi-scale time-series features. At the isomorphic modality level, unique features of multiple isomorphic modalities and shared features that fuse multiple isomorphic modal features are extracted based on the basic model. A contrastive distillation learning method is used to maximize the mutual information between the unique and shared features. The shared features fuse multiple isomorphic modal features from the 3D skeleton coordinate data, significantly reducing the number of parameters. At the heterogeneous modality level, spatiotemporal feature maps and spatiotemporal attention maps are constructed based on RGB image data and shared features, respectively. RGB image features are extracted based on these maps. A contrastive learning method is used to align the RGB image features and shared features, achieving fusion and semantic alignment of 3D skeleton coordinate data and RGB image data at the attention level. Finally, depression prediction results are output based on the RGB image features and shared features, achieving a reliable depression risk assessment that can objectively assist medical personnel in diagnosing depression.

[0019] Additional aspects and advantages of the embodiments of this application will be set forth in the following description, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 A flowchart of a depression risk assessment method based on multimodal gait feature fusion provided in this application embodiment;

[0022] Figure 2 This is a schematic diagram of the structure of the 3D skeleton coordinate feature extraction module provided in the embodiments of this application;

[0023] Figure 3 A schematic diagram of a spatial Transformer module provided in an embodiment of this application;

[0024] Figure 4 A schematic diagram of a multi-scale temporal convolution module provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of feature splicing provided for an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the heterogeneous modal feature fusion model architecture provided in the embodiments of this application;

[0027] Figure 7 This is a schematic diagram of the spatiotemporal pseudo-alignment module architecture provided in an embodiment of this application;

[0028] Figure 8 This is a comparative learning diagram of various parts provided in the embodiments of this application;

[0029] Figure 9 A schematic diagram of the structure of the depression risk assessment device based on multimodal gait feature fusion provided in the embodiments of this application;

[0030] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0032] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0034] The solutions provided in this application can be executed by any electronic device, such as a terminal device or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. Regarding the technical problems existing in the prior art, the depression risk assessment method and apparatus based on multimodal gait feature fusion provided in this application aim to solve at least one of the technical problems in the prior art.

[0035] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0036] Existing research has demonstrated a link between body posture and depression. For example, there is overlap between brain regions controlling gait movement and emotional expression, and gait characteristics in depressed patients show significant reductions in walking speed, stride length, and cycle time. Therefore, it is necessary to develop a gait-based depression risk assessment system that can provide reliable, convenient, low-cost, and non-intrusive depression risk assessment to assist in clinical diagnosis.

[0037] Currently, models based on gait-based 3D skeleton coordinate data features can be categorized into machine learning and deep learning methods. In machine learning, a representative study utilizes a Kinect camera to acquire gait representations of patients with depression, then extracts the kinetic and potential energy of the motion to obtain robust features, and employs Support Vector Machines (SVMs) for depression identification. With increased computing power, end-to-end deep learning methods have gradually become mainstream. One study uses a Spatial-Temporal Graph Convolutional Network (SPAC) to extract spatiotemporal features from gait sequences, thereby identifying the pedestrian's current emotion (angry, sad, happy, normal). Furthermore, it utilizes a Variational Autoencoder (VAE) to generate gait data for data augmentation, enhancing the model's generalization ability. Some studies utilize Gated Recurrent Units (GRUs) to acquire joint features over the entire time series, representing these features using linear units. The outputs from the arms, legs, and torso are aggregated into separate linear layers based on the characteristics of different joints. An attention mechanism is then introduced to focus on different features in different body parts, ultimately classifying pedestrian emotions. In gait research for depression, some studies have used spatiotemporal graph convolutional neural networks to build a basic model, then employed methods such as rotation, channel masking, and mutual information to augment data, improving the model's generalization performance and further enhancing diagnostic accuracy.

[0038] This application provides a possible implementation method, such as... Figure 1 The diagram shows a flowchart of a depression risk assessment method based on multimodal gait feature fusion. This method can be executed by any electronic device, optionally on a server or a terminal device.

[0039] like Figure 1 As shown, the method may include the following steps:

[0040] Step 101: Construct a basic model for depression risk assessment, which is used to extract global features of 3D skeleton coordinate data in the spatial dimension using a multi-head self-attention mechanism, and to extract multi-scale time series features using multi-scale temporal convolution.

[0041] The depression risk assessment model in this embodiment comprises two parts: a spatial multi-head self-attention module and a multi-scale temporal convolution module. The spatial multi-head self-attention module utilizes a multi-head self-attention mechanism to extract global features from 3D skeleton coordinate data in the spatial dimension, while the multi-scale temporal convolution module uses multi-scale temporal convolution to extract multi-scale temporal series features. The spatial multi-head self-attention module and the multi-scale temporal convolution module are decoupled from each other, respectively solving the problems of spatial global feature extraction and multi-scale temporal feature modeling.

[0042] Step 102: Extract unique features of multiple isomorphic modalities and shared features that integrate features of multiple isomorphic modalities based on the depression risk assessment base model, and maximize the mutual information between multiple unique features and shared features through a contrastive distillation learning method.

[0043] By using contrastive distillation learning to maximize the mutual information between multiple unique features and shared features, feature fusion can be achieved. The final shared feature is a fusion of the output features of four unique features, which can improve the accuracy of the output shared feature.

[0044] Step 103: Construct a spatiotemporal feature map and a spatiotemporal attention map based on the RGB image data and shared features, and extract RGB image features based on the spatiotemporal feature map and the spatiotemporal attention map.

[0045] In this embodiment, an RGB pre-trained model is used to extract RGB image features. By constructing a spatiotemporal attention map through shared features, the RGB pre-trained model can be guided to focus on more discriminative RGB image features.

[0046] Step 104: Use a contrastive learning method to align RGB image features and shared features, and output depression prediction results based on RGB image features and shared features.

[0047] Contrastive learning methods can align the image of each part in the RGB image features with the skeleton coordinates of each part in the shared features, ensuring semantic consistency of the output while preserving the diversity of heterogeneous modal features.

[0048] Based on the above steps, a depression prediction result is output according to RGB image features and shared features. The method proposed in this application embodiment can model spatial and temporal features to obtain a depression prediction result Y = {depression, health}. Essentially, the method in this application embodiment is based on a segment of 3D skeleton coordinate data with a certain temporal sequence, and learns the implicit mapping relationship between gait and depression by maximizing the probability P(Y|X).

[0049] In this embodiment, a basic model for depression risk assessment is constructed. This model utilizes a multi-head self-attention mechanism to extract global features from 3D skeleton coordinate data in the spatial dimension and employs multi-scale temporal convolution to extract multi-scale time-series features. At the isomorphic modality level, unique features of multiple isomorphic modalities and shared features that fuse multiple isomorphic modal features are extracted based on the basic model. A contrastive distillation learning method is used to maximize the mutual information between the unique and shared features. The shared features fuse multiple isomorphic modal features from the 3D skeleton coordinate data, significantly reducing the number of parameters. At the heterogeneous modality level, spatiotemporal feature maps and spatiotemporal attention maps are constructed based on RGB image data and shared features, respectively. RGB image features are extracted based on these maps. A contrastive learning method is used to align the RGB image features and shared features, achieving fusion and semantic alignment of 3D skeleton coordinate data and RGB image data at the attention level. Finally, depression prediction results are output based on the RGB image features and shared features, achieving a reliable depression risk assessment that can objectively assist medical personnel in diagnosing depression.

[0050] In an optional embodiment, the depression risk assessment base model includes a spatial multi-head self-attention module and a multi-scale temporal convolution module. The construction of the depression risk assessment base model includes:

[0051] A spatial multi-head self-attention module is constructed using a multi-head self-attention mechanism and a multilayer perceptron to model the global features of 3D skeleton coordinate data in the spatial dimension.

[0052] Using the output of the spatial multi-head self-attention module as input, a multi-scale temporal convolution module is constructed; the multi-scale temporal convolution module uses multiple different branches to obtain multi-scale time series features.

[0053] As a first example, given a gait sequence of T frames as 3D skeleton coordinate data, using a vertex set S = {s1, s2, ..., s...} N} represents the 3D joint position (or other isomorphic modal) of a certain frame, where N represents the number of vertices, and for each vertex s i The feature dimension is set to C. Therefore, a gait sequence with T frames can be represented as follows:

[0054] Figure 2 A schematic diagram of the 3D skeleton coordinate feature extraction module is shown. (Refer to...) Figure 2 In the spatial dimension, graph convolutional networks can only model local features of key points and their neighbors, lacking global features. Therefore, we introduce a multi-head self-attention mechanism and a multilayer perceptron (MLP) to construct a spatial multi-head self-attention module. Figure 2The spatial Transformer module within the [database / module name]. In the spatial dimension, the 3D skeleton gait coordinate data is represented by a vertex set S = {s1, s2, ..., s...}. N} represents the number of vertices, where N represents the number of vertices, and for each vertex s i The feature dimension is set to C.

[0055] In an optional embodiment, a spatial multi-head self-attention module is constructed using a multi-head self-attention mechanism and a multilayer perceptron to model global features of 3D skeleton coordinate data in the spatial dimension, including:

[0056] The embedded features are randomly position-encoded to obtain the first latent variable;

[0057] The latent variable is input into the multi-head self-attention to obtain the second latent variable;

[0058] The second latent variable is processed by the first residual and normalization process and then input into the multilayer perceptron. After the second residual and normalization process, the global features of the 3D skeleton coordinate data are output.

[0059] The specific process is as follows:

[0060] Reference Figure 3 Vertex feature S (in Figure 3 The Chinese character is represented as Feature embedding is achieved through an embedding layer, where S∈Bacthsize(Bs)×T×N×C, Batchsize is the batch size during training (abbreviated as Bs in the figure), T is the time series, N is the number of vertices, and C is the feature dimension. For ease of training, the batch size during training is modified to Bacthsize(Bs)×T. Since the spatial multi-head self-attention module uses a self-attention mechanism to replace the human body topology and locality relied upon by the graph convolutional network, the spatial location information of these key points is lost, causing the model to be unable to determine the relative and absolute location information of each key point in the human body topology. Therefore, this embodiment introduces a randomly located encoding embedding feature to generate the first latent variable, Pos, as the location encoding. Specifically, it is shown in the following formula:

[0061] H = Embedding layer (V) + Position encoding (1)

[0062] Step 1.2. Map the first latent variable to the Queries vector, Keys vector, and Values ​​vector through three preset matrices, and split the Queries vector, Keys vector, and Values ​​vector into multiple parts in the channel dimension to generate multiple self-attention heads.

[0063] Through step 1.1 above, the first latent variable H is obtained, with a feature size of N×C'. This embodiment uses three preset trainable parameter matrices. The vectors (of size C×C') are mapped to obtain Queries(Q), Keys(K), and Values(V), as shown in equation (2). In this embodiment, the above vectors are split into h parts along the channel dimension, where h represents the number of self-attention heads, and the feature shape is transformed into N×h×C', where C' = C' / h.

[0064]

[0065] Step 1.3. Calculate the correlation score between each vertex in the self-attention head and convert the correlation score into a probability distribution; where the vertex includes the skeleton joints in the 3D skeleton coordinate data.

[0066] Specifically, after multi-head splitting, the correlation score between each vertex in the i-th self-attention head is calculated, i.e. It is an h×N×N matrix. Next, the Softmax function is used to transform the score vector Score between each skeleton joint into a probability distribution between [0,1]. Specifically, the following formula is used:

[0067]

[0068] Step 1.4. Obtain the self-attention head features based on the probability distribution between vertices and the corresponding Values ​​vector.

[0069] Specifically, the probability distribution Score between each skeleton joint is multiplied by the corresponding vector V. i The probability distribution matrix Score(H×N×N) and V i Perform a dot product of (H×N×C'') to obtain the corresponding self-attention feature, as shown in the following formula:

[0070] head i =Score i ×V i (4)

[0071] Step 1.5. After concatenating the self-attention head features corresponding to multiple self-attention heads, perform feature mapping through a linear layer to obtain the feature vector.

[0072] Among them, through a linear layer W O Feature mapping is performed, and the output Z is an N×C' dimensional feature vector, as shown in equation (5):

[0073] Z = [head1, head2, ..., head kW O (5)

[0074] Step 1.6. Multi-head self-attention mechanism to obtain the second latent variable (feature vector) Z, then as follows Figure 3 As shown, the latent variable Z undergoes a first residual and normalization process through a residual and normalization layer, and is then fed into a multilayer perceptron (MLP) for a second residual and normalization process, outputting the final second latent variable Z. This is illustrated in the following equation:

[0075] Z = residual & normalization(multilayer perceptron(residual & normalization(Z))) (6)

[0076] like Figure 3 As shown, except for step 1, steps 1.2-1.6 are repeated 9 times, that is... Figure 3 In this case, l ≤ 9. Therefore, global features of the 3D skeleton gait coordinate data can be modeled.

[0077] Reference Figure 2 and Figure 4 In terms of the time dimension, to address the periodicity of temporal sequences, this application employs multi-scale temporal convolution. For example, four branches can be constructed, each using different kernel sizes, dilation coefficients, and max pooling operations to obtain multi-scale temporal features. Specifically, branch 1 uses a 5×1 convolution with a dilation coefficient of 1 to capture longer temporal features; branch 2 uses a 5×1 convolution with a dilation coefficient of 2 to capture even longer temporal features; branch 3 uses 3×1 max pooling to capture short-term features; and branch 4 uses a 1×1 convolution to preserve the original abstract features. The input to the multi-scale temporal convolution module is the output of the aforementioned spatial multi-head self-attention module. The operation is performed only on the time dimension T to obtain the output.

[0078] The depression risk assessment model in this embodiment utilizes cross-entropy loss L ce and ground truth tag Supervised prediction output y i As shown in the following formula:

[0079]

[0080] In machine learning, "ground truth" refers to the accuracy of the training set in classifying supervised learning techniques.

[0081] In an optional embodiment, unique features of multiple isomorphic modalities and shared features that fuse features from multiple isomorphic modalities are extracted based on a depression risk assessment baseline model. The mutual information between the unique and shared features is maximized using a contrastive distillation learning method, including:

[0082] Multiple unique feature models and one shared feature model are constructed using the basic model for depression risk assessment.

[0083] Multiple isomorphic modes are input into multiple unique feature models, and the unique features of the multiple isomorphic modes are output.

[0084] Multiple isomorphic modalities are concatenated and input into a shared feature model to obtain shared features as output.

[0085] A contrastive distillation learning method is used to extract correlations from multiple unique and shared features, thereby achieving feature distillation and fusion.

[0086] As a second example, such as Figure 2 As shown, in this embodiment, four isomorphic modalities—joint coordinates, bone coordinates, joint velocity, and bone velocity—are input into the unique feature model (encoder) to obtain corresponding abstract unique features. These four types of data (e.g., joint coordinates, bone coordinates, joint velocity, and bone velocity) are then concatenated in a dimensional manner. Figure 5 As shown), the input feature model (encoder) yields the corresponding abstract shared features. Among these, the unique feature models of the four isomorphic modalities extract the feature information unique to each modality, i.e., f. J f B f J-V f B-V The encoded shared feature model, which concatenates features from four modalities, yields the encoded f. concat In this embodiment of the application, it is believed that it contains richer common feature information among the four modes.

[0087] Contrastive learning is a self-supervised learning method used to learn general features of a dataset by having the model learn which data points are similar or different without labels. This results in a more uniform sample space, bringing similar samples closer together and dissimilar samples further apart. In this embodiment, through contrastive distillation learning, based on rich shared feature information, the unique feature models of four isomorphic modalities are considered as "teacher models," and the shared feature model combining features from the four modalities is considered as "student models." However, there is no teacher-student relationship between them; rather, it represents the concept of twin models in distillation learning, aiming to fuse different features. Therefore, this embodiment utilizes the contrastive learning loss function to make the rich shared features in the feature space similar to the unique features, achieving feature distillation and fusion. The specific implementation steps are as follows:

[0088] like Figure 2 As shown in the dashed box on the right, the 3D skeleton coordinate data includes four isomorphic modes and spliced ​​mode data (i.e., data that splices together the four isomorphic modes). The logits vector (f) is obtained by encoding using unique feature models and shared feature models.J f B f J-V f B-V f concat The vector is of size M×C', where M is the batch size during training and C' is the vector dimension. This vector represents the abstract features of the original data mapped into the feature space under the constraint of cross-entropy.

[0089] In an optional embodiment, a contrastive distillation learning method is employed to extract correlations from multiple unique and shared features, achieving feature distillation and fusion, including:

[0090] Shared features are paired with unique features corresponding to the same 3D skeleton coordinate data sample to form positive pairs, and shared features are paired with unique features corresponding to different 3D skeleton coordinate data samples to form negative pairs.

[0091] The correlation between shared features and unique features in positive and negative pairs is calculated using a pre-defined contrastive loss function.

[0092] Optionally, taking the 3D joint position features in isomorphic modes as an example, in a batch of size M, a certain data sample i... Vector and Forming positive pairs, along with the remaining data samples in the same batch, such as... Forming negative pairs, where i, j ∈ M, i ≠ j. We can obtain the following comparison set:

[0093]

[0094] Similarly, for a given batch size of data, the comparison sets for the other three isomorphic modes are constructed as follows:

[0095]

[0096]

[0097]

[0098] This application employs contrastive distillation learning for feature fusion. Unlike previous student-teacher knowledge distillation methods that force the output of the student model S to precisely mimic the soft / hard targets generated by the teacher model T, the main idea of ​​contrastive distillation learning is to allow the student model to learn a representation that brings similar sample pairs closer together in certain metric spaces and separates the representations between dissimilar sample pairs. This application enables the transfer of unique features from isomorphic modalities to shared feature models, extracting correlation and dissimilarity information from the output distribution of networks with multiple isomorphic modalities. Formally, the contrastive learning loss function L... cl(Contrasive Loss) is expressed in the following form:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104] Where, sim(u, v) = u T v / ||u||||v|| represents the cosine similarity between vectors u and v, and τ is the temperature coefficient.

[0105] In an optional embodiment, the RGB image data is video data. A spatiotemporal feature map and a spatiotemporal attention map are constructed based on the RGB image data and shared features, respectively. RGB image features are then extracted based on the spatiotemporal feature map and the spatiotemporal attention map, including:

[0106] Multiple frames are extracted from RGB image data at equal time intervals, and multiple pre-specified body parts are captured to construct a spatiotemporal feature map.

[0107] The shared features are subjected to average pooling and max pooling operations according to different body parts to obtain the corresponding global features and local features.

[0108] Construct a spatiotemporal attention map based on global and local features;

[0109] RGB image features are extracted based on spatiotemporal attention maps and spatiotemporal feature maps.

[0110] As a third example, see Figure 6 In this embodiment, RGB image data (video data) is processed at equal time intervals to extract T' frames, and P important body parts (hands, legs, and torso) are extracted to construct a spatiotemporal feature map, thereby reducing training time and the number of model parameters and improving efficiency. Optionally, in a specific implementation, T' = 5 and P = 5.

[0111] While spatiotemporal feature maps significantly reduce the data size, they also lose some information. Therefore, this application embodiment utilizes highly accurate output-based shared features to construct a spatiotemporal attention map. The specific implementation steps are as follows:

[0112] like Figure 6 as well as Figure 7As shown, in the spatial dimension, the shared features S obtained by the 3D skeleton coordinate feature extraction module are summed after average pooling and max pooling operations are performed on the left and right sides, left and right legs, and torso, respectively, using a spatiotemporal pseudo-alignment module to obtain S', which has a size of T×P×C'. In the temporal dimension, a one-dimensional convolution is used to map the temporal length T to T', and the resulting S" has a size of T'×P×C' to achieve pseudo-alignment with the RGB features, where T'=5 and P=5.

[0113] Next, the output features S” obtained from the above modules are subjected to average pooling and linear function normalization (min-max normalization) operations on the channels to obtain the spatiotemporal attention map. The min-max normalization formula is as follows:

[0114]

[0115] At this point, the size of the RGB image data X is T'×P×C', the size of the spatiotemporal attention map α is T'×P, and the output O = X⊙α, where ⊙ represents the Hadamard product.

[0116] In this embodiment, a robust spatiotemporal attention map is extracted based on an RGB pre-trained model, thereby guiding the RGB encoder to focus on more discriminative RGB image patches to obtain RGB image features. Optionally, a pre-trained ResNet18 model is used as the RGB pre-trained model.

[0117] In an optional embodiment, aligning the RGB image features and the shared features using a contrastive learning method includes:

[0118] The RGB image features are encoded into RGB image vectors through an encoder and a pooling layer.

[0119] The shared features are encoded into skeleton coordinate vectors through an encoder and a pooling layer;

[0120] Positive and negative pairs are formed based on the RGB image vector and the skeleton coordinate vector, and a preset contrast loss function is used to align the RGB image vector and the skeleton coordinate vector.

[0121] As a fourth example, RGB image features RGB image vectors are obtained through encoder and pooling layers, sharing features. The skeleton coordinate vector is obtained through encoding by an encoder and pooling layers. The size of the RGB image vector and / or skeleton coordinate vector can be represented as M×P×C', where M is the batch size during training, P is the number of body segments, and C' is the vector dimension. This vector represents the abstract features of the original data mapped into the feature space under the constraint of cross-entropy. Taking the left hand as an example, in a batch of size M, a data sample i... Vector and They form positive pairs with each other, and with the other data samples in the same batch, such as These pairs of positive and negative examples form negative pairs, where i, j ∈ M, and i ≠ j. The spatiotemporal feature map and spatiotemporal attention map are aligned using these positive and negative pairs, as shown below. Figure 8 As shown.

[0122] Among them, the contrastive learning loss function L for different parts of the body pcl The calculation is as follows:

[0123]

[0124]

[0125] Figure 8 In this context, Contrastive Loss refers to the contrast loss during the alignment of different body parts, i.e., the contrastive learning loss function L. pcl The value of .

[0126] By employing multi-part contrastive learning, the semantics of the head, hands, legs, and torso are implicitly aligned in the feature space between 3D skeleton coordinate features (shared features) and RGB image features, ensuring semantic consistency of the output while preserving the diversity of heterogeneous modal features.

[0127] In this embodiment of the application, the outputs of the 3D skeleton coordinate feature extraction model and the RGB pre-trained model are fused using ensemble learning. Specifically, the first classification feature (y) for classification can be obtained based on the RGB image features output by the RGB pre-trained model and the shared features output by the 3D skeleton coordinate feature extraction model. RGB ) and second classification features (y 骨架 The system uses a weighted fusion of the first and second classification features to output a depression prediction result, such as depression or health.

[0128] This application, for the first time, extends the spatiotemporal graph convolutional model to include a spatial multi-head self-attention module and a multi-scale temporal module for modeling 3D skeleton coordinate data, successfully applying it to the field of depression risk assessment. It overcomes the shortcomings of existing spatiotemporal graph convolutional models in local modeling, and its ability to model temporal gait features is greatly enhanced.

[0129] The comparative distillation learning framework used in this application integrates isomorphic modal features of 3D skeleton coordinate data, which significantly reduces the number of parameters, and the accuracy of depression risk assessment of the obtained single model is significantly improved compared with isomorphic modal models with the same number of parameters.

[0130] In addition, this application's embodiments are the first to employ a multimodal model in gait-based depression risk assessment. The proposed attention mechanism and contrastive learning-based feature fusion and semantic alignment methods achieve feature fusion and alignment of heterogeneous modalities (RGB and 3D skeleton coordinate data) at both the attention and feature levels. Through ensemble learning, the outputs of heterogeneous modalities are integrated, further improving the accuracy of depression risk assessment and enhancing model robustness.

[0131] To verify the effectiveness and advancement of the method proposed in this application, extensive experiments were conducted on a dataset of depression patients from Shenzhen People's Hospital (data from 656 subjects, including 358 healthy subjects and 298 diagnosed depression patients; unique and shared features were extracted, resulting in 9046 segments of 3D skeleton coordinates and RGB image data suitable for training). The results were compared with current state-of-the-art gait-based depression risk assessment methods. Automatic evaluation metrics (F1 Score, Accuracy) were used in the experiments. Experimental results show that the method in this application outperforms the best current gait-based depression risk assessment methods in terms of automatic evaluation metrics.

[0132] Furthermore, ablation experiments were conducted on the contrastive distillation learning framework, the multimodal attention mechanism, and the multimodal partial contrastive learning framework in this application. Without the contrastive distillation learning framework, the 3D keypoint positions (shared features, i.e., 3D skeleton coordinate features) with the highest accuracy in the isomorphic modality were used to provide an attention map for RGB image data, resulting in a decrease in the accuracy of depression risk assessment in the single RGB modality. Without the multimodal attention mechanism, compared to the RGB model guided by the attention map, the RGB model without the attention map showed a decrease in the accuracy of depression risk assessment. Without the multimodal partial contrastive learning framework, compared to the heterogeneous modality model with partial contrastive learning loss constraints, the heterogeneous modality model without partial contrastive learning loss constraints showed a decrease in the accuracy of depression risk assessment. This demonstrates the effectiveness of each module.

[0133] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application also provide a depression risk assessment device based on multimodal gait feature fusion, such as... Figure 9 As shown, the device includes:

[0134] Model building module 901 is used to build a basic model for depression risk assessment. It is used to extract global features of 3D skeleton coordinate data in the spatial dimension using a multi-head self-attention mechanism and to extract multi-scale time series features using multi-scale temporal convolution.

[0135] The feature fusion module 902 is used to extract unique features of multiple isomorphic modalities and shared features that fuse features of multiple isomorphic modalities based on the depression risk assessment base model, and to maximize the mutual information between multiple unique features and shared features through a contrastive distillation learning method.

[0136] The feature fusion and semantic alignment module 903 is used to construct a spatiotemporal feature map and a spatiotemporal attention map based on RGB image data and shared features, respectively, and to extract RGB image features based on the spatiotemporal feature map and the spatiotemporal attention map;

[0137] The feature fusion and semantic alignment module 903 is also used to align RGB image features and shared features using a contrastive learning method, and output depression prediction results based on RGB image features and shared features.

[0138] In this embodiment, the model building module constructs a basic model for depression risk assessment, which uses a multi-head self-attention mechanism to extract global features from 3D skeleton coordinate data in the spatial dimension and multi-scale temporal convolution to extract multi-scale time series features. At the isomorphic modality level, the feature fusion module extracts unique features from multiple isomorphic modalities and shared features that fuse features from multiple isomorphic modalities based on the basic model for depression risk assessment. It also maximizes the mutual information between multiple unique features and shared features through a contrastive distillation learning method. The shared features fuse multiple isomorphic modal features from the 3D skeleton coordinate data features, significantly reducing the number of parameters. At the heterogeneous modality level, the feature fusion and semantic alignment module constructs a spatiotemporal feature map and a spatiotemporal attention map based on RGB image data and shared features, respectively, and extracts RGB image features based on these maps. A contrastive learning method is used to align RGB image features and shared features, achieving fusion and semantic alignment of 3D skeleton coordinate data and RGB image data at the attention level. Finally, the depression prediction result is obtained based on the RGB image features and shared features, achieving a reliable depression risk assessment that can objectively assist medical personnel in diagnosing depression.

[0139] The depression risk assessment device based on multimodal gait feature fusion provided in this application embodiment can achieve... Figures 1 to 8 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0140] The depression risk assessment device based on multimodal gait feature fusion in this application embodiment can execute the depression risk assessment method based on multimodal gait feature fusion provided in this application embodiment. The implementation principle is similar. The actions performed by each module and unit in the depression risk assessment device based on multimodal gait feature fusion in each embodiment of this application correspond to the steps in the depression risk assessment method based on multimodal gait feature fusion in each embodiment of this application. For detailed functional descriptions of each module of the depression risk assessment device based on multimodal gait feature fusion, please refer to the descriptions in the corresponding depression risk assessment method based on multimodal gait feature fusion shown above. They will not be repeated here.

[0141] Based on the same principles as the methods shown in the embodiments of this application, embodiments of this application also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the depression risk assessment method based on multimodal gait feature fusion shown in any optional embodiment of this application by calling the computer program.

[0142] In an alternative embodiment, an electronic device, such as Figure 10 As shown, Figure 10 The illustrated electronic device 1000 can be a server, including a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the electronic device 1000 may also include a transceiver 1004. It should be noted that in practical applications, the transceiver 1004 is not limited to one type, and the structure of this electronic device 1000 does not constitute a limitation on the embodiments of this application.

[0143] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0144] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0145] The memory 1003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0146] The memory 1003 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1003 to implement the content shown in the foregoing method embodiments.

[0147] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0148] The server provided in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0149] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0150] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0151] It should be noted that the computer-readable storage medium described above in this application can also be a computer-readable signal medium or a combination of computer-readable storage media and computer-readable storage media. Computer-readable storage media can be, for example,—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0152] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0153] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0154] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the depression risk assessment method and apparatus based on multimodal gait feature fusion provided in the various optional implementations described above.

[0155] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself; for example, the model building module can also be described as "a model building module for constructing a basic model for depression risk assessment, for extracting global features of 3D skeleton coordinate data in the spatial dimension using a multi-head self-attention mechanism, and for extracting multi-scale time series features using multi-scale temporal convolution."

[0158] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A depression risk assessment method based on multi-modal gait feature fusion, characterized in that, The method comprises: constructing a depression risk assessment base model for extracting global features of 3D skeleton coordinate data in a spatial dimension using a multi-head self-attention mechanism and extracting multi-scale time series features using a multi-scale time convolution, the depression risk assessment base model comprising a spatial multi-head self-attention module and a multi-scale time convolution module, the construction of the depression risk assessment base model comprising: constructing the spatial multi-head self-attention module using a multi-head self-attention mechanism and a multi-layer perception mechanism for modeling global features of 3D skeleton coordinate data in a spatial dimension; constructing the multi-scale time convolution module using the output of the spatial multi-head self-attention module as input; the multi-scale time convolution module using multiple different branches to obtain multi-scale time series features; extracting unique features of multiple isomorphic modalities in the 3D skeleton coordinate data and shared features fused with the multiple isomorphic modalities in the 3D skeleton coordinate data according to the depression risk assessment base model, and maximizing the mutual information of the multiple unique features and the shared features through a contrastive distillation learning method; constructing a spatiotemporal feature map and a spatiotemporal attention map according to the RGB image data and the shared features, respectively, and extracting RGB image features according to the spatiotemporal feature map and the spatiotemporal attention map; aligning the RGB image features and the shared features using a contrastive learning method, and outputting a depression prediction result according to the RGB image features and the shared features.

2. The depression risk assessment method based on multi-modal gait feature fusion according to claim 1, characterized in that, The construction of the spatial multi-head self-attention module using a multi-head self-attention mechanism and a multi-layer perception mechanism for modeling global features of 3D skeleton coordinate data in a spatial dimension comprises: randomly encoding the embedded features to obtain a first latent variable; inputting the latent variable into a multi-head self-attention to obtain a second latent variable; inputting the second latent variable into the multi-layer perception mechanism after a first residual and normalization processing, and outputting the global features of the 3D skeleton coordinate data after a second residual and normalization processing.

3. The depression risk assessment method based on multi-modal gait feature fusion according to claim 1, characterized in that, The extraction of unique features of multiple isomorphic modalities in the 3D skeleton coordinate data and shared features fused with the multiple isomorphic modalities in the 3D skeleton coordinate data according to the depression risk assessment base model, and the maximization of the mutual information of the multiple unique features and the shared features through a contrastive distillation learning method comprises: constructing multiple unique feature models and a shared feature model using the depression risk assessment base model; inputting multiple isomorphic modalities into multiple unique feature models to output unique features of multiple isomorphic modalities; inputting the multiple isomorphic modalities into the shared feature model after splicing to output shared features; extracting correlations from the multiple unique features and the shared features using a contrastive distillation learning method to realize distillation and fusion of features.

4. The depression risk assessment method based on multi-modal gait feature fusion according to claim 3, characterized in that, The extraction of correlations from the multiple unique features and the shared features using a contrastive distillation learning method to realize distillation and fusion of features comprises: The shared feature and the unique feature corresponding to the same 3D skeleton coordinate data sample form a positive example pair, and the shared feature and the unique feature corresponding to different 3D skeleton coordinate data samples form a negative example pair; The correlation of the shared feature and the unique feature in the positive example pair and the negative example pair is calculated by using a preset contrast loss function.

5. The depression risk assessment method based on multi-modal gait feature fusion according to claim 1, characterized in that, The RGB image data is video data, and the spatio-temporal feature map and the spatio-temporal attention map are constructed according to the RGB image data and the shared feature respectively, and the RGB image feature is extracted according to the spatio-temporal feature map and the spatio-temporal attention map, including: Multiple frames are extracted from the RGB image data at equal intervals, and multiple pre-specified body parts are intercepted to construct a spatio-temporal feature map; The shared feature is respectively subjected to average pooling and maximum pooling operations according to different body parts to obtain corresponding global features and local features; The spatio-temporal attention map is constructed according to the global features and the local features; The RGB image feature is extracted according to the spatio-temporal attention map and the spatio-temporal feature map.

6. The depression risk assessment method based on multi-modal gait feature fusion according to claim 1, characterized in that, The RGB image feature and the shared feature are aligned by using a contrast learning method, including: The RGB image feature is encoded by an encoder and a pooling layer to obtain an RGB image vector; The shared feature is encoded by an encoder and a pooling layer to obtain a skeleton coordinate vector; The RGB image vector and the skeleton coordinate vector form a positive example pair and a negative example pair, and the RGB image vector and the skeleton coordinate vector are aligned by using a preset contrast loss function.

7. A depression risk assessment device based on multi-modal gait feature fusion, characterized in that, The device includes: A model construction module is configured to construct a depression risk assessment base model, which is configured to extract global features of 3D skeleton coordinate data in a spatial dimension by using a multi-head self-attention mechanism and extract multi-scale time series features by using a multi-scale time convolution. The depression risk assessment base model includes a spatial multi-head self-attention module and a multi-scale time convolution module. The construction of the depression risk assessment base model includes: constructing the spatial multi-head self-attention module by using a multi-head self-attention mechanism and a multi-layer perception mechanism, which is configured to model global features of 3D skeleton coordinate data in a spatial dimension; constructing the multi-scale time convolution module by using the output of the spatial multi-head self-attention module as input; and the multi-scale time convolution module is configured to obtain multi-scale time series features by using multiple different branches. A feature fusion module is configured to extract unique features of multiple isomorphic modalities in 3D skeleton coordinate data and shared features of multiple isomorphic modalities in 3D skeleton coordinate data according to the depression risk assessment base model, and maximize mutual information of the unique features and the shared features by using a contrast distillation learning method. A feature fusion and semantic alignment module is configured to construct a spatio-temporal feature map and a spatio-temporal attention map according to RGB image data and the shared feature respectively, and extract an RGB image feature according to the spatio-temporal feature map and the spatio-temporal attention map. The feature fusion and semantic alignment module also aligns the RGB image features and the shared features by using a contrast learning method, and outputs a depression prediction result according to the RGB image features and the shared features.

8. An electronic device, comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory and loadable on the processor, the processor implementing the method of any one of claims 1 to 6 when executing the program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images