Attention deficit hyperactivity disorder auxiliary identification method based on multi-modal data and related device

Abstract

The invention provides an attention deficit hyperactivity disorder auxiliary identification method based on multi-modal data and a related device. The method comprises the following steps: collecting eye movement data, head movement data, leg movement data and psychological test data of a subject; respectively converting the eye movement data, the head movement data, the leg movement data and the psychological test data into structured data; inputting the multiple pieces of structured data into a multi-modal neural network model, and outputting an attention deficit hyperactivity disorder auxiliary recognition result; wherein the multi-modal neural network model is obtained based on structured data training, the multi-modal data is combined to perform auxiliary detection on attention deficit hyperactivity disorder, the sensitivity and specificity of ADHD auxiliary diagnosis can be effectively improved, and the multi-modal neural network model is trained based on the structured data, so that the training speed is high, the efficiency is high, and the accuracy is high. The multi-modal neural network model in the invention has no strict requirement on the input data volume during use, and the accuracy of the identification classification result is further improved.

Description

Technical Field

[0001] The present invention relates to the field of smart medical technology, and in particular to an attention deficit hyperactivity disorder auxiliary identification method based on multimodal data and related devices. Background Art

[0002] Attention deficit and hyperactivity disorder (ADHD), commonly known as ADHD, is a common functional disorder in children. Clinically, it is mainly manifested as short attention span, hyperactivity, impulsivity, cognitive impairment, learning difficulties, etc. The clinical diagnosis of ADHD requires a doctor's interview and observation, parent or teacher scale assessment, cognitive and intelligence tests, and a series of auxiliary examinations. The entire diagnostic process consumes a lot of manpower and material resources and has high requirements for professionals. Among the auxiliary examinations, a series of paradigms such as continuous performance test (CPT) are used for psychological testing. The simple CPT test can be improved through multiple exercises, which will lead to inaccurate recognition results. In related technologies, other data such as head movement, eye movement and leg movement information are integrated, and a pre-trained language model based on a multi-layer Transformer encoder is used for ADHD recognition and classification. However, since the input data of the pre-trained language model based on the multi-layer Transformer encoder is time series data, the data is huge, the model training is complex, and the convergence is slow. In addition, the input data of the model is limited when it is used, which will also affect the accuracy of the recognition and classification results. Summary of the invention

[0003] The present invention provides an attention deficit hyperactivity disorder auxiliary identification method based on multimodal data and a related device, which are used to solve the defects of the traditional ADHD identification and classification method based on a pre-trained language model of a multi-layer Transformer encoder, such as complex model training and slow convergence, and limited input data when the model is used, which affects the accuracy of the identification and classification results.

[0004] The present invention provides an attention deficit hyperactivity disorder auxiliary identification method based on multimodal data, comprising: Collect the subjects' eye movement data, head movement data, leg movement data and psychological test data; converting the eye movement data, head movement data, leg movement data and psychological test data into structured data respectively; Input multiple structured data into the multimodal neural network model respectively, and output the auxiliary recognition result of attention deficit hyperactivity disorder; wherein the multimodal neural network model is trained based on the structured data.

[0005] According to the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the present invention, the psychological test data is converted into structured data, including: Establishing an XY coordinate system, wherein the positive direction of the X axis represents the reaction time when the target item appears, and the negative direction represents the reaction time when the interference item appears; the positive direction of the Y axis of the psychological test result distribution diagram represents the number of times the key is pressed, and the negative direction of the Y axis represents the number of times the key is not pressed; Analyze the psychological test data to obtain the number of correct feedbacks, number of missed alarms, number of false alarms and reaction time of the subjects; fill the number of correct feedbacks, number of missed alarms, number of false alarms and reaction time of the subjects into the corresponding quadrants of the XY coordinate system, and draw a two-dimensional distribution diagram of the psychological test results.

[0006] According to the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the present invention, the head movement data is converted into structured data, including: Initialize an initial three-dimensional head motion position distribution map with all values ​​​​as 0; The three-dimensional key points [x, y, z] of the subject's head movement are captured in real time by a camera; the three-dimensional key points are input into the initial head movement position distribution map, and the 0 values ​​at the corresponding positions of the initial head movement position distribution map are replaced by 1 to obtain a head movement position distribution map.

[0007] According to the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the present invention, the eye movement data is converted into structured data, including: Initialize an initial two-dimensional eye movement range heat map with all values ​​​​set to 0; The two-dimensional key points [x, y] of the subject's eye movement are captured in real time by a camera, and the two-dimensional key points are input into the initial eye movement range heat map. The value of the corresponding position in the initial eye movement range heat map increases with the increase in the number of times the same two-dimensional key points are input, thereby obtaining the eye movement range heat map.

[0008] According to the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the present invention, converting the leg movement data into structured data includes: Initialize an initial leg movement amplitude distribution graph with all values ​​​​as 0; The three-dimensional linear acceleration [x, y, z] of the subject's leg movement is captured in real time by a gyroscope, the three-dimensional linear acceleration is input into the initial leg movement amplitude distribution map, and the 0 value at the corresponding position in the initial leg movement amplitude distribution map is replaced by 1 to obtain the leg movement amplitude distribution map.

[0009] According to the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the present invention, the multimodal neural network model includes: Multiple structured data input channels, a feature extraction network connected to each input channel, a feature vector concatenation module, and a classification module; The input channel reads structured data and outputs two-dimensional and three-dimensional data; The feature extraction network uses a two-dimensional convolution layer, a two-dimensional batch normalization layer, and a two-dimensional downsampling layer to extract features from the two-dimensional data output by the input channel to obtain an N-dimensional feature vector, and uses a three-dimensional convolution layer, a three-dimensional batch normalization layer, and a three-dimensional downsampling layer to extract features from the three-dimensional data output by the input channel to obtain an N-dimensional feature vector; The feature vector concatenation module concatenates multiple N-dimensional feature vectors to obtain a 4N-dimensional concatenated feature vector; the classification module identifies the 4N-dimensional concatenated feature vector and outputs a classification result.

[0010] According to the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the present invention, the training method of the multimodal neural network model includes: Inputting structured training data into a multimodal neural network model and outputting a prediction classification result; the structured training data includes a psychological test result distribution map, a head movement position distribution map, an eye movement range heat map, and a leg movement amplitude distribution map; Calculate the loss value of the predicted classification result using a classification loss function; The parameters of the multimodal neural network model are optimized according to the loss value until the training end condition is met, and the trained multimodal neural network model is output.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for auxiliary identification of attention deficit hyperactivity disorder based on multimodal data as described in any one of the above items is implemented.

[0012] The present invention also provides an attention deficit hyperactivity disorder auxiliary diagnosis system based on multimodal data, comprising: Cameras, gyroscopes, and electronic devices such as those mentioned above; The camera is used to track the positions of key points on the face of the subject in real time and obtain eye movement data and head movement data; The gyroscope is used to track the subject's leg movement in real time and obtain leg movement data; The electronic device is used to present psychological test questions to the subject and receive psychological test data; The electronic device is also used to convert the eye movement data, head movement data, leg movement data and psychological test data into multimodal structured data; input the multimodal structured data into a multimodal neural network model, and output an attention deficit hyperactivity disorder auxiliary identification result.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for auxiliary identification of attention deficit hyperactivity disorder based on multimodal data described above is implemented.

[0014] The present invention provides an attention deficit hyperactivity disorder auxiliary identification method and related devices based on multimodal data, which collect eye movement data, head movement data, leg movement data and psychological test data of a subject; respectively convert the eye movement data, head movement data, leg movement data and psychological test data into structured data; respectively input a plurality of structured data into a multimodal neural network model, and output an attention deficit hyperactivity disorder auxiliary identification result; wherein the multimodal neural network model is obtained by training based on structured data, and the present invention combines multimodal data with attention deficit hyperactivity disorder auxiliary detection, which can effectively improve the sensitivity and specificity of ADHD auxiliary diagnosis, and the multimodal neural network model is trained based on structured data, with fast training speed and high efficiency, and the multimodal neural network model in the present invention has no strict requirements on the amount of input data when in use, further improving the accuracy of recognition and classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a flow chart of an attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a multimodal neural network model provided by an embodiment of the present invention; Figure 3 is a schematic diagram of an attention deficit hyperactivity disorder auxiliary identification process provided by an embodiment of the present invention; Figure 4 It is a functional structure diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Figure 1 A flowchart of an attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by an embodiment of the present invention, such as Figure 1 As shown, the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the embodiment of the present invention includes: Step 101, collecting the subject's eye movement data, head movement data, leg movement data and psychological test data; Step 102: convert the eye movement data, head movement data, leg movement data and psychological test data into structured data respectively; Step 103: input the multiple structured data into a multimodal neural network model respectively, and output an attention deficit hyperactivity disorder auxiliary identification result; wherein the multimodal neural network model is trained based on the structured data.

[0019] In the traditional auxiliary identification method of attention deficit hyperactivity disorder, ADHD identification and classification is performed by fusing head movement, eye movement and leg movement information, using a pre-trained language model based on a multi-layer Transformer encoder. However, since the input data of the pre-trained language model based on a multi-layer Transformer encoder is time series data, the data is huge, the model training is complex, and the convergence is slow. In addition, the input data of the model is limited when it is used, which will also affect the accuracy of the recognition and classification results.

[0020] The method for assisting in identifying attention deficit hyperactivity disorder based on multimodal data provided by an embodiment of the present invention collects eye movement data, head movement data, leg movement data and psychological test data of a subject; converts the eye movement data, head movement data, leg movement data and psychological test data into structured data respectively; inputs a plurality of structured data into a multimodal neural network model respectively, and outputs an assisting in identifying attention deficit hyperactivity disorder result; wherein the multimodal neural network model is obtained by training based on structured data, and the present invention combines multimodal data with assisting in detecting attention deficit hyperactivity disorder, which can effectively improve the sensitivity and specificity of assisting in diagnosis of ADHD, and the multimodal neural network model is trained based on structured data, with fast training speed and high efficiency, and the multimodal neural network model has no strict requirements on the amount of input data when in use, further improving the accuracy of identification and classification results.

[0021] Based on any of the above embodiments, the eye movement data, head movement data, leg movement data and psychological test data are converted into structured data respectively, including: For the psychological test data, a two-dimensional distribution diagram of the psychological test results is drawn based on factors such as the number of correct answers, the number of missed answers, the number of false alarms, and the reaction time of the subjects; For head movement data, since the head moves in three-dimensional space, a three-dimensional head movement position distribution map is drawn according to the movement position of the head; For eye movement data, draw a two-dimensional eye movement range heat map according to the eye movement situation; For the leg movement data, since the legs move in three-dimensional space, a three-dimensional leg movement amplitude distribution diagram is drawn according to the movement of the legs; Specifically, converting the psychological test data into structured data includes: Establishing an XY coordinate system, wherein the positive direction of the X axis represents the reaction time when the target item appears, and the negative direction represents the reaction time when the interference item appears; the positive direction of the Y axis of the psychological test result distribution diagram represents the number of times the key is pressed, and the negative direction of the Y axis represents the number of times the key is not pressed; Analyze the psychological test data to obtain the number of correct feedbacks, number of missed alarms, number of false alarms and reaction time of the subjects; fill the number of correct feedbacks, number of missed alarms, number of false alarms and reaction time of the subjects into the corresponding quadrants of the XY coordinate system, and draw a two-dimensional distribution diagram of the psychological test results.

[0022] In an embodiment of the present invention, the psychological test result distribution diagram is a four-quadrant bar graph. Because the psychological test includes a target item and a distractor, the subject needs to press a button when the target item appears, and not press a button when the distractor appears. The positive direction of the X-axis represents the reaction time when the target item appears, and the negative direction represents the reaction time when the distractor appears. The positive direction of the Y-axis represents the number of times the button is pressed, and the negative direction of the Y-axis represents the number of times the button is not pressed. Therefore, the upper right area of ​​the two-dimensional psychological test result distribution diagram is the first quadrant, which represents the time and number of times the subject presses correctly when the target item appears; the upper left area is the second quadrant, which represents the time and number of times the subject presses incorrectly when the distractor appears; the lower left area is the third quadrant, which represents the number of times the subject does not press a button when the distractor appears; the lower right area is the fourth quadrant, which represents the number of times the subject does not press a button when the target item appears. By drawing a two-dimensional psychological test result distribution diagram in this way, factors such as the correct number of times, the number of missed reports, the number of false reports, and the reaction time can be intuitively included, and the psychological test data is converted into structured data.

[0023] Converting the head movement data into structured data includes: Initialize an initial three-dimensional head motion position distribution map with all values ​​​​as 0; The three-dimensional key points [x, y, z] of the subject's head movement are captured in real time by a camera; the three-dimensional key points are input into the initial head movement position distribution map, and the 0 values ​​at the corresponding positions of the initial head movement position distribution map are replaced by 1 to obtain a head movement position distribution map.

[0024] For example, we first initialize a head movement position distribution map with a size of N×N×N (N=500) and all values ​​​​are 0. The three-dimensional key points [x, y, z] of the subject's head movement are captured in real time by a high-speed camera and input into the head movement position distribution map, and the value of the corresponding position is changed to 1. After inputting all the head movement key points, the head movement position distribution map is obtained. By drawing the head movement position distribution map, the subject's head movement can be intuitively recorded and converted into structured data.

[0025] Converting the eye movement data into structured data, including: Initialize an initial two-dimensional eye movement range heat map with all values ​​​​set to 0; The two-dimensional key points [x, y] of the subject's eye movement are captured in real time by a camera, and the two-dimensional key points are input into the initial eye movement range heat map. The value of the corresponding position in the initial eye movement range heat map increases with the increase in the number of times the same two-dimensional key points are input, thereby obtaining the eye movement range heat map.

[0026] In an embodiment of the present invention, the two-dimensional key points [x, y] of the subject's eye movement are captured in real time by a high-speed camera and input into the eye movement range heat map, and the value of the corresponding position increases with the increase in the number of times. After inputting all the eye movement key points, the eye movement range heat map is obtained. By drawing the eye movement range heat map, the subject's eye movement can be intuitively recorded and converted into structured data.

[0027] Converting the leg movement data into structured data includes: Initialize an initial leg movement amplitude distribution graph with all values ​​​​as 0; The three-dimensional linear acceleration [x, y, z] of the subject's leg movement is captured in real time by a gyroscope, the three-dimensional linear acceleration is input into the initial leg movement amplitude distribution map, and the 0 value at the corresponding position in the initial leg movement amplitude distribution map is replaced by 1 to obtain the leg movement amplitude distribution map.

[0028] In an embodiment of the present invention, a leg movement amplitude distribution map with a size of N×N×N (N=500) and all values ​​​​are initialized. The three-dimensional linear acceleration [x, y, z] of the subject's leg movement is captured in real time by the gyroscopes bound to both legs and input into the leg movement amplitude distribution map, and the value of the corresponding position is changed to 1. After inputting all the three-dimensional linear accelerations of the leg movements, the leg movement amplitude distribution map is obtained. By drawing the leg movement amplitude distribution map, the subject's leg movement can be intuitively recorded and converted into structured data.

[0029] Based on any of the above embodiments, the multimodal neural network model includes: Multiple structured data input channels, a feature extraction network connected to each input channel, a feature vector concatenation module, and a classification module; The input channel reads structured data and outputs two-dimensional and three-dimensional data; The feature extraction network uses a two-dimensional convolution layer, a two-dimensional batch normalization layer, and a two-dimensional downsampling layer to extract features from the two-dimensional data output by the input channel to obtain an N-dimensional feature vector, and uses a three-dimensional convolution layer, a three-dimensional batch normalization layer, and a three-dimensional downsampling layer to extract features from the three-dimensional data output by the input channel to obtain an N-dimensional feature vector; The feature vector concatenation module concatenates multiple N-dimensional feature vectors to obtain a 4N-dimensional concatenated feature vector; the classification module identifies the 4N-dimensional concatenated feature vector and outputs a classification result.

[0030] In the embodiment of the present invention, the ordinary neural network is modified to obtain a multimodal neural network model so that it can accept multimodal data input, such as Figure 2 As shown. For two-dimensional data, 2D convolutional layers, 2D batch normalization layers, 2D downsampling layers, etc. are used for processing; for three-dimensional data, 3D convolutional layers, 3D batch normalization layers, 3D downsampling layers, etc. are used for processing. In multimodal neural networks, for two-dimensional or three-dimensional data, networks including but not limited to Visual Geometry Group Net (VGGNet), U-shaped network (UNet), Residual Network (ResNet), etc. can be used for feature extraction. Then the data of each modality is calculated to obtain an N-dimensional (for example, 128-dimensional) feature vector, and the four N-dimensional feature vectors are spliced ​​to obtain a 4N-dimensional (for example, 512-dimensional) feature vector. Finally, this 4N-dimensional feature vector is used for identification to determine whether the subject suffers from ADHD.

[0031] Specifically, Figure 3As shown in the figure, the distribution map of psychological test results, the distribution map of head movement position, the heat map of eye movement range and the distribution map of leg movement amplitude obtained by data processing are sequentially input into the multimodal neural network. In the multimodal neural network, 2D neural network and 3D neural network are used for feature extraction respectively. Global average pooling is used at the end of each neural network to obtain 4 [1×N]-dimensional feature vectors. Then the 4 [1×N]-dimensional feature vectors are spliced ​​in sequence to obtain a [1×4N]-dimensional feature vector, and finally the predicted classification result is output.

[0032] Based on any of the above embodiments, the training method of the multimodal neural network model includes: Inputting structured training data into a multimodal neural network model and outputting a prediction classification result; the structured training data includes a psychological test result distribution map, a head movement position distribution map, an eye movement range heat map, and a leg movement amplitude distribution map; Calculate the loss value of the predicted classification result using a classification loss function; The parameters of the multimodal neural network model are optimized according to the loss value until the training end condition is met, and the trained multimodal neural network model is output.

[0033] There are three categories of predicted classification results, namely normal, suspected and confirmed, represented by labels 0, 1 and 2. The training loss uses conventional classification losses, such as cross-entropy loss and focal loss function.

[0034] The subjects of the embodiment of the present invention are children and adolescents aged 6 to 16 who may suffer from ADHD. 257 data were used, and the training set and the test set were divided into 8:2. On the test set, the sensitivity reached 90% and the specificity also reached 90%.

[0035] Compared with traditional technical solutions, the step of manually designing features is replaced by automatic feature extraction by neural networks, which can capture the deep structural information in the data and automatically extract useful features from the original data, thereby improving the accuracy of recognition.

[0036] Compared with the single CPT solution, the present invention has the following advantages: (1) It integrates more information, including the user's head movement, eye movement, and leg movement information, providing a more complete basis for diagnosis. Simple CPT can be improved through repeated practice, but direct physical reactions are difficult to control; (2) Change from manual feature extraction to automatic feature extraction. Single CPT only considers a few simple features such as the number of correct answers, the number of missed answers, the number of false alarms, and the reaction time, but the embodiment of the present invention automatically extracts high-dimensional features through a neural network, so better results can be achieved; Compared with the pre-trained language model based on the multi-layer Transformer encoder, the embodiment of the present invention has the following advantages: (1) The model has a simple structure, few parameters, fast convergence speed, convenient training, and requires few hardware resources; (2) The amount of data required for training is small, and a high-precision model can be obtained using less data.

[0037] The attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the embodiment of the present invention is based on neuropsychological test tasks, allows users to complete interactive tests, builds an interactive neuropsychological test and high-precision motion capture platform, and obtains high-precision multimodal data. For example, based on a camera and a gyroscope, high-precision capture of the subject's head movement, eye movement and leg movement can be achieved, the auxiliary diagnosis system components are streamlined, and deep learning models are used to mine deep features from multimodal data (CPT data, head movement data, eye movement data and leg movement data), which can effectively improve the sensitivity and specificity of ADHD auxiliary diagnosis.

[0038] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the server may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The memory 430 includes computer programs, operating systems and data, and the processor 410 can call the logic instructions in the memory 430 to execute the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data, the method comprising: collecting the eye movement data, head movement data, leg movement data and psychological test data of the subject; converting the eye movement data, head movement data, leg movement data and psychological test data into structured data respectively; inputting the multiple structured data into the multimodal neural network model respectively, and outputting the attention deficit hyperactivity disorder auxiliary identification result; wherein the multimodal neural network model is trained based on the structured data.

[0039] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0040] The embodiment of the present invention also provides an attention deficit hyperactivity disorder auxiliary diagnosis system based on multimodal data, comprising: a camera, a gyroscope, and an electronic device as described in the above embodiment; The camera is used to track the positions of key points on the face of the subject in real time and obtain eye movement data and head movement data; The gyroscope is used to track the subject's leg movement in real time and obtain leg movement data; The electronic device is used to present psychological test questions to the subject and receive psychological test data; The electronic device is also used to convert the eye movement data, head movement data, leg movement data and psychological test data into multimodal structured data; input the multimodal structured data into a multimodal neural network model, and output an attention deficit hyperactivity disorder auxiliary identification result.

[0041] In an embodiment of the present invention, the working principle of the ADHD auxiliary diagnosis system includes: the subject sits in front of a tablet computer, wears gyroscopes on both legs, and performs a neuropsychological test task. During the neuropsychological test task, the camera of the tablet computer captures the subject's head image, and uses a facial key point detection algorithm to track the position of the facial key points in real time to obtain eye movement data and head movement data. At the same time, the gyroscope tracks the leg movement in real time to obtain leg movement data.

[0042] The collected eye movement data, head movement data, leg movement data and psychological test data are processed because the neural network requires a unified and standardized data input format and processes data of different modalities separately.

[0043] The attention deficit hyperactivity disorder auxiliary diagnosis system based on multimodal data provided by the embodiment of the present invention is based on neuropsychological test tasks, allows users to complete interactive tests, builds interactive neuropsychological tests and high-precision motion capture platforms, and obtains high-precision multimodal data. For example, based on a camera and a gyroscope, high-precision capture of the subject's head movement, eye movement and leg movement can be achieved, the auxiliary diagnosis system components are streamlined, and deep features are mined from multimodal data (CPT data, head movement data, eye movement data and leg movement data) using a deep learning model, which can effectively improve the sensitivity and specificity of ADHD auxiliary diagnosis.

[0044] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data provided by the above-mentioned methods, the method comprising: collecting the subject's eye movement data, head movement data, leg movement data and psychological test data; converting the eye movement data, head movement data, leg movement data and psychological test data into structured data respectively; inputting multiple structured data into a multimodal neural network model respectively, and outputting attention deficit hyperactivity disorder auxiliary identification results; wherein the multimodal neural network model is trained based on the structured data.

[0045] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assisting the identification of attention deficit hyperactivity disorder based on multimodal data, characterized in that: include: Collect the subjects' eye movement data, head movement data, leg movement data and psychological test data; converting the eye movement data, head movement data, leg movement data and psychological test data into structured data respectively; Input multiple structured data into the multimodal neural network model respectively, and output the auxiliary recognition result of attention deficit hyperactivity disorder; Wherein, the multimodal neural network model is trained based on structured data.

2. The method for assisting in identifying attention deficit hyperactivity disorder based on multimodal data according to claim 1, characterized in that: Converting the psychological test data into structured data includes: Establishing an XY coordinate system, wherein the positive direction of the X axis represents the reaction time when the target item appears, and the negative direction represents the reaction time when the interference item appears; the positive direction of the Y axis of the psychological test result distribution diagram represents the number of times the key is pressed, and the negative direction of the Y axis represents the number of times the key is not pressed; Analyze the psychological test data to obtain the number of correct feedbacks, number of missed alarms, number of false alarms and reaction time of the subjects; fill the number of correct feedbacks, number of missed alarms, number of false alarms and reaction time of the subjects into the corresponding quadrants of the XY coordinate system, and draw a two-dimensional distribution diagram of the psychological test results.

3. The method for assisting in identifying attention deficit hyperactivity disorder based on multimodal data according to claim 1, characterized in that: Converting the head movement data into structured data includes: Initialize an initial three-dimensional head motion position distribution map with all values ​​​​as 0; The three-dimensional key points [x, y, z] of the subject's head movement are captured in real time by a camera; the three-dimensional key points are input into the initial head movement position distribution map, and the 0 values ​​at the corresponding positions of the initial head movement position distribution map are replaced by 1 to obtain a head movement position distribution map.

4. The method for assisting in identifying attention deficit hyperactivity disorder based on multimodal data according to claim 1, characterized in that: Converting the eye movement data into structured data, including: Initialize an initial two-dimensional eye movement range heat map with all values ​​​​set to 0; The two-dimensional key points [x, y] of the subject's eye movement are captured in real time by a camera, and the two-dimensional key points are input into the initial eye movement range heat map. The value of the corresponding position in the initial eye movement range heat map increases with the increase in the number of times the same two-dimensional key points are input, thereby obtaining the eye movement range heat map.

5. The method for assisting in identifying attention deficit hyperactivity disorder based on multimodal data according to claim 1, characterized in that: Converting the leg movement data into structured data includes: Initialize an initial leg movement amplitude distribution graph with all values ​​​​as 0; The three-dimensional linear acceleration [x, y, z] of the subject's leg movement is captured in real time by a gyroscope, the three-dimensional linear acceleration is input into the initial leg movement amplitude distribution map, and the 0 value at the corresponding position in the initial leg movement amplitude distribution map is replaced by 1 to obtain the leg movement amplitude distribution map.

6. The method for assisting in identifying attention deficit hyperactivity disorder based on multimodal data according to claim 1, characterized in that: The multimodal neural network model includes: Multiple structured data input channels, a feature extraction network connected to each input channel, a feature vector concatenation module, and a classification module; The input channel reads structured data and outputs two-dimensional and three-dimensional data; The feature extraction network uses a two-dimensional convolution layer, a two-dimensional batch normalization layer, and a two-dimensional downsampling layer to extract features from the two-dimensional data output by the input channel to obtain an N-dimensional feature vector, and uses a three-dimensional convolution layer, a three-dimensional batch normalization layer, and a three-dimensional downsampling layer to extract features from the three-dimensional data output by the input channel to obtain an N-dimensional feature vector; The feature vector concatenation module concatenates multiple N-dimensional feature vectors to obtain a 4N-dimensional concatenated feature vector; the classification module identifies the 4N-dimensional concatenated feature vector and outputs a classification result.

7. The method for assisting in identifying attention deficit hyperactivity disorder based on multimodal data according to claim 1, characterized in that: The training method of the multimodal neural network model comprises: Inputting structured training data into a multimodal neural network model and outputting a prediction classification result; the structured training data includes a psychological test result distribution map, a head movement position distribution map, an eye movement range heat map, and a leg movement amplitude distribution map; Calculate the loss value of the predicted classification result using a classification loss function; The parameters of the multimodal neural network model are optimized according to the loss value until the training end condition is met, and the trained multimodal neural network model is output.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the attention deficit hyperactivity disorder auxiliary identification method based on multimodal data as described in any one of claims 1 to 7 is implemented.

9. An attention deficit hyperactivity disorder auxiliary diagnosis system based on multimodal data, characterized in that: include: A camera, a gyroscope, and an electronic device as claimed in claim 8; The camera is used to track the positions of key points on the face of the subject in real time and obtain eye movement data and head movement data; The gyroscope is used to track the subject's leg movement in real time and obtain leg movement data; The electronic device is used to present psychological test questions to the subject and receive psychological test data; The electronic device is also used to convert the eye movement data, head movement data, leg movement data and psychological test data into multimodal structured data; input the multimodal structured data into a multimodal neural network model, and output an attention deficit hyperactivity disorder auxiliary identification result.

10. A non-transitory readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for auxiliary identification of attention deficit hyperactivity disorder based on multimodal data as described in any one of claims 1 to 7 is implemented.

Images