Prediction method of medical image reading interactive behavior based on mouse trajectory

Abstract

The present application relates to a method for predicting interactive behavior in medical image reading based on mouse trajectories. By segmenting and semantically annotating the interactive mouse trajectories during medical image reading, the local semantics of the mouse trajectories are recognized, effectively obtaining the interactive behavior information of the user at different time periods during the interactive process of medical image reading. By integrating the multi-feature information of the interactive behavior and the attention mechanism to predict the next interactive behavior, the utilization rate of the historical sequence information of the interactive behavior and the effectiveness of sequence modeling are improved, thereby enhancing the prediction performance. Starting from the perspective of optimizing the interactive process of medical image reading for doctors, the present application realizes the recognition and prediction of the user's medical image reading behavior based on the mouse interaction trajectory, providing a theoretical and experimental basis for the research and development of a new medical image reading system that can help radiologists improve the efficiency and accuracy of reading diagnosis to a greater extent.

Description

Technical Field

[0001] The present application relates to the field of cross-application of human-computer interaction, and specifically, to a method for predicting interactive behavior in medical image reading based on mouse trajectory. Background Art

[0002] In recent years, the daily workload of radiologists has increased due to factors such as increased public health awareness, an aging population, and the increasing complexity of diagnostic imaging technology. Reducing the burden of image reading for radiologists and helping them make more efficient imaging diagnostic inferences and decisions has become a hot topic of concern and holds significant social and clinical value. Digital image reading is now widely used in imaging departments at all levels of hospitals. While it has helped improve diagnostic efficiency and accuracy, it still cannot meet the growing demand for image reading. Almost all image reading tasks require doctors to perform a series of operations and judgments. If image reading systems can identify and predict user interaction intent, prepare in advance, and deliver the right information at the right time, they can better assist doctors in image diagnosis through various means, such as providing adaptive auxiliary tools and optimized interfaces. Mouse input remains the primary method for doctors to interact with image reading systems. Furthermore, collecting mouse interaction data can be easily implemented through software deployment, and the collection process is discreet and does not interfere with users' normal work. Therefore, at the current stage, analyzing the mouse interaction data during the user's image reading process to explore the human-computer interaction process is the most realistic, feasible and practical way. However, the existing technology rarely involves research on the above method. Summary of the Invention

[0003] In order to overcome at least one deficiency in the prior art, the present application provides a method for predicting medical image reading interactive behavior based on mouse trajectory.

[0004] In a first aspect, a method for predicting interactive behavior in medical image reading based on mouse trajectory is provided, comprising:

[0005] Obtaining mouse interaction data generated by the user during the medical image reading process; the mouse interaction data includes mouse interaction trajectory data and mouse interaction data attributes;

[0006] Segmenting the mouse interaction trajectory data according to the mouse interaction data attributes to obtain multiple trajectory subsequences;

[0007] Semantically annotate multiple trajectory subsequences to determine the user's semantic label sequence;

[0008] The semantic label sequence is input into the multi-feature sequence prediction network based on LSTM-Attention, and the interactive behavior prediction result is output.

[0009] In one embodiment, the mouse interaction trajectory data is segmented according to the mouse interaction data attributes to obtain multiple trajectory subsequences, including:

[0010] According to the AOI attribute in the mouse interaction data attribute, the mouse interaction trajectory data is preliminarily segmented to obtain multiple trajectory segments; the AOI attribute of all trajectory points in each trajectory segment is the same;

[0011] Each track segment is further segmented according to the Mouse Event Type attribute in the mouse interaction data attribute to obtain multiple track subsequences; all track points in each track subsequence have the same AOI attribute and the same Mouse Event Type attribute.

[0012] In one embodiment, semantic annotation is performed on multiple trajectory subsequences to determine a user's semantic label sequence;

[0013] Based on the mouse interaction data attributes and the pre-trained mouse trajectory recognition model, the coarse-grained category label of each trajectory subsequence is determined. The coarse-grained category label includes the system response behavior label, the active behavior label, and the background behavior label. All the system response behavior labels and active behavior labels are extracted as the user's coarse-grained category label.

[0014] Determine the user's fine-grained category label corresponding to the user's coarse-grained category label based on the mouse interaction data attributes;

[0015] According to the user's coarse-grained category label and the user's fine-grained category label, the user's semantic label sequence is determined.

[0016] In one embodiment, a coarse-grained category label for each trajectory subsequence is determined based on the mouse interaction data attributes and a pre-trained mouse trajectory recognition model, including:

[0017] According to the attributes of the mouse interaction data, determine whether the trajectory subsequence is a system response behavior;

[0018] If the trajectory subsequence is a system response behavior, obtain the system response behavior category as the system response behavior label of the trajectory subsequence;

[0019] If the trajectory subsequence is not a system response behavior, the pre-trained mouse trajectory recognition model is used to determine whether the trajectory subsequence is an active behavior or a background behavior. If the trajectory subsequence is an active behavior, the active behavior category of the trajectory subsequence is obtained as the active behavior label of the trajectory subsequence. If the trajectory subsequence is a background behavior, the background behavior is used as the background behavior label of the trajectory subsequence.

[0020] In one embodiment, determining whether a trajectory subsequence is an active behavior or a background behavior based on a pre-trained mouse trajectory recognition model includes:

[0021] Determine the mouse trajectory recognition task type of each trajectory subsequence according to the AOI attribute in the mouse interaction data attribute;

[0022] The trajectory subsequence is input into a pre-trained mouse trajectory recognition model corresponding to the mouse trajectory recognition task type of the trajectory subsequence to determine whether the trajectory subsequence is an active behavior or a background behavior.

[0023] In one embodiment, the semantic tag sequence is input into a multi-feature sequence prediction network based on LSTM-Attention to determine the interactive behavior prediction result, including:

[0024] The multi-feature sequence prediction network based on LSTM-Attention includes an embedding layer, a sequence modeling layer, an attention layer, and an output layer; the semantic label sequence includes the semantic label of each trajectory subsequence;

[0025] Based on the semantic label of each trajectory subsequence, multiple interactive behavior features corresponding to each trajectory subsequence are obtained. The interactive behavior features include prediction features, multiple additional features, and user ID features.

[0026] A fixed-length sliding window is used to select semantic labels of a fixed sequence length from the semantic label sequence as a semantic label subsequence. Multiple interaction behavior features corresponding to each semantic label in the semantic label subsequence are processed into a data structure, and the multiple additional features are taken as a whole and recorded as the overall feature to obtain the interaction behavior feature vector of the semantic label subsequence.

[0027] The interaction behavior feature vector of the semantic label subsequence is input into the multi-feature sequence prediction network based on LSTM-Attention, and the prediction score for each preset candidate interaction behavior is output;

[0028] The interaction behavior prediction result is determined based on the prediction score.

[0029] In one embodiment, the interaction behavior feature vector of the semantic label subsequence is input into a multi-feature sequence prediction network based on LSTM-Attention, and the prediction score for each preset candidate interaction behavior is output, including:

[0030] The interaction behavior feature vector of the semantic tag subsequence includes the interaction behavior feature corresponding to each semantic tag after data structure processing; the interaction behavior feature corresponding to each semantic tag after data structure processing includes prediction feature, overall feature and user number feature;

[0031] For each semantic tag, the predicted features and overall features are input into the embedding layer. The embedding layer embeds the predicted features to obtain an embedding result. The embedding result is multiplied by each additional feature in the overall features to obtain the multiplication result corresponding to each additional feature. The multiplication result corresponding to each additional feature is concatenated with the embedding result to obtain the encoding vector corresponding to the semantic tag.

[0032] Input the encoding vector corresponding to each semantic label into the sequence modeling layer for temporal modeling to obtain the vector after sequence modeling corresponding to each semantic label;

[0033] The sequence modeling vector corresponding to each semantic label is input into the fully connected layer of the attention layer. The tanh activation function is used to obtain the latent vector, which contains the attention probability of each semantic label. The latent vector is normalized to obtain the normalized attention probability corresponding to each semantic label. The normalized attention probability corresponding to each semantic label is used as the weight to perform weighted summation on the sequence modeling vector corresponding to each semantic label to obtain the sequence vector.

[0034] The sequence vector is input into the output layer, which concatenates the sequence vector and the user ID feature to obtain a concatenated sequence vector. The concatenated sequence vector is input into the fully connected layer of the output layer, and based on the Softmax activation function, the predicted score for each preset candidate interaction behavior is output.

[0035] In a second aspect, a device for predicting interactive behavior of medical image reading based on mouse trajectory is provided, comprising:

[0036] The mouse interaction data acquisition module is used to acquire the mouse interaction data generated by the user during the medical image reading process; the mouse interaction data includes mouse interaction trajectory data and mouse interaction data attributes;

[0037] A trajectory subsequence acquisition module is used to segment the mouse interaction trajectory data according to the mouse interaction data attributes to obtain multiple trajectory subsequences;

[0038] A semantic label sequence determination module is used to semantically annotate multiple trajectory subsequences and determine the user's semantic label sequence;

[0039] The interactive behavior prediction module is used to input the semantic label sequence into the multi-feature sequence prediction network based on LSTM-Attention and output the interactive behavior prediction results.

[0040] In one embodiment, the trajectory subsequence acquisition module is further configured to:

[0041] According to the AOI attribute in the mouse interaction data attribute, the mouse interaction trajectory data is preliminarily segmented to obtain multiple trajectory segments; the AOI attribute of all trajectory points in each trajectory segment is the same;

[0042] Each track segment is further segmented according to the Mouse Event Type attribute in the mouse interaction data attribute to obtain multiple track subsequences; all track points in each track subsequence have the same AOI attribute and the same Mouse Event Type attribute.

[0043] In one embodiment, the semantic tag sequence determination module is further configured to:

[0044] Based on the mouse interaction data attributes and the pre-trained mouse trajectory recognition model, the coarse-grained category label of each trajectory subsequence is determined. The coarse-grained category label includes the system response behavior label, the active behavior label, and the background behavior label. All the system response behavior labels and active behavior labels are extracted as the user's coarse-grained category label.

[0045] Determine the user's fine-grained category label corresponding to the user's coarse-grained category label based on the mouse interaction data attributes;

[0046] According to the user's coarse-grained category label and the user's fine-grained category label, the user's semantic label sequence is determined.

[0047] Compared with the prior art, this application has the following beneficial effects:

[0048] (1) This application starts from the perspective of optimizing the interactive process of medical image reading by doctors, and realizes the recognition and prediction of users' medical image reading behavior based on the mouse interaction trajectory, providing a theoretical and experimental basis for the research and development of a new medical image reading system that can help radiologists improve the efficiency and accuracy of medical image reading diagnosis to a greater extent.

[0049] (2) This application realizes the recognition of local semantics of mouse trajectory by segmenting and semantically annotating the mouse interaction trajectory of medical image reading, and effectively obtains the user's interactive behavior information at different time periods during the medical image reading interaction process.

[0050] (3) This application predicts the next interactive behavior by integrating multi-feature information of interactive behaviors and the attention mechanism, thereby improving the utilization rate of historical sequence information of interactive behaviors and the effectiveness of sequence modeling, and improving the prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present application may be better understood by referring to the following description in conjunction with the accompanying drawings, which together with the following detailed description are incorporated into and form a part of this specification. In the drawings:

[0052] Figure 1 A method for predicting medical image reading interaction behavior based on mouse trajectory according to an embodiment of the present application is shown;

[0053] Figure 2 The figure shows the visualization result of segmenting the mouse interaction trajectory data according to an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a process for determining a coarse-grained category label and a fine-grained category label for each trajectory subsequence according to an embodiment of the present application is shown;

[0055] Figure 4 The figure shows a structural block diagram of a device for predicting interactive behavior of medical image reading based on mouse trajectory according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] Exemplary embodiments of the present application are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.

[0057] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.

[0058] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.

[0059] The present invention discloses a method for predicting interactive behaviors in medical image reading based on mouse trajectories. The method first obtains mouse interaction data for medical image reading; segments the mouse interaction trajectories to obtain mouse structural trajectories with structural information; semantically annotates the mouse structural trajectories to obtain mouse semantic trajectories, wherein the semantic label of each trajectory subsequence is the user's fine-grained interactive behavior represented by this segment of the mouse trajectory; then, extracts a fine-grained interactive behavior sequence from the mouse semantic trajectory; and feeds the fine-grained interactive behavior sequence into a prediction network to predict the user's next fine-grained interactive behavior.

[0060] The prediction method of this application can fill the gap in the current research field of mouse interaction data in medical image reading, and provide a theoretical and experimental basis for the future research and development of new human-computer collaborative image reading systems; at the same time, this method no longer analyzes the mouse trajectory as a whole during the reading process, and realizes the recognition of the local semantics of the mouse trajectory; finally, this method can more reasonably model historical interaction sequences and improve network prediction performance.

[0061] The present invention provides a method for predicting interactive behavior of medical image reading based on mouse trajectory. Figure 1 The method for predicting medical image reading interaction behavior based on mouse trajectory according to an embodiment of the present application is shown. Figure 1 , methods include:

[0062] Step S1, obtaining mouse interaction data generated by a user during the process of reading medical images; the mouse interaction data includes mouse interaction trajectory data and mouse interaction data attributes;

[0063] In this step, the raw data generated by users during the medical image reading process is segmented to obtain raw mouse interaction data. This raw mouse interaction data is then preprocessed to obtain mouse interaction data. Here, the mouse interaction data includes multiple mouse interaction trajectory data, each representing the interaction data of a user reading a CXR image. The following method steps will only be described using one piece of mouse interaction data as an example. Mouse interaction trajectory data consists of multiple trajectory points, each of which has multiple mouse interaction data attributes, as shown in Table 1.

[0064] Table 1 Mouse interaction data attributes

[0065]

[0066]

[0067] Step S2, segmenting the mouse interaction trajectory data according to the mouse interaction data attributes to obtain multiple trajectory subsequences;

[0068] Specifically, in this step, the mouse interaction trajectory data can be preliminarily segmented according to the AOI attribute in the mouse interaction data attribute to obtain multiple trajectory segments; the AOI attribute of all trajectory points in each trajectory segment is the same; here, the AOI attribute represents the AOI type corresponding to the interaction coordinates, that is, the area of interest type.

[0069] Each trajectory segment is further segmented according to the Mouse Event Type attribute in the mouse interaction data attribute to obtain multiple trajectory subsequences, and multiple trajectory subsequences constitute a structural trajectory; all trajectory points in each trajectory subsequence have the same AOI attribute and the same Mouse Event Type attribute. Figure 2 The visualization result of segmenting the mouse interaction trajectory data according to an embodiment of the present application is shown.

[0070] Step S3, semantically annotating multiple trajectory subsequences to determine the user's semantic label sequence;

[0071] In this step, the user's semantic label sequence includes multiple semantic labels. Each semantic label corresponds to a trajectory subsequence and has four attribute values, namely, the coarse-grained category label, the fine-grained category label, the start time of the trajectory subsequence, and the end time of the trajectory subsequence.

[0072] In step S4, the semantic label sequence is input into the multi-feature sequence prediction network based on LSTM-Attention, and the interactive behavior prediction result is output.

[0073] In this embodiment, by segmenting and semantically annotating the mouse interaction trajectory during medical image reading, the local semantics of the mouse trajectory is recognized, and the user's interaction behavior information at different time periods during the medical image reading interaction process is effectively obtained; by fusing the multi-feature information of the interaction behavior and the attention mechanism to predict the next interaction behavior, the utilization rate of the historical sequence information of the interaction behavior and the effectiveness of sequence modeling are improved, thereby enhancing the prediction performance.

[0074] In one embodiment, in step S3, semantic annotation is performed on multiple trajectory subsequences to determine the user's semantic label sequence, including:

[0075] Step S31: Determine a coarse-grained category label for each trajectory subsequence based on the mouse interaction data attributes and the pre-trained mouse trajectory recognition model; the coarse-grained category label includes a system response behavior label, an active behavior label, and a background behavior label; extract all system response behavior labels and active behavior labels as the user's coarse-grained category label;

[0076] Step S32, determining the user's fine-grained category label corresponding to the user's coarse-grained category label according to the mouse interaction data attributes;

[0077] In this step, the coarse-grained category labels of each trajectory subsequence in the user's coarse-grained category labels are refined, and the coarse-grained category labels of the user are further divided into finer-grained category labels based on the system interface attributes of the mouse trajectory corresponding to the coarse-grained category labels to represent finer-grained user interaction behavior semantics. For example, in order to refine the detected image operation-related behaviors, the anatomical region bounding boxes are manually annotated on the CXR images used for the experiment. Each bounding box is defined by its upper left corner point (X in the original CXR image coordinate system). X1 ,Y Y1 ) and the lower right corner (X X2 ,Y Y2 Combined with this supplementary data, the original mouse coordinate data relative to the system interface coordinate system is mapped to the CXR image coordinate system, thereby achieving the refinement of the interactive behavior such as "browsing the image" to "browsing the heart shadow area of the image".

[0078] Step S33: Determine the user's semantic tag sequence based on the user's coarse-grained category tag and the user's fine-grained category tag.

[0079] In one embodiment, Figure 3 A schematic diagram of the process of determining the coarse-grained category label and the fine-grained category label of each trajectory subsequence according to an embodiment of the present application is shown. Figure 3 In step S31, the coarse-grained category label of each trajectory subsequence is determined based on the mouse interaction data attributes and the pre-trained mouse trajectory recognition model, including:

[0080] Step S311, judging whether the trajectory subsequence is a system response behavior based on the mouse interaction data attributes;

[0081] In this step, the AOI and Mouse Event Type attributes in the mouse interaction data properties can be used to determine whether the trajectory subsequence is a system response behavior. Here, a system response behavior is an interactive action that the software system can recognize as a specific instruction and respond accordingly, such as a button click. A non-system response behavior is an interactive action that cannot be recognized as a clear instruction by the software system, such as browsing an image.

[0082] Step S312: if the trajectory subsequence is a system response behavior, obtain the system response behavior category as the system response behavior label of the trajectory subsequence;

[0083] In this step, while judging that the trajectory subsequence is a system response behavior based on the AOI attribute and the Mouse Event Type attribute in the mouse interaction data attributes, the system response behavior category can be determined. In this step, the determined system response behavior category can be further obtained as the system response behavior label of the trajectory subsequence.

[0084] Step S313: If the trajectory subsequence is not a system response behavior, determine whether the trajectory subsequence is an active behavior or a background behavior based on the pre-trained mouse trajectory recognition model; and when the trajectory subsequence is an active behavior, obtain the active behavior category of the trajectory subsequence as the active behavior label of the trajectory subsequence; when the trajectory subsequence is a background behavior, the background behavior is used as the background behavior label of the trajectory subsequence.

[0085] In one embodiment, in step S313, determining whether a trajectory subsequence is an active behavior or a background behavior based on a pre-trained mouse trajectory recognition model includes:

[0086] First, the mouse trajectory recognition task type of each trajectory subsequence is determined based on the AOI attribute in the mouse interaction data attributes. Here, while determining the mouse trajectory recognition task type of each trajectory subsequence based on the AOI attribute in the mouse interaction data attributes, the active behavior category of the trajectory subsequence can be determined, which serves as the active behavior label of the trajectory subsequence in step S33.

[0087] Then, the trajectory subsequence is input into a pre-trained mouse trajectory recognition model corresponding to the mouse trajectory recognition task type of the trajectory subsequence to determine whether the trajectory subsequence is an active behavior or a background behavior.

[0088] In this embodiment, the mouse movements that the user may use the mouse cursor to help him process cognitive information are called active behaviors. In contrast, the mouse movements that are only for moving to a certain area or unconsciously during the user interaction process are called background behaviors.

[0089] In the above embodiment, the process of pre-training the mouse trajectory recognition model to recognize the trajectory subsequence may include:

[0090] Given a trajectory subsequence T with a length of l (i.e. the number of trajectory points is l) sub ={γ i ,γ i+1 ,…,γ i+l-1}, where γ i Represents the i-th trajectory point. After sliding window sampling and mouse motion feature extraction, n feature vectors Mat can be obtained, which can be expressed as Mat = (vec1, vec2, ..., vec i ,…,vec n ) T , vec i =(p1,p2,…,p j ) is the i-th eigenvector, p jis the jth optimal feature selected, j is the number of optimal features selected, n=(lw) / s+1 is the number of feature vectors, w is the sliding window length, and s is the sliding step size. Table 2 shows the mouse motion features.

[0091] Then, the mouse trajectory recognition model is used to perform trajectory recognition on the n feature vectors to obtain the semantic labels of the n feature vectors. sub The semantic label type b of is ultimately determined by the majority label of its feature vector, and the formula is as follows:

[0092]

[0093] Among them, l active is the number of active behavior labels, l background is the number of background behavior labels.

[0094] Table 2 Mouse movement characteristics

[0095]

[0096]

[0097] The above-mentioned pre-trained mouse trajectory recognition model is obtained through training, and the sequence forward feature selection algorithm is used to select the optimal features during the training process.

[0098] In one embodiment, in step S4, the semantic tag sequence is input into a multi-feature sequence prediction network based on LSTM-Attention, and the interactive behavior prediction result is output, including:

[0099] The multi-feature sequence prediction network based on LSTM-Attention includes an embedding layer, a sequence modeling layer, an attention layer, and an output layer; the semantic label sequence includes the semantic label of each trajectory subsequence;

[0100] In step S41, multiple interactive behavior features corresponding to each trajectory subsequence are obtained based on the semantic label of each trajectory subsequence. The interactive behavior features include prediction features, multiple additional features, and user ID features. Table 3 shows the interactive behavior features.

[0101] Table 3 Interaction behavior characteristics

[0102] property illustrate User_id User Number Duration Interaction duration Interval The time interval between an event and the last event in the sequence Coarse_id The coarse-grained interaction behavior number AOI_id AOI area number Fine_id Fine-grained interaction behavior numbering

[0103] In Table 3, Fine_id is the prediction feature, User_id is the user ID feature, and there are four additional features, namely Duration, Interval, Coarse_id, and AOI_id.

[0104] Step S42: Using a sliding window of fixed length n+1, select semantic tags of fixed sequence length n from the semantic tag sequence as a semantic tag subsequence; perform data structure processing on multiple interactive behavior features corresponding to each semantic tag in the semantic tag subsequence, and take the multiple additional features as a whole, record them as the overall feature X, and express the overall feature and the predicted feature as (X1, Y1…, X n ,Y n ), where Y n is the predicted feature corresponding to the nth semantic label, X n is the overall feature corresponding to the nth semantic tag, and the interaction behavior feature vector of the semantic tag subsequence is finally obtained, including the overall feature, prediction feature and user number feature.

[0105] Step S43: Input the interaction behavior feature vector of the semantic label subsequence into the multi-feature sequence prediction network based on LSTM-Attention, and output the predicted score for each preset candidate interaction behavior;

[0106] Step S44: Determine the interaction behavior prediction result based on the prediction score. Here, the top k preset candidate interaction behaviors with the highest scores can be selected as the final interaction behavior prediction result based on the prediction score.

[0107] In one embodiment, in step S43, the interaction behavior feature vector of the semantic label subsequence is input into a multi-feature sequence prediction network based on LSTM-Attention, and a prediction score for each preset candidate interaction behavior is output, including:

[0108] The interaction behavior feature vector of the semantic tag subsequence includes the interaction behavior feature corresponding to each semantic tag after data structure processing; the interaction behavior feature corresponding to each semantic tag after data structure processing includes prediction feature, overall feature and user number feature;

[0109] Step S431: For each semantic tag, the predicted features and the overall features are input into the embedding layer. The embedding layer embeds the predicted features to obtain an embedding result. The embedding result is multiplied by each additional feature in the overall features to obtain a multiplication result corresponding to each additional feature. The multiplication result corresponding to each additional feature is concatenated with the embedding result to obtain the encoding vector corresponding to the semantic tag.

[0110] Step S432: Input the encoding vector corresponding to each semantic tag into the sequence modeling layer for temporal modeling to obtain the sequence modeled vector corresponding to each semantic tag;

[0111] Step S433: Input the sequence modeling vector corresponding to each semantic label into the fully connected layer of the attention layer, use the tanh activation function to obtain a latent vector, which contains the attention probability of each semantic label. Normalize the latent vector to obtain the normalized attention probability corresponding to each semantic label. Use the normalized attention probability corresponding to each semantic label as a weight to perform weighted summation on the sequence modeling vector corresponding to each semantic label to obtain a sequence vector.

[0112] In step S434, the sequence vector is input into the output layer. The output layer concatenates the sequence vector and the user ID feature to obtain a concatenated sequence vector. The concatenated sequence vector is input into the fully connected layer of the output layer. Based on the Softmax activation function, the predicted score for each preset candidate interaction behavior is output.

[0113] Using the same inventive concept as the method for predicting interactive behavior in medical image reading based on mouse trajectory, this embodiment also provides a corresponding device for predicting interactive behavior in medical image reading based on mouse trajectory. Figure 4 The following is a block diagram of a medical image reading interactive behavior prediction device based on mouse trajectory according to an embodiment of the present application, including:

[0114] The mouse interaction data acquisition module 41 is used to acquire the mouse interaction data generated by the user during the medical image reading process; the mouse interaction data includes mouse interaction trajectory data and mouse interaction data attributes;

[0115] A trajectory subsequence acquisition module 42 is configured to segment the mouse interaction trajectory data according to attributes of the mouse interaction data to obtain a plurality of trajectory subsequences;

[0116] A semantic label sequence determination module 43 is used to perform semantic annotation on multiple trajectory subsequences to determine the user's semantic label sequence;

[0117] The interactive behavior prediction module 44 is used to input the semantic label sequence into the multi-feature sequence prediction network based on LSTM-Attention and output the interactive behavior prediction result.

[0118] In one embodiment, the trajectory subsequence acquisition module 42 is further configured to:

[0119] According to the AOI attribute in the mouse interaction data attribute, the mouse interaction trajectory data is preliminarily segmented to obtain multiple trajectory segments; the AOI attribute of all trajectory points in each trajectory segment is the same;

[0120] Each track segment is further segmented according to the Mouse Event Type attribute in the mouse interaction data attribute to obtain multiple track subsequences; all track points in each track subsequence have the same AOI attribute and the same Mouse Event Type attribute.

[0121] In one embodiment, the semantic tag sequence determination module 43 is further configured to:

[0122] Based on the mouse interaction data attributes and the pre-trained mouse trajectory recognition model, the coarse-grained category label of each trajectory subsequence is determined. The coarse-grained category label includes the system response behavior label, the active behavior label, and the background behavior label. All the system response behavior labels and active behavior labels are extracted as the user's coarse-grained category label.

[0123] Determine the user's fine-grained category label corresponding to the user's coarse-grained category label based on the mouse interaction data attributes;

[0124] According to the user's coarse-grained category label and the user's fine-grained category label, the user's semantic label sequence is determined.

[0125] In order to further verify the effectiveness of the method and device of the present application in the recognition of coarse-grained interactive behaviors, the experimental results on the constructed mouse interaction dataset are shown in Table 4; Table 4 shows the three evaluation scores of recall, precision and harmonic mean under the TIoU (Time-Intersection over Union) threshold setting of [0.3:0.8:0.1] in the process of semantic annotation based on mouse trajectory. Based on the mouse interaction dataset constructed by the present application, the original dataset contains a total of 1580 coarse-grained interactive behavior labels. The present application has identified a total of 1824 coarse-grained interactive behaviors, and the average values of recall, precision and harmonic mean are 0.887, 0.749 and 0.826 respectively, indicating that this method can obtain a higher recall rate while ensuring the recognition accuracy. At the same time, it means that for the task of developing a new generation of intelligent film reading systems in the future, this method can effectively understand and identify user interaction intentions on the basis of ensuring user experience, laying the foundation for the next step of realizing user interaction intention prediction.

[0126] Table 4 Experimental results of coarse-grained interactive behavior recognition

[0127] TIoU threshold Recall Precision Harmonic mean 0.3 0.921 0.783 0.866 0.4 0.912 0.779 0.859 0.5 0.899 0.771 0.849 0.6 0.878 0.752 0.828 0.7 0.853 0.731 0.805 0.8 0.794 0.68 0.749 average 0.877 0.749 0.826

[0128] To further validate the effectiveness of the present method and apparatus in predicting fine-grained interactive behavior, the present method was compared with MRU, MFU, first-order Markov, naive Bayes, deep neural network (DNN), and recurrent neural network (RNN) in an experiment predicting interactive behavior in medical image reading using a sequence length of 7. The processed fine-grained interactive behavior sequence dataset was 2056, and 30% of the data was used as the test set. The prediction results are shown in Table 5. As can be seen from Table 5, the present method (SMLA) is the best in terms of the evaluation metric Recall@k (k∈[1,6]). Therefore, the present method's interactive behavior prediction results are superior to existing methods.

[0129] Table 5 Comparison of results of multi-feature sequence prediction network based on LSTM-Attention and other methods

[0130]

[0131]

[0132] In summary, this application has the following technical effects:

[0133] (1) This application starts from the perspective of optimizing the interactive process of medical image reading by doctors, and realizes the recognition and prediction of users' medical image reading behavior based on the mouse interaction trajectory, providing a theoretical and experimental basis for the research and development of a new medical image reading system that can help radiologists improve the efficiency and accuracy of medical image reading diagnosis to a greater extent.

[0134] (2) This application realizes the recognition of local semantics of mouse trajectory by segmenting and semantically annotating the mouse interaction trajectory of medical image reading, and effectively obtains the user's interactive behavior information at different time periods during the medical image reading interaction process.

[0135] (3) This application predicts the next interactive behavior by integrating multi-feature information of interactive behaviors and the attention mechanism, thereby improving the utilization rate of historical sequence information of interactive behaviors and the effectiveness of sequence modeling, and improving the prediction performance.

[0136] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for predicting interactive behavior in medical image reading based on mouse trajectory, characterized in that: include: Obtaining mouse interaction data generated by users during the medical image reading process; The mouse interaction data includes mouse interaction trajectory data and mouse interaction data attributes; Segmenting the mouse interaction trajectory data according to the mouse interaction data attributes to obtain a plurality of trajectory subsequences; Performing semantic annotation on the multiple trajectory subsequences to determine a semantic label sequence of the user; Input the semantic label sequence into a multi-feature sequence prediction network based on LSTM-Attention, and output the interactive behavior prediction result; The mouse interaction trajectory data is segmented according to the mouse interaction data attributes to obtain multiple trajectory subsequences, including: Preliminarily segmenting the mouse interaction trajectory data according to the AOI attribute in the mouse interaction data attribute to obtain a plurality of trajectory segments; wherein all trajectory points in each trajectory segment have the same AOI attribute; Each of the trajectory segments is further segmented according to the Mouse Event Type attribute in the mouse interaction data attribute to obtain the multiple trajectory subsequences; all trajectory points in each of the trajectory subsequences have the same AOI attribute and the same Mouse Event Type attribute.

2. The method according to claim 1, wherein in, Performing semantic annotation on the multiple trajectory subsequences to determine a semantic label sequence of the user; Determining a coarse-grained category label for each of the trajectory subsequences based on the mouse interaction data attributes and a pre-trained mouse trajectory recognition model; The coarse-grained category labels include system response behavior labels, active behavior labels, and background behavior labels; Extracting all system response behavior labels and active behavior labels as coarse-grained category labels of the user; Determining a fine-grained category label of the user corresponding to the coarse-grained category label of the user according to the mouse interaction data attribute; A semantic tag sequence of the user is determined according to the coarse-grained category tag of the user and the fine-grained category tag of the user.

3. The method according to claim 2, wherein in, Determining a coarse-grained category label for each of the trajectory subsequences based on the mouse interaction data attributes and a pre-trained mouse trajectory recognition model includes: determining, based on the mouse interaction data attributes, whether the trajectory subsequence is a system response behavior; If the trajectory subsequence is a system response behavior, obtaining a system response behavior category as a system response behavior label of the trajectory subsequence; If the trajectory subsequence is not a system response behavior, determine whether the trajectory subsequence is an active behavior or a background behavior based on the pre-trained mouse trajectory recognition model; and when the trajectory subsequence is an active behavior, obtain the active behavior category of the trajectory subsequence as the active behavior label of the trajectory subsequence; when the trajectory subsequence is a background behavior, the background behavior is used as the background behavior label of the trajectory subsequence.

4. The method according to claim 2, wherein in, Determining whether the trajectory subsequence is an active behavior or a background behavior according to the pre-trained mouse trajectory recognition model includes: determining a mouse trajectory recognition task type for each of the trajectory subsequences according to the AOI attribute in the mouse interaction data attribute; The trajectory subsequence is input into a pre-trained mouse trajectory recognition model corresponding to the mouse trajectory recognition task type of the trajectory subsequence to determine whether the trajectory subsequence is an active behavior or a background behavior.

5. The method according to claim 1, wherein in, The semantic label sequence is input into a multi-feature sequence prediction network based on LSTM-Attention to determine the interactive behavior prediction result, including: The multi-feature sequence prediction network based on LSTM-Attention includes an embedding layer, a sequence modeling layer, an attention layer, and an output layer; the semantic label sequence includes a semantic label for each trajectory subsequence; Obtaining a plurality of interactive behavior features corresponding to each trajectory subsequence according to the semantic label of each trajectory subsequence, wherein the interactive behavior features include a prediction feature, a plurality of additional features, and a user ID feature; Using a fixed-length sliding window, semantic tags of a fixed sequence length are selected from the semantic tag sequence as a semantic tag subsequence; performing data structure processing on multiple interaction behavior features corresponding to each semantic tag in the semantic tag subsequence, taking the multiple additional features as a whole and recording them as a whole feature, and obtaining an interaction behavior feature vector for the semantic tag subsequence; Inputting the interaction behavior feature vector of the semantic label subsequence into the LSTM-Attention-based multi-feature sequence prediction network, and outputting a prediction score for each preset candidate interaction behavior; The interactive behavior prediction result is determined according to the prediction score.

6. The method according to claim 5, wherein in, Inputting the interactive behavior feature vector of the semantic label subsequence into the multi-feature sequence prediction network based on LSTM-Attention, and outputting a prediction score for each preset candidate interactive behavior, including: The interactive behavior feature vector of the semantic tag subsequence includes the interactive behavior feature corresponding to each semantic tag after being processed by the data structure; the interactive behavior feature corresponding to each semantic tag after being processed by the data structure includes a prediction feature, an overall feature, and a user number feature; For each semantic label, the predicted feature and the overall feature are input into the embedding layer, and the embedding layer embeds the predicted feature to obtain an embedding result; the embedding result is multiplied by each additional feature in the overall feature to obtain a multiplication result corresponding to each additional feature; the multiplication result corresponding to each additional feature is concatenated with the embedding result to obtain an encoding vector corresponding to the semantic label; Inputting the encoding vector corresponding to each semantic label into the sequence modeling layer for time series modeling to obtain the sequence modeled vector corresponding to each semantic label; Inputting the sequence modeling vector corresponding to each semantic label into the fully connected layer of the attention layer, using the tanh activation function to obtain a latent vector, wherein the latent vector contains the attention probability of each semantic label, normalizing the latent vector to obtain the normalized attention probability corresponding to each semantic label; using the normalized attention probability corresponding to each semantic label as a weight to perform weighted summation on the sequence modeling vector corresponding to each semantic label, to obtain a sequence vector; The sequence vector is input into the output layer, and the output layer concatenates the sequence vector and the user ID feature to obtain a concatenated sequence vector; the concatenated sequence vector is input into the fully connected layer of the output layer, and based on the Softmax activation function, a predicted score for each preset candidate interaction behavior is output.

7. A device for predicting interactive behavior of medical image reading based on mouse trajectory, characterized in that: include: The mouse interaction data acquisition module is used to acquire the mouse interaction data generated by the user during the medical image reading process; The mouse interaction data includes mouse interaction trajectory data and mouse interaction data attributes; a trajectory subsequence acquisition module, configured to segment the mouse interaction trajectory data according to attributes of the mouse interaction data to obtain a plurality of trajectory subsequences; a semantic label sequence determination module, configured to perform semantic annotation on the plurality of trajectory subsequences to determine a semantic label sequence for the user; An interactive behavior prediction module, configured to input the semantic label sequence into a multi-feature sequence prediction network based on LSTM-Attention and output an interactive behavior prediction result; The trajectory subsequence acquisition module is further used to: Preliminarily segmenting the mouse interaction trajectory data according to the AOI attribute in the mouse interaction data attribute to obtain a plurality of trajectory segments; wherein all trajectory points in each trajectory segment have the same AOI attribute; Each of the trajectory segments is further segmented according to the Mouse Event Type attribute in the mouse interaction data attribute to obtain the multiple trajectory subsequences; all trajectory points in each of the trajectory subsequences have the same AOI attribute and the same Mouse Event Type attribute.

8. The device according to claim 7, wherein The semantic tag sequence determination module is further configured to: Determining a coarse-grained category label for each of the trajectory subsequences based on the mouse interaction data attributes and a pre-trained mouse trajectory recognition model; the coarse-grained category label includes a system response behavior label, an active behavior label, and a background behavior label; Extracting all system response behavior labels and active behavior labels as coarse-grained category labels of the user; Determining a fine-grained category label of the user corresponding to the coarse-grained category label of the user according to the mouse interaction data attribute; A semantic tag sequence of the user is determined according to the coarse-grained category tag of the user and the fine-grained category tag of the user.

Images