Behavior recognition apparatus, method, and storage medium

Abstract

This application relates to a behavior recognition device, method, and storage medium. The device can acquire image feature information of a medical image based on the number and type of medical objects included in the image; determine the complexity parameter of the medical image based on the image feature information; select a suitable target branch in a graph neural network based on the complexity parameter; and input the medical image into the graph neural network through the target branch to obtain the corresponding medical behavior recognition result. The graph neural network includes multiple branches, each with a different number of network nodes. For medical images with higher complexity, branches with more network nodes are selected to improve the accuracy of behavior recognition; for medical images with lower complexity, branches with fewer network nodes are selected to ensure the efficiency of behavior recognition. Using this device can improve the accuracy of behavior recognition while maintaining its efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a behavior recognition device, method, computer equipment, storage medium, and computer program product. Background Technology

[0002] Neural network models have achieved better accuracy than traditional models in most disciplines and fields, and also possess good generalization ability. In recent years, with the rapid development of deep neural networks, they have gradually been applied in the medical field.

[0003] Surgical behavior category analysis (such as cutting, tissue separation, suturing, etc.) is a key issue in laparoscopic surgery video analysis. Existing research mainly uses deep learning models to learn and predict various surgical behaviors in videos. This process requires identifying various key tissues / organs and surgical instruments in the video, and analyzing the behavioral relationships between tissues and instruments based on the identification results.

[0004] Currently, the dynamic configuration / selection features of neural networks are not fully utilized in predicting surgical behavior. Furthermore, since the main context of surgical behavior is based on the detected target object, using the same neural network structure to process all behavior predictions is not optimal in terms of accuracy and efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a behavior recognition device, method, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of behavior recognition while ensuring recognition efficiency, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a behavior recognition device. The device includes:

[0007] The acquisition module is used to acquire image feature information of medical images. The image feature information is obtained based on the number and type of medical objects included in the medical images. Medical objects include medical tissue objects and medical device objects.

[0008] The processing module is used to determine the complexity parameters of medical images based on image feature information;

[0009] The matching module is used to determine the target branch in the graph neural network based on the complexity parameter; the graph neural network includes multiple branches, each branch corresponding to a different number of network nodes;

[0010] The recognition module is used to input medical images into a graph neural network through a target branch to obtain the medical behavior recognition results corresponding to the medical images; the medical behavior recognition results include medical object region identifiers and medical behavior category identifiers.

[0011] In one embodiment, the acquisition module is further configured to acquire video frame images in a medical image; the video frame image includes at least one medical object; perform feature recognition on the video frame image to acquire image feature information of the video frame image; the image feature information is obtained based on the number and type of medical objects included in the video frame image; and acquire image feature information of the medical image based on the image feature information.

[0012] In one embodiment, the acquisition module is further configured to acquire the temporal attention features and spatial attention features of the video frame image respectively; and acquire the image feature information of the video frame image based on the temporal attention features and spatial attention features.

[0013] In one embodiment, the processing module is further configured to obtain the number of medical objects included in the medical image and the feature vector corresponding to the medical image based on the image feature information; obtain the target label vector that matches the feature vector from multiple label vectors; each label vector is used to represent a medical behavior category; and determine the complexity parameter of the medical image based on the target label vector and the number of medical objects included in the medical image.

[0014] In one embodiment, the matching module is further configured to determine the number of network nodes corresponding to each branch in the graph neural network; determine the complexity label corresponding to each branch based on the number of network nodes corresponding to each branch; determine the target complexity label that matches the complexity parameter among multiple complexity labels, and take the branch corresponding to the target complexity label as the target branch.

[0015] In one embodiment, the device further includes:

[0016] The training module is used to obtain an initial graph neural network (Graph NNN). The initial Graph NNN includes multiple branches, each with a different number of network nodes. A training set is obtained, consisting of multiple training samples. Each training sample includes a medical image sample and a sample label. The sample label represents the medical object region in the medical image sample and the corresponding medical behavior category. Each training sample in the training set is input into the initial Graph NNN through each branch to train the initial Graph NNN, thus obtaining the Graph NNN.

[0017] Secondly, this application also provides a behavior recognition method. The method includes:

[0018] Obtain image feature information from medical images; image feature information is obtained based on the number and type of medical objects included in the medical images, including medical tissue objects and medical device objects;

[0019] Based on image feature information, determine the complexity parameters of medical images;

[0020] The target branch is determined in the graph neural network based on the complexity parameter; the graph neural network includes multiple branches, and each branch corresponds to a different number of network nodes;

[0021] The medical image is input into the graph neural network through the target branch to obtain the medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes the medical object area identifier and the medical behavior category identifier.

[0022] In one embodiment, obtaining image feature information of a medical image includes:

[0023] Acquire video frame images from medical images; the video frame images include at least one medical object;

[0024] Feature recognition is performed on video frame images to obtain image feature information of the video frame images; the image feature information is obtained based on the number and type of medical objects included in the video frame images;

[0025] Based on image feature information, obtain the image feature information of medical images.

[0026] In one embodiment, feature recognition is performed on the video frame image to obtain image feature information of the video frame image, including:

[0027] Temporal attention features and spatial attention features of video frame images are obtained respectively;

[0028] Based on temporal attention features and spatial attention features, image feature information of video frame images is obtained.

[0029] In one embodiment, the complexity parameter of the medical image is determined based on image feature information, including:

[0030] Based on image feature information, obtain the number of medical objects included in the medical image and the corresponding feature vector of the medical image;

[0031] Among multiple label vectors, the target label vector that matches the feature vector is obtained; each label vector is used to represent a medical behavior category.

[0032] The complexity parameter of the medical image is determined based on the target label vector and the number of medical objects included in the medical image.

[0033] In one embodiment, determining the target branch in the graph neural network based on a complexity parameter includes:

[0034] Determine the number of network nodes corresponding to each branch in the graph neural network;

[0035] The complexity label for each branch is determined based on the number of network nodes corresponding to each branch.

[0036] Among multiple complexity labels, determine the target complexity label that matches the complexity parameter, and take the branch corresponding to the target complexity label as the target branch.

[0037] In one embodiment, the graph neural network is obtained by means of:

[0038] Obtain the initial graph neural network; the initial graph neural network includes multiple branches, and each branch corresponds to a different number of network nodes;

[0039] Obtain the training set; the training set includes multiple training samples, each of which includes a medical image sample and a sample label. The sample label is used to characterize the medical object region in the medical image sample and the medical behavior category corresponding to the medical image.

[0040] Each training sample in the training set is input into the initial graph neural network through each branch of the initial graph neural network to train the initial graph neural network, resulting in the trained initial graph neural network, which is then used as the graph neural network.

[0041] In one embodiment, after training the initial graph neural network using a training set to obtain the trained initial graph neural network, the method further includes:

[0042] Obtain the test set from the training set;

[0043] The initial graph neural network after training is tested using a test set to obtain the confidence level of the initial graph neural network after training.

[0044] If the confidence level is within the preset confidence interval, the trained initial graph neural network is used as the graph neural network.

[0045] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0046] Obtain image feature information from medical images; image feature information is obtained based on the number and type of medical objects included in the medical images, including medical tissue objects and medical device objects;

[0047] Based on image feature information, determine the complexity parameters of medical images;

[0048] The target branch is determined in the graph neural network based on the complexity parameter; the graph neural network includes multiple branches, and each branch corresponds to a different number of network nodes;

[0049] The medical image is input into the graph neural network through the target branch to obtain the medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes the medical object area identifier and the medical behavior category identifier.

[0050] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0051] Obtain image feature information from medical images; image feature information is obtained based on the number and type of medical objects included in the medical images, including medical tissue objects and medical device objects;

[0052] Based on image feature information, determine the complexity parameters of medical images;

[0053] The target branch is determined in the graph neural network based on the complexity parameter; the graph neural network includes multiple branches, and each branch corresponds to a different number of network nodes;

[0054] The medical image is input into the graph neural network through the target branch to obtain the medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes the medical object area identifier and the medical behavior category identifier.

[0055] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0056] Obtain image feature information from medical images; image feature information is obtained based on the number and type of medical objects included in the medical images, including medical tissue objects and medical device objects;

[0057] Based on image feature information, determine the complexity parameters of medical images;

[0058] The target branch is determined in the graph neural network based on the complexity parameter; the graph neural network includes multiple branches, and each branch corresponds to a different number of network nodes;

[0059] The medical image is input into the graph neural network through the target branch to obtain the medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes the medical object area identifier and the medical behavior category identifier.

[0060] The aforementioned behavior recognition device, method, computer equipment, storage medium, and computer program product acquire image feature information of medical images based on the number and type of medical objects included in the medical images; determine the complexity parameters of the medical images based on the image feature information; select a suitable target branch in a graph neural network based on the complexity parameters; input the medical images into the graph neural network through the target branch to obtain the medical behavior recognition results corresponding to the medical images. The graph neural network can recognize the medical behaviors displayed in the current medical images. The graph neural network includes multiple branches, each with a different number of network nodes. For medical images with higher complexity, branches with more network nodes are selected to improve the accuracy of behavior recognition; for medical images with lower complexity, branches with fewer network nodes are selected to ensure the efficiency of behavior recognition. In this way, both the accuracy and efficiency of behavior recognition can be improved. Attached Figure Description

[0061] Figure 1 This is a diagram illustrating the application environment of the behavior recognition method in one embodiment;

[0062] Figure 2 This is a flowchart illustrating a behavior recognition method in one embodiment;

[0063] Figure 3 This is a schematic diagram of the processing flow of a behavior recognition method in one embodiment;

[0064] Figure 4 This is a schematic diagram illustrating the functional requirements of a graph neural network in one embodiment;

[0065] Figure 5 This is a schematic diagram of a set of prediction results from different branches of a graph neural network in one embodiment;

[0066] Figure 6 This is a schematic diagram of another set of prediction results for different branches of a graph neural network in one embodiment;

[0067] Figure 7 This is a schematic diagram of feature extraction in one embodiment;

[0068] Figure 8 This is a schematic diagram of dynamic selection in one embodiment;

[0069] Figure 9 This is a schematic diagram of the graph neural network structure in one embodiment;

[0070] Figure 10 This is a schematic diagram of the network structure for node relationship analysis in one embodiment;

[0071] Figure 11 This is a schematic diagram illustrating the numerical variation of the loss function in one embodiment;

[0072] Figure 12 This is a schematic diagram illustrating the changes in the accuracy parameter values ​​in one embodiment;

[0073] Figure 13 This is a schematic diagram of the precision-recall results in one embodiment;

[0074] Figure 14 This is a schematic diagram of the F1-score test results in one embodiment;

[0075] Figure 15 This is a structural block diagram of a behavior recognition device in one embodiment;

[0076] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0078] In one embodiment, such as Figure 1 As shown, a behavior recognition method is provided. This embodiment illustrates the application of this method to a computer device. It can be understood that the computer device can specifically be as follows: Figure 1 The application environment shown includes either terminal 102 or server 104. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0079] In one embodiment, such as Figure 2 As shown, a behavior recognition method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0080] Step 202: Obtain the image feature information of the medical image; the image feature information is obtained based on the number and type of medical objects included in the medical image, including medical tissue objects and medical device objects.

[0081] Among them, the medical tissue object refers to the lesion area of ​​the patient, including but not limited to various human tissues and organs; the medical device object refers to various surgical instruments, taking laparoscopic surgery as an example, including but not limited to scalpels, surgical forceps, tissue scissors, curved hemostats, straight hemostats, right-angle forceps, suture scissors, and kerat forceps.

[0082] Optionally, the terminal first acquires the medical image to be identified, and then... Figure 3 As shown, feature extraction is performed on medical images, extracting image feature information based on the number and type of medical objects included in the medical images.

[0083] Step 204: Determine the complexity parameters of the medical image based on the image feature information.

[0084] Optionally, after the terminal obtains the image feature information, such as Figure 3 As shown, the network branch is dynamically selected based on image feature information. During this dynamic selection process, the terminal first determines the complexity of the medical actions involved in the medical image based on the image feature information. These medical actions are equivalent to surgical procedures or stages. Since different surgical procedures or stages typically use different medical instruments and capture different lesion areas, and different surgical procedures or stages often have clearly defined levels of complexity, the medical actions involved in the medical image can be determined based on the image feature information, further determining the complexity parameter of the medical image. For example, if the image feature information contains a large number of medical tissue objects or medical instrument objects (i.e., a large number of medical objects), and these medical tissue objects or medical instrument objects involve very complex surgical procedures or stages, then the terminal determines the medical image to be highly complex and generates a first complexity parameter representing high complexity. Conversely, if the image feature information contains a small number of medical objects, and these medical tissue objects or medical instrument objects involve simple surgical procedures or stages, then the terminal determines the medical image to be less complex and generates a second complexity parameter representing lower complexity.

[0085] Step 206: Determine the target branch in the graph neural network based on the complexity parameter; the graph neural network includes multiple branches, and each branch corresponds to a different number of network nodes.

[0086] Among them, due to the need to process medical images, medical cameras, and medical videos, the functional requirements of graph neural networks are as follows: Figure 4As shown, the system needs to include four functions: feature extraction, target classification, node relationship mining and analysis, and time series relationship analysis, and finally output medical behavior recognition results. Optional graph neural networks include, but are not limited to, graph convolutional networks (GCN), graph attention networks, graph autoencoders, graph generative networks, and graph spatial-temporal networks.

[0087] Optionally, after obtaining the complexity parameter, the terminal continues with the step of dynamically selecting the network branch, such as... Figure 3 The diagram shows branches 1, 2, and N. The graph neural network comprises multiple branches, each with a different number of network nodes. The terminal selects a branch that matches the complexity level based on a complexity parameter. For medical images with higher complexity, a branch with more network nodes is selected to improve the accuracy of behavior recognition; for medical images with lower complexity, a branch with fewer network nodes is selected to ensure efficiency. After determining the target branch based on the complexity parameter, the terminal ends the dynamic selection process for the current medical image and then inputs the medical image into the graph neural network through the target branch.

[0088] Step 208: Input the medical image into the graph neural network through the target branch to obtain the medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes the medical object area identifier and the medical behavior category identifier.

[0089] Optionally, the terminal calls the trained graph neural network and inputs the medical image into the target branch of the graph neural network. The graph neural network will perform feature extraction, node mapping, node relationship analysis and other processing on the medical image, and output the medical behavior recognition result of the medical image. The medical behavior recognition result includes medical object region identifier and medical behavior category identifier. The medical object region identifier is usually displayed as the region box of the medical object in the medical image. The medical behavior category identifier is usually displayed as a set of data, which represents the unique medical behavior and confidence level.

[0090] like Figure 5 As shown, the left and right images represent the prediction results of complex and relatively simple graph neural network branches, respectively. The left image shows the complex network, while the right image shows the simple network. From the prediction results of surgical instruments against this fat background, it can be seen that there are differences between the complex and simple networks in terms of prediction location, confidence level, and instrument size. Differences in prediction latency are also observed in the test results. Figure 6As shown, the left and right images represent the prediction results of complex and relatively simple graph neural network branches for multiple instruments, respectively. From the comparison of the left and right images, it can be seen that the confidence value on the left is slightly higher than that on the right. This is because the pixel values ​​of the surgical instruments in the image are larger and their shape features are more obvious. Therefore, after reasonable training of the graph neural network, the difference in accuracy between the two different network branches is not significant, and both can meet the accuracy requirements of medical behavior recognition.

[0091] In the aforementioned behavior recognition method, image feature information of the medical image is obtained based on the number and type of medical objects included in the medical image; based on the image feature information, the complexity parameter of the medical image is determined; based on the complexity parameter, a suitable target branch is selected in the graph neural network; the medical image is input into the graph neural network through the target branch to obtain the medical behavior recognition result corresponding to the medical image. The graph neural network can recognize the medical behavior displayed in the current medical image. The graph neural network includes multiple branches, each with a different number of network nodes. For medical images with higher complexity, branches with more network nodes are selected to improve the accuracy of behavior recognition, while for medical images with lower complexity, branches with fewer network nodes are selected to ensure the efficiency of behavior recognition. In this way, both the accuracy and efficiency of behavior recognition can be improved.

[0092] In one embodiment, obtaining image feature information of a medical image includes: acquiring video frame images in the medical image; the video frame images include at least one medical object; acquiring temporal attention features and spatial attention features of the video frame images respectively; acquiring image feature information of the video frame images based on the temporal attention features and spatial attention features; the image feature information is obtained based on the number and type of medical objects included in the video frame images; and acquiring image feature information of the medical image based on the image feature information.

[0093] Optionally, when extracting features from medical images, the terminal can extract a single frame from the medical image and then obtain image feature information based on the features of that frame; alternatively, it can extract multiple frames from the medical image at preset time intervals and then obtain image feature information based on the features of the multiple frames; or it can obtain image feature information based on the features of each frame in the medical image. For a single video frame image, such as... Figure 7As shown, the feature extraction used by the terminal can include temporal attention and spatial attention features. The aim is to extract more comprehensive two-dimensional information from a video frame. After relevant information extraction, a feature vector fusion algorithm is used to output the final fused feature vector of the video frame. Finally, based on the final fused feature vector of each video frame, the image feature information of the medical image is obtained.

[0094] In this embodiment, attention features and spatial attention features are extracted simultaneously during feature extraction, which can more comprehensively extract the two-dimensional information of a video frame image and ensure the accuracy of image feature information.

[0095] In one embodiment, determining the complexity parameter of a medical image based on image feature information includes: obtaining the number of medical objects included in the medical image and the corresponding feature vector of the medical image based on the image feature information; obtaining a target label vector that matches the feature vector from multiple label vectors; each label vector is used to represent a medical behavior category; and determining the complexity parameter of the medical image based on the target label vector and the number of medical objects included in the medical image.

[0096] Furthermore, based on the complexity parameter, the target branch is determined in the graph neural network, including: determining the number of network nodes corresponding to each branch in the graph neural network; determining the complexity label corresponding to each branch based on the number of network nodes corresponding to each branch; among multiple complexity labels, determining the target complexity label that matches the complexity parameter, and taking the branch corresponding to the target complexity label as the target branch.

[0097] Among them, the category of medical behavior is equivalent to surgical behavior or surgical stage. Taking the division of thyroid surgery stages as an example, the types of medical behavior include, but are not limited to, incision / pneumoperitoneum establishment, anatomical exposure, thyroid lobectomy, thyroid artery and vein disconnection, parathyroid protection, recurrent laryngeal nerve protection, and tissue treatment.

[0098] Optionally, multiple label vectors and complexity labels can be pre-configured in the terminal according to actual needs. Each target label vector represents a medical behavior category. The complexity label can be a preset complexity parameter or a preset range of complexity parameters. For example, a larger complexity parameter indicates higher complexity. Branches with more nodes can correspond to complexity labels with larger complexity parameters, while branches with fewer nodes can correspond to complexity labels with smaller complexity parameters. When the complexity labels are within a preset range of complexity parameters, there is no overlap between the complexity labels. The corresponding network can be assigned based on the minimum or average value within the complexity parameter range.

[0099] like Figure 8 As shown, during the dynamic selection process, the terminal first estimates the number of targets based on the detected surgical instruments and tissue / structure feature vectors, as different surgical procedures involve different numbers of surgical instruments and organs / tissues. After estimating the relevant quantities, the L2 distance is calculated between the input feature vector and each label vector to initially classify the overall category of each target in the video frame and determine the current stage of the surgery and its corresponding probability value. The smaller the L2 distance between a feature vector and a label vector, the higher the probability of the medical behavior category represented by that label vector. Correspondingly, when selecting network branches, the label vector with the smallest L2 distance to the feature vector is taken as the target label vector. That is, the medical behavior category represented by the target label vector (e.g., incision, dissection, protection, etc.) is taken as the medical behavior category with the highest probability. Based on the medical behavior category with the highest probability and the number of medical objects, the complexity parameter of the medical image is determined.

[0100] Finally, the calculated complexity parameter is compared with each complexity label. When the complexity label is a single value, the complexity label with the closest complexity parameter is determined. When the complexity label is a range of values, a complexity label containing the complexity parameter is determined. Thus, the target complexity label that matches the complexity parameter is determined, and the branch corresponding to the target complexity label is taken as the target branch.

[0101] In this embodiment, the graph neural network includes multiple branches, each with a different number of network nodes. For medical images with higher complexity, branches with more network nodes are selected to improve the accuracy of behavior recognition. For medical images with lower complexity, branches with fewer network nodes are selected to ensure the efficiency of behavior recognition. In this way, the accuracy of behavior recognition can be improved while ensuring the efficiency of behavior recognition.

[0102] In one embodiment, the method of obtaining a graph neural network includes: obtaining an initial graph neural network; the initial graph neural network includes multiple branches, each branch corresponding to a different number of network nodes; obtaining a training set; the training set includes multiple training samples, each training sample including a medical image sample and a sample label, the sample label being used to characterize the medical object region in the medical image sample and the medical behavior category corresponding to the medical image; inputting each training sample in the training set into the initial graph neural network through each branch of the initial graph neural network to train the initial graph neural network, thereby obtaining a trained initial graph neural network; obtaining a test set based on the training set; testing the trained initial graph neural network using the test set to obtain the confidence level of the trained initial graph neural network; if the confidence level is within a preset confidence interval, using the trained initial graph neural network as the graph neural network.

[0103] Optionally, the structure of a graph neural network is as follows: Figure 9 As shown, the graph neural network first uses ResNet50 to perform feature recognition on medical images. Then, it maps the features representing each medical object to different nodes in the graph neural network. By learning and analyzing the random relationships between nodes, it performs medical behavior analysis to determine the category of the medical behavior. This analysis process is performed after determining the random relationships between network nodes, and then inputs them into the node relationship analysis network. The node relationship analysis network ultimately outputs the medical behavior recognition result. The main purpose of this network structure is to select different numbers of graph neural network convolutional layers and the final network node vector length based on the number of input nodes and the decision results of the dynamic selection module. Figure 10 As shown, this node relationship analysis network can be dynamically selected in both horizontal and vertical directions to meet the accuracy requirements of different surgical stages and operation categories. The more convolutional layers used and the larger the network node vector length, the greater the latency of the analysis will be.

[0104] During the training phase of a graph neural network, such as Figure 11 As shown in the figure, the loss function value changes during training. A larger loss function value indicates a lower overall accuracy of the model. Therefore, the goal of model training is to minimize the loss function value. The figure shows that the total training epoch is 300. During this training process, both the classification and segmentation loss function values ​​decrease, and the optimal model appears near the end of the training epoch. Figure 12 The figure shows the changes in accuracy parameters during training. The list of parameters to be detected includes commonly used parameters such as precision, recall, and mAP (mean accuracy). As can be seen from the figure, the trend of this accuracy parameter change is similar to... Figure 11The loss function shows the opposite trend, gradually increasing until the end of training, thus reflecting the objective situation of model accuracy.

[0105] During the testing phase of the graph neural network, the classification accuracy results after model training are shown in Table 1 (illustrative). The main classifications included key tissues such as the thyroid gland and trachea, as well as instruments. The accuracy of the target classification results is crucial for further graph neural network computation and behavioral result analysis. From the classification accuracy results above, it can be seen that key parameters such as precision, recall, and mAP all reached 80% accuracy. For some more complex organs, due to the influence of image quality and lighting conditions on region segmentation, the overall accuracy was slightly lower than 80%. Figure 13 As shown, the results display the precision-recall ratio on the actual test dataset. The graph shows that among the listed categories, the average precision for thyroid reached 0.733, while for parathyroid it exceeded 0.9. In this test sample, the main precision-recall distribution was between 0.6 and 0.8. Therefore, the trained model achieved good accuracy on this actual test sample. Figure 14 The figure shows the F1-score test results. The F1-score is defined as 2*(Precision*Recall) / (Precision+Recall). The graph shows the F1-score relative to the detection confidence. The main confidence interval for the detected target is between 0.75 and 0.85, indicating that the trained model achieved a high average accuracy on the test dataset.

[0106]

[0107]

[0108] Table 1

[0109] In this embodiment, the graph neural network is trained using a training set and then tested using a test set to ensure that the recognition confidence of the graph neural network meets the usage requirements and guarantees the accuracy of medical behavior recognition.

[0110] In one feasible implementation, the method of obtaining a graph neural network includes: obtaining an initial graph neural network; the initial graph neural network includes multiple branches, each branch corresponding to a different number of network nodes; obtaining a training set; the training set includes multiple training samples, each training sample including a medical image sample and two sample labels, one sample label is used to characterize the medical object region in the medical image sample and the medical behavior category corresponding to the medical image, and the other sample label is used to characterize the level of medical behavior; each training sample in the training set is input into the initial graph neural network through each branch of the initial graph neural network to train the initial graph neural network, thereby obtaining the trained initial graph neural network as the graph neural network.

[0111] In this context, "medical behavior level" refers to the degree of standardization of the current surgical procedure during the surgical action or stage. Taking thyroid surgery stages as an example, as shown in Table 2, medical behavior types include, but are not limited to, incision / pneumoperitoneum establishment, anatomical exposure, thyroid lobectomy, thyroid artery and vein transection, parathyroid gland protection, recurrent laryngeal nerve protection, and tissue handling, as listed in Table 2. The medical behavior level refers to the three operational levels under each medical behavior type in Table 2. Thus, in the application stage of the graph neural network, after processing the input medical images, the output medical behavior recognition result will include medical object region identification, medical behavior category identification, and medical behavior level identification.

[0112]

[0113]

[0114] Table 2

[0115] In one embodiment, a behavior recognition method includes:

[0116] The process involves acquiring video frame images from medical imaging, each video frame image including at least one medical object; acquiring temporal and spatial attention features of the video frame images; obtaining image feature information of the video frame images based on the temporal and spatial attention features; and obtaining image feature information of the medical image based on the number and type of medical objects included in the video frame images. The image feature information is derived from the number and type of medical objects included in the medical image, which include medical tissue objects and medical device objects.

[0117] Based on image feature information, the number of medical objects included in the medical image and the corresponding feature vector of the medical image are obtained; among multiple label vectors, the target label vector that matches the feature vector is obtained; each label vector is used to represent a medical behavior category; based on the target label vector and the number of medical objects included in the medical image, the complexity parameter of the medical image is determined.

[0118] The process involves determining the number of network nodes corresponding to each branch in the graph neural network; based on the number of network nodes corresponding to each branch, determining the complexity label corresponding to each branch; among multiple complexity labels, identifying the target complexity label that matches the complexity parameter, and selecting the branch corresponding to the target complexity label as the target branch. The graph neural network includes multiple branches, each with a different number of network nodes.

[0119] The medical image is input into the graph neural network through the target branch to obtain the medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes the medical object area identifier and the medical behavior category identifier.

[0120] In one embodiment, a method for training a graph neural network includes:

[0121] Obtain the initial graph neural network; the initial graph neural network includes multiple branches, each branch corresponding to a different number of network nodes; obtain the training set; the training set includes multiple training samples, each training sample includes a medical image sample and a sample label, the sample label is used to characterize the medical object region in the medical image sample, and the medical behavior category corresponding to the medical image; input each training sample in the training set into the initial graph neural network through each branch of the initial graph neural network to train the initial graph neural network, and obtain the trained initial graph neural network.

[0122] Obtain a test set from the training set; use the test set to test the trained initial graph neural network and obtain the confidence score of the trained initial graph neural network; if the confidence score is within a preset confidence interval, the trained initial graph neural network is used as the graph neural network.

[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0124] Based on the same inventive concept, this application also provides a behavior recognition device for implementing the behavior recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more behavior recognition device embodiments provided below can be found in the limitations of the behavior recognition method described above, and will not be repeated here.

[0125] In one embodiment, such as Figure 15 As shown, a behavior recognition device 1500 is provided, including: an acquisition module 1501, a processing module 1502, a matching module 1503, and a recognition module 1504, wherein:

[0126] The acquisition module 1501 is used to acquire image feature information of medical images; the image feature information is obtained based on the number and type of medical objects included in the medical images, including medical tissue objects and medical device objects;

[0127] The processing module 1502 is used to determine the complexity parameters of medical images based on image feature information;

[0128] The matching module 1503 is used to determine the target branch in the graph neural network based on the complexity parameter; the graph neural network includes multiple branches, and each branch corresponds to a different number of network nodes;

[0129] The recognition module 1504 is used to input medical images into a graph neural network through a target branch to obtain the medical behavior recognition results corresponding to the medical images; the medical behavior recognition results include medical object region identifiers and medical behavior category identifiers.

[0130] In one embodiment, the acquisition module 1501 is further configured to acquire video frame images in a medical image; the video frame image includes at least one medical object; perform feature recognition on the video frame image to acquire image feature information of the video frame image; the image feature information is obtained based on the number and type of medical objects included in the video frame image; and acquire image feature information of the medical image based on the image feature information.

[0131] In one embodiment, the acquisition module 1501 is further configured to acquire the temporal attention features and spatial attention features of the video frame image respectively; and acquire the image feature information of the video frame image based on the temporal attention features and spatial attention features.

[0132] In one embodiment, the processing module 1502 is further configured to obtain the number of medical objects included in the medical image and the feature vector corresponding to the medical image based on the image feature information; obtain the target label vector that matches the feature vector from multiple label vectors; each label vector is used to represent a medical behavior category; and determine the complexity parameter of the medical image based on the target label vector and the number of medical objects included in the medical image.

[0133] In one embodiment, the matching module 1503 is further configured to determine the number of network nodes corresponding to each branch in the graph neural network; determine the complexity label corresponding to each branch based on the number of network nodes corresponding to each branch; determine the target complexity label that matches the complexity parameter among multiple complexity labels, and take the branch corresponding to the target complexity label as the target branch.

[0134] In one embodiment, the apparatus further includes:

[0135] Training module 1505 is used to obtain an initial graph neural network. The initial graph neural network includes multiple branches, each with a different number of network nodes. A training set is obtained, consisting of multiple training samples. Each training sample includes a medical image sample and a sample label. The sample label is used to characterize the medical object region in the medical image sample and the corresponding medical behavior category. Each training sample in the training set is input into the initial graph neural network through each branch of the initial graph neural network to train the initial graph neural network, resulting in a trained initial graph neural network, which serves as the graph neural network.

[0136] In one embodiment, the apparatus further includes:

[0137] The testing module 1506 is used to obtain a test set based on the training set; to test the trained initial graph neural network using the test set and obtain the confidence level of the trained initial graph neural network; and to use the trained initial graph neural network as the graph neural network if the confidence level is within a preset confidence interval.

[0138] Each module in the aforementioned behavior recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0139] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 16 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores medical image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a behavior recognition method.

[0140] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0141] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring image feature information of a medical image; the image feature information is obtained based on the number and type of medical objects included in the medical image, including medical tissue objects and medical device objects; determining a complexity parameter of the medical image based on the image feature information; determining a target branch in a graph neural network based on the complexity parameter; the graph neural network includes multiple branches, each branch corresponding to a different number of network nodes; inputting the medical image into the graph neural network through the target branch to obtain a medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes a medical object region identifier and a medical behavior category identifier.

[0142] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring a video frame image in a medical image; the video frame image includes at least one medical object; performing feature recognition on the video frame image to acquire image feature information of the video frame image; the image feature information is obtained based on the number and type of medical objects included in the video frame image; and acquiring image feature information of the medical image based on the image feature information.

[0143] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring the temporal attention features and spatial attention features of the video frame image respectively; and acquiring the image feature information of the video frame image based on the temporal attention features and spatial attention features.

[0144] In one embodiment, when the processor executes the computer program, it further performs the following steps: based on image feature information, it obtains the number of medical objects included in the medical image and the feature vector corresponding to the medical image; among multiple label vectors, it obtains a target label vector that matches the feature vector; each label vector is used to represent a medical behavior category; and based on the target label vector and the number of medical objects included in the medical image, it determines the complexity parameter of the medical image.

[0145] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the number of network nodes corresponding to each branch in the graph neural network; determining the complexity label corresponding to each branch based on the number of network nodes corresponding to each branch; determining a target complexity label that matches the complexity parameter among multiple complexity labels, and taking the branch corresponding to the target complexity label as the target branch.

[0146] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining an initial graph neural network; the initial graph neural network includes multiple branches, each branch corresponding to a different number of network nodes; obtaining a training set; the training set includes multiple training samples, each training sample including a medical image sample and a sample label, the sample label being used to characterize the medical object region in the medical image sample and the medical behavior category corresponding to the medical image; inputting each training sample in the training set into the initial graph neural network through each branch of the initial graph neural network to train the initial graph neural network, thereby obtaining the trained initial graph neural network as the graph neural network.

[0147] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a test set based on the training set; testing the trained initial graph neural network using the test set to obtain the confidence level of the trained initial graph neural network; and using the trained initial graph neural network as a graph neural network if the confidence level is within a preset confidence interval.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: acquiring image feature information of a medical image; the image feature information is obtained based on the number and type of medical objects included in the medical image, including medical tissue objects and medical device objects; determining a complexity parameter of the medical image based on the image feature information; determining a target branch in a graph neural network based on the complexity parameter; the graph neural network includes multiple branches, each branch corresponding to a different number of network nodes; inputting the medical image into the graph neural network through the target branch to obtain a medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes a medical object region identifier and a medical behavior category identifier.

[0149] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring a video frame image in a medical image; the video frame image includes at least one medical object; performing feature recognition on the video frame image to acquire image feature information of the video frame image; the image feature information is obtained based on the number and type of medical objects included in the video frame image; and acquiring image feature information of the medical image based on the image feature information.

[0150] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring the temporal attention features and spatial attention features of the video frame image respectively; and acquiring the image feature information of the video frame image based on the temporal attention features and spatial attention features.

[0151] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the number of medical objects included in the medical image and the feature vector corresponding to the medical image based on image feature information; obtaining a target label vector that matches the feature vector from multiple label vectors; each label vector is used to represent a medical behavior category; and determining the complexity parameter of the medical image based on the target label vector and the number of medical objects included in the medical image.

[0152] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the number of network nodes corresponding to each branch in the graph neural network; determining the complexity label corresponding to each branch based on the number of network nodes corresponding to each branch; determining the target complexity label that matches the complexity parameter among multiple complexity labels, and taking the branch corresponding to the target complexity label as the target branch.

[0153] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining an initial graph neural network; the initial graph neural network includes multiple branches, each branch corresponding to a different number of network nodes; obtaining a training set; the training set includes multiple training samples, each training sample including a medical image sample and a sample label, the sample label being used to characterize the medical object region in the medical image sample and the medical behavior category corresponding to the medical image; inputting each training sample in the training set into the initial graph neural network through each branch of the initial graph neural network to train the initial graph neural network, thereby obtaining the trained initial graph neural network as the graph neural network.

[0154] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a test set based on the training set; testing the trained initial graph neural network using the test set to obtain the confidence level of the trained initial graph neural network; and using the trained initial graph neural network as a graph neural network if the confidence level is within a preset confidence interval.

[0155] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A behavior recognition apparatus, characterized by comprising: The device comprises: An acquisition module, configured to acquire image feature information of a medical image; the image feature information is obtained according to a number and a type of medical objects included in the medical image, the medical objects including medical tissue objects and medical instrument objects; A processing module, configured to determine a complexity parameter of the medical image according to the image feature information; A matching module, configured to determine a target branch in a graph neural network according to the complexity parameter; the graph neural network comprises a plurality of branches, and a number of network nodes corresponding to each branch is different; An identification module, configured to input the medical image into the graph neural network through the target branch to obtain a medical behavior identification result corresponding to the medical image; the medical behavior identification result includes medical object region identification and medical behavior category identification; The processing module is further configured to acquire a number of medical objects included in the medical image and a feature vector corresponding to the medical image according to the image feature information; in a plurality of label vectors, a target label vector matching the feature vector is acquired; each label vector is used to represent a medical behavior category; the complexity parameter of the medical image is determined according to the target label vector and the number of medical objects included in the medical image.

2. The apparatus of claim 1, wherein, The acquisition module is further configured to acquire a video frame image in the medical image; the video frame image includes at least one medical object; feature identification is performed on the video frame image to acquire image feature information of the video frame image; the image feature information is obtained according to a number and a type of medical objects included in the video frame image; and the image feature information of the medical image is acquired according to the image feature information.

3. The apparatus of claim 2, wherein, The acquisition module is further configured to acquire time attention features and spatial attention features of the video frame image respectively; and the image feature information of the video frame image is acquired according to the time attention features and the spatial attention features.

4. The apparatus of claim 1, wherein, The matching module is further configured to determine a number of network nodes corresponding to each branch in the graph neural network; a complexity label corresponding to each branch is determined according to the number of network nodes corresponding to each branch; in a plurality of complexity labels, a target complexity label matching the complexity parameter is determined, and a branch corresponding to the target complexity label is taken as the target branch.

5. The apparatus of claim 1, wherein, The device further comprises: A training module, configured to acquire an initial graph neural network; the initial graph neural network comprises a plurality of branches, and a number of network nodes corresponding to each branch is different; a training set is acquired; the training set comprises a plurality of training samples, each training sample includes a medical image sample and a sample label, the sample label is used to represent a medical object region in the medical image sample and a medical behavior category corresponding to the medical image; each training sample in the training set is input into the initial graph neural network through each branch of the initial graph neural network respectively to train the initial graph neural network, and the graph neural network is obtained.

6. A behavior recognition method characterized by, The method comprises: acquire image feature information of a medical image; the image feature information is obtained according to a number and a type of medical objects included in the medical image, the medical objects including a medical tissue object and a medical instrument object; determine a complexity parameter of the medical image according to the image feature information; determine a target branch in a graph neural network according to the complexity parameter; the graph neural network includes a plurality of branches, and a number of network nodes corresponding to each branch is different; input the medical image into the graph neural network through the target branch to obtain a medical behavior recognition result corresponding to the medical image; the medical behavior recognition result includes a medical object region identifier and a medical behavior category identifier; the determining the complexity parameter of the medical image according to the image feature information includes: acquiring a number of medical objects included in the medical image and a feature vector corresponding to the medical image according to the image feature information; acquiring a target label vector matching the feature vector from a plurality of label vectors; each label vector is used to represent a medical behavior category; determining the complexity parameter of the medical image according to the target label vector and the number of medical objects included in the medical image.

7. The method of claim 6, wherein, the determining the target branch in the graph neural network according to the complexity parameter includes: determining a number of network nodes corresponding to each branch in the graph neural network; determining a complexity label corresponding to each branch according to the number of network nodes corresponding to each branch; determining a target complexity label matching the complexity parameter from a plurality of complexity labels, and taking a branch corresponding to the target complexity label as the target branch.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, the computer program is executed by a processor to implement the steps of the method in any one of claims 6 to 7.

Images