A multimodal data fusion management system and method based on deep learning

Abstract

The present invention discloses a multimodal data fusion management system and method based on deep learning, which belongs to the field of deep learning technology. It includes: obtaining a surgical video of a patient during surgery, converting the surgical video into a picture part and an audio part; extracting effective operations of the picture part and characteristic information of the audio part respectively; obtaining effective information video in the surgical video according to the effective operations and characteristic information; classifying effective operations in the effective information video, and storing the effective information video according to effective operations of different classification levels. The present invention extracts and stores effective video information during the patient's surgery, and classifies the video data, thereby solving the problem that a large amount of repeated and useless medical data occupies a large amount of memory space, and when it is necessary to extract patient video data, it will also increase the extraction time and difficulty.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and specifically to a multimodal data fusion management system and method based on deep learning. Background Art

[0002] The monitoring and recording of the whole process of complex large-scale surgery is of great significance in many aspects. First, from the perspective of medical quality and safety, monitoring records can serve as objective evidence of the medical process. When medical disputes occur or the effect of surgery needs to be evaluated, it can provide clear and accurate information to help determine responsibilities and improve surgical procedures. Secondly, it is a valuable resource for teaching and training. Young doctors can learn surgical skills and operating standards by watching surgical records and improve their professional level. Furthermore, monitoring records help hospitals to conduct internal quality control and management. It can be used to evaluate the collaborative efficiency and operating standardization of the surgical team, find problems and make timely improvements. However, a large amount of video data that occupies a large amount of memory is generated during complex surgery, and there is no shortage of a large amount of repeated and useless video data, such as some preparation stages and the time to adjust the equipment, which may not be of research value due to different patient conditions. If all the videos of the surgical process are stored without data screening and extraction, a large amount of repeated and useless medical data information will occupy most of the memory space, and when it is necessary to extract patient video data, it will also increase the extraction time and difficulty. Summary of the invention

[0003] The purpose of the present invention is to provide a multimodal data fusion management system and method based on deep learning to solve the problems raised in the above background technology.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] A multimodal data fusion management method based on deep learning, comprising the following steps:

[0006] Step S100: Acquire a surgical video of a patient during surgery, and convert the surgical video into a picture part and an audio part;

[0007] Step S200: By analyzing the picture part, the audio part, and the patient data, respectively extracting the effective operations of the picture part and the characteristic information of the audio part;

[0008] Step S300: obtaining a valid information video in the surgical video according to the valid operation and feature information;

[0009] In order to obtain the completeness of the surgical record, ensure that the visual and auditory information of the surgery is complete and accurately corresponds, provide comprehensive and authentic information for subsequent review, teaching and research, as well as medical quality assessment, evaluate the synchronization and coordination of surgical operations, such as the coordination of the doctor's actions with relevant instructions or prompt sounds, and judge whether the surgical process is standardized and efficient. It is necessary to extract the information of the picture and audio parts to obtain the effective information video in the surgical video, that is, to obtain the complete event process;

[0010] Step S400: Classify the valid operations in the valid information video, and store the valid information video according to the valid operations at different classification levels.

[0011] Further, the step of extracting the valid operation of the screen part in step S200 includes:

[0012] Step S210: converting the image portion into an image sequence consisting of a plurality of image frames, and obtaining a ROI region and a target feature region in each image frame, wherein the ROI region is the surgical region of the patient, and the target feature region is the surgical instrument region directly in contact with the surgical region of the patient;

[0013] The ROI area is the surgical area of ​​the patient, which is determined according to the name of the patient's surgery, such as appendectomy, kidney transplantation, etc., and the appendix, kidney and other specific body parts are used as the surgical area; wherein the surgical instruments in this embodiment include: scalpel, tweezers, suture needle, and the target feature area is the surgical instrument area that directly contacts the patient's surgical area, which in this embodiment includes: scalpel head, tweezers head, needle head, because the scalpel head, tweezers head, and needle head are directly in contact with human body tissues during the surgical operation, and their position and shape are crucial to the accuracy and safety of the surgery, so they are used as the target feature area;

[0014] Step S220: Starting from the first image frame, search backwards in order. If the target feature region TReg exists in the mth image frame, m ROI area Reg ROI , and satisfy Then start from the mth image frame and search backwards in order. If the target feature region TReg exists in the nth image frame n ROI area Reg ROI , and satisfy

[0015] Then obtain the target feature region TReg corresponding to the n-1th and n+1th image frames respectively n-1 , and TReg n+1 ;

[0016] Step S230: Randomly obtain the target feature region TReg n-1, TReg n and TReg n+1 The coordinates of several pixels in the image are calculated, and the average coordinates are obtained to obtain the target feature point P. n-1 , P n and P n+1 , and then get the vector and vector The vector The starting point is P n-1 , the end point is P n ,vector The starting point is P n , the end point is P n+1 , and get the angle between the two vectors If satisfied Then, the image frames between the m-th image frame and the n-th image frame are collected to obtain a valid image frame set, the surgical operation corresponding to the valid image frame set is the valid operation, the time corresponding to the m-th image frame and the n-th image frame is respectively used as the start time and end time of the valid operation, and the range between the start time and the end time is the valid operation time range;

[0017] vector The starting point is P n-1 , the end point is P n ,vector The starting point is P n , the end point is P n+1 The target feature points obtained are random points of the surgical knife head, forceps head and needle head. The target feature points are random points that can reflect the moving direction and position of the surgical instrument. According to the target feature points, it can be judged whether the effective operation is completed.

[0018] Step S240: taking the target feature area corresponding to the valid image frame set as the operation area, obtaining each operation area, and if there is an overlapped area between two temporally adjacent operation areas, and the area of ​​the overlapped area is greater than the area threshold, marking the overlapped area, and summarizing the overlapped areas that are continuous in time and greater than the time threshold and are marked, to obtain the valid operation area corresponding to the valid operation;

[0019] The effective operation area is the area in the patient's surgical area that needs to be focused on. This is usually because the treatment results of these areas have a crucial impact on the success of the operation, the patient's recovery and prognosis, such as the main location of the patient's lesions, the location that requires frequent or delicate operations, and the high-risk and susceptible location. For example, in cardiac surgery, the anastomosis site of the coronary artery is an effective operation area, which requires delicate operations and close attention to ensure the success of the operation and the recovery of the patient's heart function after surgery;

[0020] Step S250: repeat steps S220 to S250 to obtain all valid operations and corresponding valid operation areas, and establish a valid operation time set.

[0021] Furthermore, extracting characteristic information from the audio part in step S200 includes: converting the audio part into a time domain graph, acquiring all time periods in which the audio amplitude is greater than the amplitude threshold, and establishing an audio time period set as characteristic information of the audio part.

[0022] Furthermore, step S300 includes:

[0023] Step S310: obtaining several audio time ranges in the audio time period set that are not in the valid operation time set. Taking the audio time range ad as an example, the corresponding time range is The end time of the last valid operation corresponding to range ad is Tp p , the start time of the next valid operation is Tp q , if satisfied The end time Tp q The time between the range ad is taken as the effective transition time, if it satisfies The range ad and the start time Tp q The time between is taken as the effective transition time, and then the effective transition time set is obtained;

[0024] Step S320: The corresponding times in the valid operation time set, the audio time period set and the valid transition time set are respectively used as TIFR IMF ,TIFR AUF and TIFR EFT , and will meet TIFR IMF ∪TIFR AUF ∪TIFR EFT The video corresponding to the time range is taken as the valid information video.

[0025] Furthermore, step S400 includes:

[0026] Step S410: Obtain the effective operation area corresponding to each effective operation, and gather all effective operation areas in one image. If the number of effective operation areas corresponding to a certain pixel in the image is greater than the area number threshold, mark the pixel, and gather all marked pixels to obtain the surgical important area Reg im ;

[0027] Step S420: Taking the effective operation eo as an example, the effective operation region corresponding to the effective operation eo is Reg eo , the corresponding effective operation time range is TIFR eo , effective operating area Reg eoReg im The area of ​​the intersection is Area eo ∩im , the operational characteristic coefficient of the effective operation eo is obtained as: Among them, β eo is the correlation coefficient of effective operation eo, Area eo Reg is the effective operating area eo area;

[0028] Step S430: Classify the effective operations into different levels according to the operation characteristic coefficients of the effective operations, and store the effective information videos according to the effective operations of different levels.

[0029] In this embodiment, the operation characteristic coefficients are normalized, and the levels of these effective operations are divided into important operation levels, general operation levels and minor operation levels.

[0030] A multimodal data fusion management system based on deep learning, including the following modules:

[0031] Video conversion module: used to obtain the surgical video of the patient during surgery and convert the surgical video into the picture part and the audio part;

[0032] Analysis and extraction module: used to analyze the image part, the audio part, and the patient data to extract the effective operation of the image part and the characteristic information of the audio part;

[0033] Effective information video module: used to obtain effective information video in the surgical video according to the effective operation and feature information;

[0034] Operation classification storage module: used to classify the valid operations in the valid information video, and store the valid information video according to the valid operations of different classification levels.

[0035] Furthermore, the analysis and extraction module includes the following units:

[0036] Acquisition area unit: used to convert the picture part into an image sequence consisting of several image frames, and acquire the ROI area and target feature area in each image frame;

[0037] Image frame unit: used to search and analyze the image frames in order to obtain the mth and nth image frames that meet the conditions;

[0038] An effective operation unit: used for collecting image frames between the mth image frame and the nth image frame to obtain an effective image frame set, and the surgical operation corresponding to the effective image frame set is a effective operation;

[0039] Valid operation area unit: used to obtain the valid operation area corresponding to the valid operation according to the valid operation;

[0040] Valid operation time set unit: used to obtain all valid operations and corresponding valid operation areas, and establish a valid operation time set.

[0041] Furthermore, the effective information video module includes the following units:

[0042] An effective transition time set unit: used to obtain an effective transition time set according to the effective operation and feature information;

[0043] Valid information video unit: used to obtain valid information video according to a valid operation time set, an audio time period set and a valid transition time set.

[0044] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention provides a multimodal data fusion management system and method based on deep learning, including: obtaining the surgical video of the patient during the operation, converting the surgical video into a picture part and an audio part; extracting the effective operation of the picture part and the characteristic information of the audio part respectively; obtaining the effective information video in the surgical video according to the effective operation and the characteristic information; classifying the effective operation in the effective information video, and storing the effective information video according to the effective operation of different classification levels. The present invention extracts and stores the effective video information of the patient during the operation, and classifies the video data, thereby solving the problem that a large amount of repeated and useless medical data occupies a large amount of memory space, and when the patient video data needs to be extracted, it will also increase the extraction time and difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0046] Figure 1 It is a flowchart of a multimodal data fusion management method based on deep learning of the present invention;

[0047] Figure 2 It is a structural diagram of a multimodal data fusion management system based on deep learning in the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] See also Figure 1-Figure 2 The present invention provides a multimodal data fusion management system and method based on deep learning, and the corresponding flowchart of the multimodal data fusion management method based on deep learning is as follows: Figure 1 As shown. The following steps are included:

[0050] Step S100: Acquire a surgical video of a patient during surgery, and convert the surgical video into a picture part and an audio part;

[0051] The surgical video during the operation on the patient is recorded, and the surgical video is converted into the picture part and the audio part through the existing video processing software (such as Adobe Premiere Pro) or Python language programming library (such as opencv-python for processing the picture part and moviepy for extracting the audio part). The separation process is the existing technology and will not be repeated here.

[0052] Step S200: By analyzing the picture part, the audio part, and the patient data, respectively extracting the effective operations of the picture part and the characteristic information of the audio part;

[0053] The steps to extract the effective operation of the picture part include the following:

[0054] Step S210: converting the image portion into an image sequence consisting of a plurality of image frames, and obtaining a ROI region and a target feature region in each image frame, wherein the ROI region is the surgical region of the patient, and the target feature region is the surgical instrument region directly in contact with the surgical region of the patient;

[0055] The ROI area is the surgical area of ​​the patient, which is determined according to the name of the patient's surgery, such as appendectomy, kidney transplantation, etc., and specific body parts such as the appendix and kidney are used as the surgical area; wherein the surgical instruments in this embodiment include: scalpels, tweezers, and suture needles, and the target feature area is the surgical instrument area that directly contacts the patient's surgical area, which in this embodiment includes: scalpel heads, tweezer heads, and needles, because the scalpel heads, tweezer heads, and needles are in direct contact with human body tissues during the surgical operation, and their position and shape are crucial to the accuracy and safety of the surgery, so they are used as the target feature area.

[0056] Step S220: Starting from the first image frame, search backwards in order. If the target feature region TReg exists in the mth image frame, m ROI area Reg ROI , and satisfy Then start from the mth image frame and search backwards in order. If the target feature region TReg exists in the nth image frame n ROI area Reg ROI , and satisfy Then obtain the target feature region TReg corresponding to the n-1th and n+1th image frames respectively n-1 , and TReg n+1 ;

[0057] Step S230: Randomly obtain the target feature region TReg n-1 , TReg n and TReg n+1 The coordinates of several pixels in the image are calculated, and the average coordinates are obtained to obtain the target feature point P. n-1 , P n and P n+1 , and then get the vector and vector The vector The starting point is P n-1 , the end point is P n ,vector The starting point is P n , the end point is P n+1 , and get the angle between the two vectors If satisfied Then, the image frames between the m-th image frame and the n-th image frame are collected to obtain a valid image frame set, the surgical operation corresponding to the valid image frame set is the valid operation, the time corresponding to the m-th image frame and the n-th image frame is respectively used as the start time and end time of the valid operation, and the range between the start time and the end time is the valid operation time range;

[0058] The target feature points obtained are random points of the surgical knife head, forceps head and needle. The target feature points are random points that can reflect the movement direction and position of the surgical instrument. According to the target feature points, it can be judged whether the effective operation is completed.

[0059] Step S240: Take the corresponding target feature area in the valid image frame set as the operation area to obtain each operation area. If there is an overlapping area between two temporally adjacent operation areas, and the area of ​​the overlapping area is greater than the area threshold, the overlapping area is marked, and the overlapping areas that are continuous in time and greater than the time threshold and are all marked are summarized to obtain the effective operation area corresponding to the effective operation.

[0060] Since the areas of two temporally adjacent operation areas are the same, the area threshold in this embodiment is the operation area multiplied by 0.7, and the time threshold in this embodiment is the effective operation time range multiplied by 0.5.

[0061] The effective operation area is the area in the patient's surgical area that needs to be focused on. This is usually because the treatment results of these areas have a crucial impact on the success of the operation, the patient's recovery and prognosis, such as the main location of the patient's lesions, the location that requires frequent or delicate operations, and the high-risk and susceptible location. For example, in cardiac surgery, the anastomosis site of the coronary artery is an effective operation area that requires delicate operations and close attention to ensure the success of the operation and the recovery of the patient's heart function after surgery.

[0062] Step S250: repeat steps S220 to S250 to obtain all valid operations and corresponding valid operation areas, and establish a valid operation time set.

[0063] Step S260: extracting characteristic information from the audio part includes: converting the audio part into a time domain graph, obtaining all time periods in which the audio amplitude is greater than the amplitude threshold, and establishing an audio time period set as the characteristic information of the audio part.

[0064] The time period when the amplitude is greater than the amplitude threshold is usually speech, the sound of equipment or abnormal sound, and these data are of great significance to the turning point of the operation. In this embodiment, the amplitude threshold is 40dB.

[0065] Step S300: Obtaining effective information video in the surgical video according to the effective operation and feature information.

[0066] In order to obtain the completeness of the surgical record, ensure that the visual and auditory information of the surgery is complete and accurately corresponds, provide comprehensive and authentic data for subsequent review, teaching and research, as well as medical quality assessment, evaluate the synchronization and coordination of surgical operations, such as the coordination of the doctor's actions with relevant instructions or prompt sounds, and judge whether the surgical process is standardized and efficient. It is necessary to extract the information of the picture and audio parts to obtain the effective information video in the surgical video, that is, to obtain the complete event process.

[0067] Step S310: obtaining several audio time ranges in the audio time period set that are not in the valid operation time set. Taking the audio time range ad as an example, the corresponding time range is The end time of the last valid operation corresponding to range ad is Tp p , the start time of the next valid operation is Tp q , if satisfied The end time Tp p and

[0068] The time between range ad is taken as the effective transition time, if it satisfies The range ad and the start time Tp q The time between is taken as the effective transition time, and then the effective transition time set is obtained;

[0069] In this embodiment, k sp =k qf =0.3.

[0070] Step S320: The corresponding times in the valid operation time set, the audio time period set and the valid transition time set are respectively used as TIFR IMF ,TIFR AUF and TIFR EFT , and will meet TIFR IMF ∪TIFR AUF ∪TIFR EFT The video corresponding to the time range is taken as the valid information video.

[0071] Step S400: Classify the valid operations in the valid information video, and store the valid information video according to the valid operations at different classification levels.

[0072] Step S410: Obtain the effective operation area corresponding to each effective operation, and gather all effective operation areas in one image. If the number of effective operation areas corresponding to a certain pixel in the image is greater than the area number threshold, mark the pixel, and gather all marked pixels to obtain the surgical important area Reg im ;

[0073] Through existing software, it is possible to gather all valid operating areas in one image and obtain the number of valid operating areas corresponding to any pixel in the image. In this embodiment, the area number threshold is the number of valid operating areas multiplied by 0.7.

[0074] Step S420: Taking the effective operation eo as an example, the effective operation region corresponding to the effective operation eo is Reg eo , the corresponding effective operation time range is TIFR eo , effective operating area Reg eo Reg im The area of ​​the intersection is Area eo ∩im , the operational characteristic coefficient of the effective operation eo is obtained as: Among them, β eo is the correlation coefficient of effective operation eo, Area eo Reg is the effective operating area eo area.

[0075] Reflects the effective operating area Reg eo In the surgically important area Reg im If the effective operation area Reg eo All in the important surgical areaReg im This indicates that this operation is important and has a significant impact on the surgery.

[0076] Step S430: Classify the effective operations into different levels according to the operation characteristic coefficients of the effective operations, and store the effective information videos according to the effective operations of different levels.

[0077] In this embodiment, the operation characteristic coefficients are normalized, and the levels of these effective operations are divided into important operation levels, general operation levels, and minor operation levels. Among them, the important operation level is an effective operation that takes time and requires attention in an operation, and the operation characteristic coefficient is [0.7, 1]; the general operation level is an operation that is not as important as an important and effective operation in an operation, but still has a certain positive impact on the smooth progress and effect of the operation, such as some routine tissue processing, wound cleaning, etc., and the operation characteristic coefficient is [0.4, 0.7); the minor operation level is an operation that is relatively auxiliary in an operation and has little impact on the core goals and results of the operation, such as some non-core parts or operations that take less time, and the operation characteristic coefficient is [0, 0.4).

[0078] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0079] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal data fusion management method based on deep learning, characterized in that: The following steps are involved: Step S100: Acquire a surgical video of a patient during surgery, and convert the surgical video into a picture part and an audio part; Step S200: By analyzing the picture part, the audio part, and the patient data, respectively extracting the effective operations of the picture part and the characteristic information of the audio part; Step S300: obtaining a valid information video in the surgical video according to the valid operation and feature information; Step S400: classifying the valid operations in the valid information video, and storing the valid information video according to the valid operations of different classification levels; Step S210: converting the image portion into an image sequence consisting of a plurality of image frames, and obtaining a ROI region and a target feature region in each image frame, wherein the ROI region is the surgical region of the patient, and the target feature region is the surgical instrument region directly in contact with the surgical region of the patient; Step S220: Starting from the first image frame, search backwards in order. If the target feature region exists in the mth image frame, With ROI area , and satisfy , then start with the mth image frame and search backwards in order. If the nth image frame contains the target feature area With ROI area , and satisfy , then obtain the target feature areas corresponding to the n-1th and n+1th image frames respectively ,and ; Step S230: Randomly obtain target feature areas , and The coordinates of several pixels in the image are calculated and the average coordinates are used to obtain the target feature points. , and , and then get the vector and vector , the vector The starting point is , the end point is ,vector The starting point is , the end point is , and get the angle between the two vectors , if satisfied , the image frames between the m-th image frame and the n-th image frame are collected to obtain a valid image frame set, the surgical operation corresponding to the valid image frame set is the valid operation, the time corresponding to the m-th image frame and the n-th image frame is respectively used as the start time and end time of the valid operation, and the range between the start time and the end time is the valid operation time range; Step S240: taking the target feature area corresponding to the valid image frame set as the operation area, obtaining each operation area, and if there is an overlapped area between two temporally adjacent operation areas, and the area of ​​the overlapped area is greater than the area threshold, marking the overlapped area, and summarizing the overlapped areas that are continuous in time and greater than the time threshold and are marked, to obtain the valid operation area corresponding to the valid operation; Step S250: repeat steps S220 to S250 to obtain all valid operations and corresponding valid operation areas, and establish a valid operation time set.

2. According to the multimodal data fusion management method based on deep learning in claim 1, it is characterized in that: Extracting characteristic information from the audio part in step S200 includes: converting the audio part into a time domain graph, acquiring all time periods in which the audio amplitude is greater than the amplitude threshold, and establishing an audio time period set as characteristic information of the audio part.

3. The multimodal data fusion management method based on deep learning according to claim 2 is characterized in that: Step S300 includes: Step S310: obtaining a number of audio time ranges in the audio time range set that are not in the valid operation time set, and using the audio time range For example, the corresponding time range is ,scope The corresponding end time of the last valid operation is , the start time of the next valid operation is , if satisfied , the end time With range The time between is taken as the effective transition time, if it satisfies , the range With start time The time between is taken as the effective transition time, and then the effective transition time set is obtained; Step S320: The corresponding times in the valid operation time set, the audio time period set and the valid transition time set are respectively used as , and , and will satisfy The video corresponding to the time range is taken as the valid information video.

4. The multimodal data fusion management method based on deep learning according to claim 3 is characterized in that: Step S400 includes: Step S410: Obtain the effective operation area corresponding to each effective operation, and gather all effective operation areas in one image. If the number of effective operation areas corresponding to a certain pixel in the image is greater than the area number threshold, the pixel is marked, and all marked pixels are gathered to obtain the important surgical area. ; Step S420: Effective operation For example, effective operation The corresponding effective operating area is , the corresponding effective operation time range is , effective operating area Important surgical areas The area of ​​intersection is , get effective operation The operating characteristic coefficient is: ,in, It’s about effective operation The correlation coefficient of Effective operating area area; Step S430: Classify the effective operations into different levels according to the operation characteristic coefficients of the effective operations, and store the effective information videos according to the effective operations of different levels.

5. A multimodal data fusion management system for executing a multimodal data fusion management method based on deep learning as described in any one of claims 1 to 4, characterized in that: Includes the following modules: Video conversion module: used to obtain the surgical video of the patient during surgery and convert the surgical video into the picture part and the audio part; Analysis and extraction module: used to analyze the image part, the audio part, and the patient data to extract the effective operation of the image part and the characteristic information of the audio part; Effective information video module: used to obtain effective information video in the surgical video according to the effective operation and feature information; Operation classification storage module: used to classify the valid operations in the valid information video, and store the valid information video according to the valid operations of different classification levels.

6. The multimodal data fusion management system according to claim 5, characterized in that: The analysis and extraction module includes the following units: Acquisition area unit: used to convert the picture part into an image sequence consisting of several image frames, and acquire the ROI area and target feature area in each image frame; Image frame unit: used to search and analyze the image frames in order to obtain the mth and nth image frames that meet the conditions; An effective operation unit: used for collecting image frames between the mth image frame and the nth image frame to obtain an effective image frame set, and the surgical operation corresponding to the effective image frame set is a effective operation; Valid operation area unit: used to obtain the valid operation area corresponding to the valid operation according to the valid operation; Valid operation time set unit: used to obtain all valid operations and corresponding valid operation areas, and establish a valid operation time set.

7. The multimodal data fusion management system according to claim 6, characterized in that: The effective information video module includes the following units: An effective transition time set unit: used to obtain an effective transition time set according to the effective operation and feature information; Valid information video unit: used to obtain valid information video according to a valid operation time set, an audio time period set and a valid transition time set.

Images