Electronic endoscope diagnosis and treatment data processing system and method based on multi-modal data

Abstract

The invention relates to the technical field of data processing, and discloses an electronic endoscope diagnosis and treatment data processing system and method based on multi-modal data, and the system comprises a diagnosis and treatment data integration display module, a diagnosis and treatment data adjustment module, and a diagnosis and treatment data adjustment result output module. According to the real-time image information and the physiological parameters of the digestive tract of the patient, independent and accurate data fusion processing is conducted on the real-time image information and the physiological parameters of the digestive tract of the patient by combining video editing software; according to the alimentary canal diagnosis and treatment data adjustment demand standard word identification data, combining an intelligent search algorithm with different types of scientifically stored alimentary canal diagnosis and treatment data adjustment demand control instructions, carrying out accurate matching processing on the alimentary canal diagnosis and treatment data adjustment demand control instructions of the patient; the regulation requirement instructed scientific analysis of the display position, the arrangement mode, the display size and the display effect of the physiological parameters in the real-time image of the digestive tract is realized, and the intelligence and the flexibility of electronic endoscope diagnosis and treatment data processing are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a system and method for processing electronic endoscope diagnosis and treatment data based on multimodal data. Background Art

[0002] During gastrointestinal surgery, doctors are required to use a gastrointestinal electronic endoscope to collect real-time images of the patient's gastrointestinal tract and to dynamically monitor the patient's physiological parameters such as blood pressure, heart rate, and blood oxygen saturation through a monitor. Currently, the real-time images of the gastrointestinal tract collected by the gastrointestinal electronic endoscope and the physiological parameters of the patient such as blood pressure, heart rate, and blood oxygen saturation collected by the monitor are independently displayed on the display screen. Therefore, it is not conducive to doctors to accurately and efficiently obtain the real-time images and physiological parameters of the patient's gastrointestinal tract during the patient's gastrointestinal surgery. Existing gastrointestinal surgeries cannot realize dynamic synchronous integrated display of real-time images of the gastrointestinal tract and physiological parameters, nor can they conveniently and intelligently adjust the display position, display arrangement, display size and display effect of the physiological parameters in the real-time images of the gastrointestinal tract, thereby increasing the risk of gastrointestinal surgery.

[0003] The Chinese invention patent application with publication number CN119622629A discloses an intelligent monitoring method and system for the wearing status of miners' clothing based on physiological feature fusion, which collects the wearer's heart rate, body temperature and respiratory rate signals; obtains the wearer's motion state data; based on the heart rate, body temperature and respiratory rate signals, uses an edge computing module to perform local data processing to obtain physiological feature data; based on the physiological feature data and motion state data, uses a multimodal data fusion algorithm to generate fused feature data; based on the fused feature data, uses a deep learning algorithm to establish a personalized physiological baseline model; based on the personalized physiological baseline model, determines whether the wearer's wearing status is abnormal; when the wearing status is determined to be abnormal, triggers a vibration reminder; transmits the wearing status and warning information to the supervision center through a wireless communication module, and can dynamically adjust the monitoring parameters according to the working environment and task characteristics; however, the above technical solution cannot realize the dynamic fusion output of the wearer's physiological parameters and motion state image parameters, and cannot achieve synchronous feedback of the wearer's physiological characteristics. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In order to solve the problem that the above-mentioned existing gastrointestinal surgery cannot realize dynamic synchronous integrated display of real-time images and physiological parameters of the digestive tract, and cannot conveniently and intelligently adjust the display position, display arrangement, display size and display effect of physiological parameters in the real-time images of the digestive tract, thereby increasing the risk of gastrointestinal surgery, the purpose of realizing the dynamic synchronous integrated display of the above-mentioned patient physiological parameters and digestive tract images, intelligently adjusting the display position, arrangement, size and effect of the patient physiological parameters in the digestive tract images, and increasing the efficiency and quality of gastrointestinal surgery is achieved.

[0006] (2) Technical solution

[0007] The present invention is implemented through the following technical solution: a system and method for processing electronic endoscope diagnosis and treatment data based on multimodal data, the method comprising the following steps:

[0008] S1. Collecting real-time image data of the patient's digestive tract and the patient's physiological characteristics data;

[0009] S2. Performing data fusion processing on the patient's digestive tract real-time image data and the patient's physiological characteristic data based on the patient's digestive tract real-time image data and the patient's physiological parameter data to generate real-time fusion data of the patient's digestive tract image and physiological parameter;

[0010] S3, outputting the patient's digestive tract real-time image information and physiological parameter fusion data according to the patient's digestive tract image and physiological parameter real-time fusion data;

[0011] S4. Collecting digestive tract diagnosis and treatment data adjustment demand text data and performing digestive tract diagnosis and treatment data adjustment demand standard word recognition processing on the patient's digestive tract diagnosis and treatment data adjustment demand standard word of different types to generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data;

[0012] S5. Matching the patient's digestive tract diagnosis and treatment data adjustment demand control instruction with the digestive tract diagnosis and treatment data adjustment demand standard word recognition data and different types of digestive tract diagnosis and treatment data adjustment demand control instruction control instructions to generate digestive tract diagnosis and treatment data adjustment demand control instruction matching data;

[0013] S6, performing digestive tract diagnosis and treatment data adjustment processing on the patient based on the real-time fusion data of the patient's digestive tract image and physiological parameters and the digestive tract diagnosis and treatment data adjustment demand control instruction matching data, to generate real-time adjustment data of the patient's digestive tract image and physiological parameters;

[0014] S7. Construct the digestive tract diagnosis and treatment data adjustment result data and execute the operation of outputting the adjustment result by fusing the patient's digestive tract real-time image information with physiological parameters.

[0015] Preferably, the steps of collecting the patient's digestive tract real-time image data and the patient's physiological characteristic data are as follows:

[0016] S11, collecting real characteristic images of the digestive tract of a patient undergoing digestive tract surgery online through a digestive tract electronic endoscope, and generating real-time image data Z of the patient's digestive tract;

[0017] The physiological parameters of the patient during the digestive tract surgery are collected online through a monitor, and the patient's physiological characteristic data Y is generated. The patient's physiological characteristic data includes blood pressure physiological characteristic data, heart rate physiological characteristic data and blood oxygen saturation physiological characteristic data of patients with digestive tract diseases.

[0018] Preferably, the steps of performing data fusion processing of the patient's digestive tract real-time image data and the patient's physiological characteristic data to generate the patient's digestive tract image and physiological parameter real-time fusion data are as follows:

[0019] S21. Use video editing software to map and fuse the blood pressure physiological characteristic data, heart rate physiological characteristic data and blood oxygen saturation physiological characteristic data in the patient's physiological characteristic data Y into the image corresponding to the patient's digestive tract real-time image data Z in an orderly horizontal or vertical arrangement according to the physiological parameter type, and generate real-time fusion data P of the patient's digestive tract image and physiological parameters, where P = (Z, Y). The video editing software includes any one of Adobe Premiere, After Effects, Final Cut Pro, and Da Vinci Resolve.

[0020] Preferably, the steps for outputting the patient's digestive tract real-time image information and physiological parameter fusion data according to the patient's digestive tract image and physiological parameter real-time fusion data are as follows:

[0021] S31. Input the real-time fusion data P of the patient's digestive tract image and physiological parameters into a wearable AR display device and feed it back to the digestive tract surgery personnel in real time, and perform the real-time output operation of the patient's digestive tract real-time image information and physiological parameter fusion data. The wearable AR display device includes any one of head-mounted AR glasses and mobile AR devices.

[0022] Preferably, the steps of collecting digestive tract diagnosis and treatment data adjustment demand text data and performing digestive tract diagnosis and treatment data adjustment demand standard word recognition processing on the patient with different types of digestive tract diagnosis and treatment data adjustment demand standard words to generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data are as follows:

[0023] S41. Online collection of textual feedback and suggestions from digestive tract surgery personnel regarding the output results of the digestive tract real-time image information and physiological parameter fusion information corresponding to the real-time fusion data P of the patient's digestive tract image and physiological parameters through a voice input module installed in a wearable AR display device, and generation of digestive tract diagnosis and treatment data adjustment demand text data O, wherein the digestive tract diagnosis and treatment data adjustment demand text data includes display position adjustment text information, display arrangement adjustment text information, display size adjustment text information, and display effect adjustment text information of the patient's physiological parameters in the digestive tract real-time image; the display position adjustment text information includes upper left position adjustment text information, left middle position adjustment text information, lower left position adjustment text information, upper middle position adjustment text information, middle position adjustment text information, and lower middle position adjustment text information of the physiological parameters in the digestive tract real-time image. Adjustment text information, upper right position adjustment text information, right middle position adjustment text information and lower right position adjustment text information; display arrangement adjustment text information includes horizontal arrangement adjustment text information and vertical arrangement adjustment text information of physiological parameters in real-time images of the digestive tract; display size adjustment text information includes doubling adjustment text information, doubling adjustment text information, tripling adjustment text information, quadrupling adjustment text information, quintuple adjustment text information, doubling adjustment text information, tripling adjustment text information, quadruple adjustment text information and quintuple adjustment text information of physiological parameters in real-time images of the digestive tract; display effect adjustment text information includes transparent display effect adjustment text information and normal display effect adjustment text information of physiological parameters in real-time images of the digestive tract;

[0024] S42. Establish a standard word set A = (a1, ..., a m ,…,a σ ), m=1,2,3,…,σ; where a m represents the different types of digestive tract diagnosis and treatment data adjustment requirement standard words corresponding to the m-th digestive tract diagnosis and treatment data output adjustment requirement type, σ represents the maximum number of digestive tract diagnosis and treatment data output adjustment requirement types, the digestive tract diagnosis and treatment data output adjustment requirement type represents an index data type mainly formed by combining the physiological parameter display position adjustment requirement text information, display arrangement adjustment requirement text information, display size adjustment requirement text information and display effect adjustment requirement text information in the digestive tract real-time image, and is used to search for specific digestive tract diagnosis and treatment data adjustment requirement standard words; the different types of digestive tract diagnosis and treatment data adjustment requirement standard words represent the standard text information of the data display status of the adjustment physiological parameters in the digestive tract real-time image set according to different types of digestive tract diagnosis and treatment data output adjustment requirement standards;

[0025] S43, comparing the digestive tract diagnosis and treatment data adjustment requirement text data O with the different types of digestive tract diagnosis and treatment data adjustment requirement standard word set A. m Perform digestive tract diagnosis and treatment data adjustment demand character matching to search for the digestive tract diagnosis and treatment data adjustment demand text data O corresponding to the different types of digestive tract diagnosis and treatment data adjustment demand standard words a m , and generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data a through data identification mubiao , execute to generate the digestive tract diagnosis and treatment data adjustment demand standard word recognition data a mubiao The specific steps are as follows:

[0026] S431, setting the standard words a for adjusting the requirements of different types of digestive tract diagnosis and treatment data m Assume that T=T0 is the current solution, set the initial temperature T0 for starting annealing in the search space of the standard word set A for adjusting the requirements of different types of digestive tract diagnosis and treatment data, and randomly generate a standard word a for adjusting the requirements of different types of digestive tract diagnosis and treatment data that matches the text data O for adjusting the requirements of digestive tract diagnosis and treatment data in the search space of the standard word set A for adjusting the requirements of different types of digestive tract diagnosis and treatment data. m Initial solution Θ r , and calculate the objective function value Φ(Θ r );

[0027] S432, generating the standard words a for adjusting the requirements of different types of digestive tract diagnosis and treatment data m New interpretation, and the standard word for adjusting the requirements of different types of digestive tract diagnosis and treatment data a m The difference of the current solution, according to the different types of digestive tract diagnosis and treatment data, adjust the demand standard word a m For the current solution Θ i Perform disturbance to generate a standard word a for the different types of digestive tract diagnosis and treatment data adjustment requirements text data O. m New solution Θ j , calculate the objective function value Φ(Θ i ) and the objective function value Φ(Θ j ), we get the objective function variation ΔΦ=Φ(Θ j )-Φ(Θ i );

[0028] S433, determine the different types of digestive tract diagnosis and treatment data adjustment requirements standard word a m Whether the new solution is accepted, if the objective function change ΔΦ<0, then the different types of digestive tract diagnosis and treatment data adjustment demand standard word a m New solution Θ jAccepted; if the objective function change ΔΦ≥0, then the different types of digestive tract diagnosis and treatment data adjustment demand standard word a m New solution Θ j Accepted with the following probability, T i represents the current temperature; the probability formula is Where T i represents the current temperature, and exp represents a random function with a value range of (0, 1);

[0029] S434, when the different types of digestive tract diagnosis and treatment data adjustment demand standard word a m When the new interpretation is accepted, the standard word for the adjustment of the requirements for different types of digestive tract diagnosis and treatment data a m New solution Θ j It is regarded as the standard word for the adjustment requirements of different types of digestive tract diagnosis and treatment data a m Current solution;

[0030] S435, loop through step S431 to step S434, at temperature T i Under these conditions, the perturbation and acceptance process is repeated Λ times, and then the next step is executed;

[0031] S436. Finally find the global optimal solution: determine whether T has reached the termination temperature T zui If yes, the algorithm is terminated and the standard word a of the different types of digestive tract diagnosis and treatment data adjustment requirements that best matches the digestive tract diagnosis and treatment data adjustment requirements text data O is output. m If not, go to loop step S435 and continue executing;

[0032] S437, the output of step S436 for adjusting the demand standard words for different types of digestive tract diagnosis and treatment data a m Generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data a through data identification mubiao .

[0033] Preferably, the control instruction matching processing of the patient's digestive tract diagnosis and treatment data adjustment demand is performed based on the digestive tract diagnosis and treatment data adjustment demand standard word recognition data and different types of digestive tract diagnosis and treatment data adjustment demand control instructions, and the operating steps for generating digestive tract diagnosis and treatment data adjustment demand control instruction matching data are as follows:

[0034] S51, establish different types of digestive tract diagnosis and treatment data adjustment demand control instruction set B = (b1, ..., b m ,…,b σ ), where b mIndicates different types of digestive tract diagnosis and treatment data adjustment demand control instructions corresponding to the mth type of digestive tract diagnosis and treatment data output adjustment demand type, wherein the different types of digestive tract diagnosis and treatment data adjustment demand control instructions represent video editing software control instruction information for adjusting the display status of physiological parameters in a real-time digestive tract image according to different types of digestive tract diagnosis and treatment data output adjustment demand standards;

[0035] S52, using an iterative deepening search algorithm to adjust the digestive tract diagnosis and treatment data to the required standard word recognition data a mubiao The different types of digestive tract diagnosis and treatment data adjustment demand control instructions b in the different types of digestive tract diagnosis and treatment data adjustment demand control instructions set B m Perform keyword matching on the digestive tract diagnosis and treatment data output adjustment requirement, and search for the digestive tract diagnosis and treatment data adjustment requirement standard word recognition data a mubiao The corresponding different types of digestive tract diagnosis and treatment data adjustment demand control instructions b m , and generate digestive tract diagnosis and treatment data adjustment demand control instruction matching data b through data identification mubiao .

[0036] Preferably, the patient's digestive tract diagnosis and treatment data is adjusted based on the real-time fusion data of the patient's digestive tract image and physiological parameters and the digestive tract diagnosis and treatment data adjustment demand control instruction matching data to generate the patient's digestive tract image and physiological parameters real-time adjustment data in the following steps:

[0037] S61, using video editing software to adjust the demand control instruction matching data b according to the digestive tract diagnosis and treatment data mubiao The corresponding gastrointestinal diagnosis and treatment data adjustment demand control instructions orderly perform data adjustment processing on the display position, display arrangement, display size and display effect of the patient's physiological characteristic data Y in the patient's gastrointestinal image and physiological parameter real-time fusion data P in the patient's gastrointestinal real-time image data Z, and generate the patient's gastrointestinal image and physiological parameter real-time adjustment data L.

[0038] Preferably, the steps of constructing the digestive tract diagnosis and treatment data adjustment result data and outputting the operation of fusing the patient's digestive tract real-time image information with physiological parameters to adjust the result are as follows:

[0039] S71, constructing digestive tract diagnosis and treatment data adjustment result data F by data identification of the generated patient digestive tract image and physiological parameter real-time adjustment data L;

[0040] S72. Feedback the digestive tract diagnosis and treatment data adjustment result data F to the digestive tract surgery personnel in real time through the wearable AR display device, and perform the operation of outputting the adjustment result by fusing the patient's digestive tract real-time image information with physiological parameters.

[0041] An electronic endoscope diagnosis and treatment data processing system based on multimodal data, used to implement the electronic endoscope diagnosis and treatment data processing method based on multimodal data, the system comprising a diagnosis and treatment data integrated display module, a diagnosis and treatment data adjustment module, and a diagnosis and treatment data adjustment result output module;

[0042] The diagnosis and treatment data integrated display module includes a patient digestive tract real-time image acquisition unit, a patient physiological parameter acquisition unit, a patient digestive tract diagnosis and treatment data fusion unit, and a patient digestive tract diagnosis and treatment data output unit;

[0043] The patient's digestive tract real-time image acquisition unit acquires real-time image data of the patient's digestive tract through a digestive tract electronic endoscope; the patient's physiological parameter acquisition unit acquires the patient's physiological characteristic data through a monitor; the patient's digestive tract diagnosis and treatment data fusion unit performs data fusion processing of the patient's digestive tract real-time image information and physiological parameters based on the patient's digestive tract real-time image data and the patient's physiological characteristic data in combination with video editing software to generate real-time fusion data of the patient's digestive tract image and physiological parameters; the patient's digestive tract diagnosis and treatment data output unit performs an operation of outputting the patient's digestive tract real-time image information and physiological parameter fusion data based on the patient's digestive tract real-time fusion data and the wearable AR display device;

[0044] The diagnosis and treatment data adjustment module includes a digestive tract diagnosis and treatment data adjustment demand information collection unit, a digestive tract diagnosis and treatment data adjustment demand standard word storage unit, a digestive tract diagnosis and treatment data adjustment demand standard word recognition unit, a digestive tract diagnosis and treatment data adjustment demand control instruction storage unit, a digestive tract diagnosis and treatment data adjustment demand control instruction matching unit, and a digestive tract diagnosis and treatment data adjustment operation execution unit;

[0045] The digestive tract diagnosis and treatment data adjustment demand information acquisition unit collects digestive tract diagnosis and treatment data adjustment demand text data through the voice input module installed in the wearable AR display device; the different types of digestive tract diagnosis and treatment data adjustment demand standard word storage unit is used to store different types of digestive tract diagnosis and treatment data adjustment demand standard words; the digestive tract diagnosis and treatment data adjustment demand standard word recognition unit performs digestive tract diagnosis and treatment data adjustment demand standard word recognition processing on the patient based on the digestive tract diagnosis and treatment data adjustment demand text data and different types of digestive tract diagnosis and treatment data adjustment demand standard words, and generates digestive tract diagnosis and treatment data adjustment demand standard word recognition data; the different types of digestive tract diagnosis and treatment data adjustment demand control instruction storage unit is used to store storing different types of digestive tract diagnosis and treatment data adjustment demand control instructions; the digestive tract diagnosis and treatment data adjustment demand control instruction matching unit performs control instruction matching processing on the patient's digestive tract diagnosis and treatment data adjustment demand control instructions according to the digestive tract diagnosis and treatment data adjustment demand standard word recognition data and different types of digestive tract diagnosis and treatment data adjustment demand control instructions, and generates digestive tract diagnosis and treatment data adjustment demand control instruction matching data; the digestive tract diagnosis and treatment data adjustment operation execution unit performs digestive tract diagnosis and treatment data adjustment processing on the patient based on the real-time fusion data of the patient's digestive tract image and physiological parameters, the digestive tract diagnosis and treatment data adjustment demand control instruction matching data and video editing software, and generates real-time adjustment data of the patient's digestive tract image and physiological parameters;

[0046] The diagnosis and treatment data adjustment result output module includes a digestive tract diagnosis and treatment data adjustment result information acquisition unit and a digestive tract diagnosis and treatment data adjustment result output unit;

[0047] The digestive tract diagnosis and treatment data adjustment result information acquisition unit is used to construct the digestive tract diagnosis and treatment data adjustment result data; the digestive tract diagnosis and treatment data adjustment result output unit performs the patient's digestive tract real-time image information and physiological parameter fusion adjustment result output operation based on the digestive tract diagnosis and treatment data adjustment result data in combination with the wearable AR display device.

[0048] (3) Beneficial effects

[0049] The present invention provides a system and method for processing electronic endoscope diagnosis and treatment data based on multimodal data. It has the following beneficial effects:

[0050] 1. Dynamically and accurately obtain the real-time image information and physiological parameter information of the patient's digestive tract through the gastrointestinal electronic endoscope and monitor, providing real data support for the dynamic synchronous display of the patient's physiological parameters and digestive tract images; autonomously and accurately fuse the patient's digestive tract real-time image information and physiological parameters with video editing software based on the patient's digestive tract real-time image information and physiological parameters, and realize the dynamic intelligent data fusion of the patient's physiological parameters and digestive tract real-time images during digestive tract surgery; autonomously execute the patient's digestive tract real-time image information and physiological parameter fusion data output operation based on the real-time fusion data of the patient's digestive tract images and physiological parameters combined with wearable AR display equipment, and realize the intuitive and dynamic integrated display of the patient's digestive tract status images and physiological parameters during digestive tract surgery, thereby improving the efficiency of electronic endoscope diagnosis and treatment data processing.

[0051] 2. Install a voice entry module in the wearable AR display device to efficiently and accurately obtain the information on the adjustment of digestive tract diagnosis and treatment data, and provide reliable data support for the scientific adjustment of the display status of the patient's physiological parameters in the real-time image of the digestive tract; Based on the information on the adjustment of digestive tract diagnosis and treatment data, combine the artificial intelligence recognition algorithm with the different types of digestive tract diagnosis and treatment data adjustment demand standard words based on big data storage to accurately identify and process the patient's digestive tract diagnosis and treatment data adjustment demand standard words, and realize the scientific and standardized translation of digestive tract diagnosis and treatment data adjustment demand information based on natural language processing; Based on the recognition data of the digestive tract diagnosis and treatment data adjustment demand standard words, combine the intelligent search algorithm with the different types of digestive tract diagnosis and treatment data adjustment demand words stored scientifically The control instructions are used to accurately match and process the patient's gastrointestinal diagnosis and treatment data adjustment demand control instructions, realize the scientific analysis of the adjustment demand instructions of the display position, arrangement, display size and display effect of the physiological parameters in the real-time image of the digestive tract, and improve the accuracy of the electronic endoscope diagnosis and treatment data processing; based on the real-time fusion data of the patient's digestive tract images and physiological parameters, the digestive tract diagnosis and treatment data adjustment demand control instruction matching data combined with the video editing software, the patient's digestive tract diagnosis and treatment data adjustment process is carried out independently and scientifically, realize the intelligent and precise adjustment of the display position, arrangement, display size and display effect of the patient's physiological parameters in the real-time image of the digestive tract, and improve the intelligence and flexibility of the electronic endoscope diagnosis and treatment data processing.

[0052] 3. By adjusting the data based on the patient's gastrointestinal images and physiological parameters in real time, the gastrointestinal diagnosis and treatment data adjustment result data can be timely and accurately constructed; at the same time, combined with wearable AR display equipment, the patient's gastrointestinal real-time image information and physiological parameters are integrated and adjusted to output the result, realizing the intuitiveness and applicability of electronic endoscope diagnosis and treatment data processing, and improving the efficiency and quality of gastrointestinal surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1A schematic diagram of the modules of the electronic endoscope diagnosis and treatment data processing system based on multimodal data provided by the present invention;

[0054] Figure 2 This is a flowchart of the method for processing electronic endoscope diagnosis and treatment data based on multimodal data provided by the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] The embodiments of the electronic endoscope diagnosis and treatment data processing system and method based on multimodal data are as follows:

[0057] Example 1:

[0058] See also Figure 1 - Figure 2 , an electronic endoscope diagnosis and treatment data processing system and method based on multimodal data, the method comprising the following steps:

[0059] S1. Collecting real-time image data of the patient's digestive tract and the patient's physiological characteristics data;

[0060] S2. Performing data fusion processing on the patient's digestive tract real-time image data and the patient's physiological characteristic data based on the patient's digestive tract real-time image data and the patient's physiological parameter data to generate real-time fusion data of the patient's digestive tract image and physiological parameter;

[0061] S3, outputting the patient's digestive tract real-time image information and physiological parameter fusion data according to the patient's digestive tract image and physiological parameter real-time fusion data;

[0062] S4. Collecting digestive tract diagnosis and treatment data adjustment demand text data and performing digestive tract diagnosis and treatment data adjustment demand standard word recognition processing on the patient's digestive tract diagnosis and treatment data adjustment demand standard word of different types to generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data;

[0063] S5. Matching the patient's digestive tract diagnosis and treatment data adjustment demand control instruction with the digestive tract diagnosis and treatment data adjustment demand standard word recognition data and different types of digestive tract diagnosis and treatment data adjustment demand control instruction control instructions to generate digestive tract diagnosis and treatment data adjustment demand control instruction matching data;

[0064] S6, performing digestive tract diagnosis and treatment data adjustment processing on the patient based on the real-time fusion data of the patient's digestive tract image and physiological parameters and the digestive tract diagnosis and treatment data adjustment demand control instruction matching data, to generate real-time adjustment data of the patient's digestive tract image and physiological parameters;

[0065] S7. Construct the digestive tract diagnosis and treatment data adjustment result data and execute the operation of outputting the adjustment result by fusing the patient's digestive tract real-time image information with physiological parameters.

[0066] For further information, see Figure 1 - Figure 2 The steps for collecting the patient's digestive tract real-time image data and the patient's physiological characteristic data are as follows:

[0067] S11, collecting real characteristic images of the digestive tract of a patient undergoing digestive tract surgery online through a digestive tract electronic endoscope, and generating real-time image data Z of the patient's digestive tract;

[0068] The physiological parameters of the patient during the digestive tract surgery are collected online through a monitor, and the patient's physiological characteristic data Y is generated. The patient's physiological characteristic data includes blood pressure physiological characteristic data, heart rate physiological characteristic data and blood oxygen saturation physiological characteristic data of patients with digestive tract diseases.

[0069] The steps of performing data fusion processing of the patient's digestive tract real-time image data and the patient's physiological characteristic data to generate real-time fusion data of the patient's digestive tract image and physiological parameters are as follows:

[0070] S21. Use video editing software to map and fuse the blood pressure physiological characteristic data, heart rate physiological characteristic data and blood oxygen saturation physiological characteristic data in the patient's physiological characteristic data Y into the image corresponding to the patient's digestive tract real-time image data Z in an orderly horizontal or vertical arrangement according to the physiological parameter type, and generate real-time fusion data P of the patient's digestive tract image and physiological parameters, where P = (Z, Y). The video editing software includes any one of Adobe Premiere, After Effects, Final Cut Pro, and Da Vinci Resolve.

[0071] The steps for outputting the patient's digestive tract real-time image information and physiological parameter fusion data according to the patient's digestive tract image and physiological parameter real-time fusion data are as follows:

[0072] S31. Input the real-time fusion data P of the patient's digestive tract image and physiological parameters into a wearable AR display device and feed it back to the digestive tract surgery personnel in real time, and perform the real-time output operation of the patient's digestive tract real-time image information and physiological parameter fusion data. The wearable AR display device includes any one of head-mounted AR glasses and mobile AR devices.

[0073] Through the cooperation of the patient's digestive tract real-time image acquisition unit and the patient's physiological parameter acquisition unit, the digestive tract electronic endoscope and the monitor are used respectively to dynamically and accurately obtain the patient's digestive tract real-time image information and physiological parameter information, providing real data support for the dynamic synchronous display of the patient's physiological parameters and digestive tract images; the patient's digestive tract diagnosis and treatment data fusion unit, based on the patient's digestive tract real-time image information and physiological parameters combined with video editing software, performs autonomous and accurate data fusion processing of the patient's digestive tract real-time image information and physiological parameters, realizing dynamic intelligent data fusion of the patient's physiological parameters and digestive tract real-time images during digestive tract surgery; the patient's digestive tract diagnosis and treatment data output unit, based on the patient's digestive tract image and physiological parameter real-time fusion data combined with the wearable AR display device, autonomously performs the patient's digestive tract real-time image information and physiological parameter fusion data output operation, realizing the intuitive and dynamic integrated display of the patient's digestive tract status image and physiological parameters during digestive tract surgery, and improving the efficiency of electronic endoscope diagnosis and treatment data processing.

[0074] For further information, see Figure 1 - Figure 2 The steps of collecting digestive tract diagnosis and treatment data adjustment demand text data and performing digestive tract diagnosis and treatment data adjustment demand standard word recognition processing on the patient with different types of digestive tract diagnosis and treatment data adjustment demand standard words to generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data are as follows:

[0075] S41. Online collection of textual feedback and suggestions from digestive tract surgery personnel regarding the output results of the digestive tract real-time image information and physiological parameter fusion information corresponding to the real-time fusion data P of the patient's digestive tract image and physiological parameters through a voice input module installed in a wearable AR display device, and generation of digestive tract diagnosis and treatment data adjustment demand text data O, wherein the digestive tract diagnosis and treatment data adjustment demand text data includes display position adjustment text information, display arrangement adjustment text information, display size adjustment text information, and display effect adjustment text information of the patient's physiological parameters in the digestive tract real-time image; the display position adjustment text information includes upper left position adjustment text information, left middle position adjustment text information, lower left position adjustment text information, upper middle position adjustment text information, middle position adjustment text information, and lower middle position adjustment text information of the physiological parameters in the digestive tract real-time image. Adjustment text information, upper right position adjustment text information, right middle position adjustment text information and lower right position adjustment text information; display arrangement adjustment text information includes horizontal arrangement adjustment text information and vertical arrangement adjustment text information of physiological parameters in real-time images of the digestive tract; display size adjustment text information includes doubling adjustment text information, doubling adjustment text information, tripling adjustment text information, quadrupling adjustment text information, quintuple adjustment text information, doubling adjustment text information, tripling adjustment text information, quadruple adjustment text information and quintuple adjustment text information of physiological parameters in real-time images of the digestive tract; display effect adjustment text information includes transparent display effect adjustment text information and normal display effect adjustment text information of physiological parameters in real-time images of the digestive tract;

[0076] S42. Establish a standard word set A = (a1, ..., a m ,…,a σ ), m=1,2,3,…,σ; where a m represents the different types of digestive tract diagnosis and treatment data adjustment requirement standard words corresponding to the m-th digestive tract diagnosis and treatment data output adjustment requirement type, σ represents the maximum number of digestive tract diagnosis and treatment data output adjustment requirement types, the digestive tract diagnosis and treatment data output adjustment requirement type represents an index data type mainly formed by combining the physiological parameter display position adjustment requirement text information, display arrangement adjustment requirement text information, display size adjustment requirement text information and display effect adjustment requirement text information in the digestive tract real-time image, and is used to search for specific digestive tract diagnosis and treatment data adjustment requirement standard words; the different types of digestive tract diagnosis and treatment data adjustment requirement standard words represent the standard text information of the data display status of the adjustment physiological parameters in the digestive tract real-time image set according to different types of digestive tract diagnosis and treatment data output adjustment requirement standards;

[0077] S43, comparing the digestive tract diagnosis and treatment data adjustment requirement text data O with the different types of digestive tract diagnosis and treatment data adjustment requirement standard word set A. m Perform digestive tract diagnosis and treatment data adjustment demand character matching to search for the digestive tract diagnosis and treatment data adjustment demand text data O corresponding to the different types of digestive tract diagnosis and treatment data adjustment demand standard words a m , and generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data a through data identification mubiao , execute to generate the digestive tract diagnosis and treatment data adjustment demand standard word recognition data a mubiao The specific steps are as follows:

[0078] S431, setting the standard words a for adjusting the requirements of different types of digestive tract diagnosis and treatment data m Assume that T=T0 is the current solution, set the initial temperature T0 for starting annealing in the search space of the standard word set A for adjusting the requirements of different types of digestive tract diagnosis and treatment data, and randomly generate a standard word a for adjusting the requirements of different types of digestive tract diagnosis and treatment data that matches the text data O for adjusting the requirements of digestive tract diagnosis and treatment data in the search space of the standard word set A for adjusting the requirements of different types of digestive tract diagnosis and treatment data. m Initial solution Θ r , and calculate the objective function value Φ(Θ r );

[0079] S432, generating the standard words a for adjusting the requirements of different types of digestive tract diagnosis and treatment data m New interpretation, and the standard word for adjusting the requirements of different types of digestive tract diagnosis and treatment data a m The difference of the current solution, according to the different types of digestive tract diagnosis and treatment data, adjust the demand standard word a m For the current solution Θ i Perform disturbance to generate a standard word a for the different types of digestive tract diagnosis and treatment data adjustment requirements text data O. m New solution Θ j , calculate the objective function value Φ(Θ i ) and the objective function value Φ(Θ j ), we get the objective function variation ΔΦ=Φ(Θ j )-Φ(Θ i );

[0080] S433, determine the different types of digestive tract diagnosis and treatment data adjustment requirements standard word a m Whether the new solution is accepted, if the objective function change ΔΦ<0, then the different types of digestive tract diagnosis and treatment data adjustment demand standard word a m New solution Θ jAccepted; if the objective function change ΔΦ≥0, then the different types of digestive tract diagnosis and treatment data adjustment demand standard word a m New solution Θ j Accepted with the following probability, T i represents the current temperature; the probability formula is Where T i represents the current temperature, and exp represents a random function with a value range of (0, 1);

[0081] S434, when the different types of digestive tract diagnosis and treatment data adjustment demand standard word a m When the new interpretation is accepted, the standard word for the adjustment of the requirements for different types of digestive tract diagnosis and treatment data a m New solution Θ j It is regarded as the standard word for the adjustment requirements of different types of digestive tract diagnosis and treatment data a m Current solution;

[0082] S435, loop through step S431 to step S434, at temperature T i Under these conditions, the perturbation and acceptance process is repeated Λ times, and then the next step is executed;

[0083] S436. Finally find the global optimal solution: determine whether T has reached the termination temperature T zui If yes, the algorithm is terminated and the standard word a of the different types of digestive tract diagnosis and treatment data adjustment requirements that best matches the digestive tract diagnosis and treatment data adjustment requirements text data O is output. m If not, go to loop step S435 and continue executing;

[0084] S437, the output of step S436 for adjusting the demand standard words for different types of digestive tract diagnosis and treatment data a m Generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data a through data identification mubiao .

[0085] The steps for performing control instruction matching processing on the patient's digestive tract diagnosis and treatment data adjustment demand control instruction according to the digestive tract diagnosis and treatment data adjustment demand standard word recognition data and different types of digestive tract diagnosis and treatment data adjustment demand control instructions to generate digestive tract diagnosis and treatment data adjustment demand control instruction matching data are as follows:

[0086] S51, establish different types of digestive tract diagnosis and treatment data adjustment demand control instruction set B = (b1, ..., b m ,…,b σ ), where b mIndicates different types of digestive tract diagnosis and treatment data adjustment demand control instructions corresponding to the mth type of digestive tract diagnosis and treatment data output adjustment demand type, wherein the different types of digestive tract diagnosis and treatment data adjustment demand control instructions represent video editing software control instruction information for adjusting the display status of physiological parameters in a real-time digestive tract image according to different types of digestive tract diagnosis and treatment data output adjustment demand standards;

[0087] S52, using an iterative deepening search algorithm to adjust the digestive tract diagnosis and treatment data to the required standard word recognition data a mubiao The different types of digestive tract diagnosis and treatment data adjustment demand control instructions b in the different types of digestive tract diagnosis and treatment data adjustment demand control instructions set B m Perform keyword matching on the digestive tract diagnosis and treatment data output adjustment requirement, and search for the digestive tract diagnosis and treatment data adjustment requirement standard word recognition data a mubiao The corresponding different types of digestive tract diagnosis and treatment data adjustment demand control instructions b m , and generate digestive tract diagnosis and treatment data adjustment demand control instruction matching data b through data identification mubiao .

[0088] The steps of performing digestive tract diagnosis and treatment data adjustment processing on the patient based on the real-time fusion data of the patient's digestive tract image and physiological parameters and the digestive tract diagnosis and treatment data adjustment demand control instruction matching data to generate the real-time adjustment data of the patient's digestive tract image and physiological parameters are as follows:

[0089] S61, using video editing software to adjust the demand control instruction matching data b according to the digestive tract diagnosis and treatment data mubiao The corresponding gastrointestinal diagnosis and treatment data adjustment demand control instructions orderly perform data adjustment processing on the display position, display arrangement, display size and display effect of the patient's physiological characteristic data Y in the patient's gastrointestinal image and physiological parameter real-time fusion data P in the patient's gastrointestinal real-time image data Z, and generate the patient's gastrointestinal image and physiological parameter real-time adjustment data L.

[0090] Through the digestive tract diagnosis and treatment data adjustment demand information collection unit, the voice input module installed in the wearable AR display device is used to efficiently and accurately obtain the digestive tract diagnosis and treatment data adjustment demand information, providing reliable data support for the scientific adjustment of the patient's physiological parameters in the real-time image of the digestive tract; the digestive tract diagnosis and treatment data adjustment demand standard word recognition unit, based on the digestive tract diagnosis and treatment data adjustment demand information combined with artificial intelligence recognition algorithm and different types of digestive tract diagnosis and treatment data adjustment demand standard words based on big data storage, accurately recognizes and processes the patient's digestive tract diagnosis and treatment data adjustment demand standard words, and realizes scientific and standardized translation of digestive tract diagnosis and treatment data adjustment demand information based on natural language processing; the digestive tract diagnosis and treatment data adjustment demand control instruction matching unit, based on the digestive tract diagnosis and treatment data adjustment demand standard word recognition data combined with intelligent search algorithm The method and scientific storage of different types of gastrointestinal diagnosis and treatment data adjustment demand control instructions are used to accurately match and process the patient's gastrointestinal diagnosis and treatment data adjustment demand control instructions, realize the scientific analysis of the adjustment demand instructions of the display position, arrangement, display size and display effect of the physiological parameters in the real-time image of the digestive tract, and improve the accuracy of the electronic endoscope diagnosis and treatment data processing; the gastrointestinal diagnosis and treatment data adjustment operation execution unit, based on the real-time fusion data of the patient's digestive tract images and physiological parameters, the gastrointestinal diagnosis and treatment data adjustment demand control instruction matching data combined with video editing software, independently and scientifically adjusts the patient's gastrointestinal diagnosis and treatment data, realizes the intelligent and precise adjustment of the display position, arrangement, display size and display effect of the patient's physiological parameters in the real-time image of the digestive tract, and improves the intelligence and flexibility of the electronic endoscope diagnosis and treatment data processing.

[0091] For further information, see Figure 1 - Figure 2 The steps for constructing the digestive tract diagnosis and treatment data adjustment result data and performing the output operation of fusing the patient's digestive tract real-time image information and physiological parameters to adjust the result are as follows:

[0092] S71, constructing digestive tract diagnosis and treatment data adjustment result data F by data identification of the generated patient digestive tract image and physiological parameter real-time adjustment data L;

[0093] S72. Feedback the digestive tract diagnosis and treatment data adjustment result data F to the digestive tract surgery personnel in real time through the wearable AR display device, and perform the operation of outputting the adjustment result by fusing the patient's digestive tract real-time image information with physiological parameters.

[0094] Through the cooperation between the gastrointestinal diagnosis and treatment data adjustment result information acquisition unit and the gastrointestinal diagnosis and treatment data adjustment result output unit, the gastrointestinal diagnosis and treatment data adjustment result data is timely and accurately constructed based on the patient's gastrointestinal image and physiological parameter real-time adjustment data; at the same time, combined with the wearable AR display device, the patient's gastrointestinal real-time image information and physiological parameter fusion adjustment result output operation is accurately visualized, realizing the intuitiveness and applicability of electronic endoscope diagnosis and treatment data processing, and improving the efficiency and quality of gastrointestinal surgery.

[0095] Example 2:

[0096] See also Figure 1 - Figure 2 , an electronic endoscope diagnosis and treatment data processing system based on multimodal data, used to implement the electronic endoscope diagnosis and treatment data processing method based on multimodal data, the system includes a diagnosis and treatment data integrated display module, a diagnosis and treatment data adjustment module, and a diagnosis and treatment data adjustment result output module;

[0097] The diagnosis and treatment data integrated display module includes a patient digestive tract real-time image acquisition unit, a patient physiological parameter acquisition unit, a patient digestive tract diagnosis and treatment data fusion unit, and a patient digestive tract diagnosis and treatment data output unit;

[0098] The patient's digestive tract real-time image acquisition unit acquires real-time image data of the patient's digestive tract through a digestive tract electronic endoscope; the patient's physiological parameter acquisition unit acquires the patient's physiological characteristic data through a monitor; the patient's digestive tract diagnosis and treatment data fusion unit performs data fusion processing of the patient's digestive tract real-time image information and physiological parameters based on the patient's digestive tract real-time image data and the patient's physiological characteristic data in combination with video editing software to generate real-time fusion data of the patient's digestive tract image and physiological parameters; the patient's digestive tract diagnosis and treatment data output unit performs an operation of outputting the patient's digestive tract real-time image information and physiological parameter fusion data based on the patient's digestive tract real-time fusion data and the wearable AR display device;

[0099] The diagnosis and treatment data adjustment module includes a digestive tract diagnosis and treatment data adjustment demand information collection unit, a digestive tract diagnosis and treatment data adjustment demand standard word storage unit, a digestive tract diagnosis and treatment data adjustment demand standard word recognition unit, a digestive tract diagnosis and treatment data adjustment demand control instruction storage unit, a digestive tract diagnosis and treatment data adjustment demand control instruction matching unit, and a digestive tract diagnosis and treatment data adjustment operation execution unit;

[0100] The digestive tract diagnosis and treatment data adjustment demand information acquisition unit collects digestive tract diagnosis and treatment data adjustment demand text data through the voice input module installed in the wearable AR display device; the different types of digestive tract diagnosis and treatment data adjustment demand standard word storage unit is used to store different types of digestive tract diagnosis and treatment data adjustment demand standard words; the digestive tract diagnosis and treatment data adjustment demand standard word recognition unit performs digestive tract diagnosis and treatment data adjustment demand standard word recognition processing on the patient based on the digestive tract diagnosis and treatment data adjustment demand text data and different types of digestive tract diagnosis and treatment data adjustment demand standard words, and generates digestive tract diagnosis and treatment data adjustment demand standard word recognition data; the different types of digestive tract diagnosis and treatment data adjustment demand control instruction storage unit is used to store storing different types of digestive tract diagnosis and treatment data adjustment demand control instructions; the digestive tract diagnosis and treatment data adjustment demand control instruction matching unit performs control instruction matching processing on the patient's digestive tract diagnosis and treatment data adjustment demand control instructions according to the digestive tract diagnosis and treatment data adjustment demand standard word recognition data and different types of digestive tract diagnosis and treatment data adjustment demand control instructions, and generates digestive tract diagnosis and treatment data adjustment demand control instruction matching data; the digestive tract diagnosis and treatment data adjustment operation execution unit performs digestive tract diagnosis and treatment data adjustment processing on the patient based on the real-time fusion data of the patient's digestive tract image and physiological parameters, the digestive tract diagnosis and treatment data adjustment demand control instruction matching data and video editing software, and generates real-time adjustment data of the patient's digestive tract image and physiological parameters;

[0101] The diagnosis and treatment data adjustment result output module includes a digestive tract diagnosis and treatment data adjustment result information acquisition unit and a digestive tract diagnosis and treatment data adjustment result output unit;

[0102] The digestive tract diagnosis and treatment data adjustment result information acquisition unit is used to construct the digestive tract diagnosis and treatment data adjustment result data; the digestive tract diagnosis and treatment data adjustment result output unit performs the patient's digestive tract real-time image information and physiological parameter fusion adjustment result output operation based on the digestive tract diagnosis and treatment data adjustment result data in combination with the wearable AR display device.

[0103] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing electronic endoscope diagnosis and treatment data based on multimodal data, characterized in that: The method comprises the following steps: S1. Collecting real-time image data of the patient's digestive tract and the patient's physiological characteristics data; S2. Perform data fusion processing on the patient's digestive tract real-time image information and physiological parameters to generate real-time fusion data of the patient's digestive tract image and physiological parameters; S3, performing an operation of outputting the patient's digestive tract real-time image information and physiological parameter fusion data according to the patient's digestive tract image and physiological parameter real-time fusion data; S4. Collecting digestive tract diagnosis and treatment data adjustment demand text data and performing digestive tract diagnosis and treatment data adjustment demand standard word recognition processing on the patient's digestive tract diagnosis and treatment data adjustment demand standard word of different types to generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data; S5. Perform control instruction matching processing on the patient's digestive tract diagnosis and treatment data adjustment requirements to generate digestive tract diagnosis and treatment data adjustment requirements control instruction matching data; S6. Adjust and process the patient's digestive tract diagnosis and treatment data to generate real-time adjustment data of the patient's digestive tract image and physiological parameters; S7. Construct the digestive tract diagnosis and treatment data adjustment result data and execute the operation of outputting the adjustment result by fusing the patient's digestive tract real-time image information with physiological parameters.

2. The method for processing electronic endoscope diagnosis and treatment data based on multimodal data according to claim 1, characterized in that: Said S1 comprises the following steps: S11, collecting real characteristic images of the digestive tract of a patient undergoing digestive tract surgery online through a digestive tract electronic endoscope, and generating real-time image data Z of the patient's digestive tract; The physiological parameters of the patient during the digestive tract surgery are collected online through the monitor, and the patient's physiological characteristic data Y is generated.

3. The method for processing electronic endoscope diagnosis and treatment data based on multimodal data according to claim 2, characterized in that: The S2 comprises the following steps: S21. Use video editing software to map and fuse the blood pressure physiological characteristic data, heart rate physiological characteristic data, and blood oxygen saturation physiological characteristic data in Y into the image corresponding to Z in an orderly horizontal or vertical arrangement according to the physiological parameter type, and generate real-time fusion data P of the patient's digestive tract image and physiological parameters.

4. The method for processing electronic endoscope diagnosis and treatment data based on multimodal data according to claim 3, characterized in that: The S3 includes the following steps: S31. Input the P into the wearable AR display device and feed it back to the digestive tract surgery personnel in real time, and perform the real-time output operation of the patient's digestive tract real-time image information and physiological parameter fusion data.

5. The method for processing electronic endoscope diagnosis and treatment data based on multimodal data according to claim 4, characterized in that: The S4 comprises the following steps: S41. Online collection of textual feedback and suggestions from digestive tract surgeons regarding an output result of the real-time digestive tract image information and physiological parameter fusion information corresponding to P, through a voice input module installed in the wearable AR display device, and generation of digestive tract diagnosis and treatment data adjustment requirement text data O, wherein the digestive tract diagnosis and treatment data adjustment requirement text data includes textual information for adjusting the display position, display arrangement, display size, and display effect of the patient's physiological parameters in the real-time digestive tract image; S42. Establish a standard word set A = (a1, ..., a m ,…,a σ ), m=1,2,3,…,σ; where a m represents the standard words for different types of digestive tract diagnosis and treatment data adjustment requirements corresponding to the mth digestive tract diagnosis and treatment data output adjustment requirement type, and σ represents the maximum number of digestive tract diagnosis and treatment data output adjustment requirement types; S43, the O and the a in the A m Perform digestive tract diagnosis and treatment data adjustment requirement character matching and search for the a corresponding to the O m , and generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data a through data identification mubiao , execute to generate the digestive tract diagnosis and treatment data adjustment demand standard word recognition data a mubiao The specific steps are as follows: S431, set the a m Assume T=T0 as the current solution, set the initial temperature T0 for starting annealing in the search space of A, and randomly generate a matching a in the search space of A. m Initial solution Θ r , and calculate the objective function value Φ(Θ r ); S432, generate the a m New solution, and m The difference of the current solution, according to the a m For the current solution Θ i Perturb and generate a m New solution Θ j , calculate the objective function value Φ(Θ i ) and the objective function value Φ(Θ j ), and obtain the objective function variation ΔΦ; S433, determine the a m Whether the new solution is accepted, if the objective function change ΔΦ<0, then the a m New solution Θ j Accepted; if the objective function change ΔΦ≥0, then the a m New solution Θ j Accepted with the following probability, T i Indicates the current temperature; S434, when the a m When the new solution is accepted, the a m New solution Θ j is regarded as the m Current solution; S435, loop through step S431 to step S434, at temperature T i Under these conditions, the perturbation and acceptance process is repeated Λ times, and then the next step is executed; S436. Finally find the global optimal solution: determine whether T has reached the termination temperature T zui If so, terminate the algorithm and output the a that best matches O. m If not, go to loop step S435 and continue executing; S437, output the a in step S436 m Generate digestive tract diagnosis and treatment data adjustment demand standard word recognition data a through data identification mubiao .

6. The method for processing electronic endoscope diagnosis and treatment data based on multimodal data according to claim 5, characterized in that: The S5 comprises the following steps: S51, establish different types of digestive tract diagnosis and treatment data adjustment demand control instruction set B = (b1, ..., b m ,…,b σ ), where b m Indicates different types of digestive tract diagnosis and treatment data adjustment demand control instructions corresponding to the mth type of digestive tract diagnosis and treatment data output adjustment demand type; S52, using iterative deepening search algorithm to mubiao With the B in b m Perform digestive tract diagnosis and treatment data output adjustment demand keyword matching and search for the a mubiao The corresponding b m , and generate digestive tract diagnosis and treatment data adjustment demand control instruction matching data b through data identification mubiao .

7. The method for processing electronic endoscope diagnosis and treatment data based on multimodal data according to claim 6, characterized in that: The S6 comprises the following steps: S61, use video editing software according to b mubiao The corresponding gastrointestinal diagnosis and treatment data adjustment demand control instructions orderly perform data adjustment processing on the display position, display arrangement, display size and display effect of the Y in the P in the Z, and generate real-time adjustment data L of the patient's gastrointestinal image and physiological parameters.

8. The method for processing electronic endoscope diagnosis and treatment data based on multimodal data according to claim 7, characterized in that: The S7 comprises the following steps: S71, constructing digestive tract diagnosis and treatment data adjustment result data F by data identification of the generated L; S72: Feedback the F to the digestive tract surgery personnel in real time through the wearable AR display device, and perform the operation of outputting the result of the fusion adjustment of the patient's digestive tract real-time image information and physiological parameters.

9. An electronic endoscope diagnosis and treatment data processing system based on multimodal data, for implementing the electronic endoscope diagnosis and treatment data processing method based on multimodal data according to any one of claims 1 to 8, characterized in that: The system includes a diagnosis and treatment data integration display module, a diagnosis and treatment data adjustment module, and a diagnosis and treatment data adjustment result output module.

Images