Ultrasonic endoscope intelligent training system

Abstract

The invention discloses an ultrasonic endoscope intelligent training system, and relates to ultrasonic endoscope training data processing, and the technical scheme is characterized in that a data collection module is used for collecting an action image set of a student and organ feedback data and body operation data of an experimental subject in the training process; the data processing module is used for preprocessing the collected action image set to obtain a high-quality image set, and performing feature processing on organ feedback data and body operation data of the experimental subject to obtain a feature data set; the scheme generation module is used for inputting the high-quality image set and the feature data set after data fusion into the trained GoogLeNet network learning model to obtain a training scheme of the trainee; all the modules are connected in a wired and / or wireless mode, all the modules are connected in a wired and / or wireless mode, and personalized scheme generation of ultrasonic endoscope intelligent training is achieved.

Description

Technical Field

[0001] The present invention relates to ultrasonic endoscope training data processing, and more particularly to an ultrasonic endoscope intelligent training system. Background Art

[0002] Endoscopic Ultrasound (EUS) is a medical device that combines endoscopy and ultrasound technology, and is used to examine and treat diseases of the digestive system and other related organs. Endoscopic ultrasound plays an irreplaceable role in minimally invasive surgery, but the risk of internal surgery is extremely high, so professional training is extremely important. Endoscopic ultrasound is a necessary training content for medical students, involving a wide range of medical knowledge. Traditional endoscopy training is generally traditional classroom teaching. Due to the lack of practical operation, theoretical knowledge is disconnected from actual operation, and it is difficult for students to apply what they have learned in real scenarios. In addition, teachers have limited energy and lack of interaction with students, making it difficult to provide personalized guidance based on the specific needs of each student.

[0003] With the development of science and technology, AI has been combined with the education and training industry, and intelligent endoscopic ultrasound training has emerged. Intelligent endoscopic ultrasound training is a kind of training and simulation for doctors and medical practitioners using advanced technologies such as artificial intelligence and virtual reality. It improves the skill level and operation accuracy of medical personnel, reduces the learning curve, and improves patient safety and treatment effects. However, due to the expensive equipment, high-quality physical simulators are expensive, maintenance costs are high, and the number of simulators is limited, so trainees are limited in their time of use. Therefore, it is necessary to maximize the quality and efficiency of training under limited conditions.

[0004] During the intelligent training of ultrasonic endoscopy, due to scene limitations and equipment factors, the accuracy of motion recognition during trainees' training is not high, there is a problem of poor image quality, and it is difficult to highlight in complex application scenarios, resulting in interference in image feature analysis and deep information mining; the key point detection and feature recognition accuracy of trainees' motion features during training needs to be improved; and there are certain difficulties in extracting image features in complex environments, which affects the targeted recognition effect; there is also a lack of continuity analysis of dynamic changes, and it is impossible to mark poor or non-standard training movements well, which in turn affects the training progress. There is also a lack of personalized training, and it is impossible to meet the diverse needs of different users at different training progress. Therefore, it is necessary to design intelligent training for ultrasonic endoscopy. Summary of the invention

[0005] In view of the problems existing in the prior art, the purpose of the present invention is to provide an ultrasonic endoscope intelligent training system to achieve personalized program generation.

[0006] To achieve the above object, the present invention provides the following technical solution: an ultrasonic endoscope intelligent training system, the ultrasonic endoscope intelligent training system comprising:

[0007] Data collection module: used to collect the trainees' action image sets and the experimental subjects' organ feedback data and body operation data during the training process;

[0008] Data processing module: used to pre-process the collected action image set to obtain a high-quality image set, and perform feature processing on the organ feedback data and body operation data of the experimental subject to obtain a feature data set;

[0009] Solution generation module: build and train the GoogLeNet network learning model, input the high-quality image set and feature data set after data fusion into the trained GoogLeNet network learning model, and obtain the training plan for the trainees;

[0010] The modules are connected to each other via wired and / or wireless means.

[0011] Preferably, the method for acquiring the student's action image set includes:

[0012] During the training time t, multiple grayscale images are continuously and uninterruptedly acquired at a fixed time frequency p, which are collected synchronously by the endoscope's built-in camera and the external camera. Organ feedback data include blood flow parameters, peristaltic frequency and amplitude of tissues in the surgical site. Body operation data include heart rate, body temperature, and brain waves, which can be directly obtained through medical devices connected to the experimental subjects.

[0013] Preferably, the preprocessing of the collected action image set includes image quality enhancement, and the image quality enhancement specifically includes:

[0014] Construct an image matrix: Get each image I in the action image set, set the size of image I to M×N, soften the maximum pixel in the image, and obtain the reconstructed image I 1 , reconstructed image I 1 The size of is also M×N. Through the symmetric expansion method, the reconstructed image I 1 Copy the pixel values ​​on the boundary to expand the image and obtain the reconstructed image I 2 The size of is (M+r)×(N+r);

[0015] Image decomposition: Set the order of Coiflets wavelet, the order and parameters of B-spline scaling function, and reconstruct the image I 2 Perform a two-dimensional discrete wavelet transform on the image matrix to decompose the image into sub-bands of multiple scales and directions, and the sub-bands include low-frequency and high-frequency components;

[0016] Threshold processing: Apply threshold processing to the high-frequency subband, set the coefficients less than the threshold to zero, and keep the coefficients greater than the threshold. Set the threshold to: σ is the standard deviation of the noise;

[0017] Wavelet reconstruction: The processed coefficients are subjected to a two-dimensional discrete wavelet inverse transform to reconstruct the image, and image quality enhancement is applied to each image in the action image set to obtain a high-quality image set.

[0018] Preferably, the method for softening the maximum pixel in the image comprises:

[0019] Suppose that the image I is divided into small blocks of size m×n, and equalization is applied to the grayscale histogram of each small block.

[0020]

[0021] Hij(v) 1 represents the frequency of gray level v in the image block (i, j) after equalization, Hij(v) represents the frequency of gray level v in the image block (i, j) before softening, the value range of i is [0, M / m], and the value range of j is [0, N / n]; θ represents the threshold of histogram clipping, and L represents the total number of gray levels;

[0022] Reconstruct each small block after softening into a complete image, which is recorded as reconstructed image I 1 .

[0023] Preferably, the threshold acquisition method for histogram clipping includes:

[0024]

[0025] Smax represents the number of brightest pixels in the image before softening, Smin represents the number of darkest pixels in the image before softening, S represents the total number of pixels in the image before softening, and λ is the cropping coefficient.

[0026] Preferably, the symmetric expansion method is used to reconstruct the image I 1 Copy pixel values ​​on the boundary to expand the image. The specific steps are as follows:

[0027] Horizontal expansion: symmetrically copy the left and right sides of the image by r rows respectively;

[0028] Vertical expansion: symmetrically copy the upper and lower sides of the image by r rows respectively;

[0029] Reconstructed Image I 1 After processing by the symmetric expansion method, the reconstructed image I is obtained 2 The size of is (M+r)×(N+r), and the reconstructed image I2 The two-dimensional matrix representation of is:

[0030]

[0031] Among them, I 1 k1，k2 Represents the pixel value of the k1th row and k2th column, and k1=1,...,M+r, k2=1,...,N+r.

[0032] Preferably, the performing feature processing on the organ feedback data and body operation data of the experimental subject to obtain the feature data set includes: performing standardization processing on the organ feedback data and body operation data of the experimental subject, and calculating the state coefficient of the experimental subject:

[0033] Qf=f1×xl+f2×rd+f3×fd;

[0034] Sy=g1×cl+g2×tw+g3×np;

[0035] Among them, Qf represents the organ state coefficient, xl, rd and fd represent the blood flow parameters, peristalsis frequency and amplitude of the surgical site tissue respectively; Sy represents the body state coefficient, cl, tw, np represent the body operation data including heart rate, body temperature and brain wave respectively, f1, f2, f3, g1, g2 and g3 represent weight coefficients;

[0036] The organ state coefficient and the body state coefficient are respectively encoded with IEEE754 floating point numbers to obtain two groups of binary characters. The two groups of binary characters are combined and time-stamped to obtain a string of characteristic binary characters. All characteristic binary characters are combined to form a characteristic data set.

[0037] Preferably, the data fusion of the high-quality image set and the feature data set includes:

[0038] The trainee's action image set is a set of b grayscale images acquired continuously and uninterruptedly at a fixed time frequency p during the training time t. The training time t is discretized into b time points in chronological order. Each time point corresponds to an image. The timestamp of each image in the high-quality image set is extracted and filled into the corresponding time point.

[0039] Get the time label of each string of feature binary characters in the feature data set, match them one by one with the discrete time points, and then match each string of feature binary characters with each image in the high-quality image set to obtain fused data points. Each fused data point includes the feature binary characters and images at the corresponding time points. All fused data points within the training time t constitute the fused data set.

[0040] Preferably, the method for constructing the GoogLeNet network learning model includes:

[0041] Four parallel paths are used, and the four paths are set to 1×1, 3×3, 5×5 and 1×1 convolution kernels in sequence. The 1×1 convolution kernel is set before the 5×5 and 3×3 convolution kernels;

[0042] The number of Inception modules is set to 3, the number of Inception convolution kernels is set to 512, two fully connected layers are added above the softmax output layer, the fully connected layers use ReLU as the activation function, and the number of neurons in the fully connected layers is set to 2048 and 1024 respectively.

[0043] The GoogLeNet network learning model defines the loss function as,

[0044]

[0045] Where X is the total number of fused data sets in each subset, and x is the index; Output the time in the training plan for the GoogLeNet network learning model, The output training scheme of the GoogLeNet network learning model occupies the running memory, t is the time in the actual training scheme, nc is the actual training scheme occupies the running memory, and w1 and w2 are weight coefficients.

[0046] Preferably, the GoogLeNet network learning model training process includes:

[0047] S11, setting the sample set to include o groups of subsets, each subset including X groups of fused data sets;

[0048] S12, dividing the subset into a training set and a validation set according to the ratio, and inputting the subsets in the training set into the GoogLeNet network learning model in sequence for forward propagation;

[0049] S13, obtain the predicted output of the GoogLeNet network learning model, calculate the loss value using the loss function, and update the weights and biases of the model through the back propagation algorithm;

[0050] S14, repeat S12 and S13 until the loss function value of the GoogLeNet network learning model no longer changes, and then import the verification set for verification;

[0051] S15. When the verification is successful, obtain the trained GoogLeNet network learning model. When the verification is unsuccessful, repeat S12.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] In the present invention, the present invention performs image quality enhancement preprocessing on the collected action image set, realizes image denoising by wavelet transform in image quality enhancement, performs softening processing on the maximum value pixels, retains more details and features for the brightest and darkest pixels to reduce the loss of details in the denoising process, and expands the image by copying pixel values ​​on the reconstructed image boundary through a symmetric expansion method to avoid introducing artifacts and distortion during boundary processing, thereby ensuring the effectiveness of wavelet transform, and increases the richness of image features and data information by performing feature processing on the organ feedback data and body operation data of the experimental subject, and ensures that the data is in a unified time axis through data fusion. It improves consistency in terms of performance, reduces the computing power of data analysis, and achieves high-quality data input, improves the effect of subsequent data processing and model training, as well as the retrieval and response speed, and provides an accurate data foundation for subsequent model optimization; accelerates feature extraction efficiency, injects new momentum into the agility and flexibility of data processing. The system implements a GoogLeNet network learning model to output personalized training plans for ultrasonic endoscopy intelligent training; the adaptive regulation and optimization mechanism continuously improves the overall efficiency of training, and multiple modules collaborate with each other to achieve comprehensive data collection, processing and analysis, and provide real-time and accurate data support. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention proposes a structural schematic diagram of an ultrasonic endoscope intelligent training system;

[0055] Figure 2 The figure is a schematic diagram of the method steps applied to the ultrasonic endoscope intelligent training system in the present invention. DETAILED DESCRIPTION

[0056] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0057] Embodiment 1

[0058] Reference Figure 1 and Figure 2 Embodiment 1 further illustrates an ultrasonic endoscope intelligent training system proposed by the present invention.

[0059] The ultrasonic endoscopy intelligent training system provides a realistic endoscopic examination and operation environment. It allows trainees to practice in a virtual environment, simulate real operations, analyze trainees' operation performance through machine learning algorithms, provide instant feedback, adjust the difficulty and content according to the progress of trainees, generate personalized training plans, record trainees' operation data during each training process, and generate detailed reports to help trainees understand their progress. This system also includes a rich case library covering various complex clinical situations. Therefore, the advantages of this system are that it operates in a virtual environment, analyzes the operation data during the training process, pays attention to non-standard actions, and generates personalized training plans for the next training operation process until the trainees master all skills, reducing the loss and consumption of actual operation equipment, saving training time and resources, and targeted and personalized training. It is suitable for all kinds of trainees, realizing teaching according to their aptitude, and improving learning efficiency for trainees of different levels through instant feedback and personalized guidance.

[0060] The ultrasonic endoscopy intelligent training system has broad application prospects in the field of medical education and training. By simulating complex cases, it can improve the trainees' ability to deal with actual situations. It can not only be used to train novice doctors, but also for the continuing education and skill improvement of working doctors. In addition, this system is also helpful for medical research. By collecting and analyzing operation data, it provides support for the improvement and innovation of medical technology, greatly improves the training quality of endoscopic operations, and cultivates more high-quality professionals for the medical industry.

[0061] The hardware architecture of the ultrasonic endoscopy intelligent training system can include high-performance computers, VR / AR devices, sensors and input devices to support image processing, algorithm operation, and data analysis; VR / AR devices such as Oculus Rift, HTC Vive, and HoloLens are used to connect virtual scenes, and sensors and input devices such as tactile feedback gloves and motion capture cameras are used for data and image acquisition. Therefore, high-performance computers and graphics processing units (GPUs) are configured to ensure the ability to process large amounts of image data and complex AI algorithms, and the compatibility of VR / AR devices with the training system.

[0062] The design scheme of the present invention is to collect the trainees' action images during the training process, as well as the feedback data generated by the corresponding surgical parts (organs) when the actions act on the experimental subjects, so that the trainees' surgical actions are matched with the real-time body operation data of the experimental subjects, and the impact and feedback generated by the surgical actions are collected in real time. By analyzing the impact data generated by the actions, it is judged whether the actions meet the training standards. That is, if the physical data of the experimental subjects is poor, it means that there is a problem with the corresponding actions, otherwise it means that the actions are qualified, thereby enhancing the practicality of the training process. According to the data collection and related processing in the training process, the machine learning model is used to adaptively output the training improvement plan, so as to realize the real-time updating of the training plan.

[0063] Data collection module: used to collect the trainees' action image sets and the experimental subjects' organ feedback data and body operation data during the training process;

[0064] The trainee's action image set is a set of multiple grayscale images acquired continuously and uninterruptedly at a fixed time frequency p within the training time t. Each time the trainee performs an operation, the images are collected synchronously through the endoscope's built-in camera and the external camera. The organ feedback data include blood flow parameters, peristalsis frequency and amplitude of the tissue in the surgical site, and the body operation data include heart rate, body temperature, and brain waves, which can all be directly obtained through medical devices connected to the experimental subjects.

[0065] Data processing module: used to pre-process the collected action image set, including image quality enhancement to obtain a high-quality image set, and perform feature processing on the organ feedback data and body operation data of the experimental subject to obtain a feature data set.

[0066] The images collected during ultrasound endoscopy operations may be affected by environmental interference, equipment noise, etc., resulting in poor image quality. Denoising can help reduce these interferences and noises and improve image clarity and quality. Clear images are the basis for accurately analyzing trainees' operations and identifying errors. Denoising can effectively improve image quality and help with subsequent analysis and identification work.

[0067] Wavelet transform can decompose an image into sub-bands of different scales and directions, and then perform denoising on each sub-band to reconstruct the image. Wavelet denoising can better preserve the edges and details of the image. Wavelet basis functions (mother wavelets) and scale functions play a vital role in wavelet transform, which affect the performance and applicability of wavelet transform. Wavelet basis functions (Mother Wavelets) determine the frequency characteristics of wavelet transform, and different wavelet basis functions have different frequency responses. For example, high-frequency wavelet basis functions are suitable for analyzing rapidly changing signals or image details, while low-frequency wavelet basis functions are suitable for analyzing smooth signals or image structures. Scale functions describe the behavior of wavelet transform at different scales and determine the multi-scale analysis capability of wavelet transform. Different scale functions can provide different frequency resolutions and scale ranges, thus being suitable for signal or image analysis at different scales.

[0068] Image enhancement using Coiflets wavelet and B-spline scaling function for wavelet transform includes:

[0069] Construct an image matrix: Get each image I in the action image set, set the size of image I to M×N, soften the maximum pixel in the image, and obtain the reconstructed image I 1 , reconstructed image I 1The size of is also M×N. Before wavelet transform, in order to avoid introducing artifacts and distortion during boundary processing, it is often necessary to expand the image to ensure the effectiveness of wavelet transform. The present invention uses a symmetric expansion method to reconstruct the image I 1 Copy pixel values ​​on the boundary to expand the image. The specific steps are as follows:

[0070] Horizontal expansion: symmetrically copy the left and right sides of the image by r rows. For example, if r=3, and the left side of the image has pixel values ​​[10, 20, 30], the expanded left side will become [30, 20, 10, 10, 20, 30]. The value of r can be set based on experience or experimental analysis.

[0071] Vertical expansion: symmetrically copy the top and bottom of the image by r rows. For example, if the top of the image has pixel values ​​[10, 20, 30], the expanded top will become [30, 20, 10, 10, 20, 30].

[0072] Therefore, the reconstructed image I 1 After processing by the symmetric expansion method, the reconstructed image I is obtained 2 The size of is (M+r)×(N+r). By symmetrically replicating the boundary pixel values ​​of the image, the continuity and smoothness of the image can be maintained during wavelet transform, avoiding artifacts and distortion that may be introduced during boundary processing.

[0073] An image is usually composed of pixels, each pixel has a grayscale value. We can use a matrix to represent the values ​​of these pixels. For a grayscale image, the value of each pixel is represented as a grayscale value. 2 The two-dimensional matrix representation of is:

[0074] Among them, I 1 k1，k2 Represents the pixel value of the k1th row and k2th column, and k1=1,…,M+r, k2=1,…,N+r.

[0075] The softening process for the maximum pixel in the image includes:

[0076] Before image denoising, preprocessing the brightest and darkest pixels can help retain more details and features to reduce the loss of details during the denoising process.

[0077] Suppose the image I is of size M×N and is divided into small blocks of size m×n. The number of small blocks is (M / m)×(N / n). Equalization is applied to the grayscale histogram of each small block.

[0078]

[0079] The frequency of each gray level is then limited so that it does not exceed a threshold. The frequency of the gray level is obtained by counting the number of times each gray value appears in the image. The histogram can be obtained through the OpenCV function in Python.

[0080] Hij(v) 1 represents the frequency (i.e., the number of occurrences) of gray level v in the image block (i, j) after equalization, Hij(v) represents the frequency (i.e., the number of occurrences) of gray level v in the image block (i, j) before softening, the value range of i is [0, M / m], the value range of j is [0, N / n], and θ represents the threshold of histogram clipping, which is used to limit the maximum frequency of each gray level to avoid excessive contrast enhancement;

[0081]

[0082] Smax represents the number of brightest pixels in the image before softening, Smin represents the number of darkest pixels in the image before softening, S represents the total number of pixels in the image before softening, and λ is the cropping coefficient.

[0083] L represents the total number of gray levels, which is set to 256 (corresponding to an 8-bit grayscale image) in the present invention. The bit depth of a grayscale image determines the number of gray levels. This means that each pixel can have 28 = 256 different gray levels. The higher the bit depth, the more gray levels there are, and the more delicate the brightness levels that the image can express.

[0084] Reconstruct each small block after softening into a complete image, which is recorded as reconstructed image I 1 .

[0085] Image decomposition: Set the order of Coiflets wavelet, the order and parameters of B-spline scaling function, and reconstruct the image I 2 A two-dimensional discrete wavelet transform is performed to decompose the image into sub-bands of multiple scales and directions, and the sub-bands include low-frequency and high-frequency components.

[0086] Use Coiflets wavelet basis and B-spline scaling function to perform 2D-DWT on the image. The specific steps are as follows:

[0087] Perform discrete wavelet transform (DWT) on the rows:

[0088] Among them, ψ j,k (n) is the Coiflets wavelet basis function or the B-spline scaling function, is the result of DWT on the rows.

[0089] Perform discrete wavelet transform (DWT) on the columns:

[0090] in, The result of performing discrete wavelet transform DWT on the column is shown in Figure 1. Finally, the sub-bands obtained include: a low-frequency sub-band (LL) and three high-frequency sub-bands (LH, HL, HH).

[0091] Low frequency subband LL: represents the low frequency components of rows and columns. It contains the main information and large-scale features of the image.

[0092] High frequency subband LH: represents the low frequency components of rows and high frequency components of columns, capturing horizontal edge information.

[0093] High frequency subband HL: represents the high frequency components of rows and the low frequency components of columns, capturing vertical edge information.

[0094] High frequency subband HH: represents the high frequency components of rows and columns, capturing the detailed information in the diagonal direction.

[0095] The Coiflets wavelet basis and B-spline scaling function essentially decompose the image row by row and column by column, so the obtained subband structures (LL, LH, HL, HH) are the same.

[0096] Threshold processing: Apply threshold processing to the high-frequency subbands (LH, HL, HH), set the coefficients less than the threshold to zero, and keep the coefficients greater than the threshold. Set the threshold to:

[0097] Wavelet reconstruction: The processed coefficients are subjected to a two-dimensional discrete wavelet inverse transform to reconstruct the image and finally obtain a denoised image. Image quality enhancement is applied to each image in the action image set to obtain a high-quality image set.

[0098] In summary, wavelet basis functions and scaling functions play a complementary role in wavelet transform, and jointly determine the performance and applicability of wavelet transform. This system selects Coiflets wavelet and B-spline scaling function for image denoising. Coiflets wavelet has better smoothness, which helps to extract subtle features in the image while maintaining the smoothness of the image. B-spline scaling function has certain smoothness and locality, which complements the characteristics of Coiflets wavelet and can provide better performance in image processing. The combination of Coiflets wavelet and B-spline scaling function can simultaneously retain the detail information and overall structure in the image.

[0099] The organ feedback data and body operation data of the experimental subject are subjected to feature processing, and the feature data set is obtained, including: standardizing the organ feedback data and body operation data of the experimental subject, and calculating the state coefficient of the experimental subject:

[0100] Qf=f1×xl+f2×rd+f3×fd;

[0101] Sy=g1×cl+g2×tw+g3×np;

[0102] Among them, Qf represents the organ state coefficient, xl, rd and fd represent the blood flow parameters, peristaltic frequency and amplitude of the tissue in the surgical site respectively; Sy represents the body state coefficient, cl, tw, np represent the body operation data including heart rate, body temperature and brain wave respectively, and f1, f2, f3, g1, g2 and g3 all represent weight coefficients.

[0103] The organ state coefficient and body state coefficient are respectively encoded with IEEE754 floating point numbers to obtain two groups of binary characters. The two groups of binary characters are combined and time-tagged to obtain a string of characteristic binary characters. During the training time t, the organ feedback data and body operation data of the experimental subjects are continuously and uninterruptedly collected at a fixed time frequency p to obtain a string of characteristic binary characters to constitute a feature data set.

[0104] Solution generation module: used to input the high-quality image set and feature data set after data fusion into the trained GoogLeNet network learning model to obtain the training plan for the students.

[0105] Data fusion of high-quality image sets and feature datasets includes:

[0106] The trainee's action image set is a set of b grayscale images acquired continuously and uninterruptedly at a fixed time frequency p within the training time t. The training time t is discretized into b time points in chronological order, each time point corresponds to an image, and the timestamp of each image in the high-quality image set is extracted and filled into the corresponding time point. For example, the training time is set to t=10 minutes, the number of grayscale images acquired is 100, the number of discretized time points is b=100, and each time point corresponds to an image.

[0107] Get the time label of each string of feature binary characters in the feature data set, match them one by one with the discrete time points, and then match each string of feature binary characters with each image in the high-quality image set to obtain fused data points. Each fused data point includes the feature binary characters and images at the corresponding time points. All fused data points within the training time t constitute the fused data set.

[0108] During the training time, since the image acquisition frequency and the organ feedback data and body operation data are the same, that is to say, images, organ feedback data and body operation data are collected at the same time point, that is, a=b.

[0109] Build the GoogLeNet network learning model framework:

[0110] Four parallel paths are used, and the four paths are set to 1×1, 3×3, 5×5 and 1×1 convolution kernels in sequence. The 1×1 convolution kernel is set before the 5×5 and 3×3 convolution kernels;

[0111] The number of Inception modules is set to 3, the number of Inception convolution kernels is set to 512, and two fully connected layers are added above the softmax output layer. The fully connected layers use ReLU as the activation function, and the number of neurons in the fully connected layers is set to 2048 and 1024 respectively.

[0112] The GoogLeNet network learning model defines the loss function as,

[0113]

[0114] Where X is the total number of fused data sets in each subset, and x is the index; Output the time in the training plan for the GoogLeNet network learning model, is the output training solution of the GoogLeNet network learning model, t is the time in the actual training solution, and nc is the actual training solution in the running memory.

[0115] The GoogLeNet network learning model training process includes:

[0116] S11, setting the sample set to include o groups of subsets, each subset including X groups of fused data sets;

[0117] S12, dividing the subset into a training set and a validation set according to the ratio, and inputting the subsets in the training set into the GoogLeNet network learning model in sequence for forward propagation;

[0118] S13, obtain the predicted output of the GoogLeNet network learning model, calculate the loss value using the loss function, and update the weights and biases of the model through the back propagation algorithm;

[0119] S14. Repeat S12 and S13 until the loss function value of the GoogLeNet network learning model no longer changes, and then import the validation set for validation.

[0120] S15. When the verification is successful, obtain the trained GoogLeNet network learning model. When the verification is unsuccessful, repeat S12.

[0121] The fused data set obtained by the current students at this time is imported into the trained GoogLeNet network learning model to obtain the training plan for the students at the next training time, thereby realizing personalized training for the students.

[0122] In this embodiment, the present invention performs image quality enhancement preprocessing on the collected action image set, and realizes image denoising by wavelet transform in image quality enhancement. It performs softening processing on the maximum value pixels, retains more details and features for the brightest and darkest pixels, so as to reduce the loss of details in the denoising process, and expands the image by copying pixel values ​​on the boundary of the reconstructed image by a symmetric expansion method, so as to avoid introducing artifacts and distortion during boundary processing, thereby ensuring the effectiveness of wavelet transform. It increases the richness of image features and data information by performing feature processing on the organ feedback data and body operation data of the experimental subject, and ensures the data is integrated at the same time through data fusion. The consistency on the axis reduces the computing power of data analysis, while achieving high-quality data input, improving the effect of subsequent data processing and model training, as well as the retrieval and response speed, providing an accurate data foundation for subsequent model optimization; accelerating the efficiency of feature extraction, injecting new momentum into the agility and flexibility of data processing. The system implements the GoogLeNet network learning model to output personalized training plans for ultrasonic endoscopy intelligent training; the adaptive regulation and optimization mechanism continuously improves the overall efficiency of training, and multiple modules collaborate with each other to achieve comprehensive data collection, processing and analysis, and provide real-time and accurate data support.

[0123] Embodiment 2

[0124] Reference Figure 1 and Figure 2 , Example 2 further illustrates an ultrasonic endoscope intelligent training system proposed by the present invention.

[0125] The method for an ultrasonic endoscope intelligent training system is applied to an ultrasonic endoscope intelligent training system, and comprises the following steps:

[0126] Step S1: collecting the action image set of the current trainee within the training time t and the organ feedback data and body operation data of the experimental subject;

[0127] Step S2: pre-processing the collected action image set to enhance image quality, obtaining a high-quality image set, and performing feature processing on the organ feedback data and body operation data of the experimental subject to obtain a feature data set;

[0128] Step S3: Fuse the high-quality image set and the feature data set to obtain a fused data set.

[0129] Step S4: construct a GoogLeNet network learning model framework and perform training to obtain a trained GoogLeNet network learning model;

[0130] Step S5: Import the fused data set into the trained GoogLeNet network learning model to obtain the training plan for the next training time of the trainee.

[0131] The ultrasonic endoscope intelligent training system comprises:

[0132] Data collection module: used to collect the trainees' action image sets and the experimental subjects' organ feedback data and body operation data during the training process;

[0133] Data processing module: used to pre-process the collected action image set to obtain a high-quality image set, and perform feature processing on the organ feedback data and body operation data of the experimental subject to obtain a feature data set;

[0134] Solution generation module: used to input the high-quality image set and feature data set after data fusion into the trained GoogLeNet network learning model to obtain the training plan for the students;

[0135] The modules are connected to each other via wired and / or wireless means.

[0136] In addition, according to the implementation of the present application, the process described in the drawings of an ultrasonic endoscope intelligent training system can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided in the present application. Of course, the architecture shown in the drawings of an ultrasonic endoscope intelligent training system is only exemplary. When implementing different devices, adaptive selection or adjustment can be performed according to actual needs.

[0137] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0138] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An ultrasonic endoscope intelligent training system, characterized in that: The ultrasonic endoscope intelligent training system comprises: Data collection module: used to collect the trainees' action image sets and the experimental subjects' organ feedback data and body operation data during the training process; Data processing module: used to pre-process the collected action image set to obtain a high-quality image set, and perform feature processing on the organ feedback data and body operation data of the experimental subject to obtain a feature data set; Solution generation module: build and train the GoogLeNet network learning model, input the high-quality image set and feature data set after data fusion into the trained GoogLeNet network learning model, and obtain the training plan for the trainees; The modules are connected to each other via wired and / or wireless means.

2. The ultrasonic endoscope intelligent training system according to claim 1, characterized in that: The method for acquiring the action image set of the trainee comprises: During the training time t, multiple grayscale images are continuously and uninterruptedly acquired at a fixed time frequency p, which are collected synchronously by the endoscope's built-in camera and the external camera. Organ feedback data include blood flow parameters, peristaltic frequency and amplitude of tissues in the surgical site. Body operation data include heart rate, body temperature, and brain waves, which can be directly obtained through medical devices connected to the experimental subjects.

3. The ultrasonic endoscope intelligent training system according to claim 2, characterized in that: The preprocessing of the collected action image set includes image quality enhancement, and the image quality enhancement specifically includes: Construct an image matrix: Get each image I in the action image set, set the size of image I to M×N, soften the maximum pixel in the image, and obtain the reconstructed image I 1 , reconstructed image I 1 The size of is also M×N. Through the symmetric expansion method, the reconstructed image I 1 Copy the pixel values ​​on the boundary to expand the image and obtain the reconstructed image I 2 The size of is (M+r)×(N+r); Image decomposition: Set the order of Coiflets wavelet, the order and parameters of B-spline scaling function, and reconstruct the image I 2 Perform a two-dimensional discrete wavelet transform on the image matrix to decompose the image into sub-bands of multiple scales and directions, and the sub-bands include low-frequency and high-frequency components; Threshold processing: Apply threshold processing to the high-frequency subband, set the coefficients less than the threshold to zero, and keep the coefficients greater than the threshold. Set the threshold to: σ is the standard deviation of the noise; Wavelet reconstruction: The processed coefficients are subjected to a two-dimensional discrete wavelet inverse transform to reconstruct the image, and image quality enhancement is applied to each image in the action image set to obtain a high-quality image set.

4. The ultrasonic endoscope intelligent training system according to claim 3, characterized in that: The method for softening the maximum pixel in the image comprises: Suppose that the image I is divided into small blocks of size m×n, and equalization is applied to the grayscale histogram of each small block. Hij(v) 1 represents the frequency of gray level v in the image block (i, j) after equalization, Hij(v) represents the frequency of gray level v in the image block (i, j) before softening, the value range of i is [0, M / m], and the value range of j is [0, N / n]; θ represents the threshold of histogram clipping, and L represents the total number of gray levels; Reconstruct each small block after softening into a complete image, which is recorded as reconstructed image I 1 .

5. The ultrasonic endoscope intelligent training system according to claim 4, characterized in that: The threshold acquisition method for histogram clipping includes: Smax represents the number of brightest pixels in the image before softening, Smin represents the number of darkest pixels in the image before softening, S represents the total number of pixels in the image before softening, and λ is the cropping coefficient.

6. The ultrasonic endoscope intelligent training system according to claim 5, characterized in that: The symmetric expansion method is used to reconstruct the image I 1 Copy pixel values ​​on the boundary to expand the image. The specific steps are as follows: Horizontal expansion: symmetrically copy the left and right sides of the image by r rows respectively; Vertical expansion: symmetrically copy the upper and lower sides of the image by r rows respectively; Reconstructed Image I 1 After processing by the symmetric expansion method, the reconstructed image I is obtained 2 The size of is (M+r)×(N+r), and the reconstructed image I 2 The two-dimensional matrix representation of is: Among them, I 1 k1，k2 Represents the pixel value of the k1th row and k2th column, and k1=1,...,M+r, k2=1,...,N+r.

7. The ultrasonic endoscope intelligent training system according to claim 6, characterized in that: The feature processing of the organ feedback data and body operation data of the experimental subject to obtain the feature data set includes: standardizing the organ feedback data and body operation data of the experimental subject, and calculating the state coefficient of the experimental subject: Qf=f1×xl+f2×rd+f3×fd; Sy=g1×cl+g2×tw+g3×np; Among them, Qf represents the organ state coefficient, xl, rd and fd represent the blood flow parameters, peristalsis frequency and amplitude of the surgical site tissue respectively; Sy represents the body state coefficient, cl, tw, np represent the body operation data including heart rate, body temperature and brain wave respectively, f1, f2, f3, g1, g2 and g3 represent weight coefficients; The organ state coefficient and the body state coefficient are respectively encoded with IEEE754 floating point numbers to obtain two groups of binary characters. The two groups of binary characters are combined and time-stamped to obtain a string of characteristic binary characters. All characteristic binary characters are combined to form a characteristic data set.

8. The ultrasonic endoscope intelligent training system according to claim 7, characterized in that: The data fusion of the high-quality image set and the feature data set includes: The trainee's action image set is a set of b grayscale images acquired continuously and uninterruptedly at a fixed time frequency p during the training time t. The training time t is discretized into b time points in chronological order. Each time point corresponds to an image. The timestamp of each image in the high-quality image set is extracted and filled into the corresponding time point. Get the time label of each string of feature binary characters in the feature data set, match them one by one with the discrete time points, and then match each string of feature binary characters with each image in the high-quality image set to obtain fused data points. Each fused data point includes the feature binary characters and images at the corresponding time points. All fused data points within the training time t constitute the fused data set.

9. The ultrasonic endoscope intelligent training system according to claim 8, characterized in that: The method for constructing the GoogLeNet network learning model includes: Four parallel paths are used, and the four paths are set to 1×1, 3×3, 5×5 and 1×1 convolution kernels in sequence. The 1×1 convolution kernel is set before the 5×5 and 3×3 convolution kernels; The number of Inception modules is set to 3, the number of Inception convolution kernels is set to 512, two fully connected layers are added above the softmax output layer, the fully connected layers use ReLU as the activation function, and the number of neurons in the fully connected layers is set to 2048 and 1024 respectively. The GoogLeNet network learning model defines the loss function as, Among them, X is the total number of fused data sets in each subset, and x is the index; Output the time in the training plan for the GoogLeNet network learning model, The output training scheme of the GoogLeNet network learning model occupies the running memory, t is the time in the actual training scheme, nc is the actual training scheme occupies the running memory, and w1 and w2 are weight coefficients.

10. The ultrasonic endoscope intelligent training system according to claim 9, characterized in that: The GoogLeNet network learning model training process includes: S11, setting the sample set to include o groups of subsets, each subset including X groups of fused data sets; S12, dividing the subset into a training set and a validation set according to the ratio, and inputting the subsets in the training set into the GoogLeNet network learning model in sequence for forward propagation; S13, obtain the predicted output of the GoogLeNet network learning model, calculate the loss value using the loss function, and update the weights and biases of the model through the back propagation algorithm; S14, repeat S12 and S13 until the loss function value of the GoogLeNet network learning model no longer changes, and then import the verification set for verification; S15. When the verification is successful, obtain the trained GoogLeNet network learning model. When the verification is unsuccessful, repeat S12.

Images