Evaluation method and device for observing ultrasonic image, equipment and storage medium
By converting eye movement data into heat maps and fusing it with ultrasound images, the method objectively evaluates a trainee's observation, improving the accuracy of ultrasound image evaluation.
Patent Information
- Application Number
- CN202411185819.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-27
AI Technical Summary
The accuracy of evaluating a doctor's method for observing ultrasound images is low due to subjective judgment by experienced experts, leading to varying observation and diagnosis results.
An evaluation method that converts the eye movement video stream of a trainee into an eye movement heat map video stream, fuses it with the ultrasound video stream to generate a fused video stream, and uses this to objectively determine the trainee's observation position, thereby improving evaluation accuracy.
The method provides an objective evaluation of the trainee's observation position, enhancing the accuracy of ultrasound image evaluation results.
Smart Images

Figure CN119069088A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical teaching auxiliary technology, and in particular to an evaluation method, device, equipment and storage medium for observing ultrasonic images. Background Art
[0002] Medical ultrasound has important clinical examination value. Doctors can make a preliminary diagnosis of patients by observing the patient's ultrasound image. However, different doctors observe ultrasound images in different ways, which will lead to different observation and diagnosis results. Therefore, it is necessary to evaluate the doctor's method or way of observing ultrasound images to determine whether the doctor's method or way of observing ultrasound images is correct. The existing technology mainly relies on experienced experts to judge whether the doctor's method or way of observing ultrasound images is correct to obtain the doctor's evaluation result. The evaluation results obtained by the existing technology are subjective to a certain extent, resulting in low accuracy of the evaluation results.
[0003] In summary, the accuracy of the evaluation results of ultrasound images observed in the prior art is low.
[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides an evaluation method, device, equipment and storage medium for observing ultrasonic images, which solves the problem of low accuracy of evaluation results of observing ultrasonic images in the prior art.
[0006] To achieve the above object, the present invention adopts the following technical solutions:
[0007] In a first aspect, the present invention provides an evaluation method for observing an ultrasound image, comprising:
[0008] Displaying an ultrasound video stream, collecting an eye movement video stream of a trainee when observing the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the observation position of the trainee;
[0009] Generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image;
[0010] An evaluation result of the trainee is obtained according to the fused video stream.
[0011] In one implementation, the step of displaying the ultrasound video stream and collecting the eye movement video stream of the trainee when observing the ultrasound video stream includes:
[0012] Two independent processes are used to display the ultrasound video stream and collect the eye movement video stream respectively.
[0013] In one implementation, generating a fused video stream based on the eye movement heat map video stream and the ultrasound video stream includes:
[0014] Determine each frame of thermal image in the eye movement thermal map video stream, determine all frames of ultrasound image in the ultrasound video stream, and all frames of ultrasound image are the same image;
[0015] Adjusting the size of each frame of the thermal image and adjusting the ultrasonic image corresponding to the frame of the thermal image so that the thermal image of the frame and the ultrasonic image corresponding to the frame of the thermal image have the same size;
[0016] A fused image is generated based on the thermal image and the ultrasonic image of the same size, and a fused video stream is generated by combining all the fused images.
[0017] In one implementation, generating a fused image based on the thermal image and the ultrasound image of the same size includes:
[0018] Binarizing the thermal image to obtain a binary thermal image;
[0019] Determine a first pixel point position with a pixel value of 255 and a second pixel point position with a pixel value of 0 on the thermal binary image;
[0020] Determine a target ultrasound pixel value at a position on the ultrasound image corresponding to the first pixel point position, and determine a target thermal pixel value at a position on the thermal image corresponding to the second pixel point position;
[0021] A fused image is generated according to the target ultrasound pixel value and the target thermal pixel value.
[0022] In one implementation, the pixel points with a pixel value of 255 on the thermal binary image are recorded as non-visible pixel points, and the non-visible pixel points correspond to the pixel points on the thermal image whose pixel values are less than a threshold value; the pixel points with a pixel value of 0 on the thermal binary image are recorded as visible pixel points, and the visible pixel points correspond to the pixel points on the thermal image whose pixel values are greater than or equal to the threshold value.
[0023] In one implementation, obtaining the evaluation result of the trainee according to the fused video stream includes:
[0024] The annotated video stream is displayed, where the annotated video stream is used to mark the position of the lesion area that needs to be observed on the ultrasound image.
[0025] Applying a graph wheel algorithm to the annotated video stream and the fused video stream to obtain an observation order similarity, and using the observation order similarity as an evaluation result;
[0026] Alternatively, determining the continuous same frame fused images and the first synthesized image and each frame fused image in the fused video stream;
[0027] Counting the total number of image frames included in the continuous same-frame fusion images, and obtaining the time length for the trainee to observe the lesion area corresponding to the continuous same-frame fusion images according to the total number of image frames;
[0028] Determining a first lesion region included in the first composite image;
[0029] Determining each observed lesion area corresponding to each frame of the fused image;
[0030] The evaluation result of the trainee is obtained according to the duration, the first lesion area and each of the observed lesion areas.
[0031] In one implementation, the method further includes:
[0032] Determine each frame of eye movement thermal image in the eye movement thermal image video stream;
[0033] According to each frame of the eye movement thermal image, an observation order of each lesion area on the ultrasound image observed by the trainee is obtained, wherein the ultrasound image is each frame of the ultrasound image included in the ultrasound video stream, and each frame of the ultrasound image is the same image;
[0034] Obtaining a reference observation sequence, and obtaining the accuracy of the observation method adopted by the trainee based on the observation sequence and the reference observation sequence;
[0035] A final evaluation result is obtained based on the accuracy and the evaluation result.
[0036] In a second aspect, an embodiment of the present invention further provides an evaluation device for observing ultrasound images, wherein the device includes the following components:
[0037] A video stream acquisition module, used for displaying an ultrasound video stream, and acquiring an eye movement video stream when a trainee observes the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to characterize the observation position of the trainee;
[0038] A fusion module, used to generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image;
[0039] The evaluation module is used to obtain the evaluation result of the trainee according to the fused video stream.
[0040] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device comprises a memory, a processor, and an evaluation program for observing ultrasonic images stored in the memory and executable on the processor, and when the processor executes the evaluation program for observing ultrasonic images, the steps of the above-mentioned method for evaluating ultrasonic images are implemented.
[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which is stored an evaluation program for observing ultrasonic images. When the evaluation program for observing ultrasonic images is executed by a processor, the steps of the above-mentioned evaluation method for observing ultrasonic images are implemented.
[0042] Beneficial effects: The present invention first converts the eye movement video stream into an eye movement heat map video stream to obtain the trainee's eye observation position, and then fuses the eye movement heat map video stream and the ultrasound video stream to obtain a fused video stream, which records the position of each frame of the ultrasound image observed by the trainee on the ultrasound video stream. Therefore, the fused video stream can accurately determine whether the trainee's observation position of the ultrasound image is correct, thereby obtaining the trainee's evaluation result. From the above analysis, the present invention can objectively determine whether the trainee's observation position of the ultrasound image is correct by generating a fused video stream, thereby improving the accuracy of the evaluation result. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is the overall flow chart of the present invention;
[0044] Figure 2 is a schematic diagram of a display interface in an embodiment of the present invention;
[0045] Figure 3 is a design flow chart of a diagnostic training platform in an embodiment of the present invention;
[0046] Figure 4 It is the real-time monitoring system interface in the embodiment of the present invention;
[0047] Figure 5 is a structural diagram of scoring indicators in an embodiment of the present invention;
[0048] Figure 6 Schematic diagram of the observation sequence in the embodiment of the present invention;
[0049] Figure 7 A structural diagram of an evaluation device for observing ultrasonic images provided by the present invention;
[0050] Figure 8 This is a block diagram of the internal structure principle of the terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The following is a clear and complete description of the technical solution of the present invention in combination with the embodiments and the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] Research has found that medical ultrasound has important clinical examination value. Doctors can make a preliminary diagnosis of patients by observing the patient's ultrasound image. However, different doctors observe ultrasound images in different ways, which will lead to different observation and diagnosis results. Therefore, it is necessary to evaluate the doctor's method or way of observing ultrasound images to determine whether the doctor's method or way of observing ultrasound images is correct. The existing technology mainly uses experienced experts to judge whether the doctor's method or way of observing ultrasound images is correct to obtain the doctor's evaluation result. The evaluation result obtained by the existing technology has a certain degree of subjectivity, resulting in low accuracy of the evaluation result.
[0053] In order to solve the above technical problems, the present invention provides an evaluation method, device, equipment and storage medium for observing ultrasonic images, which solves the problem of low accuracy of evaluation results of observing ultrasonic images in the prior art. In specific implementation, the ultrasonic video stream is first displayed, and the eye movement video stream of the trainee when observing the ultrasonic video stream is collected, and the eye movement video stream is converted into an eye movement heat map video stream, the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasonic video stream, and the eye movement heat map video stream is used to characterize the observation position of the trainee; then, based on the eye movement heat map video stream and the ultrasonic video stream, a fused video stream is generated, and the fused video stream is used to characterize the position of each frame of the ultrasonic image of the trainee observing the ultrasonic video stream; finally, based on the fused video stream, the evaluation result of the trainee is obtained.
[0054] For example, an ultrasound video stream is composed of ten ultrasound images (these ten ultrasound images are the same ultrasound image). These ten ultrasound images are displayed in sequence. Each time a frame of ultrasound image is displayed, the trainee is prompted to observe the frame of ultrasound image. At the same time, the trainee's eye image is collected. The ten ultrasound images correspond to ten eye images. The ten eye images are sorted according to the collection time to form an eye movement video stream. The eye movement video stream records the eye movement trajectory. The rotation trajectory represents the position of the trainee observing each frame of ultrasound image, that is, whether the trainee's eyes observe the position of the actual lesion area on the ultrasound image. The eye movement video stream is converted into an eye movement thermal map video stream. The eye movement thermal map video stream also contains ten frames of thermal images. Since the first frame of ultrasound image and the first frame of thermal image have the same size, a new image (the new image is the first frame of fused image) is derived from the first frame of ultrasound image and the first frame of thermal image. The first frame of fused image records the position where the trainee observes the first frame of ultrasound image. Using the same method, the second frame of fused image is derived from the second frame of ultrasound image and the second frame of thermal image, and the third frame of fused image is derived from the third frame of ultrasound image and the third frame of thermal image, and so on, the third frame of fused image, the fourth frame of fused image, the fifth frame of fused image, the sixth frame of fused image, the seventh frame of fused image, the eighth frame of fused image, the ninth frame of fused image, and the tenth frame of fused image are derived. These ten frames of fused images form a fused video stream. Through the ten frames of fused images, it can be determined whether the position observed by the trainee is the position of the lesion area on the ultrasound image, so as to obtain the evaluation result of the trainee.
[0055] The evaluation method for observing ultrasound images of this embodiment can be applied to a terminal device, and the terminal device can be a terminal product with a video playback function, such as a computer. Figure 1 As shown in , the evaluation method for observing ultrasound images specifically includes the following steps:
[0056] S100, displaying an ultrasound video stream, collecting an eye movement video stream when the trainee observes the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to characterize the observation position of the trainee.
[0057] Among them, converting eye movement video stream into eye movement heat map video stream is an existing technology.
[0058] S200, generating a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image.
[0059] S300, obtaining an evaluation result of the trainee according to the fused video stream.
[0060] The invention is based on the evaluation method of S100 to S300 and is applied to the diagnosis training platform. The diagnosis training platform includes a real-time monitoring module, a user evaluation module, and a scoring interface module. The user evaluation module is used to Figure 2 The ultrasound image shown is loaded onto the platform, and the progress of the real-time monitoring module is controlled by the user evaluation module. The specific implementation of the user evaluation module is as follows: by giving three pictures containing AC, NT, etc. (such as Figure 3 NT and AC as shown, by traversing each picture in the contents of each folder, the image pixels of each picture are stored in each element in the list in the form of a matrix. When display is needed, just call it directly in the list. By setting a button click slot function and accepting the name of the click event object, such as clicking the nt button, the slot function will get the information that the button was clicked and that the button clicked was the nt button, so the list containing the information of the nt pictures is displayed on the label as the current main list, and the user will watch it and click the switch button to add or subtract one to the current picture index to switch to the next picture and switch to the previous picture. By setting a full-screen hidden label, when the user clicks watching, the ultrasound picture will be displayed in full screen to achieve full-screen viewing. The user's eye movement video and the eye movement video of the expert library are input through the graph theory algorithm for comparison to obtain the analysis result.
[0061] The real-time monitoring module is used to display the ultrasound video stream formed by the loaded ultrasound image, and the real-time monitoring module is also used to display the eye movement heat map video stream and the fusion video stream, and also display the annotated video stream as a benchmark. The scoring interface module is used to obtain the evaluation result in step S300.
[0062] In the first embodiment, the step S100 of displaying the ultrasound video stream and collecting the eye movement video stream when the trainee observes the ultrasound video stream specifically includes: using two independent processes to respectively display the ultrasound video stream and collect the eye movement video stream.
[0063] This embodiment not only independently displays the ultrasound video stream and the eye movement video stream, but also independently displays the annotated video stream and the fused video stream, that is, four processes are used to independently collect the ultrasound video stream, the eye movement video stream, the annotated video stream and the fused video stream, and these four processes are used to independently display these four video streams, and two event signals control the simultaneous start and end of these four processes, ensuring that the four video stream data are sampled and displayed simultaneously within an almost error-free range. Among them, the pictures of various organ parts annotated in advance by the detection model (annotated video streams) are displayed on the interface at the same time at a rate of thirty frames per second, so that the evaluator (that is, a doctor with rich experience in viewing ultrasound images) can use the annotated video stream as a benchmark to observe the accuracy of the user viewing the ultrasound image.
[0064] In this embodiment, while the display interface displays the ultrasound video stream, eye movement video stream, annotated video stream and fused video stream, the display is recorded to obtain a fifth video stream, that is, a screen recording video stream. When the user finishes the training, the five video streams will be saved to the current software directory to provide original eye movement data for scoring, analysis and review. The five video stream data sources sampled by ffmpeg will also be sent to a separate video saving process, in which five videowriters will simultaneously write image data at thirty frames per second, and finally generate five videos.
[0065] This embodiment uses ffmpeg to drive the capture card and screen capture, samples at a sampling rate of thirty frames per second, and then displays it on the interface.
[0066] The display interface of this embodiment includes an Eyetracking display area, an Ultrasonic display area, a Synthesis display area, and a Rectangle display area.
[0067] The Eyetracking display area uses ffmpeg to drive the capture card to read and display the user's eye movement data in real time. The specific implementation operation is to first define the ffmpeg command line in python and then set the parameters of this ffmpeg command line, namely self.ffmpeg_command1 = ['ffmpeg','-f','dshow','-i','video=USB VideoDevice','-s','1088x612','-r','30','-pix_fmt','rgb24','-f','rawvideo','pipe:1'], and then use the Popen function in a subprocess class to use this code to drive the capture card to read the entire screen data of another computer at thirty frames per second. The screen of the other computer will display the current user's eye movement heat map in real time (TobiiEyetracking comes with software for real-time display of eye movement heat maps).
[0068] The Ultrasonic display area uses ffmpeg screen recording to read and display the ultrasound image currently viewed by the doctor in real time.
[0069] The Synthesis display area aligns and integrates eye movement data and ultrasound images in time and space to generate a fused video stream.
[0070] The Rectangle display area displays the labeled pictures in real time (several frames of labeled pictures constitute a labeled video stream), which is used to monitor the ultrasound image area that the user is looking at in real time, that is, to monitor whether the ultrasound image area observed by the user is an area on the labeled picture, where the labeled picture is used as the reference picture.
[0071] like Figure 4 As shown, the real-time data display of the Eyetracking display area, Ultrasonic display area, Synthesis display area and Rectangle display area of the display interface is executed by four processes at the same time, and accompanied by a dedicated recording process in the background, the contents of these four display areas and the entire interface will be recorded synchronously at the same time, generating five independent video files, recording the entire monitoring process from all directions and angles. The Start Recording and Stop Recording buttons at the bottom of the display interface are used to respond to the signals from the user evaluation interface to execute the start and end recording functions, ensuring that all important information is fully recorded and ready for subsequent data analysis and feedback.
[0072] Embodiment 2, based on embodiment 1, in this embodiment, step S200 includes the following specific steps S201 to S206:
[0073] S201, determining each frame of thermal image in the eye movement thermal map video stream, and determining all frames of ultrasound image in the ultrasound video stream, wherein all frames of ultrasound image are the same image;
[0074] S202, adjusting the size of each frame of the thermal image, and adjusting the ultrasonic image corresponding to the frame of the thermal image, so that the thermal image of the frame and the ultrasonic image corresponding to the frame of the thermal image have the same size.
[0075] S203, binarizing the thermal image to obtain a binary thermal image.
[0076] S204, determining a first pixel point position with a pixel value of 255 and a second pixel point position with a pixel value of 0 on the thermal binary image.
[0077] S205, determining a target ultrasound pixel value at a position on the ultrasound image corresponding to the first pixel point position, and determining a target thermal pixel value at a position on the thermal image corresponding to the second pixel point position.
[0078] S206: Generate a fused image according to the target ultrasound pixel value and the target thermal pixel value.
[0079] The specific implementation process from S203 to S206 is as follows:
[0080] The corresponding mask is generated by using the inRange function (image threshold function) of the cv library, that is, all the pixels in the range of (0, 0, 0)-(30, 30, 30) in image 1 are assigned a value of 255, and the rest are 0. Then the where function of the np library is used, that is, np.where(mask[:,:,None].astype(bool),rgb_frame2,rgb_frame1), that is, the area with a pixel value of 255 in the mask will be retained as the pixels corresponding to rgb_fram2 (ultrasound image), and the rest of the positions will be pixels of rgb_frame1 (thermal image), thereby achieving synthesis.
[0081] Wherein, 30 is the threshold value, and the pixel points with a pixel value of 255 on the thermal binary image are recorded as non-visible pixel points, and the non-visible pixel points correspond to the pixel points on the thermal image with a pixel value less than the threshold value; the pixel points with a pixel value of 0 on the thermal binary image are recorded as visible pixel points, and the visible pixel points correspond to the pixel points on the thermal image with a pixel value greater than or equal to the threshold value.
[0082] That is, the pixel points in the range of (0, 0, 0)-(30, 30, 30) on the thermal image are assigned a value of 255, that is, the area with a pixel value of 255 is the area not seen by the trainee, and the pixel points in other ranges are assigned a value of 0, that is, the area with a pixel value of 0 is the area seen by the trainee, so as to convert the thermal image into a thermal binary image. A third blank image with the same size as the thermal binary image and the ultrasonic image is established, and the first position area corresponding to the pixel point with a pixel value of 255 in the thermal binary image is found on the blank image, and the first position area of the blank image is assigned the pixel value of the ultrasonic image, and the second position area of the blank image is found on the blank image corresponding to the pixel point with a pixel value of 0 in the thermal binary image, and the second position area of the blank image is assigned a value of 0 (0 is the pixel value of the thermal binary image in this position area). Since the pixel values in the range of (0, 0, 0)-(30, 30, 30) are black pixels, that is, the positions of the pixel values in this range on the image are not seen by the trainee's eyes, while the positions of other pixel value ranges are seen by the trainee's eyes. Therefore, the other pixel values on the thermal binary image are retained to indicate the positions observed by the trainee.
[0083] For example, the size of the thermal image and the ultrasonic image are both AxA, the pixel value of the upper left corner area of the thermal image (that is, the target thermal pixel value) is (r1, g1, b1), and the pixel values of other areas of the thermal image are (r1′, g1′, b1′), and (r1, g1, b1) are in the range of (0, 0, 0)-(30, 30, 30), so the pixel values of the upper left corner area of the thermal image are all set to 255, and the pixel values of other areas of the thermal image are set to 0, so as to convert the thermal image into a thermal binary image. Create a third blank image, the size of which is also AxA. If the pixel value of the upper left corner area of the ultrasound image (that is, the target ultrasound pixel value) is (r2, g2, b2), then the pixel value of the upper left corner area of the third blank image is set to (r2, g2, b2), and the pixel values of other areas of the third blank image are set to (r1′, g1′, b1′) to generate a fused image. The other areas on the fused image are the positions on the ultrasound image observed by the trainee mapped to the positions on the fused image. Therefore, the fused image can be used to know which position of the ultrasound image the trainee is observing, so as to determine whether the observation position is correct.
[0084] Embodiment 3, based on embodiment 1 or embodiment 2, in this embodiment, step S300 includes the following specific steps: displaying a labeled video stream, the labeled video stream is used to mark the position of the lesion area to be observed on the ultrasound image. Applying a graph wheel algorithm to the labeled video stream and the fused video stream to obtain an observation sequence similarity, and using the observation sequence similarity as an evaluation result.
[0085] For example, the annotated video stream includes five annotated images, and the annotated positions (the positions represent the positions that need to be viewed and observed) are positions s11, s12, s13, s14, and s15 on the image. The order of these five positions constitutes the baseline observation order. The fused video stream also includes five fused images. The positions on the fused images that represent the positions observed by the trainee are positions s21, s22, s23, s24, and s25. The order of these five positions constitutes the experimental observation order. The graph wheel algorithm is applied to positions s11, s12, s13, s14, s15, s21, s22, s23, s24, and s25 to obtain the observation order similarity between the baseline observation order and the experimental observation order. Figure 5 As shown in the figure, the graph wheel algorithm includes weighted edge set similarity, maximum weight matching similarity, node transition probability-based similarity, and shared node similarity.
[0086] The trained eye movement data (i.e., fused video stream) and the eye movement data of the expert library (i.e., labeled video stream) are compared above, so as to give objective and accurate feedback scores to guide user learning.
[0087] In this embodiment, step S300 may also include the following specific steps: determining the continuous same-frame fused images and the first synthetic image and each frame fused image in the fused video stream; counting the total number of image frames contained in the continuous same-frame fused images, and based on the total number of image frames, obtaining the time duration for the trainee to observe the lesion area corresponding to the continuous same-frame fused images; determining the first lesion area contained in the first synthetic image; determining each observed lesion area corresponding to each frame of the fused image; and obtaining the evaluation result of the trainee based on the time duration and the first lesion area and each observed lesion area.
[0088] The continuous same frame fusion image represents the fusion image generated by observing the same lesion area on the continuous frame ultrasound image, and the total number of frames of the fusion image represents the time the trainee observes the lesion area. This embodiment will count the total time the trainee uses to observe all lesion areas, and combine the time and the total time and the first lesion area to determine whether the trainee's way or method of observing the ultrasound image is correct, so as to obtain the evaluation result of the trainee.
[0089] In this embodiment, the similarity between the observed organ and the reference can also be determined to obtain an evaluation result.
[0090] Embodiment 4, based on embodiment 1 or embodiment 2 or embodiment 3, includes the following specific steps S401 to S404:
[0091] S401, determining each frame of eye movement heat image in the eye movement heat map video stream.
[0092] The pixel position corresponding to the pixel value greater than the threshold (the threshold, that is, the value of the RGB three channels is 30) on the eye movement thermal image is the observation position of the trainee on the image. Since the eye movement thermal image and the ultrasound image have the same size, the observation position represents the lesion area on the ultrasound image observed by the trainee.
[0093] S402, obtaining the observation order of the trainee observing each lesion area on the ultrasound image based on each frame of the eye movement thermal image, wherein the ultrasound image is each frame of ultrasound image included in the ultrasound video stream, and each frame of ultrasound image is the same image.
[0094] S403, obtaining a reference observation sequence, and obtaining the accuracy of the observation method adopted by the trainee according to the observation sequence and the reference observation sequence.
[0095] Figure 6 The numbers on the a diagram in FIG. 1 represent the reference observation order formed by the positions that should be observed on the ultrasound image. Figure 6 Each number on the b diagram in FIG. 1 represents the observation order formed by the observed positions on the ultrasound image. By comparing the observation order with the reference order, it can be determined whether the trainee's observation method is correct.
[0096] S404: Obtain a final evaluation result according to the accuracy and the evaluation result.
[0097] The accuracy and evaluation results are weighted to obtain the final evaluation results.
[0098] like Figure 6 As shown in the figure, the present invention uses the similarity of observed organs, the ranking of attention concentration (similarity of the comprehensive time of observing each organ), the correctness of the starting organ, the decision speed and the observation order as the evaluation dimensions of reading behavior, wherein the Markov chain is introduced for sequence modeling in the dimension of observation order, and the reading order is evaluated based on the similarity of nodes and edges through the graph theory algorithm. At the same time, the eye movement data of ultrasound physicians are introduced to analyze the reading behavior of physicians of different years.
[0099] The present invention first starts working in the video reading and initialization module, specifically processing the black screen eye movement video data obtained by the monitoring system. This stage involves a strict video synchronization and frame alignment mechanism to ensure the spatial correspondence between each frame image and the ultrasound image, and achieve consistency in spatial dimensions through precise size matching. Subsequently, a series of image preprocessing steps are performed, including but not limited to noise reduction, grayscale and contrast enhancement, aiming to optimize image quality and provide clear basic data for subsequent change detection. The core of the algorithm is to adopt an efficient inter-frame difference analysis method, which effectively identifies the changing area of the visual focus over time by calculating the absolute pixel difference between adjacent frames. On this basis, the algorithm incorporates an advanced contour detection algorithm to identify significant contours in the difference area, and through geometric center positioning, determines the most important change point in each frame, that is, the focus of the doctor's gaze. This strategy not only improves the sensitivity and accuracy of change detection, but also provides reliable positioning information for the continuous tracking of eye movement trajectories. By continuously tracking the above-mentioned change points, the algorithm constructs a sequence model of the doctor's eye movement trajectory during ultrasound image observation. This process not only reveals the dynamic distribution of visual attention, but also generates a detailed sequence of user-observed organs by integrating all change point information, intuitively showing the degree of physician's attention to each part of the ultrasound image. It is of great value for understanding physician's image interpretation strategy, evaluating education and training effects, and guiding the optimization of ultrasound image interface.
[0100] In summary, the present invention innovatively integrates eye tracking technology, graph theory algorithms, and multi-process technology to construct an eye tracking training platform. Based on these two, two sets of pre-recorded eye movement data can be scored separately using graph theory algorithms and eye tracking and multi-process acquisition using deep learning instead of graph theory algorithms, which may avoid this design scheme in this way. Alternative solutions to the core technology may include: in terms of data collection; screen recording software in other languages may be used instead, and the python script calls the ffmpeg library method to achieve the simultaneous recording of five video streams. Then the already labeled ultrasound images may be replaced with ultrasound images of other organ parts that have been labeled. At the same time, the graph theory algorithm is used to rate the similarity between the eye movement data graph and the expert graph. It is possible that competitors will use deep learning methods to evaluate the similarity of the two images.
[0101] This embodiment also provides an evaluation device for observing ultrasound images, such as Figure 7 As shown, the device comprises the following components:
[0102] The video stream acquisition module 01 is used to display the ultrasound video stream, and to acquire the eye movement video stream of the trainee when observing the ultrasound video stream, and to convert the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to characterize the observation position of the trainee;
[0103] A fusion module 02, used for generating a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used for marking the position where the trainee observes the ultrasound image;
[0104] The evaluation module 03 is used to obtain the evaluation result of the trainee according to the fused video stream.
[0105] Based on the above embodiments, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Figure 8 As shown. The terminal device includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for observing and evaluating ultrasound images is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.
[0106] Those skilled in the art will understand that Figure 8 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0107] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and an evaluation program for observing ultrasound images stored in the memory and executable on the processor. When the processor executes the evaluation program for observing ultrasound images, the following operation instructions are implemented:
[0108] Displaying an ultrasound video stream, collecting an eye movement video stream of a trainee when observing the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the observation position of the trainee;
[0109] Generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image;
[0110] An evaluation result of the trainee is obtained according to the fused video stream.
[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for observing and evaluating ultrasound images, characterized in that: include: Displaying an ultrasound video stream, collecting an eye movement video stream of a trainee when observing the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the observation position of the trainee; Generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image; An evaluation result of the trainee is obtained according to the fused video stream.
2. The method for evaluating ultrasound images according to claim 1, wherein: The displaying of the ultrasound video stream and collecting the eye movement video stream of the trainee when observing the ultrasound video stream includes: Two independent processes are used to display the ultrasound video stream and collect the eye movement video stream respectively.
3. The method for evaluating ultrasound images according to claim 1, wherein: The step of generating a fused video stream based on the eye movement heat map video stream and the ultrasound video stream comprises: Determine each frame of thermal image in the eye movement thermal map video stream, determine all frames of ultrasound image in the ultrasound video stream, and all frames of ultrasound image are the same image; Adjusting the size of each frame of the thermal image and adjusting the ultrasonic image corresponding to the frame of the thermal image so that the thermal image of the frame and the ultrasonic image corresponding to the frame of the thermal image have the same size; A fused image is generated based on the thermal image and the ultrasonic image of the same size, and a fused video stream is generated by combining all the fused images.
4. The method for evaluating ultrasound images according to claim 3, wherein: The step of generating a fused image based on the thermal image and the ultrasonic image of the same size comprises: Binarizing the thermal image to obtain a binary thermal image; Determine a first pixel point position with a pixel value of 255 and a second pixel point position with a pixel value of 0 on the thermal binary image; Determine a target ultrasound pixel value at a position on the ultrasound image corresponding to the first pixel point position, and determine a target thermal pixel value at a position on the thermal image corresponding to the second pixel point position; A fused image is generated according to the target ultrasound pixel value and the target thermal pixel value.
5. The method for evaluating ultrasound images according to claim 4, wherein: The pixel points with a pixel value of 255 on the thermal binary image are recorded as non-visible pixel points, and the non-visible pixel points correspond to the pixel points on the thermal image whose pixel values are less than a threshold value; the pixel points with a pixel value of 0 on the thermal binary image are recorded as visible pixel points, and the visible pixel points correspond to the pixel points on the thermal image whose pixel values are greater than or equal to the threshold value.
6. The method for evaluating ultrasound images according to claim 1, wherein: The step of obtaining the evaluation result of the trainee according to the fused video stream includes: The annotated video stream is displayed, where the annotated video stream is used to mark the position of the lesion area that needs to be observed on the ultrasound image. Applying a graph theory algorithm to the annotated video stream and the fused video stream to obtain an observation order similarity, and using the observation order similarity as an evaluation result; Alternatively, determining the continuous same frame fused images and the first synthesized image and each frame fused image in the fused video stream; Counting the total number of image frames included in the continuous same-frame fusion images, and obtaining the time length for the trainee to observe the lesion area corresponding to the continuous same-frame fusion images according to the total number of image frames; Determining a first lesion region included in the first composite image; Determining each observed lesion area corresponding to each frame of the fused image; The evaluation result of the trainee is obtained according to the duration, the first lesion area and each of the observed lesion areas.
7. The method for evaluating ultrasound images according to claim 1, wherein: Also includes: Determine each frame of eye movement thermal image in the eye movement thermal image video stream; According to each frame of the eye movement thermal image, an observation order of each lesion area on the ultrasound image observed by the trainee is obtained, wherein the ultrasound image is each frame of the ultrasound image included in the ultrasound video stream, and each frame of the ultrasound image is the same image; Obtaining a reference observation sequence, and obtaining the accuracy of the observation method adopted by the trainee based on the observation sequence and the reference observation sequence; A final evaluation result is obtained based on the accuracy and the evaluation result.
8. An evaluation device for observing ultrasonic images, characterized in that: The device comprises the following components: A video stream acquisition module, used for displaying an ultrasound video stream, and acquiring an eye movement video stream when a trainee observes the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to characterize the observation position of the trainee; A fusion module, used to generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image; The evaluation module is used to obtain the evaluation result of the trainee according to the fused video stream.
9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and an evaluation program for observing ultrasonic images stored in the memory and executable on the processor. When the processor executes the evaluation program for observing ultrasonic images, the steps of the method for observing ultrasonic images as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an evaluation program for observing ultrasonic images. When the evaluation program for observing ultrasonic images is executed by a processor, the steps of the evaluation method for observing ultrasonic images according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Method, device and system for determining image target area
CN112885435A
Method for improving accuracy of ultrasonic image recognition model through doctor eye movement data
CN115641496A
Eye movement attention driving-based cardiac ultrasound image domain adaptive segmentation method
CN118154627A
Method and System of Using Eye Tracking to Evaluate Subjects
US20150282705A1
