An evaluation method, device and equipment for observing an ultrasound image and a storage medium

Abstract

The present application relates to the technical field of medical teaching auxiliary, in particular to a kind of observation ultrasonic image's evaluation method, device, equipment and storage medium.The present application first converts eye movement video stream into eye movement heat map video stream, to obtain the eye observation position of trainee, then eye movement heat map video stream and the ultrasonic video stream are fused, obtain fusion video stream, fusion video stream records the position of each frame ultrasonic image observed on ultrasonic video stream by trainee, therefore through fusion video stream, whether the position observed by trainee ultrasonic image can be accurately judged correct, to obtain the evaluation result of trainee.From the above analysis, the present application can objectively judge whether the position observed by trainee ultrasonic image is correct by generating fusion video stream, to improve the accuracy of evaluation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical teaching assistance, in particular to an observation ultrasound image evaluation method and device, equipment and storage medium. BACKGROUND

[0002] Medical ultrasound has important clinical examination value, doctors can make preliminary diagnosis on patients by observing ultrasound images of patients. However, different doctors observe ultrasound images in different ways, which will lead to different observation and diagnosis results, so it is necessary to evaluate the method or way of doctors observing ultrasound images to determine whether the way or method of doctors observing ultrasound images is correct. The prior art mainly judges whether the way or method of doctors observing ultrasound images is correct by experienced experts to obtain the evaluation result of the doctor, and the evaluation result obtained by the prior art has a certain subjectivity, thereby resulting in low accuracy of the evaluation result.

[0003] In summary, the prior art has low accuracy of the evaluation result of observing ultrasound images.

[0004] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0005] To solve the above technical problems, the present application provides an observation ultrasound image evaluation method, device, equipment and storage medium, which solves the problem of low accuracy of the evaluation result of observing ultrasound images in the prior art.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides an observation ultrasound image evaluation method, comprising:

[0008] displaying an ultrasound video stream, collecting an eye movement video stream of a trainee observing the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, the eye movement video stream being used to record the eye movement trajectory of the trainee observing the ultrasound video stream, and the eye movement heat map video stream being used to represent the observation position of the trainee;

[0009] generating a fusion video stream according to the eye movement heat map video stream and the ultrasound video stream, the fusion video stream being used to mark the position of the trainee observing the ultrasound image;

[0010] obtaining the evaluation result of the trainee according to the fusion video stream.

[0011] In an implementation mode, the method of displaying an ultrasound video stream and collecting an eye movement video stream of a trainee observing the ultrasound video stream comprises:

[0012] Two independent processes are used to display the ultrasound video stream and collect the eye movement video stream respectively.

[0013] In one implementation, generating a fused video stream based on the eye movement heat map video stream and the ultrasound video stream includes:

[0014] Determine each frame of thermal image in the eye movement thermogram video stream, and determine all frames of ultrasonic image in the ultrasonic video stream, wherein all frames of ultrasonic image are the same image;

[0015] Adjusting the size of each frame of the thermal image and adjusting the ultrasound image corresponding to the frame of the thermal image so that the thermal image and the ultrasound image corresponding to the frame of the thermal image have the same size;

[0016] A fused image is generated based on the thermal image and the ultrasonic image of the same size, and a fused video stream is generated by combining all frames of the fused image.

[0017] In one implementation, generating a fused image based on the thermal image and the ultrasound image of the same size includes:

[0018] Binarizing the thermal image to obtain a binary thermal image;

[0019] Determine a first pixel point position with a pixel value of 255 and a second pixel point position with a pixel value of 0 on the thermal binary image;

[0020] Determining a target ultrasound pixel value at a position on the ultrasound image corresponding to the first pixel position, and determining a target thermal pixel value at a position on the thermal image corresponding to the second pixel position;

[0021] A fusion image is generated according to the target ultrasound pixel value and the target thermal pixel value.

[0022] In one implementation, the pixel points with a pixel value of 255 on the thermal binary image are recorded as non-visible pixel points, and the non-visible pixel points correspond to the pixel points on the thermal image whose pixel values ​​are less than a threshold; the pixel points with a pixel value of 0 on the thermal binary image are recorded as visible pixel points, and the visible pixel points correspond to the pixel points on the thermal image whose pixel values ​​are greater than or equal to the threshold.

[0023] In one implementation, obtaining the evaluation result of the trainee based on the fused video stream includes:

[0024] The annotated video stream is displayed, where the annotated video stream is used to mark the position of the lesion area that needs to be observed on the ultrasound image.

[0025] Applying a graph wheel algorithm to the labeled video stream and the fused video stream to obtain an observation sequence similarity, and using the observation sequence similarity as an evaluation result;

[0026] Alternatively, determining the continuous same-frame fused images and the first synthesized image and the fused images of each frame in the fused video stream;

[0027] Counting the total number of image frames included in the continuous same-frame fused images, and obtaining the time duration for the trainee to observe the lesion area corresponding to the continuous same-frame fused images based on the total number of image frames;

[0028] Determining a first lesion region included in the first composite image;

[0029] Determining each observed lesion area corresponding to each frame of the fused image;

[0030] An evaluation result of the trainee is obtained based on the duration, the first lesion area, and each of the observed lesion areas.

[0031] In one implementation, the method further includes:

[0032] Determining the eye movement thermal image of each frame in the eye movement thermal map video stream;

[0033] Obtaining an observation order of the trainee observing each lesion area on the ultrasound image based on each frame of the eye movement thermal image, wherein the ultrasound image is each frame of the ultrasound image included in the ultrasound video stream, and each frame of the ultrasound image is the same image;

[0034] Obtaining a benchmark observation sequence, and obtaining the accuracy of the observation method adopted by the trainee based on the observation sequence and the benchmark observation sequence;

[0035] A final evaluation result is obtained based on the accuracy and the evaluation result.

[0036] In a second aspect, an embodiment of the present invention further provides an evaluation device for observing ultrasound images, wherein the device includes the following components:

[0037] a video stream acquisition module, configured to display an ultrasound video stream, acquire an eye movement video stream of a trainee observing the ultrasound video stream, and convert the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the trainee's observation position;

[0038] a fusion module, configured to generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark a position where the trainee observes the ultrasound image;

[0039] An evaluation module is used to obtain an evaluation result of the trainee based on the fused video stream.

[0040] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device includes a memory, a processor, and an evaluation program for observing ultrasonic images stored in the memory and runnable on the processor, and when the processor executes the evaluation program for observing ultrasonic images, the steps of the above-mentioned evaluation method for observing ultrasonic images are implemented.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which an evaluation program for observing ultrasound images is stored. When the evaluation program for observing ultrasound images is executed by a processor, the steps of the above-mentioned evaluation method for observing ultrasound images are implemented.

[0042] Beneficial Effects: The present invention first converts an eye movement video stream into an eye movement heat map video stream to obtain the trainee's eye observation position, and then fuses the eye movement heat map video stream with the ultrasound video stream to obtain a fused video stream. The fused video stream records the position of each frame of the ultrasound image observed by the trainee on the ultrasound video stream. Therefore, the fused video stream can accurately determine whether the trainee's observation position of the ultrasound image is correct, thereby obtaining the trainee's evaluation results. From the above analysis, the present invention can objectively determine whether the trainee's observation position of the ultrasound image is correct by generating a fused video stream, thereby improving the accuracy of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall flow chart of the present invention;

[0044] Figure 2 Schematic diagram of a display interface in an embodiment of the present invention;

[0045] Figure 3 This is a design flow chart of the diagnostic training platform in an embodiment of the present invention;

[0046] Figure 4 This is the real-time monitoring system interface in the embodiment of the present invention;

[0047] Figure 5 This is a structural diagram of scoring indicators in an embodiment of the present invention;

[0048] Figure 6 Schematic diagram of the observation sequence in an embodiment of the present invention;

[0049] Figure 7 A structure diagram of an evaluation device for observing an ultrasound image is provided for the present application.

[0050] Figure 8 An internal structure principle block diagram of a terminal device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the present application are described clearly and completely in combination with the embodiments and the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0052] It is found through research that medical ultrasound has important clinical examination value, and doctors can preliminarily diagnose patients by observing ultrasound images of the patients. However, different observation methods of ultrasound images by different doctors will lead to different observation and diagnosis results, and therefore, the observation method or way of ultrasound images by doctors needs to be evaluated to determine whether the observation way or method of ultrasound images by doctors is correct. The prior art mainly determines whether the observation way or method of ultrasound images by doctors is correct through experienced experts to obtain the evaluation result of the doctor, and the evaluation result obtained by the prior art has certain subjectivity, thereby leading to low accuracy of the evaluation result.

[0053] To solve the above technical problems, the present application provides an evaluation method, device and equipment for observing ultrasound images and a storage medium, which solves the problem of low accuracy of the evaluation result of observing ultrasound images in the prior art. In specific implementation, first, an ultrasound video stream is displayed, and an eye movement video stream of an observer when observing the ultrasound video stream is collected, and the eye movement video stream is converted into an eye movement heat map video stream, the eye movement video stream is used to record the eye movement track of the observer when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the observation position of the observer; then, a fusion video stream is generated according to the eye movement heat map video stream and the ultrasound video stream, the fusion video stream is used to represent the position of each frame of ultrasound image observed by the observer; finally, the evaluation result of the observer is obtained according to the fusion video stream.

[0054] For example, if the ultrasound video stream is composed of ten ultrasound images (the ten ultrasound images are the same ultrasound image), the ten ultrasound images are displayed in turn, and the observer is prompted to observe each ultrasound image, and the eye image of the observer is collected at the same time. The ten ultrasound images correspond to ten eye images, and the ten eye images are sorted according to the collection time to form an eye movement video stream. The eye movement video stream records the eye movement trajectory, which represents the position of the observer on each ultrasound image, that is, whether the observer observes the position of the real lesion area on the ultrasound image. The eye movement video stream is converted into an eye movement heat map video stream, and the eye movement heat map video stream also contains ten heat map images. Since the first ultrasound image and the first heat map image have the same size, a new image (the first fusion image) is derived from the first ultrasound image and the first heat map image. The first fusion image records the position of the observer observing the first ultrasound image. In the same way, the second fusion image is derived from the second ultrasound image and the second heat map image, and the third fusion image is derived from the third ultrasound image and the third heat map image. In turn, the third fusion image, the fourth fusion image, the fifth fusion image, the sixth fusion image, the seventh fusion image, the eighth fusion image, the ninth fusion image, and the tenth fusion image are derived. The ten fusion images form a fusion video stream, and the position of the observer can be determined by the ten fusion images to obtain the evaluation result of the observer.

[0055] The evaluation method for observing ultrasound images in the embodiment can be applied to a terminal device, which can be a terminal product with a video playing function, such as a computer. Figure 1 The evaluation method for observing ultrasound images in the embodiment specifically includes the following steps:

[0056] S100, display an ultrasound video stream, collect an eye movement video stream of an observer observing the ultrasound video stream, and convert the eye movement video stream into an eye movement heat map video stream. The eye movement video stream is used to record the eye movement trajectory of the observer observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the observation position of the observer.

[0057] The conversion of the eye movement video stream into the eye movement heat map video stream is a prior art.

[0058] S200, generate a fusion video stream according to the eye movement heat map video stream and the ultrasound video stream. The fusion video stream is used to mark the position of the observer observing the ultrasound image.

[0059] S300: Obtaining evaluation results of the trainee based on the fused video stream.

[0060] The invention is based on the evaluation method of S100 to S300 and is applied to the diagnosis training platform, which includes a real-time monitoring module, a user evaluation module, and a scoring interface module. The user evaluation module is used to Figure 2 The ultrasound images shown are loaded onto the platform, and the progress of the real-time monitoring module is controlled by the user evaluation module. The specific implementation of the user evaluation module is as follows: by giving three pictures containing AC, NT, etc. (such as Figure 3 The path of the folders (NT and AC) shown in the figure is obtained by traversing each picture in the contents of each folder, and storing the image pixels of each picture in each element in the list in the form of a matrix. When it needs to be displayed, it can be called directly in the list. By setting a button click slot function and accepting the name of the click event object, such as clicking the nt button, the slot function will obtain the information that the button was clicked and that the button clicked was the nt button, so the list containing the information of the nt pictures is displayed on the label as the current main list, and the user can watch it and click the switch button to add or subtract one to the current picture index to switch to the next picture and switch to the previous picture. By setting a full-screen hidden label, when the user clicks watching, the ultrasound picture will be displayed in full screen to achieve full-screen viewing. The user's eye movement video and the eye movement video of the expert library are input through the graph theory algorithm for comparison to obtain the analysis results.

[0061] The real-time monitoring module is used to display the ultrasound video stream formed by the loaded ultrasound image. The real-time monitoring module is also used to display the eye movement heat map video stream and the fused video stream, as well as the annotated video stream used as a reference. The scoring interface module is used to obtain the evaluation results in step S300.

[0062] In the first embodiment, step S100 of displaying the ultrasound video stream and collecting the eye movement video stream of the trainee observing the ultrasound video stream specifically includes: using two independent processes to display the ultrasound video stream and collect the eye movement video stream respectively.

[0063] This embodiment not only independently displays the ultrasound video stream and the eye movement video stream, but also independently displays the annotated video stream and the fused video stream. Specifically, four processes are used to independently collect the ultrasound video stream, the eye movement video stream, the annotated video stream, and the fused video stream, and these four processes are used to independently display these four video streams. Two event signals control the simultaneous start and end of these four processes, ensuring that the four video streams are sampled and displayed simultaneously within a virtually error-free range. Images of various organ parts (annotated video streams), pre-annotated by the detection model, are displayed simultaneously on the interface at a rate of 30 frames per second, allowing evaluators (i.e., doctors experienced in viewing ultrasound images) to use this annotated video stream as a benchmark to assess the accuracy of the user's viewing of the ultrasound image.

[0064] In this embodiment, while the display interface displays the ultrasound video stream, eye movement video stream, annotated video stream, and fused video stream, the display is recorded to generate a fifth video stream, the screen recording video stream. When the user completes the training, these five video streams are saved to the current software directory, providing raw eye movement data for scoring, analysis, and review. Similarly, the five video stream data sources sampled by ffmpeg are sent to a separate video saving process, where five video writers simultaneously write image data at 30 frames per second, ultimately generating five videos.

[0065] This embodiment uses ffmpeg to drive the capture card and screen capture, samples at a sampling rate of thirty frames per second, and then displays it on the interface.

[0066] The display interface of this embodiment includes an Eyetracking display area, an Ultrasonic display area, a Synthesis display area, and a Rectangle display area.

[0067] The Eyetracking display area uses ffmpeg to drive the capture card to read and display the user's eye movement data in real time. The specific implementation operation is to first define the ffmpeg command line in Python and then set the parameters of this ffmpeg command line, namely self.ffmpeg_command1 = ['ffmpeg','-f','dshow','-i','video=USB VideoDevice','-s','1088x612','-r','30','-pix_fmt','rgb24','-f','rawvideo','pipe:1'], and then use the Popen function in a subprocess class to use this code to drive the capture card to read the entire screen data of another computer at 30 frames per second. The screen of the other computer will display the current user's eye movement heat map in real time (TobiiEyetracking comes with software for real-time display of eye movement heat maps).

[0068] The Ultrasonic display area uses ffmpeg screen recording to read and display the ultrasound image currently viewed by the doctor in real time.

[0069] The Synthesis display area aligns and integrates eye movement data and ultrasound images in time and space to generate a fused video stream.

[0070] The Rectangle display area displays the labeled pictures in real time (several frames of labeled pictures constitute a labeled video stream), which is used to monitor the ultrasound image area that the user is viewing in real time, that is, to monitor whether the ultrasound image area observed by the user is on the area of ​​the labeled picture, where the labeled picture serves as the reference picture.

[0071] like Figure 4 As shown, the real-time data display of the Eyetracking, Ultrasonic, Synthesis, and Rectangle display areas on the display interface is executed simultaneously by four processes. A dedicated background recording process simultaneously records the contents of these four display areas and the entire interface, generating five independent video files, providing a comprehensive, multi-angle record of the entire monitoring process. The Start Recording and Stop Recording buttons below the display interface respond to signals from the user evaluation interface to start and end recording, ensuring that all important information is fully recorded and prepared for subsequent data analysis and feedback.

[0072] Example 2 is based on Example 1. In this example, step S200 includes the following specific steps S201 to S206:

[0073] S201, determining each frame of thermal image in the eye movement thermogram video stream, and determining all frames of ultrasonic image in the ultrasonic video stream, wherein all frames of ultrasonic image are the same image;

[0074] S202 , adjusting the size of each frame of the thermal image and adjusting the ultrasound image corresponding to the frame of the thermal image so that the thermal image and the ultrasound image corresponding to the frame of the thermal image have the same size.

[0075] S203 , binarizing the thermal image to obtain a binary thermal image.

[0076] S204 , determining a first pixel point position with a pixel value of 255 and a second pixel point position with a pixel value of 0 on the thermodynamic binary image.

[0077] S205 , determining a target ultrasound pixel value at a position corresponding to the first pixel point on the ultrasound image, and determining a target thermal pixel value at a position corresponding to the second pixel point on the thermal image.

[0078] S206: Generate a fusion image according to the target ultrasound pixel value and the target thermal pixel value.

[0079] The specific implementation process from S203 to S206 is as follows:

[0080] By using the inRange function (image threshold function) of the cv library to generate the corresponding mask, that is, all the pixels in the range of (0, 0, 0)-(30, 30, 30) in image 1 are assigned a value of 255, and the rest are 0. Then, the where function of the np library is used, that is, np.where(mask[:,:,None].astype(bool),rgb_frame2,rgb_frame1), that is, the area with a pixel value of 255 in the mask will be retained as the pixels corresponding to rgb_fram2 (ultrasound image), and the rest will be pixels of rgb_frame1 (thermal image), thereby achieving synthesis.

[0081] Wherein, 30 is the threshold value, and the pixel points with a pixel value of 255 on the thermal binary image are recorded as invisible pixel points, and the invisible pixel points correspond to the pixel points on the thermal image with a pixel value less than the threshold value; the pixel points with a pixel value of 0 on the thermal binary image are recorded as visible pixel points, and the visible pixel points correspond to the pixel points on the thermal image with a pixel value greater than or equal to the threshold value.

[0082] That is, the pixel points in the thermal image with a value of 255 in the range of (0, 0, 0)-(30, 30, 30) are assigned a value of 255, that is, the area with a pixel value of 255 is the area not seen by the trainee, and the pixel points with a pixel value in other ranges are assigned a value of 0, that is, the area with a pixel value of 0 is the area seen by the trainee, so as to convert the thermal image into a thermal binary image. A third blank image with the same size as the thermal binary image and the ultrasound image is established, and a first position area corresponding to a pixel point with a pixel value of 255 in the thermal binary image is found on the blank image, and the first position area of ​​the blank image is assigned the pixel value of the ultrasound image. A second position area corresponding to a pixel point with a pixel value of 0 in the thermal binary image is found on the blank image, and the second position area of ​​the blank image is assigned a value of 0 (0 is the pixel value of the thermal binary image in this position area). Since the pixel values ​​in the range (0, 0, 0)-(30, 30, 30) are black pixels, that is, the pixel values ​​in this range are located in the image where the trainee's eyes do not see, while the pixel values ​​in other ranges are located where the trainee's eyes see. Therefore, the other pixel values ​​in the thermal binary image are retained to indicate the locations observed by the trainee.

[0083] For example, the size of the thermal image and the ultrasound image are both AxA, the pixel value of the upper left corner area of ​​the thermal image (that is, the target thermal pixel value) is (r1, g1, b1), and the pixel values ​​of the other areas of the thermal image are (r1′, g1′, b1′), and (r1, g1, b1) are within the range of (0, 0, 0)-(30, 30, 30). Therefore, the pixel values ​​of the upper left corner area of ​​the thermal image are all set to 255, and the pixel values ​​of the other areas of the thermal image are set to 0 to convert the thermal image into a thermal binary image. Create a third blank image, the size of which is also AxA. If the pixel value of the upper left corner area of ​​the ultrasound image (that is, the target ultrasound pixel value) is (r2, g2, b2), then the pixel value of the upper left corner area of ​​the third blank image is set to (r2, g2, b2), and the pixel values ​​of other areas of the third blank image are set to (r1′, g1′, b1′) to generate a fused image. The other areas on the fused image are the positions on the ultrasound image observed by the trainee mapped to the positions on the fused image. Therefore, the fused image can be used to know which position of the ultrasound image the trainee is observing, so as to determine whether the observation position is correct.

[0084] Example 3, based on Example 1 or Example 2, in this example, step S300 includes the following specific steps: displaying a labeled video stream, the labeled video stream being used to mark the location of the lesion area to be observed on the ultrasound image; applying a graph wheel algorithm to the labeled video stream and the fused video stream to obtain an observation sequence similarity, and using the observation sequence similarity as an evaluation result.

[0085] For example, the annotated video stream includes five frames of annotated images, and the annotated positions (the positions that need to be viewed and observed) are positions s11, s12, s13, s14, and s15 on the image. The order of these five positions constitutes the baseline observation order. The fused video stream also includes five frames of fused images. The positions on the fused images that represent the positions observed by the trainees are positions s21, s22, s23, s24, and s25. The order of these five positions constitutes the experimental observation order. The graph wheel algorithm is applied to positions s11, s12, s13, s14, s15, s21, s22, s23, s24, and s25 to obtain the observation order similarity between the baseline observation order and the experimental observation order. Among them, Figure 5 As shown in the figure, the graph wheel algorithm includes weighted edge set similarity, maximum weight matching similarity, node transition probability-based similarity, and shared node similarity.

[0086] The above-mentioned eye movement data to be trained (that is, the fused video stream) is compared with the eye movement data of the expert library (that is, the labeled video stream), so as to give objective and accurate feedback scores to guide user learning.

[0087] In this embodiment, step S300 may also include the following specific steps: determining the continuous same-frame fused images and the first synthetic image and each frame fused image in the fused video stream; counting the total number of image frames contained in the continuous same-frame fused images, and based on the total number of image frames, obtaining the time duration for the trainee to observe the lesion area corresponding to the continuous same-frame fused images; determining the first lesion area contained in the first synthetic image; determining each observed lesion area corresponding to each frame of the fused image; and obtaining the evaluation result of the trainee based on the time duration and the first lesion area and each observed lesion area.

[0088] The consecutive identical-frame fused images represent fused images generated by observing the same lesion region on consecutive ultrasound image frames, and the total number of frames in the fused images represents the time the trainee observed the lesion region. This embodiment calculates the total time the trainee observed all lesion regions, and combines the time, total time, and the first lesion region to determine whether the trainee's method or approach for observing the ultrasound image was correct, thereby obtaining an evaluation result for the trainee.

[0089] In this embodiment, the similarity between the observed organ and the reference can also be determined to obtain an evaluation result.

[0090] Embodiment 4, based on embodiment 1, embodiment 2, or embodiment 3, includes the following specific steps S401 to S404:

[0091] S401, determining each frame of eye movement heat image in the eye movement heat map video stream.

[0092] The pixel position corresponding to the pixel value greater than the threshold (the threshold value is 30 for the three RGB channels) on the eye movement thermal image is the observation position of the trainee on the image. Since the eye movement thermal image and the ultrasound image have the same size, the observation position represents the lesion area on the ultrasound image observed by the trainee.

[0093] S402, based on the eye movement thermal image of each frame, obtain the observation order of the trainee observing each lesion area on the ultrasound image, the ultrasound image is each frame of ultrasound image included in the ultrasound video stream, and each frame of ultrasound image is the same image.

[0094] S403: Obtain a reference observation sequence, and obtain the accuracy of the observation method adopted by the trainee based on the observation sequence and the reference observation sequence.

[0095] Figure 6 The numbers on the figure a represent the reference observation order formed by the positions that should be observed on the ultrasound image. Figure 6 The numbers in Figure b represent the observation order formed by the observed positions on the ultrasound image. By comparing the observation order with the reference order, it can be determined whether the trainee's observation method is correct.

[0096] S404: Obtain a final evaluation result based on the accuracy and the evaluation result.

[0097] The accuracy and evaluation results are weighted to obtain the final evaluation results.

[0098] like Figure 6 As shown, the present invention uses the similarity of observed organs, ranking of attention concentration (similarity of the combined duration of observation of each organ), correctness of the starting organ, decision speed, and observation order as evaluation dimensions for image reading behavior. The observation order dimension uses Markov chains for sequence modeling, and graph theory algorithms are used to evaluate the image reading order based on node and edge similarity. Furthermore, eye movement data from ultrasound physicians is used to analyze the image reading behavior of physicians of different years of experience.

[0099] The present application first works in the video reading and initialization module, which is specially used for processing the black screen eye movement video data obtained by the monitoring system. This stage involves strict video synchronization and frame alignment mechanism, which ensures the spatial correspondence of each frame image and ultrasound image, and realizes the consistency of spatial dimension through accurate size matching. Then, a series of image preprocessing steps are performed, including but not limited to noise reduction, grayscale and contrast enhancement, aiming to optimize the image quality and provide clear basic data for subsequent change detection. The core of the algorithm is to use an efficient inter-frame difference analysis method to effectively identify the change area of visual attention points over time by calculating the absolute pixel difference of adjacent frames. On this basis, the algorithm integrates an advanced contour detection algorithm to identify the significant contour in the difference area and determine the most important change point in each frame, i.e. the focus of the doctor's gaze through geometric center positioning. This strategy not only improves the sensitivity and accuracy of change detection, but also provides reliable positioning information for continuous tracking of eye movement trajectories. By continuously tracking the above change points, the algorithm constructs an eye movement trajectory sequence model of the doctor during the observation of ultrasound images. This process not only reveals the dynamic distribution of visual attention, but also generates a detailed user organ observation sequence by integrating all change point information, which intuitively shows the attention degree of the doctor to each part of the ultrasound image. It has important value for understanding the image interpretation strategy of the doctor, evaluating the education and training effect and guiding the optimization of ultrasound image interface.

[0100] In summary, the present application innovatively integrates eye tracking technology, graph algorithm, and multi-process technology to construct an eye tracking training platform. These two technologies can be used to score two pre-recorded eye movement data separately using graph algorithm and eye tracking and multi-process collection using deep learning, without using graph algorithm. This method may avoid the design scheme. The alternative scheme of the core technology may include: in data collection; other language screen recording software may be used instead of the method of calling ffmpeg library by python script to realize the recording of five video streams at the same time. Then the ultrasound images already labeled may be replaced by ultrasound images of other organ parts already labeled. The similarity of eye movement data graph and expert graph may be evaluated using graph algorithm, and competitors may use deep learning method to evaluate the similarity of two images.

[0101] The present embodiment also provides an evaluation device for observing ultrasound images, as shown in Figure 7 The device comprises the following components:

[0102] Video stream acquisition module 01 is used to display the ultrasound video stream, collect the trainee's eye movement video stream when observing the ultrasound video stream, and convert the eye movement video stream into an eye movement heat map video stream. The eye movement video stream is used to record the trainee's eye movement trajectory when observing the ultrasound video stream. The eye movement heat map video stream is used to represent the trainee's observation position;

[0103] A fusion module 02 is configured to generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image;

[0104] The evaluation module 03 is used to obtain the evaluation result of the trainee based on the fused video stream.

[0105] Based on the above embodiment, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Figure 8 As shown. The terminal device includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for observing and evaluating ultrasound images is implemented. The display screen of the terminal device can be a liquid crystal display or an electronic ink display.

[0106] Those skilled in the art will understand that Figure 8 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0107] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and an evaluation program for observing ultrasound images stored in the memory and executable on the processor. When the processor executes the evaluation program for observing ultrasound images, the following operating instructions are implemented:

[0108] displaying an ultrasound video stream, collecting an eye movement video stream of a trainee when observing the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the trainee's observation position;

[0109] generating a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image;

[0110] An evaluation result of the trainee is obtained based on the fused video stream.

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for observing and evaluating ultrasound images, characterized in that: include: displaying an ultrasound video stream, collecting an eye movement video stream of a trainee when observing the ultrasound video stream, and converting the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the trainee's observation position; generating a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark the position where the trainee observes the ultrasound image; Obtaining an evaluation result of the trainee based on the fused video stream; The generating of a fused video stream based on the eye movement heat map video stream and the ultrasound video stream includes: Determine each frame of thermal image in the eye movement thermogram video stream, and determine all frames of ultrasonic image in the ultrasonic video stream, wherein all frames of ultrasonic image are the same image; Adjusting the size of each frame of the thermal image and adjusting the ultrasound image corresponding to the frame of the thermal image so that the thermal image and the ultrasound image corresponding to the frame of the thermal image have the same size; Generate a fused image based on the thermal image and the ultrasonic image of the same size, and generate a fused video stream from all frames of the fused image; Generating a fused image based on the thermal image and the ultrasound image of the same size includes: Binarizing the thermal image to obtain a binary thermal image; Determine a first pixel point position with a pixel value of 255 and a second pixel point position with a pixel value of 0 on the thermal binary image; Determining a target ultrasound pixel value at a position on the ultrasound image corresponding to the first pixel position, and determining a target thermal pixel value at a position on the thermal image corresponding to the second pixel position; generating a fused image according to the target ultrasound pixel value and the target thermal pixel value; The pixel points with a pixel value of 255 on the thermal binary image are recorded as invisible pixel points, and the invisible pixel points correspond to the pixel points on the thermal image whose pixel values ​​are less than a threshold value; the pixel points with a pixel value of 0 on the thermal binary image are recorded as visible pixel points, and the visible pixel points correspond to the pixel points on the thermal image whose pixel values ​​are greater than or equal to the threshold value.

2. The method for evaluating ultrasound images according to claim 1, wherein: The displaying of the ultrasound video stream and collecting the eye movement video stream of the trainee when observing the ultrasound video stream includes: Two independent processes are used to display the ultrasound video stream and collect the eye movement video stream respectively.

3. The method for evaluating ultrasound images according to claim 1, wherein: Obtaining the evaluation result of the trainee based on the fused video stream includes: Displaying a marked video stream, wherein the marked video stream is used to mark the position of the lesion area that needs to be observed on the ultrasound image; Applying a graph theory algorithm to the labeled video stream and the fused video stream to obtain an observation sequence similarity, and using the observation sequence similarity as an evaluation result; Alternatively, determining the continuous same-frame fused images and the first synthesized image and the fused images of each frame in the fused video stream; Counting the total number of image frames included in the continuous same-frame fused images, and obtaining the time duration for the trainee to observe the lesion area corresponding to the continuous same-frame fused images based on the total number of image frames; Determining a first lesion region included in the first composite image; Determining each observed lesion area corresponding to each frame of the fused image; An evaluation result of the trainee is obtained based on the duration, the first lesion area, and each of the observed lesion areas.

4. The method for evaluating ultrasound images according to claim 1, wherein: Also includes: Determining the eye movement thermal image of each frame in the eye movement thermal map video stream; Obtaining an observation order of the trainee observing each lesion area on the ultrasound image based on each frame of the eye movement thermal image, wherein the ultrasound image is each frame of the ultrasound image included in the ultrasound video stream, and each frame of the ultrasound image is the same image; Obtaining a benchmark observation sequence, and obtaining the accuracy of the observation method adopted by the trainee based on the observation sequence and the benchmark observation sequence; A final evaluation result is obtained based on the accuracy and the evaluation result.

5. An evaluation device for observing ultrasonic images, characterized in that: The device comprises the following components: a video stream acquisition module, configured to display an ultrasound video stream, acquire an eye movement video stream of a trainee observing the ultrasound video stream, and convert the eye movement video stream into an eye movement heat map video stream, wherein the eye movement video stream is used to record the eye movement trajectory of the trainee when observing the ultrasound video stream, and the eye movement heat map video stream is used to represent the trainee's observation position; a fusion module, configured to generate a fused video stream based on the eye movement heat map video stream and the ultrasound video stream, wherein the fused video stream is used to mark a position where the trainee observes the ultrasound image; An evaluation module, configured to obtain an evaluation result of the trainee based on the fused video stream; The generating of a fused video stream based on the eye movement heat map video stream and the ultrasound video stream includes: Determine each frame of thermal image in the eye movement thermogram video stream, and determine all frames of ultrasonic image in the ultrasonic video stream, wherein all frames of ultrasonic image are the same image; Adjusting the size of each frame of the thermal image and adjusting the ultrasound image corresponding to the frame of the thermal image so that the thermal image and the ultrasound image corresponding to the frame of the thermal image have the same size; Generate a fused image based on the thermal image and the ultrasonic image of the same size, and generate a fused video stream from all frames of the fused image; Generating a fused image based on the thermal image and the ultrasound image of the same size includes: Binarizing the thermal image to obtain a binary thermal image; Determine a first pixel point position with a pixel value of 255 and a second pixel point position with a pixel value of 0 on the thermal binary image; Determining a target ultrasound pixel value at a position on the ultrasound image corresponding to the first pixel position, and determining a target thermal pixel value at a position on the thermal image corresponding to the second pixel position; generating a fused image according to the target ultrasound pixel value and the target thermal pixel value; The pixel points with a pixel value of 255 on the thermal binary image are recorded as invisible pixel points, and the invisible pixel points correspond to the pixel points on the thermal image whose pixel values ​​are less than a threshold value; the pixel points with a pixel value of 0 on the thermal binary image are recorded as visible pixel points, and the visible pixel points correspond to the pixel points on the thermal image whose pixel values ​​are greater than or equal to the threshold value.

6. A terminal device, characterized in that: The terminal device includes a memory, a processor, and an evaluation program for observing ultrasonic images stored in the memory and executable on the processor. When the processor executes the evaluation program for observing ultrasonic images, the steps of the method for observing ultrasonic images as described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an evaluation program for observing ultrasonic images. When the evaluation program for observing ultrasonic images is executed by a processor, the steps of the evaluation method for observing ultrasonic images according to any one of claims 1 to 4 are implemented.

Images