Remote ultrasound image annotation method, terminal device and storage medium

Abstract

The present invention relates to the field of image processing technology, and in particular to a remote ultrasound image annotation method, terminal device and storage medium, which are used to solve the problem of how to improve the work efficiency and diagnostic accuracy of remote consultation. The present application performs coarse positioning after annotating the first ultrasound image to obtain a coarse contour area with a preset shape, and displays the coarse contour area synchronously with the first terminal device through the server. The first terminal device performs a fine edge extraction operation on the target object in the coarse contour area in the second ultrasound image, and sends the second ultrasound image and the obtained fine contour of the target object to the first terminal device to synchronize the fine contour of the target object. Since the fine contour is displayed synchronously in the first terminal device and the second terminal device, and there is no need for user voice communication to determine the fine contour, the work efficiency and diagnostic accuracy of the remote consultation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of ultrasonic image processing, and in particular to a remote ultrasonic image annotation method, terminal equipment, and storage medium. Background Art

[0002] During a remote consultation, expert doctors will instruct primary care physicians to focus on areas of interest, such as organs to be examined and lesions to be discovered. If the expert doctor is on a remote terminal device and the primary care physician is on a local terminal device, they can only provide instructions through voice, which reduces work efficiency and affects diagnostic accuracy. Therefore, improving the efficiency and diagnostic accuracy of remote consultations is an urgent issue. Summary of the Invention

[0003] The present application discloses a remote ultrasound image annotation method, terminal device and storage medium, which are used to solve the problem of how to improve the working efficiency and diagnosis accuracy of remote consultation.

[0004] In a first aspect, the present application proposes a remote ultrasound image annotation method, the method comprising:

[0005] displaying a first ultrasound image;

[0006] In response to a labeling operation on a target object in the first ultrasound image, coarsely locating the target object to obtain a coarse contour region of the target object, wherein the coarse contour region has a preset shape;

[0007] Sending the rough outline area to a server, so that the server sends the rough outline area to the first terminal device;

[0008] receiving a fine contour of the target object and a second ultrasound image sent by the first terminal device; the fine contour is obtained by the first terminal device performing a fine edge extraction operation on the target object in the coarse contour area of ​​the second ultrasound image;

[0009] The fine outline of the target object is marked and displayed in the second ultrasound image.

[0010] In one embodiment, in response to the labeling operation on the target object in the first ultrasound image, roughly locating the target object to obtain a rough contour area of ​​the target object includes:

[0011] Based on the operation position of the marking operation, the basic outline of the target object is determined by any one of the following methods:

[0012] Watershed algorithm, flood filling algorithm, image segmentation method based on edge detection, image segmentation method based on morphological operation;

[0013] A minimum rectangular area surrounding the basic outline is found as the rough outline area of ​​the target object.

[0014] In one embodiment, it further includes:

[0015] receiving a marking color for the fine outline sent by the first terminal device;

[0016] The marking of a fine outline of the target object in the second ultrasound image includes:

[0017] The fine outline of the target object is marked in the second ultrasound image using the marking color.

[0018] In one embodiment, displaying the second ultrasound image with the fine outline of the target object marked includes:

[0019] converting the position coordinates of the fine contour in the second ultrasound image into display coordinates required for display, and adapting the resolution of the second ultrasound image to the resolution required for display;

[0020] The second ultrasound image marked with a fine outline of the target object is displayed based on the display coordinates and the resolution required for the display.

[0021] In a second aspect, the present application proposes a remote ultrasound image annotation method, the method comprising:

[0022] receiving a coarse outline region sent by the server, wherein the coarse outline region is determined by the second terminal device in response to a labeling operation on the target object in the first ultrasound image, and the coarse outline region has a preset shape;

[0023] performing a fine edge extraction operation on the target object in the coarse contour area of ​​the second ultrasound image to obtain a fine contour of the target object;

[0024] The second ultrasound image and the fine contour are synchronized to the second terminal device through the server.

[0025] In one embodiment, performing a fine edge extraction operation on the target object in the coarse contour area of ​​the second ultrasound image to obtain a fine contour of the target object includes:

[0026] In the coarse contour area of ​​the second ultrasound image, the fine contour of the target object is extracted by using any one of the following methods:

[0027] Level set segmentation algorithm, image segmentation model based on deep learning.

[0028] In one embodiment, the method further comprises:

[0029] The marking color of the fine outline is synchronized to the second terminal device.

[0030] A remote ultrasound image annotation system includes:

[0031] a second terminal device configured to display a first ultrasound image; in response to a marking operation on a target object in the first ultrasound image, roughly locate the target object to obtain a coarse contour region of the target object, wherein the coarse contour region has a preset shape; send the coarse contour region to a server; receive a fine contour of the target object and a second ultrasound image sent by the first terminal device; mark the fine contour of the target object in the second ultrasound image and display the result;

[0032] The first terminal device is configured to receive a coarse contour region sent by a server, wherein the coarse contour region is determined by the second terminal device in response to a labeling operation on a target object in a first ultrasound image, and the coarse contour region has a preset shape; perform a fine edge extraction operation on the target object within the coarse contour region of a second ultrasound image to obtain a fine contour of the target object; and synchronize the second ultrasound image and the fine contour to the second terminal device via the server;

[0033] The first terminal device or the second terminal device is connected to an ultrasound device and is used to control the ultrasound device to acquire ultrasound images.

[0034] In a third aspect, the present application proposes a terminal device, comprising:

[0035] processor;

[0036] a memory for storing instructions executable by the processor;

[0037] The processor is configured to execute the instructions to implement the method as described in any one of the first aspect and the second aspect.

[0038] In a fourth aspect, the present application proposes a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by a processor of a terminal device, enables the terminal device to execute any of the methods provided in the first and second aspects of the present application.

[0039] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any one of the methods provided in the first and second aspects of the present application.

[0040] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0041] The present application performs coarse positioning after annotating the first ultrasound image to obtain a coarse contour area with a preset shape. At this time, the coarse contour area is sent to the first terminal device through the server to achieve the effect of synchronously displaying the coarse contour area. The first terminal device performs a fine edge extraction operation on the target object in the coarse contour area in the second ultrasound image to obtain a fine contour for the target object, and sends the second ultrasound image and the fine contour of the target object to the first terminal device to synchronize the fine contour of the target object. Since only the first ultrasound image needs to be annotated, the coarse positioning and fine edge extraction operations are performed by the terminal device, thereby improving the efficiency and accuracy of annotating the ultrasound image. Since the fine contour is displayed synchronously in the first terminal device and the second terminal device, there is no need for the user to communicate by voice to confirm, thereby further improving the work efficiency of remote consultation and the accuracy of diagnosis.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0046] Figure 3 This is a flowchart of a remote ultrasound image annotation method provided in an embodiment of the present application;

[0047] Figure 4 This is a schematic diagram of the marking operation effect provided in the embodiment of this application;

[0048] Figure 5 This is a schematic diagram of a rough outline area provided in an embodiment of the present application;

[0049] Figure 6 This is a schematic diagram of the fine contour area provided in the embodiments of this application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0051] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0052] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0053] During a remote consultation, an expert doctor will instruct the primary care physician to focus on areas of interest, such as organs to be examined and lesions to be discovered. However, since the expert doctor is located on the first terminal device and the primary care physician is located on the second terminal device, they can only provide instructions through voice commands, which reduces efficiency and easily leads to unclear communication, affecting diagnostic accuracy. Therefore, improving the efficiency and diagnostic accuracy of remote consultations is an urgent problem that needs to be solved.

[0054] In view of this, the present application proposes a remote ultrasound image annotation method.

[0055] like Figure 1 FIG2 is a schematic diagram of an application scenario provided by an embodiment of the present application, including an ultrasound device 201, a local terminal device 202, a remote terminal device 204, and a server 203. The local terminal device 202 is used to control the ultrasound device 201 to acquire ultrasound images and send the ultrasound images to the remote terminal device 204 via the server 203. The remote terminal device 204 is used to receive and display the synchronized ultrasound images.

[0056] In an embodiment of the present application, either the local terminal device 202 or the remote terminal device 204 can be configured to perform a coarse segmentation of a region of interest in response to an annotation operation on an ultrasound image, and send the coarse segmentation result to the other terminal device via a server. Taking the remote terminal device 204 as an example, a user on the remote terminal device 204 can annotate the ultrasound image displayed thereon to obtain a coarse segmentation result of the region of interest, and then send the coarse segmentation result to the local terminal device 202 via the server 203. The local terminal device 202 then performs a fine segmentation of the region of interest in any collected ultrasound image based on the coarse segmentation result to obtain a fine contour of the region of interest, and synchronizes the fine contour and the corresponding ultrasound image to the remote terminal device 204 via the server 203, so that the terminal device can display the fine contour of the region of interest in the corresponding ultrasound image.

[0057] Therefore, in the embodiment of the present application, the user can simply mark the region of interest, and the rough segmentation of the region of interest is achieved through the user's marking operation. Then, fine segmentation is used to achieve fine segmentation of the contour of the region of interest for each frame of the ultrasound image, thereby achieving adaptive fine segmentation of the ultrasound image. The fine segmentation result is synchronized to the terminal device at the other end, which can ensure that the regions of interest displayed by the terminal devices on both sides are consistent, which is conducive to communication between medical staff on both sides of the terminal devices and improves work efficiency. In addition, the traditional method is to have the user draw a fine contour in the region of interest by himself or determine the fine contour through voice communication. The error is large, and the fine contours drawn by different users are different. The present application obtains the fine contour of the region of interest through the coarse positioning and fine edge extraction operations of the terminal device and displays it in real time. The accuracy of the fine contour calculated by the terminal device is higher than the result of the user's voice communication. Therefore, the present application can improve the accuracy of the calculated fine contour of the region of interest.

[0058] like Figure 2 FIG. 1 is a schematic structural diagram of a terminal device 100 .

[0059] The following embodiment is specifically described using the terminal device 100 as an example. It should be understood that Figure 2 The terminal device 100 shown is only an example, and the terminal device 100 may have more Figure 2 The more or less components shown in the figure can be combined with two or more components, or can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0060] Figure 2 FIG1 shows a block diagram of the hardware configuration of the terminal device 100 according to an exemplary embodiment. The structures of the first terminal device and the second terminal device in the embodiment of the present application are as follows: Figure 2 The structure shown. Figure 2 As shown, the terminal device 100 includes: a radio frequency (RF) circuit 110, a memory 120, a display unit 130, a camera 140, a sensor 150, an audio circuit 160, a wireless fidelity (Wi-Fi) module 170, a processor 180, a Bluetooth module 181, and a power supply 190 and other components.

[0061] RF circuit 110 can be used to receive and transmit signals during information transmission or calls. It can receive downlink data from the base station and pass it to processor 180 for processing; it can also send uplink data to the base station. Typically, RF circuits include but are not limited to antennas, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and other components.

[0062] The memory 120 can be used to store software programs and data. The processor 180 executes various functions and data processing of the terminal device 100 by running the software programs or data stored in the memory 120. The memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 120 stores the operating system that enables the terminal device 100 to run. In the present application, the memory 120 can store the operating system and various application programs, and may also store code for executing the remote ultrasound image annotation method described in the embodiments of the present application.

[0063] The display unit 130 may be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device 100. Specifically, the display unit 130 may include a touch screen 131 disposed on the front of the terminal device 100, which may collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0064] The display unit 130 can also be used to display information input by the user or provided to the user, as well as a graphical user interface (GUI) of various menus of the terminal 100. Specifically, the display unit 130 may include a display screen 132 disposed on the front of the terminal device 100. The display screen 132 may be configured in the form of a liquid crystal display, a light-emitting diode, etc. The display unit 130 can be used to display the various graphical user interfaces described in this application.

[0065] The touch screen 131 can be covered on the display screen 132, or the touch screen 131 and the display screen 132 can be integrated to realize the input and output functions of the terminal device 100. The integrated touch screen can be simply called a touch display screen. In this application, the display unit 130 can display applications and corresponding operation steps.

[0066] Camera 140 can be used to capture still images or video. The lens generates an optical image of an object and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to processor 180 for conversion into a digital image signal.

[0067] The terminal device 100 may further include at least one sensor 150, such as an acceleration sensor 151, a distance sensor 152, a fingerprint sensor 153, and a temperature sensor 154. The terminal device 100 may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0068] The audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the terminal device 100. The audio circuit 160 can transmit the electrical signal converted from the received audio data to the speaker 161, which is converted into a sound signal for output. The terminal device 100 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then outputs the audio data to the RF circuit 110 to be sent to, for example, another terminal, or outputs the audio data to the memory 120 for further processing. In this application, the microphone 162 can capture the user's voice.

[0069] Wi-Fi is a short-range wireless transmission technology. The terminal device 100 can help users send and receive emails, browse web pages, and access streaming media through the Wi-Fi module 170, which provides users with wireless broadband Internet access.

[0070] The processor 180 is the control center of the terminal device 100. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing software programs stored in the memory 120 and calling data stored in the memory 120, it performs various functions of the terminal device 100 and processes data. In some embodiments, the processor 180 may include one or more processing units; the processor 180 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 180. In this application, the processor 180 can run the operating system, application programs, user interface display and touch response, as well as the remote ultrasound image annotation method described in the embodiments of this application. In addition, the processor 180 is coupled to the display unit 130.

[0071] The Bluetooth module 181 is used to exchange information with other Bluetooth devices having a Bluetooth module through the Bluetooth protocol. For example, the terminal device 100 can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 181 to exchange data.

[0072] The terminal device 100 also includes a power supply 190 (e.g., a battery) that supplies power to various components. The power supply can be logically connected to the processor 180 via a power management system, thereby enabling the power management system to manage functions such as charging, discharging, and power consumption. The terminal device 100 may also be configured with a power button for turning the terminal on and off, as well as locking the screen.

[0073] by Figure 1 The illustrated application scenario introduces the remote ultrasound image annotation system of the present application, using a remote terminal device as a first terminal device and a local terminal device as a second terminal device as an example. The first terminal device is configured to control the ultrasound device to acquire ultrasound images, receive the first and second ultrasound images transmitted by the ultrasound device 201, and transmit the received first and second ultrasound images to the server 203 in the order in which they were received.

[0074] The server 203 is connected to the first terminal device and the second terminal device, and is configured to share the received first ultrasound image with the second terminal device.

[0075] The second terminal device is configured to display the first ultrasound image, and in response to a labeling operation on a target object in the first ultrasound image, roughly locate the target object, obtain a rough outline region of the target object, wherein the rough outline region has a preset shape, and send the rough outline region to the server 203.

[0076] The server 203 is further configured to send the received location information of the rough outline area to the first terminal device.

[0077] The first terminal device is further configured to receive a coarse contour area sent by the server, wherein the coarse contour area is determined by the second terminal device in response to a labeling operation on the target object in the first ultrasound image, and the coarse contour area has a preset shape.

[0078] The first terminal device is further configured to perform a fine edge extraction operation on the target object in the coarse contour area of ​​the second ultrasound image to obtain a fine contour of the target object, and send the second ultrasound image and the fine contour to the server 203 .

[0079] The server 203 is further configured to share the received second ultrasound image and fine contour with the second terminal device.

[0080] The second terminal device is further configured to receive the fine contour of the target object and the second ultrasound image sent by the first terminal device through the server 203 , and mark the fine contour of the target object in the second ultrasound image and display it.

[0081] The positions of the first terminal device and the second terminal device may be exchanged, which is not limited in this application.

[0082] Next, taking the example of the user of the second terminal device being an expert doctor, the expert doctor selects a target object in the first ultrasound image, and the terminal device roughly locates the target object to obtain a coarse contour area, wherein the coarse contour area includes position information of the coarse contour area in the first ultrasound image.

[0083] The second terminal device sends the coarse contour region to the first terminal device via server 203. The first terminal device then obtains a second ultrasound image from ultrasound device 201. Based on the coarse contour region's position in the ultrasound image, the first terminal device performs a fine edge extraction operation on the coarse contour region in the second ultrasound image to obtain a fine contour of the target object. The first terminal device then sends the second ultrasound image and the fine contour within the second ultrasound image to the second terminal device via server 203. At this point, the first and second terminal devices share the same ultrasound image and fine contour region.

[0084] Next, combine Figure 3 The steps shown introduce the remote ultrasound image annotation method of the present application.

[0085] The following is an example of the second terminal device completing the coarse segmentation of the region of interest and the first terminal device completing the fine segmentation of the region of interest. It should be noted that, as mentioned above, in the embodiment of the present application, both the remote terminal device and the local terminal device can achieve the coarse segmentation of the region of interest. Therefore, the second terminal device in the embodiment of the present application can be either a remote terminal device or a local terminal device. Figure 3 FIG. 1 is a flow chart of a marking method executed by a second terminal device, comprising the following steps:

[0086] In step 301, a first ultrasound image is displayed in a second terminal device.

[0087] In step 302, the second terminal device performs coarse positioning of the target object in response to the labeling operation on the target object in the first ultrasound image, and obtains a coarse contour area of ​​the target object, wherein the coarse contour area has a preset shape.

[0088] In the embodiments of this application, Figure 4 As shown in the left figure, the user's annotation operation can be to double-click a location point in the target object (i.e., the area of ​​interest) on the screen (e.g. Figure 4 the white dot in the target object in the left image), or Figure 4 The right figure shows the sliding operation on the target object on the screen (for example Figure 4 (The white line in the target object in the right image). After detecting a user annotation operation, if the user double-clicks the target object on the screen, the user's annotated point is read. If the user swipes the target object on the screen, the multiple pixels covered by the swipe operation are read.

[0089] Therefore, based on the operation position of the marking operation, any one of the following methods is used to determine the basic outline of the target object: watershed algorithm, flood filling algorithm, image segmentation method based on edge detection, image segmentation method based on morphological operation.

[0090] Through the above method, the minimum rectangular area surrounding the basic outline is found as the rough outline area of ​​the target object. In this application, due to the different shapes of the target objects, other preset shapes can also be used as the rough outline area of ​​the target object. For example, if the target object is a heart, the preset shape can be set to a heart shape.

[0091] Taking the watershed algorithm as an example, in the embodiment of the present application, the operation position of the marking operation is used as the reference. If the user double-clicks, the position point of the double-click operation is used as the reference. If the user slides, the center point of the multiple pixels covered by the sliding operation is used as the reference. Through the watershed algorithm, the gradient of the grayscale interval of the pixel points in the first ultrasound image is calculated with the reference as the center. Since the grayscale value in the target object is significantly different from the grayscale value in the rest of the area, the gradient change at the boundary of the target object is most obvious. At this time, the watershed algorithm uses the place where the gradient change is most obvious as the boundary of the target object, and uses points to depict the place where the gradient change is most obvious in a clockwise or counterclockwise manner to obtain the boundary of the target object.

[0092] Since the grayscale value change at the boundary of the target object in the first ultrasound image gradually increases, the boundary of the target object is a closed curve with a certain width. In order to make the target object completely contained in the point, the present application establishes a two-dimensional coordinate system with any point in the rough contour area as the center after determining the boundary of the target object, and obtains the coordinates of each point on the boundary of the target object. Or after determining the boundary of the target object, the coordinates of each point on the boundary of the target object are obtained using the image coordinate system in the ultrasound image. Taking the preset shape as a rectangle as an example, the minimum value of the horizontal coordinate of the point is used as the horizontal coordinate of the two vertices on the left side of the circumscribed rectangle, the maximum value of the horizontal coordinate of the point is used as the horizontal coordinate of the two vertices on the right side of the circumscribed rectangle, the minimum value of the vertical coordinate of the point is used as the vertical coordinate of the two vertices on the lower side of the circumscribed rectangle, and the maximum value of the vertical coordinate of the point is used as the vertical coordinate of the two vertices on the upper side of the circumscribed rectangle. The horizontal or vertical circumscribed rectangle of the target object is determined based on the horizontal and vertical coordinates of the four vertices, and as shown in the figure, Figure 5 As shown, the four vertices of the circumscribed rectangle are drawn at corresponding positions of the first ultrasound image, and adjacent vertices in the horizontal direction and the vertical direction are connected.

[0093] In step 303 , the second terminal device sends the rough outline area to the server, and in step 304 , the server sends the rough outline area to the first terminal device.

[0094] In step 305, the first terminal device receives the rough outline area sent by the server.

[0095] In an embodiment of the present application, the first terminal device receives position information of a preset shape, determines the position of the preset shape in the second ultrasound image, and uses the image inside the preset shape in the second ultrasound image as a rough outline area of ​​the target object.

[0096] In step 306 , the first terminal device performs a fine edge extraction operation on the target object within the coarse contour area of ​​the second ultrasound image to obtain a fine contour of the target object.

[0097] In an embodiment of the present application, the image acquired in real time by the first terminal device at this time is the second ultrasound image, and the position information of the coarse contour area sent by the second ultrasound device is received. The fine contour of the target object is extracted from the coarse contour area of ​​the second ultrasound image using any of the following methods: level set segmentation algorithm, image segmentation model based on deep learning.

[0098] Taking the level set segmentation algorithm as an example, before processing, the grayscale information in the coarse contour area of ​​the second ultrasound image is selected as the energy driving the evolution, and an energy function as shown in formula (1) is constructed.

[0099] E(C)=λ o ∫ inside (I(x,y) 2 -C o )dxdy

[0100] +λ b ∫ outside (I(x,y) 2 -C b )dxdy+μL(C)+γS([ns[de(C)) (1)

[0101] In formula (1), I(x, y) represents the grayscale value of the pixel point in the coarse contour area of ​​the second ultrasound image, λ o and λ b Represent the weight coefficients of the energy terms, C o represents the average grayscale value of the inner area in the rough outline area of ​​the second ultrasound image, C b represents the average grayscale value of the outer area in the rough contour area of ​​the second ultrasound image. μ represents the length coefficient of the preset shape, γ represents the area parameter of the rough contour area, and E(C) represents the overall energy information in the rough contour area. o ∫ inside (I(x,y) 2 -C o )dxdy represents the grayscale energy of the inner area of ​​the coarse contour area, λ b ∫ outside (I(x,y) 2 -C b )dxdy represents the grayscale energy of the outer area of ​​the coarse contour area, μL(C) represents the length energy of the coarse contour area, and γS(inside(C)) represents the area energy of the coarse contour area.

[0102] The evolution function of the level set method model is determined according to formula (2), formula (3) and formula (4), and the minimum value obtained according to the evolution function is the fine contour.

[0103]

[0104]

[0105]

[0106] In formula (2), formula (3) and formula (4), is the iteration step, div is the divergence, is the derivative, and I(x) is the grayscale value. is the Dirac function, is the Heaviside function, when hour, when hour,

[0107] like Figure 6 As shown, the fine contour is marked in the second ultrasound image.

[0108] In step 307 , the first terminal device sends the second ultrasound image and the fine contour to the server, and in step 308 , the second ultrasound image and the fine contour are sent to the second terminal device via the server.

[0109] In the embodiment of the present application, when the first terminal device marks the fine contour, the marking color can be selected as needed. For example, in order to make the fine contour more obvious, the following color can be used: Figure 6 The white marker shown is displayed, and the fine outline marker color is synchronized to the second terminal device.

[0110] In an embodiment of the present application, in order to achieve different display effects, the user can set the mark color of the fine outline by himself.

[0111] In step 309 , the second terminal device receives the fine contour of the target object and the second ultrasound image sent by the first terminal device.

[0112] In step 310 , the second terminal device marks the fine outline of the target object in the second ultrasound image and displays it.

[0113] In an embodiment of the present application, the second terminal device receives the marking color for the fine contour and the second ultrasound image sent by the first terminal device, and marks the fine contour of the target object in the second ultrasound image using the marking color.

[0114] In another embodiment of the present application, if the screen sizes and resolutions of the first and second terminal devices are different, it is also necessary to convert the coordinates of the location of the fine contour in the second ultrasound image into the display coordinates required for display on the second terminal device, and adapt the resolution of the second ultrasound image to the resolution required for display on the second terminal device. After converting the coordinates and resolution, the second terminal device displays the second ultrasound image marked with the fine contour of the target object based on the display coordinates and the required display resolution.

[0115] For example, if the screen size of the first terminal device is one-fourth of the screen size of the second terminal device, and the coordinates of the fine contour are established with the screen center of the first terminal device as the coordinate origin, then in the second terminal device, the screen center of the second terminal device is used as the coordinate origin, and the horizontal and vertical coordinate values ​​of the fine contour are expanded to twice the original values.

[0116] In this embodiment of the present application, the coarse positioning and fine edge extraction operations are performed separately by the first and second terminal devices, eliminating the need for user voice communication, thereby improving the efficiency of remote ultrasound image annotation. Because the fine contours are annotated on the second ultrasound image acquired in real time and shared with the second terminal device, both the first and second terminal devices can display the fine contours in real time, improving diagnostic accuracy.

[0117] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0121] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A remote ultrasound image annotation method, characterized in that: The method comprises: displaying a first ultrasound image; In response to a labeling operation on a target object in the first ultrasound image, coarsely locating the target object to obtain a coarse contour region of the target object, wherein the coarse contour region has a preset shape; the coarsely locating the target object to obtain the coarse contour region of the target object in response to the labeling operation on the target object in the first ultrasound image comprises: upon detecting a user labeling operation on the target object in the first ultrasound image, if the user double-clicks the target object on the screen, using the location of the double-click operation as a reference; if the user performs a sliding operation on the target object on the screen, using the center point of multiple pixels covered by the sliding operation as a reference; using a watershed algorithm, with the reference as the center, calculating the gradient of the grayscale interval of the pixel points in the first ultrasound image, and using points to depict the location with the most obvious gradient change in a clockwise or counterclockwise manner to obtain the boundary of the target object; after determining the boundary of the target object, obtaining the coordinates of each point on the boundary of the target object using the image coordinate system in the first ultrasound image; and determining the coarse contour region of the target object based on the preset shape and the coordinates of each point on the boundary of the target object; Sending the rough outline area to a server, so that the server sends the rough outline area to the first terminal device; receiving a fine contour of the target object and a second ultrasound image sent by the first terminal device; the fine contour is obtained by the first terminal device performing a fine edge extraction operation on the target object in the coarse contour area of ​​the second ultrasound image; The fine outline of the target object is marked and displayed in the second ultrasound image.

2. The method according to claim 1, characterized in that Also includes: receiving a marking color for the fine outline sent by the first terminal device; The marking of a fine outline of the target object in the second ultrasound image includes: The fine outline of the target object is marked in the second ultrasound image using the marking color.

3. The method according to any one of claims 1-2, characterized in that Displaying a second ultrasound image with a fine outline of the target object marked thereon, comprising: converting the position coordinates of the fine contour in the second ultrasound image into display coordinates required for display, and adapting the resolution of the second ultrasound image to the resolution required for display; The second ultrasound image marked with a fine outline of the target object is displayed based on the display coordinates and the resolution required for the display.

4. A remote ultrasound image annotation method, characterized in that: The method comprises: Receive a coarse contour area sent by the server, wherein the coarse contour area is determined by the second terminal device in response to a labeling operation on a target object in the first ultrasound image, and the coarse contour area has a preset shape; the coarse contour area is determined by the second terminal device based on the preset shape and the coordinates of each point on the boundary of the target object; the coordinates of each point on the boundary of the target object are obtained by the second terminal device using the image coordinate system in the first ultrasound image after determining the boundary of the target object; the boundary of the target object is determined by the second terminal device detecting that the user has double-clicked the target object on the screen after the user has labelled the target object in the first ultrasound image, and the position point of the double-click operation is used as a reference; if the user performs a sliding operation on the target object on the screen, the center point of the multiple pixels covered by the sliding operation is used as a reference; using a watershed algorithm, with the reference as the center, the gradient of the grayscale interval of the pixel points in the first ultrasound image is calculated, and the place where the gradient change is most obvious is depicted with points in a clockwise or counterclockwise manner; performing a fine edge extraction operation on the target object in the coarse contour area of ​​the second ultrasound image to obtain a fine contour of the target object; The second ultrasound image and the fine contour are synchronized to the second terminal device through the server.

5. The method according to claim 4, characterized in that The performing a fine edge extraction operation on the target object in the coarse contour area of ​​the second ultrasound image to obtain a fine contour of the target object includes: In the coarse contour area of ​​the second ultrasound image, the fine contour of the target object is extracted by using any one of the following methods: Level set segmentation algorithm, image segmentation model based on deep learning.

6. The method according to claim 4 or 5, characterized in that The method further comprises: The marking color of the fine outline is synchronized to the second terminal device.

7. A remote ultrasound image annotation system, characterized in that: include: a second terminal device, configured to display the first ultrasound image; In response to a marking operation on a target object in the first ultrasound image, the target object is roughly located to obtain a coarse outline region of the target object, wherein the coarse outline region has a preset shape; the coarse outline region is sent to a server; a fine outline of the target object and a second ultrasound image are received from the first terminal device; the fine outline of the target object is marked in the second ultrasound image and displayed; the second terminal device is configured to, upon detecting a user marking operation on the target object in the first ultrasound image, use the location point of the double-click operation as a reference if the user double-clicks the target object on the screen; If a user performs a sliding operation on a target object on the screen, a center point of multiple pixels covered by the sliding operation is used as a reference; a watershed algorithm is used to calculate the gradient of the grayscale interval of the pixel points in the first ultrasound image with the reference as the center, and points are used to depict the most obvious gradient changes in a clockwise or counterclockwise manner to obtain the boundary of the target object; after determining the boundary of the target object, the coordinates of each point on the boundary of the target object are obtained using the image coordinate system in the first ultrasound image; and a coarse contour area of ​​the target object is determined based on the preset shape and the coordinates of each point on the boundary of the target object; The first terminal device is configured to receive a coarse contour region sent by a server, wherein the coarse contour region is determined by the second terminal device in response to a labeling operation on a target object in a first ultrasound image, and the coarse contour region has a preset shape; perform a fine edge extraction operation on the target object within the coarse contour region of a second ultrasound image to obtain a fine contour of the target object; and synchronize the second ultrasound image and the fine contour to the second terminal device via the server; The first terminal device or the second terminal device is connected to an ultrasound device and is used to control the ultrasound device to acquire ultrasound images.

8. A terminal device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of a terminal device, the terminal device is enabled to execute any one of the methods provided in claims 1-6.

Images