Neural network model training method and liver right lobe maximum oblique diameter determination method

Abstract

According to the training method of the neural network model and the determination method of the maximum oblique diameter of the right lobe of the liver provided by the embodiment of the invention, the ultrasonic video of the liver is obtained by training the neural network model; inputting the ultrasonic video into a neural network model frame by frame to determine a mask and a liver trailing edge envelope key point of a liver region in each frame of image; determining the contour of the liver region based on the mask of the liver region in each frame of image; determining the maximum oblique diameter value of the right lobe of the liver in each frame of image based on the contour of the liver region in each frame of image and the liver trailing edge envelope key point; and determining the target maximum oblique diameter value of the right lobe of the liver based on the maximum oblique diameter value of the right lobe of the liver of each frame of image, so that the maximum oblique diameter of the right lobe of the liver can be automatically determined, and the accuracy of determining the maximum oblique diameter of the right lobe of the liver can be improved.

Description

Technical Field

[0001] The present application belongs to the field of medical technology, and in particular relates to a training method for a neural network model and a method for determining the maximum oblique diameter of the right lobe of the liver. Background Art

[0002] During a liver ultrasound examination, the patient's breathing causes the diaphragm to contract and relax rhythmically, causing the position of the liver to change. In order to accurately measure the maximum oblique diameter of the right liver lobe, the doctor needs to adjust the probe to the maximum oblique diameter standard section scanning position and hold it for a few seconds, then save the video of these few seconds, drag the video progress bar on the operation panel, find the frame with the largest oblique diameter and then measure it. In related technologies, the method completely relies on the doctor to judge the maximum oblique diameter section and manual measurement. During measurement, it is necessary to subjectively judge the vertical relationship between the measurement line and the anterior capsule of the right liver lobe. This method will lead to the problem of longer liver examination time and lower efficiency. At the same time, screening the frame with the largest maximum oblique diameter value and determining the position of the measurement line perpendicular to the anterior capsule of the liver are greatly affected by the doctor's subjective factors and clinical experience, resulting in a large final measurement error. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application provide a training method for a neural network model and a method for determining the maximum oblique diameter of the right lobe of the liver. The maximum oblique diameter of the right lobe of the liver can be automatically measured through the neural network model, thereby improving inspection efficiency and detection accuracy.

[0004] In a first aspect, an embodiment of the present application provides a method for training a neural network model, comprising:

[0005] Acquire a sample training set, wherein each sample data in the sample training set includes: a sample ultrasound image and label information of the sample ultrasound image, wherein the label information includes: a sample mask of a liver structure and key points of a sample liver posterior capsule;

[0006] The sample ultrasound image in each sample data is used as input, and the label information corresponding to each sample data is used as output to train the neural network model.

[0007] In some embodiments, the neural network model includes: a backbone network and a detection head connected to the backbone network, the backbone network is used to encode the ultrasound image, and the detection head includes: a first decoder and a second decoder, the first decoder is used to segment the contour of the liver structure, and the second decoder is used to perform key point detection.

[0008] In a second aspect, an embodiment of the present application provides a method for determining the maximum oblique diameter of the right lobe of the liver, comprising:

[0009] Obtain an ultrasound video of the liver;

[0010] Inputting the ultrasound video frame by frame into the neural network model to determine the mask of the liver area and the key points of the posterior liver capsule in each frame image;

[0011] Determine the contour of the liver region based on the mask of the liver region in each frame image;

[0012] Determine the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image and the key point of the posterior liver capsule;

[0013] The maximum oblique diameter value of the target right lobe of the liver is determined based on the maximum oblique diameter value of the right lobe of each frame image.

[0014] In some embodiments, determining the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image and the key point of the posterior liver capsule includes:

[0015] Determine, from the contour of the liver region in each frame image, the contour point with the shortest distance from the key point of the liver posterior edge capsule;

[0016] Determine the contour point as the capsule measurement point at the rear edge of the right liver lobe of each frame image;

[0017] Determine the anterior capsule contour line of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image;

[0018] The maximum oblique diameter value of the right liver lobe in each frame image is determined based on the anterior capsule contour line of the right liver lobe and the capsule measurement points at the posterior edge of the right liver lobe in each frame image.

[0019] In some embodiments, each frame image is fan-shaped, and determining the contour line of the front capsule of the right liver lobe of each frame image based on the contour of the liver region in each frame image includes:

[0020] Mapping the contour of each frame image to a rectangular area, and mapping the contour of the liver area in each frame image to the rectangular area to obtain a mapping contour;

[0021] Determine the target point corresponding to each frame image from the mapping contour based on the rectangular area corresponding to each frame image, wherein the target point includes: a first target point and a second target point, the first target point has a minimum distance from the upper left end point of the rectangular area, and the second target point has a minimum distance from the upper right end point of the rectangular area;

[0022] The contour line of the anterior capsule of the right liver lobe of each frame image is determined based on the first target point and the second target point corresponding to each frame image.

[0023] In some embodiments, determining the contour line of the anterior capsule of the right liver lobe of each frame of image based on the first target point and the second target point corresponding to each frame of image includes:

[0024] Dividing the contour points of the liver region in each frame of image into two contour point subsets based on the first target point and the second target point corresponding to each frame of image;

[0025] Calculate the average value of the ordinates of the contour points in each contour point subset;

[0026] The subset of contour points with the smallest average value is determined as the contour line of the anterior capsule of the right liver lobe in each frame image.

[0027] In some embodiments, determining the maximum oblique diameter value of the right liver lobe in each frame of image based on the anterior capsule contour line of the right liver lobe and the capsule measurement point at the posterior edge of the right liver lobe in each frame of image includes:

[0028] Traversing the contour points in the contour line of the anterior capsule of the right liver lobe in each frame image, performing fitting based on the preset contour points before and after each contour point, to obtain fitting line segments of each contour point in each frame image;

[0029] Draw a perpendicular line through the capsule measurement point at the rear edge of the right liver lobe in each frame image to the fitting line segment of each contour point in each frame image;

[0030] The value corresponding to the vertical line having the longest length and intersecting with the fitting line segment is determined as the maximum oblique diameter value of the right liver lobe in each frame image.

[0031] In some embodiments, the method further comprises:

[0032] Determine a target image corresponding to the maximum oblique diameter value of the target right liver lobe;

[0033] The target image is output, wherein the target image displays the maximum oblique diameter value of the target right liver lobe.

[0034] In a third aspect, an embodiment of the present application provides a device for determining the maximum oblique diameter of the right lobe of the liver, comprising:

[0035] A first acquisition module is used to acquire an ultrasound video of the liver;

[0036] A first determination module is used to input the ultrasound video frame by frame into the neural network model to determine the mask of the liver area and the coordinates of the key points of the posterior liver capsule in each frame image;

[0037] A second determination module is used to determine the contour of the liver region based on the mask of the liver region in each frame image;

[0038] A third determination module is used to determine the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image and the coordinates of the key points of the posterior liver capsule;

[0039] The fourth determination module is used to determine the maximum oblique diameter value of the target right lobe of the liver based on the maximum oblique diameter value of the right lobe of each frame image.

[0040] In a fourth aspect, an embodiment of the present application provides a training device for a neural network model, comprising:

[0041] A second acquisition module is used to acquire a sample training set, wherein each sample data in the sample training set includes: a sample ultrasound image and label information of the sample ultrasound image, wherein the label information includes: a sample mask of a liver structure and a key point of a sample liver posterior edge capsule;

[0042] The training module is used to train the neural network model by taking the sample ultrasound image in each sample data as input and the label information corresponding to each sample data as output.

[0043] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method provided in the first aspect described above is implemented.

[0044] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the second aspect above when executing the computer program.

[0045] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the methods described above.

[0046] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the electronic device to execute any of the methods described above.

[0047] The embodiments of the present application provide a method for training a neural network model and a method for determining the maximum oblique diameter of the right lobe of the liver. By training the neural network model, when acquiring an ultrasound video of the liver; inputting the ultrasound video frame by frame into the neural network model to determine the mask of the liver region and the key points of the posterior liver capsule in each frame image; determining the contour of the liver region based on the mask of the liver region in each frame image; determining the maximum oblique diameter value of the right lobe of the liver in each frame image based on the contour of the liver region and the key points of the posterior liver capsule in each frame image; determining the maximum oblique diameter value of the target right lobe of the liver based on the maximum oblique diameter value of the right lobe of the liver in each frame image, the maximum oblique diameter of the right lobe of the liver can be automatically determined, and the accuracy of the determined maximum oblique diameter of the right lobe of the liver can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 A schematic diagram of the implementation process of a manual measurement method for the maximum oblique diameter of the right liver lobe provided in the related art;

[0050] Figure 2 A schematic diagram of an implementation flow of a training method for a neural network model provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the implementation flow of a method for determining the maximum oblique diameter of the right lobe of the liver provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of obtaining the contour of a liver region and key points of the posterior liver capsule provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of each frame image provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram after mapping provided in an embodiment of the present application;

[0056] Figure 8 A schematic diagram of displaying a target image provided by an embodiment of the present application;

[0057] Fig. 9A schematic diagram of the structure of a device for determining the maximum oblique diameter of the right lobe of the liver provided in an embodiment of the present application;

[0058] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0059] In the drawings, the same reference numerals are used for the same components, and the drawings are not drawn to scale. DETAILED DESCRIPTION

[0060] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0061] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0062] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0063] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrases "if it is determined" or "if it is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce detected" or "in response to detecting" depending on the context.

[0064] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0065] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0066] Before introducing the embodiments of the present application, a brief introduction to the problems in the related art is given. In a liver ultrasound examination, the patient's breathing during the examination causes the diaphragm to contract and relax rhythmically, thereby causing the position of the liver to change. In order to accurately measure the maximum oblique diameter of the right lobe of the liver, the doctor needs to adjust the probe to the maximum oblique diameter standard section scanning position and hold it for a few seconds, then save the video of these few seconds, drag the video progress bar on the operation panel, find the frame with the largest oblique diameter, and then measure it. In an examination, when the probe position remains unchanged, save two frames in the video. Due to the change in the position of the liver caused by the traction of the diaphragm, the morphology of the liver in the two frames that are close in time has a large change, and the measured value of the maximum oblique diameter will also have a large difference accordingly. In a maximum oblique diameter measurement, the doctor needs to review the video at least once, and repeatedly compare several similar frames of data to determine the maximum oblique diameter frame for measurement.

[0067] After successfully determining the maximum oblique diameter frame, the next step is to determine the measurement point and measure the maximum oblique diameter value. When measuring, only the standard section is considered. The anterior liver capsule of the standard section is clear and relatively flat, and the measurement line is roughly perpendicular to the upper and lower boundaries of the screen, making it easy to determine the position of the measurement point. However, in clinical practice, the liver position in some images will be slightly tilted, and the anterior boundary of the right liver lobe also has a certain curvature, making it difficult to determine the position of the measurement line perpendicular to the anterior and posterior capsules. Therefore, in actual measurement, the doctor first determines the farthest point of the posterior edge of the right liver lobe as the measurement point of the posterior edge, and then determines a point on the right liver lobe so that the line connecting the two points is roughly perpendicular to the anterior boundary of the right liver lobe, and then records the distance between the two measurement points as the measurement value of the maximum oblique diameter of the right liver lobe. Figure 1 The following is a schematic diagram of the implementation process of the manual measurement method of the maximum oblique diameter of the right liver lobe provided in the related art. Figure 1As shown in the figure, after acquiring the ultrasound video of the right liver lobe, the maximum oblique diameter frame is manually screened, and then two measurement points are determined to obtain the maximum oblique diameter measurement value of the right liver lobe. It can be seen that this method, which completely relies on the doctor's judgment of the maximum oblique diameter section and manual measurement, requires the doctor to repeatedly drag the progress bar to select the frame suitable for the maximum oblique diameter measurement before measuring, and the measurement requires subjective judgment of the vertical relationship between the measurement line and the anterior capsule of the right liver lobe. This method will lead to the problems of long liver examination time and low efficiency. At the same time, the selection of the frame with the largest maximum oblique diameter value and the determination of the position of the measurement line perpendicular to the anterior liver capsule are greatly affected by the doctor's subjective factors and clinical experience, resulting in a large final measurement error.

[0068] Based on the technical problems of the related technology, the embodiments of the present application provide a training method for a neural network model that can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0069] The present application embodiment provides a method for training a neural network model. Figure 2 A schematic diagram of the implementation flow of a training method for a neural network model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, including:

[0070] Step S101, obtaining a sample training set, wherein each sample data in the sample training set includes: a sample ultrasound image and label information of the sample ultrasound image, wherein the label information includes: a sample mask of the liver structure and key points of the sample liver posterior edge capsule.

[0071] In the embodiment of the present application, the sample training set is used to train the machine learning or deep learning model. Ultrasound images with label information can be collected from a medical database. The label information may need to be manually annotated by a medical expert, or semi-automatic tools may be used to assist in the annotation.

[0072] Step S102, taking the sample ultrasound image in each sample data as input and the label information corresponding to each sample data as output to train the neural network model.

[0073] In an embodiment of the present application, the ultrasound image in each sample data can be used as input, and the corresponding label information (sample mask of liver structure and key points of the posterior liver capsule) can be used as output to train the neural network model. The training process involves presenting a large amount of sample data to the model and adjusting the internal parameters of the model to minimize the difference between the predicted output and the true label. After the training is completed, the model is able to receive new ultrasound images as input and output the predicted mask of the liver structure and the key points of the posterior liver capsule.

[0074] During the training process, a portion of the data needs to be reserved as a validation set and a test set to evaluate the performance of the model. The validation set is used to adjust the hyperparameters of the neural network model during the training process. Hyperparameters can include: learning rate, number of iterations, number of neurons, etc. The test set is used to evaluate the final performance of the model after the training is completed. Once the model is trained and verified, it can be deployed in an actual medical diagnostic system to automatically identify and locate the mask of the liver structure and the key points of the posterior liver capsule in ultrasound images.

[0075] In some embodiments, the neural network model includes: a backbone network and a detection head connected to the backbone network, the backbone network is used to encode the ultrasound image, the detection head includes: a first decoder and a second decoder, the first decoder is used to segment the mask of the liver structure, and the second decoder is used to perform key point detection.

[0076] Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the backbone network may be EfficientNetV2, the first decoder may be a decoder with a Unet structure, responsible for mask segmentation of the liver structure, and the second decoder may be a decoder with an Upernet structure, responsible for key point detection.

[0077] Based on the neural network model obtained through training. The embodiment of the present application provides a method for determining the maximum oblique diameter of the right lobe of the liver. The embodiment of the present application provides a method for determining the maximum oblique diameter of the right lobe of the liver, which can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA) and other electronic devices. The electronic device can be an ultrasonic scanning device, and the embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0078] The present application provides a method for determining the maximum oblique diameter of the right lobe of the liver. Figure 4 A schematic diagram of the implementation flow of a method for determining the maximum oblique diameter of the right lobe of the liver provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, including:

[0079] Step S401, obtaining an ultrasound video of the liver.

[0080] In the embodiment of the present application, the ultrasound video is a continuous sequence of ultrasound images, which is usually acquired by an ultrasound scanning device and is used to dynamically display the changes of the liver over time or its structure at different viewing angles.

[0081] Step S402: Input the ultrasound video frame by frame into the neural network model to determine the mask of the liver area and the key points of the posterior liver capsule in each frame image.

[0082] In the embodiment of the present application, the neural network model is the neural network model trained in the above embodiment. Ultrasound video can be regarded as a collection of a series of static images (frames). Each frame is an independent ultrasound image, showing a static view of the liver at a certain point in time. Mask of the liver area: This is a structural area representation of the liver in the ultrasound image, which is used to identify the precise position and shape of the liver. The posterior liver capsule is the interface between the posterior part of the liver and the surrounding tissue. Key points are points with specific significance selected on this interface, such as boundary points, feature points, etc.

[0083] In an embodiment of the present application, the ultrasound video can be input into the neural network model frame by frame to output the mask of the liver area and the key points of the posterior liver capsule in each frame image.

[0084] Figure 5 A schematic diagram of obtaining a mask of a liver region and key points of the liver posterior capsule provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, each frame image can be input into the algorithm model to obtain the mask of the liver area and the key points of the posterior liver capsule.

[0085] Step S403: determining the contour of the liver region based on the mask of the liver region in each frame image.

[0086] In the embodiment of the present application, contour detection can be performed on the mask of each frame of the image, and the contour detection algorithm in the image processing library (such as using the findContours function after Canny edge detection) can be used to extract the contour of the liver area.

[0087] Step S404, determining the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region and the key points of the liver posterior capsule in each frame of image.

[0088] In the present application, the maximum oblique diameter of the right liver lobe is an important size parameter of the liver, which generally refers to the longest straight-line distance from the front end of the right liver lobe to its bottom edge (generally including a part of the capsule at the rear edge of the liver). This value is often used in medical diagnosis to assess the health and size of the liver.

[0089] In the embodiment of the present application, the maximum oblique diameter value of the right lobe of the liver in each frame image can be calculated using the identified liver contour and liver posterior capsule key points.

[0090] Step S405, determining the maximum oblique diameter value of the target right lobe of the liver based on the maximum oblique diameter value of the right lobe of each frame image.

[0091] In the embodiment of the present application, a maximum value is calculated based on the maximum oblique diameter value of the right lobe of the liver measured in each frame image.

[0092] In an embodiment of the present application, a statistical analysis is performed on the maximum oblique diameter value of the right liver lobe calculated in each frame image to obtain the maximum maximum oblique diameter value of the right liver lobe in the entire ultrasound video, and the maximum maximum oblique diameter value of the right liver lobe is the target right lobe maximum oblique diameter value.

[0093] The method provided in the embodiment of the present application, by training a neural network model, when acquiring an ultrasound video of the liver; inputting the ultrasound video into the neural network model frame by frame to determine the mask of the liver area and the key points of the posterior liver capsule in each frame image; determining the contour of the liver area based on the mask of the liver area in each frame image; determining the maximum oblique diameter value of the right lobe of the liver in each frame image based on the contour of the liver area and the key points of the posterior liver capsule in each frame image; determining the maximum oblique diameter value of the target right lobe of the liver based on the maximum oblique diameter value of the right lobe of the liver in each frame image, can automatically determine the maximum oblique diameter of the right lobe of the liver, and can improve the accuracy of the determined maximum oblique diameter of the right lobe of the liver.

[0094] In some embodiments, step S404 may be implemented by the following steps:

[0095] Step S4041, determining the contour point with the shortest distance from the key point of the liver posterior edge capsule from the contour of the liver region in each frame image.

[0096] In the embodiment of the present application, for each frame of image, the distance from each point on the liver region contour to the key point of the liver posterior capsule is first calculated, and then the point with the smallest distance is selected as the measurement point of the liver posterior capsule of the right lobe.

[0097] In the embodiment of the present application, the contour extraction method provided by OpenCV can be used to obtain the contour point set of the liver area, and then the contour point set is traversed to find the contour point with the smallest Euclidean distance to the key point of the posterior liver capsule.

[0098] Step S4042: determine the contour point as the capsule measurement point at the rear edge of the right liver lobe of each frame image.

[0099] In the embodiment of the present application, the measurement point of the capsule at the posterior edge of the right liver lobe is a specific point determined from the contour of the liver region. It is closest to the key point of the capsule at the posterior edge of the liver and is therefore used as a reference point for measuring the maximum oblique diameter of the right liver lobe.

[0100] Step S4043, determining the contour line of the anterior capsule of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image.

[0101] In the embodiment of the present application, the contour line of the anterior capsule of the right liver lobe is the line in the liver region contour that represents the anterior capsule of the right liver lobe. It usually starts from the front end of the liver and extends along the anterior edge of the right liver lobe.

[0102] In the embodiment of the present application, step S4043 can be implemented by the following steps:

[0103] Step S1, mapping the contour of each frame image to a rectangular area, and mapping the contour of the liver area in each frame image to the rectangular area to obtain a mapping contour.

[0104] In the embodiment of the present application, each frame image is usually fan-shaped, and the fan-shaped area needs to be mapped to a rectangular area. The fan-shaped outline can be mapped to a rectangle through a mapping formula to obtain a rectangular area. Figure 6 A schematic diagram of each frame image provided by an embodiment of the present application, in which point O is the center of curvature shared by the upper and lower arcs of the sector, r is the radius of the upper arc of the sector, and R is the radius of the lower arc of the sector. A plane rectangular coordinate system is established with the center O as the origin. Any point P (x, y) in the sector can be mapped to a point P' in the rectangular area. The horizontal coordinate of P' is the product of the angle θ (horizontal radians) between the line connecting the point and the center of the curvature and the vertical coordinate and the maximum radius R of the sector plus the horizontal coordinate of the center of the curvature; the vertical coordinate is the distance from point P to the center of the curvature. The mapping formula can be expressed as:

[0105] In the embodiment of the present application, after mapping, the outline of the liver area is mapped into the rectangular area. Figure 7 A schematic diagram after mapping is provided in an embodiment of the present application, such as Figure 7 As shown, contour 100 is the sector area before mapping, contour 200 is the contour of the liver area before mapping, contour 300 is the rectangular area after the sector area is mapped, and contour 400 is the mapping contour of the liver area after the contour is mapped. It can be seen that the boundary feature on the contour of the liver area mapped into the rectangular area is more obvious.

[0106] Step S2, based on the rectangular area corresponding to each frame image, determine the target point corresponding to each frame image from the mapping contour, wherein the target point includes: a first target point and a second target point, the distance between the first target point and the upper left endpoint of the rectangular area is the smallest, and the distance between the second target point and the upper right endpoint of the rectangular area is the smallest.

[0107] In an embodiment of the present application, the mapping points in the mapping contour can be traversed to obtain the indexes of the two points closest to the upper left endpoint and the upper right endpoint of the rectangular area in the point set, and the points in the original liver contour corresponding to these two points can be obtained according to the indexes as the two endpoints of the upper boundary of the anterior capsule of the liver, namely, the first target point and the second target point.

[0108] Step S3, determining the anterior capsule contour line of the right liver lobe of each frame image based on the first target point and the second target point corresponding to each frame image.

[0109] In the embodiment of the present application, the contour line of the anterior capsule of the right liver lobe is actually the upper boundary of the contour of the liver area in the image. The contour of the liver area is an irregular closed figure composed of discrete points. The critical point of the upper boundary is difficult to determine. In order to improve the accuracy of upper boundary extraction, in the embodiment of the present application, the upper boundary is defined as the two points closest to the upper left end point and the upper right end point of the rectangular area in the contour line point set within the rectangular area, which are the starting and ending points of the upper boundary of the contour, namely the first target point and the second target point.

[0110] In an embodiment of the present application, the contour points of the liver area in each frame image can be divided into two contour point subsets based on the first target point and the second target point corresponding to each frame image; the average value of the vertical coordinates of the contour points in each contour point subset is calculated; and the contour point subset with the smallest average value is determined as the contour line of the anterior capsule of the right liver lobe in each frame image.

[0111] In an embodiment of the present application, for the contour points of the liver area, when the coordinates of the points are stored, they are stored based on the positional relationship. The stored contour points can be traversed to find the first target point and the second target point, and the contour points between the first target point and the second target point, the first target point and the second target are divided into a contour point subset, and the other contour points are another contour point subset, thereby obtaining two contour point subsets, so that the average value of the longitudinal coordinates of the contour points in each contour point subset can be calculated, and the contour point subset with the smallest average value is selected to be determined as the contour line of the anterior capsule of the right liver lobe of each frame image.

[0112] Step S4044, determining the maximum oblique diameter value of the right liver lobe in each frame of image based on the anterior capsule contour line of the right liver lobe and the capsule measurement point at the posterior edge of the right liver lobe in each frame of image.

[0113] In an embodiment of the present application, the contour points in the contour line of the anterior capsule of the right liver lobe in each frame image can be traversed, and fitting can be performed based on the preset contour points before and after each contour point to obtain the fitting line segments of each contour point in each frame image; a perpendicular line can be drawn from the capsule measurement point at the posterior edge of the right liver lobe in each frame image to the fitting line segments of each contour point in each frame image; and the value corresponding to the perpendicular line that has an intersection with the fitting line segment and has the largest length is determined as the maximum oblique diameter value of the right liver lobe in each frame image.

[0114] In an embodiment of the present application, preset contour points can be configured, for example, the preset number is 30, and the least squares method can be used to fit the preset contour points before and after each contour point to obtain a fitting line segment.

[0115] The method provided in the embodiment of the present application can accurately determine the maximum oblique diameter value of the right lobe of the liver from the ultrasound image, providing important information for medical diagnosis and treatment.

[0116] In some embodiments, after step S405, the method further includes:

[0117] Step S406, determining the target image corresponding to the maximum oblique diameter value of the target right liver lobe.

[0118] In an embodiment of the present application, the maximum value can be found from the maximum oblique diameter values ​​of the right liver lobe corresponding to each frame image, and the maximum value is the maximum oblique diameter value of the target right liver lobe. When the maximum oblique diameter value of the target right liver lobe is determined, the image corresponding to the maximum oblique diameter value of the target right liver lobe is the target image.

[0119] Step S407, outputting a target image, wherein the target image displays the maximum oblique diameter value of the target right liver lobe.

[0120] In the embodiment of the present application, an annotation is added on the selected target image using image processing software or tools to display the maximum oblique diameter value of the target right liver lobe. Figure 8 A display schematic diagram of a target image provided in an embodiment of the present application is shown as follows: Figure 8 As shown, you can draw a line segment on the image representing the maximum slope and add a text label containing the measurement value next to or above it.

[0121] In the embodiment of the present application, the processed target image is saved as an image file (such as JPEG, PNG, etc.), or directly displayed to a doctor or medical expert through the interface of medical image processing software or system. The output format can be used in medical reports, medical records or remote medical consultation.

[0122] The method provided in the embodiment of the present application can accurately determine the target image corresponding to the maximum oblique diameter value of the target right liver lobe, and output it for viewing and analysis by doctors or medical experts.

[0123] The method provided in the embodiment of the present application can fully automatically obtain the key frame and the measurement result of the maximum oblique diameter of the right lobe of the liver on the frame. The doctor only needs to confirm the result without any other unnecessary operations, which simplifies the liver scanning process and improves the doctor's scanning efficiency and the objectivity of the measurement results.

[0124] Based on the foregoing embodiments, the present application provides a device for determining the maximum oblique diameter of the right lobe of the liver. The modules included in the device for determining the maximum oblique diameter of the right lobe of the liver, and the units included in the modules, can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), a digital signal processor (DSP, Digital Signal Processing) or a field programmable gate array (FPGA, Field Programmable Gate Array), etc. Fig. 9 A schematic diagram of the structure of a device for determining the maximum oblique diameter of the right lobe of the liver provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device 900 for determining the maximum oblique diameter of the right liver lobe includes:

[0125] A first acquisition module 901 is used to acquire an ultrasound video of the liver;

[0126] A first determination module 902 is used to input the ultrasound video frame by frame into the neural network model to determine the mask of the liver region and the coordinates of the key points of the posterior liver capsule in each frame image;

[0127] A second determination module 903 is used to determine the contour of the liver region based on the mask of the liver region in each frame image;

[0128] A third determination module 904 is used to determine the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image and the coordinates of the key points of the posterior liver capsule;

[0129] The fourth determination module 905 is used to determine the maximum oblique diameter value of the target right lobe of the liver based on the maximum oblique diameter value of the right lobe of each frame image.

[0130] In some embodiments, the second determining module includes:

[0131] A first determining unit is used to determine, from the contour of the liver region in each frame image, a contour point with the shortest distance from the key point of the liver posterior edge capsule;

[0132] A second determination unit is used to determine the contour point as a capsule measurement point at the rear edge of the right liver lobe of each frame image;

[0133] A third determining unit is used to determine the contour line of the anterior capsule of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image;

[0134] The fourth determining unit is used to determine the maximum oblique diameter value of the right liver lobe in each frame image based on the anterior capsule contour line of the right liver lobe and the capsule measurement point at the posterior edge of the right liver lobe in each frame image.

[0135] In some embodiments, each frame image is in a fan shape, and the third determining unit includes:

[0136] A mapping subunit, used for mapping the contour of each frame image to a rectangular area, and mapping the contour of the liver area in each frame image to the rectangular area to obtain a mapping contour;

[0137] A first determining subunit is used to determine the target point corresponding to each frame image from the mapping contour based on the rectangular area corresponding to each frame image, wherein the target point includes: a first target point and a second target point, the first target point has a minimum distance from the upper left end point of the rectangular area, and the second target point has a minimum distance from the upper right end point of the rectangular area;

[0138] The second determination subunit is used to determine the contour line of the anterior capsule of the right liver lobe of each frame image based on the first target point and the second target point corresponding to each frame image.

[0139] In some embodiments, the second determination subunit is used to divide the contour points of the liver area in each frame image into two contour point subsets based on the first target point and the second target point corresponding to each frame image; calculate the average value of the vertical coordinates of the contour points in each contour point subset; and determine the contour point subset with the smallest average value as the contour line of the anterior capsule of the right liver lobe in each frame image.

[0140] In some embodiments, the fourth determining unit includes:

[0141] A fitting subunit is used to traverse the contour points in the contour line of the anterior capsule of the right liver lobe in each frame image, and perform fitting based on the preset contour points before and after each contour point to obtain a fitting line segment of each contour point in each frame image;

[0142] A vertical line subunit is used to draw a vertical line through the capsule measurement point at the rear edge of the right liver lobe in each frame image to the fitting line segment of each contour point in each frame image;

[0143] The third determination subunit is used to determine the value corresponding to the vertical line that has an intersection with the fitting line segment and has the longest length as the maximum oblique diameter value of the right liver lobe in each frame image.

[0144] In some embodiments, the device 900 for determining the maximum oblique diameter of the right liver lobe further includes:

[0145] A fifth determination module is used to determine a target image corresponding to the maximum oblique diameter value of the target right liver lobe;

[0146] The output module is used to output a target image, wherein the target image displays the maximum oblique diameter value of the target right liver lobe.

[0147] The present application embodiment further provides a training method for a neural network model, including:

[0148] A second acquisition module is used to acquire a sample training set, wherein each sample data in the sample training set includes: a sample ultrasound image and label information of the sample ultrasound image, wherein the label information includes: a sample mask of the liver structure and a key point of the sample liver posterior edge capsule;

[0149] The training module is used to train the neural network model by taking the sample ultrasound image in each sample data as input and the label information corresponding to each sample data as output.

[0150] In some embodiments, the neural network model includes: a backbone network and a detection head connected to the backbone network, the backbone network is used to encode the ultrasound image, the detection head includes: a first decoder and a second decoder, the first decoder is used to segment the contour of the liver structure, and the second decoder is used to perform key point detection.

[0151] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, the electronic device 3 of this embodiment may include: at least one processor 30 ( Fig.10 Only one processor 30 is shown in the figure), a memory 31, and a computer program 32 stored in the memory 31 and executable on at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the above-mentioned method embodiments are implemented; or, when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned system embodiments are implemented.

[0152] Exemplarily, the computer program 32 may be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and are executed by the processor 30 to complete the present application. One or more modules / units may be a series of computer program 32 instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3.

[0153] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program 32. When the computer program 32 is executed by the processor 30, the steps in the above-mentioned method embodiments can be implemented.

[0154] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. According to this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program 32, and the computer program 32 can be stored in a computer-readable storage medium, and the computer program 32 can implement the steps of the above-mentioned various method embodiments when executed by the processor 30. Among them, the computer program 32 includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the terminal, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0156] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0157] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0158] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A training method for a neural network model, characterized in that: include: Acquire a sample training set, wherein each sample data in the sample training set includes: a sample ultrasound image and label information of the sample ultrasound image, wherein the label information includes: a sample mask of a liver structure and key points of a sample liver posterior capsule; The sample ultrasound image in each sample data is used as input, and the label information corresponding to each sample data is used as output to train the neural network model.

2. The method according to claim 1, characterized in that The neural network model includes: a backbone network and a detection head connected to the backbone network, the backbone network is used to encode the ultrasound image, the detection head includes: a first decoder and a second decoder, the first decoder is used to segment the mask of the liver structure, and the second decoder is used to perform key point detection.

3. A method for determining the maximum oblique diameter of the right lobe of the liver, characterized in that: include: Obtain an ultrasound video of the liver; Inputting the ultrasound video frame by frame into the neural network model to determine the mask of the liver area and the key points of the posterior liver capsule in each frame image; Determine the contour of the liver region based on the mask of the liver region in each frame image; Determine the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image and the key point of the posterior liver capsule; The maximum oblique diameter value of the target right lobe of the liver is determined based on the maximum oblique diameter value of the right lobe of each frame image.

4. The method according to claim 3, characterized in that: The determining the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image and the key point of the posterior liver capsule includes: Determine, from the contour of the liver region in each frame image, the contour point with the shortest distance from the key point of the liver posterior edge capsule; Determine the contour point as the capsule measurement point at the rear edge of the right liver lobe of each frame image; Determine the anterior capsule contour line of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image; The maximum oblique diameter value of the right liver lobe in each frame image is determined based on the anterior capsule contour line of the right liver lobe and the capsule measurement points at the posterior edge of the right liver lobe in each frame image.

5. The method according to claim 4, characterized in that Each frame image is fan-shaped, and determining the contour line of the front capsule of the right liver lobe of each frame image based on the contour of the liver region in each frame image includes: Mapping the contour of each frame image to a rectangular area, and mapping the contour of the liver area in each frame image to the rectangular area to obtain a mapping contour; Determine the target point corresponding to each frame image from the mapping contour based on the rectangular area corresponding to each frame image, wherein the target point includes: a first target point and a second target point, the first target point has a minimum distance from the upper left end point of the rectangular area, and the second target point has a minimum distance from the upper right end point of the rectangular area; The contour line of the anterior capsule of the right liver lobe of each frame image is determined based on the first target point and the second target point corresponding to each frame image.

6. The method according to claim 5, characterized in that The determining of the anterior capsule contour line of the right liver lobe of each frame image based on the first target point and the second target point corresponding to each frame image includes: Dividing the contour points of the liver region in each frame of image into two contour point subsets based on the first target point and the second target point corresponding to each frame of image; Calculate the average value of the ordinates of the contour points in each contour point subset; The subset of contour points with the smallest average value is determined as the contour line of the anterior capsule of the right liver lobe in each frame image.

7. The method according to claim 4, characterized in that The method of determining the maximum oblique diameter value of the right liver lobe in each frame of image based on the anterior capsule contour line of the right liver lobe and the capsule measurement point at the posterior edge of the right liver lobe in each frame of image comprises: Traversing the contour points in the contour line of the anterior capsule of the right liver lobe in each frame image, performing fitting based on the preset contour points before and after each contour point, to obtain fitting line segments of each contour point in each frame image; Draw a perpendicular line through the capsule measurement point at the rear edge of the right liver lobe in each frame image to the fitting line segment of each contour point in each frame image; The value corresponding to the vertical line having the longest length and intersecting with the fitting line segment is determined as the maximum oblique diameter value of the right liver lobe in each frame image.

8. The method according to claim 3, characterized in that The method further comprises: Determine a target image corresponding to the maximum oblique diameter value of the target right liver lobe; The target image is output, wherein the target image displays the maximum oblique diameter value of the target right liver lobe.

9. A device for determining the maximum oblique diameter of the right lobe of the liver, characterized in that: include: A first acquisition module is used to acquire an ultrasound video of the liver; A first determination module is used to input the ultrasound video frame by frame into the neural network model to determine the mask of the liver area and the coordinates of the key points of the posterior liver capsule in each frame image; A second determination module is used to determine the contour of the liver region based on the mask of the liver region in each frame image; A third determination module is used to determine the maximum oblique diameter value of the right liver lobe in each frame of image based on the contour of the liver region in each frame of image and the coordinates of the key points of the posterior liver capsule; The fourth determination module is used to determine the maximum oblique diameter value of the target right lobe of the liver based on the maximum oblique diameter value of the right lobe of each frame image.

10. A training device for a neural network model, characterized in that: include: A second acquisition module is used to acquire a sample training set, wherein each sample data in the sample training set includes: a sample ultrasound image and label information of the sample ultrasound image, wherein the label information includes: a sample mask of the liver structure and coordinates of key points of the sample liver posterior capsule; The training module is used to train the neural network model by taking the sample ultrasound image in each sample data as input and the label information corresponding to each sample data as output.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images