Device and method for determining image object information in blood vessel cavity and storage medium

Abstract

The invention discloses a device and method for determining image object information in a blood vessel cavity and a storage medium. The device comprises a medical image acquisition module, an object information acquisition module and a measurement parameter acquisition module, the medical image acquisition module is used for acquiring a medical image; the object information acquisition module is used for acquiring object information of a medical object in the medical image; the measurement parameter acquisition module acquires measurement parameters of the medical subject based on the subject information, the measurement parameters including a size and a position of the medical subject. Under the condition, the detection frame is used as the object information of the medical object, so that the use of labeling and identification of the pixel level can be reduced, and the labor cost and the time cost can be reduced. In addition, an automatic scheme is used, so that the recognition and analysis process has high speed and certainty, and the influence of subjective factors can be reduced.

Description

Technical Field

[0001] The present disclosure generally relates to the field of medical devices, and more particularly to an apparatus, method and storage medium for determining image object information within a blood vessel cavity. Background Art

[0002] Intravascular ultrasound (IVUS) and vascular optical coherence tomography (IVOCT) can effectively help doctors observe the characteristics of lesions inside blood vessels. For example, the intravascular images obtained by IVUS and IVOCT can help determine the location and severity of vascular calcification lesions before surgery, and help analyze the nature of the lesions.

[0003] At present, the methods for processing intravascular images obtained by IVUS and IVOCT include identification and analysis by experienced doctors and recognition based on deep learning image segmentation models. For example, Patent Document 1 with Publication No. CN114727802A discloses a method for processing intravascular ultrasound images, which includes: collecting multiple ultrasound images of blood vessels; segmenting the ultrasound images with a processor. Patent Document 1 needs to adopt a segmentation method based on a deep neural network, so it is necessary to annotate the intravascular image objects at the pixel level.

[0004] However, identification and analysis by experienced doctors may be susceptible to human subjective factors, and may be time-consuming and have low repeatability. Image segmentation models based on deep learning require pixel-level annotation and recognition, which may require high labor and time costs. Summary of the invention

[0005] The present disclosure is proposed in view of the above-mentioned prior art conditions, and its purpose is to provide a method, device and storage medium for determining image object information in a vascular cavity, which can improve recognition speed and reduce manpower cost and time cost.

[0006] To this end, the first aspect of the present disclosure provides a device for determining object information of an intravascular image, the device comprising: a medical image acquisition module, an object information acquisition module and a measurement parameter acquisition module; the medical image acquisition module is used to acquire medical images, the medical images comprising intravascular images; the object information acquisition module is used to acquire object information of medical objects in the medical images, the object information comprising a detection frame having a preset shape for selecting the medical object, and shape parameters and position parameters of the detection frame; the measurement parameter acquisition module acquires measurement parameters of the medical object based on the object information, the measurement parameters comprising the size and position of the medical object.

[0007] In the first aspect of the present disclosure, the measurement parameters of the medical object can be acquired based on the detection frame. Among them, the medical image acquisition module, the object information acquisition module and the measurement parameter acquisition module cooperate with each other to achieve the acquisition and analysis of medical images, which can not only reduce the influence of subjective factors, but also realize automation, and at the same time make the recognition and analysis process have higher speed and certainty. In addition, since the general outline of the medical object located in the vascular cavity has a certain regularity, for example, the medical object may be distributed on the vascular wall, therefore, the use of a detection frame with a preset shape can conform to the general outline of the medical object, so that the shape parameters and position parameters of the detection frame can be used to characterize the characteristics of the medical object. In this case, using the detection frame as the object information of the medical object can reduce the use of pixel-level annotation and recognition in the early training, or can not perform pixel-level annotation and recognition, thereby reducing labor costs and time costs.

[0008] In addition, in the device involved in the first aspect of the present disclosure, optionally, the intravascular image includes image data acquired by intravascular ultrasound and vascular optical coherence tomography and image data of the same blood vessel acquired by intravascular ultrasound and vascular optical coherence tomography, and the image data includes one frame of image data, multiple frames of image data and image retrieval data. In this case, it can be applicable to different types of image data acquired by intravascular ultrasound and vascular optical coherence tomography, and has a wide applicability.

[0009] In addition, in the device involved in the first aspect of the present disclosure, optionally, the object information acquisition module includes an object information determination model, and the object information determination model includes a trained deep learning model, and the deep learning model is trained using training samples with the object information. In this case, compared with the traditional machine learning image processing model, the use of modern machine learning methods (deep learning models) can be trained through large data sets, can reduce dependence on manually designed features, can improve the robustness and accuracy of recognition and analysis, and improve the efficiency of recognition and analysis.

[0010] In addition, in the device involved in the first aspect of the present disclosure, optionally, the deep learning model includes a sector fast regional convolutional network (Sector Faster R-CNN), a sector retina U-net (Sector Retina U-Net), a sector center network (Sector CenterNet) and a sector residual network (Sector ResNet). In this case, a variety of neural networks are used to obtain a deep learning model, which can be selected according to the requirements of the specific scenario for accuracy and speed and the actual situation of the running device.

[0011] In addition, in the device involved in the first aspect of the present disclosure, optionally, the training method of the object information determination model includes: manually identifying and marking the medical objects in the training samples; extracting the object information of the training samples based on the marking results, and taking the object information of the training samples obtained after extraction as the gold standard result; inputting the training samples into the deep learning model to obtain the prediction results of the object information of the training samples; calculating the loss between the prediction results and the gold standard results; adjusting the network parameters of the deep learning model based on the loss and continuing to train the deep learning model until the loss is below a preset threshold or convergence is reached. In this case, training the machine learning model based on the results of manual identification and marking can make full use of the experience of manual identification. At the same time, using detection boxes to mark medical objects can reduce the use of pixel-level annotation and identification, further reducing labor costs.

[0012] In addition, in the device involved in the first aspect of the present disclosure, optionally, the detection frame is a fan-shaped detection frame. In this case, for a medical object (such as a vascular calcification lesion) located on the vascular wall, from the cross-section of the blood vessel, the fan-shaped detection frame can completely select the medical object and clearly display the physical properties of the medical object (such as length and size, etc.). In addition, there are fewer parameters related to the shape and position of the fan-shaped detection frame, which can simplify the recognition process and reduce time costs.

[0013] In addition, in the device involved in the first aspect of the present disclosure, optionally, the measurement parameters also include the length, grade and score of the medical object. In this case, obtaining the length of the medical object can help determine the specific nature of the medical object, obtaining the grade of the medical object can help determine the severity of the medical object, and obtaining the score of the medical object can more comprehensively determine the situation of the medical object, thereby being able to meet the needs of auxiliary medical practice and research.

[0014] In addition, in the device involved in the first aspect of the present disclosure, optionally, the measurement parameter acquisition module includes a measurement parameter calculation model, and the measurement parameter calculation model converts the object information into the measurement parameter based on the conversion relationship. In this case, the measurement parameter is obtained using the parameter calculation model, and the measurement parameter can be automatically obtained.

[0015] The second aspect of the present disclosure also provides a method for determining object information of an intravascular image, the method comprising: acquiring a medical image, the medical image comprising an intravascular image; acquiring object information of a medical object in the medical image, the object information comprising a detection frame having a preset shape for selecting the medical object, and shape parameters and position parameters of the detection frame; and acquiring measurement parameters of the medical object based on the object information, the measurement parameters comprising the size and position of the medical object.

[0016] In the second aspect of the present disclosure, the measurement parameters of the medical object can be obtained based on the detection frame. Among them, the medical image acquisition module, the object information acquisition module and the measurement parameter acquisition module cooperate with each other to achieve the acquisition and analysis of medical images, which can not only reduce the influence of subjective factors, but also realize automation, and at the same time make the recognition and analysis process have higher speed and certainty. In addition, since the general outline of the medical object located in the vascular cavity has a certain regularity, for example, the medical object may be distributed on the vascular wall, therefore, the use of a detection frame with a preset shape can conform to the general outline of the medical object, so that the shape parameters and position parameters of the detection frame can be used to characterize the characteristics of the medical object. In this case, the detection frame is used as the object information of the medical object. In the early training, the use of pixel-level annotation and recognition can be reduced, or pixel-level annotation and recognition can be omitted, thereby reducing labor costs and time costs.

[0017] The third aspect of the present disclosure further provides a storage medium, wherein the storage medium stores computer program instructions, and when a processor runs the computer program instructions, the method described in the second aspect of the present disclosure is executed. In this case, the method involved in the second aspect of the present disclosure can be applied to a system that can run a computer program.

[0018] According to the present disclosure, a method, device and storage medium for determining image object information within a blood vessel cavity are provided, which can improve recognition speed and reduce manpower cost and time cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present disclosure will now be explained in further detail by way of example only with reference to the accompanying drawings, in which:

[0020] Figure 1 It is a schematic diagram showing the application scenario of the intravascular ultrasound system involved in the example of the present disclosure.

[0021] Figure 2 is a system block diagram showing an apparatus for determining intravascular image object information according to an example of the present disclosure.

[0022] Figure 3 is a general flow chart showing the acquisition method involved in the example of the present disclosure.

[0023] Figure 4 is a schematic diagram showing a fan-shaped detection box prediction model based on a fan-shaped fast region convolutional network involved in the example of the present disclosure.

[0024] Figure 5 is a schematic diagram showing the fan-shaped proposal generation process of the fan-shaped fast region convolutional network involved in the example of the present disclosure.

[0025] Figure 6 is a schematic diagram showing a fan-shaped detection box prediction model based on a fan-shaped residual network involved in the example of the present disclosure.

[0026] Figure 7 is a schematic diagram showing a fan-shaped detection box prediction model of a fan-shaped retinal U-shaped network involved in the example of the present disclosure.

[0027] Figure 8 is a schematic diagram showing a fan-shaped detection box prediction model of a fan-shaped center network involved in the example of the present disclosure.

[0028] Fig. 9 is a flow chart showing a training method involved in the examples of the present disclosure.

[0029] Fig. 10A is a schematic diagram of a training sample of vascular calcification lesions before labeling involved in the examples of the present disclosure.

[0030] Fig. 10B is a schematic diagram of a training sample of labeled vascular calcification lesions involved in the examples of the present disclosure.

[0031] Fig. 10C Schematic diagram of a training sample of a blood vessel branch before labeling involved in the example of the present disclosure.

[0032] Fig. 10D Schematic diagram of a training sample of labeled blood vessel branches involved in the example of the present disclosure.

[0033] Fig.11 It is a schematic diagram showing the process of obtaining measurement parameters of a medical image containing calcified plaques involved in the example of the present disclosure.

[0034] Fig.12It is a schematic diagram showing the process of obtaining measurement parameters of a medical image containing blood vessel branches involved in the example of the present disclosure.

[0035] Fig.13 It is a schematic diagram showing a process of obtaining measurement parameters of a medical image containing uneven rotation deformation artifacts involved in an example of the present disclosure.

[0036] Description of reference numerals:

[0037] Intravascular ultrasound system…100, imaging device…110, transmission shaft…120, motor…130, signal processing system…140, device for determining intravascular image object information…200, medical image acquisition module…210, object information acquisition module…220, measurement parameter acquisition module…230, input image…401, basic network convolution layer…402, feature image…403, fan proposal…404, sliding window…501, N fan anchor frames…502, residual network…601, object detection module…602, position prediction network of fan detection frame arc length bisector…6021, length prediction network of fan detection frame half arc length…6022, U-shaped feature pyramid network…701 , convolutional subnetwork module…702, convolutional fan-shaped anchor box classification subnetwork…7021, convolutional fan-shaped anchor box regression subnetwork…7022, semantic segmentation module…703, arc mask…A, calcified plaque…B, arc…C, vascular branch…D, medical image containing calcified plaque…1101, object information determination model…1102, medical image containing calcified lesions with labeled information…1103, measurement parameter calculation model…1104, medical image containing vascular branches…1201, medical image containing vascular branches with labeled information…1202, medical image containing uneven rotation deformation artifacts…1301, medical image containing uneven rotation deformation artifacts with labeled information…1302. DETAILED DESCRIPTION

[0038] Hereinafter, the preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same symbols are assigned to the same components, and repeated descriptions are omitted. In addition, the accompanying drawings are only schematic diagrams, and the ratio of the dimensions of the components or the shapes of the components may be different from the actual ones.

[0039] It should be noted that the terms "including" and "having" and any variations thereof in the present disclosure, such as a process, method, system, product or device that includes or has a series of steps or units, are not necessarily limited to those steps or units clearly listed, but may include or have other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] The intravascular images involved in the examples of the present disclosure may be medical images of the blood vessels obtained using intravascular ultrasound (IVUS) technology and vascular optical coherence tomography (IVOCT) technology. For example, the intravascular images may be obtained using the intravascular ultrasound system and vascular optical coherence tomography system involved in the examples of the present disclosure.

[0041] Figure 1 is a schematic diagram showing an application scenario of the intravascular ultrasound system 100 involved in the example of the present disclosure.

[0042] In some examples, such as Figure 1 As shown, the intravascular ultrasound system 100 may include an imaging device 110, a transmission shaft 120, a motor 130, and a signal processing system 140. The imaging device 110 may be placed in a blood vessel and transmit imaging signals to the blood vessel wall while rotating and receive imaging signals reflected by the blood vessel wall to obtain a vascular ultrasound image in the blood vessel. The transmission shaft 120 may be connected to the imaging device 110 and extend outside the body along the blood vessel. The motor 130 may be located outside the body and may be connected to the transmission shaft 120, and the motor 130 actuates the imaging device 110 via the transmission shaft 120.

[0043] In some examples, the imaging device 110 can be introduced into a blood vessel and can transmit an imaging signal to the blood vessel wall. In some examples, the imaging device 110 can be an ultrasonic transducer, and the imaging signal can be an ultrasonic signal (e.g., an ultrasonic pulse). In some examples, the ultrasonic transducer can transmit an ultrasonic pulse to the blood vessel wall along the radial direction of the blood vessel, and receive an ultrasonic echo with blood vessel information reflected by the blood vessel wall. In other embodiments, the imaging signal can also be an optical signal, an electromagnetic wave signal, etc.

[0044] In some examples, the motor 130 can actuate the imaging device 110 via the transmission shaft 120. In some examples, the motor 130 can actuate the imaging device 110 to rotate via the transmission shaft 120, and the motor 130 can actuate the imaging device 110 to retract along the extension direction of the blood vessel via the transmission shaft 120.

[0045] In some examples, the imaging device 110 can transmit the imaging signal to the signal processing system 140. In some examples, the signal processing system 140 can process the imaging signal, for example, processing the imaging signal into an intravascular image or image data associated with the intravascular image. In some examples, the intravascular image can be a cross-sectional image of the blood vessel. In some examples, the signal processing system 140 can also be connected to a display to display the intravascular image on the display.

[0046] In some examples, the signal processing system 140 may process the imaging signal into a frame of image, multiple frames of image, or image retrieval data. In some examples, the image retrieval data may represent an image sequence composed of multiple frames of image that are continuous in time and in order (that is, it may be understood as a data similar to a video stream, used to describe the cross-sectional information of a blood vessel).

[0047] In some examples, the intravascular lumen image or image data associated with the intravascular lumen image acquired by the intravascular ultrasound system 100 can be used to obtain physiological or medical information related to the blood vessel, such as qualitative or quantitative properties of vascular branches and vascular lesions, etc. In some examples, the vascular lesions can include vascular calcification lesions.

[0048] In addition, a vascular optical coherence tomography system (not shown) can also be used to obtain intravascular images (i.e., cross-sectional images of blood vessels). The form and function of the intravascular images obtained by the system are similar to those described above for the intravascular images obtained by the intravascular ultrasound system 100, and are not described in detail here.

[0049] Figure 2 is a system block diagram showing an apparatus 200 for determining image object information within a blood vessel cavity according to an example of the present disclosure.

[0050] In some examples, such as Figure 2 As shown, the device 200 for determining the intravascular image object information (may be simply referred to as the device 200 ) may include a medical image acquisition module 210 , an object information acquisition module 220 , and a measurement parameter acquisition module 230 .

[0051] In some examples, the medical image acquisition module 210 may be used to acquire medical images. In some examples, the medical images may include intravascular images and image data associated with the intravascular images.

[0052] In some examples, as described above, the intravascular images acquired by the medical image acquisition module 210 may include image data acquired by intravascular ultrasound (e.g., intravascular ultrasound imaging system 100) and vascular optical coherence tomography, as well as image data of the same blood vessel acquired by intravascular ultrasound and vascular optical coherence tomography. The image data includes one frame of image data, multiple frames of image data, and image retrieval data. In this case, it can be applicable to different types of image data acquired by intravascular ultrasound and vascular optical coherence tomography, and has a wide range of applicability.

[0053] In some examples, the object information acquisition module 220 can be used to acquire object information of medical objects in medical images. As described above, in some examples, medical objects can include vascular branches (also referred to as vascular side branch structures), vascular lesions, and non-uniform rotational distortion (NURD) artifacts. In some examples, vascular lesions can include vascular calcification lesions (i.e., vascular calcification plaques), thrombi, and ruptured plaques.

[0054] In some examples, when there is no ambiguity, a vascular calcified lesion may also be referred to as a calcified lesion. Accordingly, a vascular calcified plaque may also be referred to as a calcified plaque or plaque.

[0055] In some examples, the object information may include a detection box having a preset shape for selecting a medical object, shape parameters of the detection box, and position parameters of the detection box. In some examples, the detection box may be a bounding box surrounding an area of ​​interest (i.e., an area where the medical object is located). In some examples, the form of the detection box may include a rectangle, a rotated rectangle, a circle, a sector, etc. In some examples, the shape parameters of the detection box may be the shape of the detection box. In some examples, the position parameters of the detection box may include the position of the detection box in the medical image expressed in coordinates. In some examples, the coordinates may include Cartesian coordinates and polar coordinates, etc.

[0056] In some examples, the detection frame may be a sector-shaped detection frame (also referred to as an object sector-shaped detection frame). In this case, from the cross-section of the blood vessel, for a medical object (such as a vascular calcified lesion) located on the blood vessel wall, the sector-shaped detection frame can be used to completely select it, and the physical properties of the medical object (such as length and size, etc.) can be clearly displayed. In addition, there are fewer parameters related to the shape and position of the sector-shaped detection frame, which can simplify the recognition process and reduce time costs.

[0057] In some examples, the measurement parameter acquisition module 230 may acquire measurement parameters of the medical object based on the object information. In some examples, the measurement parameters may include the size and position of the medical object.

[0058] In some examples, the size of the medical object may be the length of the medical object on the circumference formed by the cross section of the blood vessel wall in the blood vessel cross-sectional image (i.e., the arc length) or the angle corresponding to the arc (i.e., the angle is calculated based on the proportion of the arc length on the circumference formed by the entire blood vessel wall cross section). In some examples, the size of the medical object may also be referred to as the coverage of the medical object.

[0059] In some examples, the position of the medical object may include the position of the medical object on the circumference formed by the cross section of the blood vessel wall in the blood vessel cross-sectional image, including the starting point and the end point of the medical object on the circumference formed by the cross section of the blood vessel wall. In some examples, the position of the medical object may also include the position of a point on the medical object (e.g., a point that divides the arc length equally) on the circumference formed by the blood vessel wall.

[0060] In some examples, the location of the medical object is determined by dividing the circumference formed by the cross section of the blood vessel wall into several segments, and then taking the segment where the arc length bisector passing through the arc dividing point and the center of the circumference is located as the location of the medical object. In some examples, the circumference can be divided into 12 segments, and each segment is referred to by the position of the hour hand of a clock (for example, the segment at the bottom of the circumference can be called 6 o'clock).

[0061] In some examples, the measurement parameters may also include the length, grade, and score of the medical object.

[0062] In some examples, the length of the medical object may be the length of the medical object in the direction of blood flow or in the direction opposite to the direction. In some examples, the length of the medical object may be based on the length obtained from multiple frames of medical images or pullback data of medical images. In some examples, different lengths of several medical objects in the entire pullback data may be calculated on a time axis or a length axis based on the condition of the medical object (e.g., vascular calcification lesions) in each frame of the pullback data.

[0063] In some examples, the classification can be a classification specified based on recognized standards. Taking vascular calcification lesions as an example, the classification of vascular calcification lesions can refer to the vascular calcification lesion classification method of the "Chinese Expert Consensus on the Application of Intravascular Ultrasound in Coronary Artery Disease (2018)" and the "Chinese Expert Consensus on the Diagnosis and Treatment of Coronary Artery Calcification Lesions (2021 Edition)", and convert the size and other properties of the vascular calcification lesions into the vascular calcification lesion classification.

[0064] In some examples, the score of the medical object may be a score obtained by comprehensively considering the size, position, length and classification of the medical object described above. In some examples, each of the above different aspects of the properties of the medical object has a scoring standard, and the sub-score corresponding to each of the above properties can be obtained based on the scoring standard. In some examples, the score of the medical object may be the sum of all the above sub-scores. In some examples, the score of the medical object may also be the smallest or largest item among all the above sub-scores.

[0065] In some examples, the method for calculating the score of vascular calcification lesions can refer to a calcification score calculation method described in "Intravascular Ultrasound-Derived Calcium Score to Predict Stent Expansion in Severely Calcified Lesions", which involves scoring methods related to calcification lesion angle, calcification lesion length, calcification nodules and blood vessel diameter.

[0066] Among the measurement parameters involved in the present disclosure, obtaining the length of a medical object can help determine the specific nature of the medical object, obtaining the grade of a medical object can help determine the severity of the medical object, and obtaining the score of a medical object can more comprehensively determine the condition of the medical object. Thus, it can meet the needs of auxiliary medical practice and research.

[0067] The present disclosure also provides a method for determining image object information within a blood vessel cavity.

[0068] The following describes in detail a method for acquiring object information and measurement parameters of a medical object by using the apparatus 200 for determining intravascular image object information (hereinafter referred to as an acquisition method).

[0069] Figure 3 is a general flow chart showing the acquisition method involved in the example of the present disclosure.

[0070] like Figure 3 As shown, the specific steps of the acquisition method may include: acquiring a medical image (step S100), acquiring object information of a medical object in the medical image (step S200), and acquiring measurement parameters of the medical object based on the object information (step S300).

[0071] In some examples, in step S100, medical images may be acquired using the medical image acquisition module 210. In some examples, the medical image acquisition module 210 may be connected to an imaging device of a medical image to acquire the medical image. In some examples, the medical image acquisition module 210 may be connected to the signal processing system 140 of the intravascular ultrasound system 100 to acquire an intravascular image or image data associated with the intravascular image after the signal processing system 140 processes the imaging signal. In some examples, the medical image acquired by the medical image acquisition module 210 from the signal processing system 140 may include real-time image data or non-real-time image data.

[0072] In some examples, the medical image acquisition module 210 can be connected to a device storing medical images to acquire medical images. In some examples, the medical image acquisition module 210 can be connected to one or more microcomputers (e.g., desktop computers storing medical images in a hospital or research institution) to acquire medical images. In other examples, the medical image acquisition module 210 can be connected to one or more servers storing medical images to acquire medical images.

[0073] In some examples, in step S200, object information in the medical image may be acquired using the object information acquisition module 220. In some examples, as described above, the object information may include a sector-shaped detection frame.

[0074] In some examples, the object information acquisition module 220 may include an object information determination model 1102. In some examples, the object information determination model 1102 includes a trained deep learning model. The deep learning model is trained using training samples with object information. In this case, compared to the image processing model of traditional machine learning, the use of modern machine learning methods (deep learning models) can be trained through large data sets, can reduce the reliance on manually designed features, can improve the robustness and accuracy of recognition and analysis, and improve the efficiency of recognition and analysis.

[0075] In some examples, the deep learning models include Sector Faster R-CNN, Sector Retina U-Net, Sector CenterNet, and Sector ResNet. In some examples, the above deep learning models may also be referred to as sector detection box prediction models.

[0076] In some examples, the above-mentioned fan-shaped detection frame prediction model can be a modified version of the corresponding neural network used by the model for predicting the fan-shaped detection frame. That is, the fan-shaped fast regional convolutional network corresponds to the fast regional convolutional network (Faster R-CNN); the fan-shaped retinal U-type network corresponds to the retinal U-type network (Retina U-Net); the fan-shaped center network corresponds to the center network (CenterNet); and the fan-shaped residual network corresponds to the residual network (ResNet). In this case, a variety of neural networks are used to obtain a deep learning model, which can be selected according to the requirements of the specific scene for accuracy and speed and the actual situation of the running device.

[0077] The fan-shaped detection box prediction model is described in detail below in conjunction with the accompanying drawings.

[0078] Figure 4is a schematic diagram showing a fan-shaped detection box prediction model based on a fan-shaped fast region convolutional network involved in the example of the present disclosure. Figure 5 is a schematic diagram showing a process of generating a fan-shaped proposal 404 of a fan-shaped fast region convolutional network involved in an example of the present disclosure.

[0079] In some examples, the fan-shaped detection box prediction model can include a deep full convolutional network model and a fan-shaped fast region convolutional network classification model. In some examples, the deep full convolutional network model includes a fan-shaped proposal network, which can be used to generate fan-shaped proposals 404.

[0080] In some examples, the fan-shaped proposal 404 may be a fan-shaped anchor box (i.e., a preset fan-shaped detection box with different sizes and positions, not a fan-shaped annotation area or a gold standard fan-shaped box) with a classification value and a regression value (described later). In some examples, the fan-shaped fast region convolution network classification model may determine the fan-shaped detection box based on the fan-shaped proposal 404. In some examples, the area where the medical object is located in the medical image may also be referred to as a region of interest.

[0081] In some examples, such as Figure 4 As shown, after the input image 401 is input into the fan-shaped detection frame prediction model, the feature image 403 is extracted by the basic network convolution layer 402.

[0082] In some examples, such as Figure 5 As shown, the fan proposal network uses a sliding window 501 to scan each position of the feature image 403, and each position can correspond to N fan-shaped anchor boxes 502, identify whether the N fan-shaped anchor boxes 502 at each position contain a medical object, and use probability p to represent the probability that each fan-shaped anchor box in the N fan-shaped anchor boxes 502 contains a medical object.

[0083] In some examples, there are 2 classification values ​​and 2 regression values ​​corresponding to 1 fan-shaped anchor box.

[0084] In some examples, the classification value may include a probability p that a fan-shaped anchor box contains a medical object and a probability 1-p that the fan-shaped anchor box does not contain a medical object.

[0085] In some examples, the regression value includes an angle θ between an arc length bisector of a sector-shaped anchor frame and a 0-angle line and a half arc length L of the sector-shaped anchor frame.

[0086] In some examples, the data obtained by scanning the sliding window 501 can be separated into 128-dimensional features through the intermediate layer, and the 128-dimensional features can be separated into 2N classification values ​​and 2N regression values ​​corresponding to N sector scanning frames 502 through the classification layer and the regression layer, respectively, and the sector proposal 404 is obtained from the above 2N classification values ​​and 2N regression values.

[0087] Continue to refer Figure 4 The region of interest pooling process can extract the sector region features of the feature image 403 based on the sector proposal 404, and pool the sector region features into a feature vector with d dimensions. The d-dimensional feature vector enters the next region of interest classification process, further identifying whether the sector proposal 404 contains a medical object, and adjusting the position of the sector proposal detection frame to better surround the medical object to obtain a sector detection frame.

[0088] Figure 6 is a schematic diagram showing a fan-shaped detection box prediction model based on a fan-shaped residual network involved in the example of the present disclosure.

[0089] In some examples, such as Figure 6 As shown, after the input image 401 is input into the sector residual network, the residual network 601 performs a series of convolution processing to extract convolution features. The object detection module 602 can identify the sector detection frame based on the convolution features.

[0090] In some examples, the object detection module 602 may include two submodules: a position prediction network 6021 of a sector-shaped detection frame arc length bisector and a length prediction network 6022 of a half arc length of the sector-shaped detection frame.

[0091] In some examples, the position prediction network 6021 of the arc length bisector of the fan-shaped detection frame can predict the arc length bisector of the fan-shaped detection frame of the possible medical object. In some examples, the length prediction network 6022 of the half arc length of the fan-shaped detection frame can predict the length of the half arc length of the fan-shaped detection frame of the possible medical object. In some examples, the fan-shaped detection frame can be obtained based on the prediction results of the above two submodules.

[0092] Figure 7 is a schematic diagram showing a fan-shaped detection box prediction model of a fan-shaped retinal U-shaped network involved in the example of the present disclosure.

[0093] In some examples, such as Figure 7 As shown, after the input image 401 is input into the fan-shaped retinal U-shaped network, the U-shaped feature pyramid network 701 performs a series of convolution processing on the input image 401 to extract convolution features. The convolution sub-network module 702 can identify the fan-shaped detection frame based on the convolution features.

[0094] In some examples, the convolutional subnetwork module 702 may include two submodules: a convolutional fan-shaped anchor box classification model 7021 and a convolutional fan-shaped anchor box regression model 7022.

[0095] In some examples, the convolutional fan-shaped anchor box classification model 7021 can classify the fan-shaped anchor box. In some examples, the convolutional fan-shaped anchor box regression model 7022 can match the position and angle of the fan-shaped anchor box with the position and angle of the gold standard fan-shaped detection box as much as possible.

[0096] In some examples, the semantic segmentation module 703 can provide semantic segmentation supervision, so that the U-shaped feature pyramid network 701 can better extract features, further promoting the classification and regression of the fan-shaped anchor box. In some examples, the fan-shaped module classification and regression results at different feature scales can be post-processed, including result fusion and non-maximum suppression, to determine the final fan-shaped detection box.

[0097] Figure 8 is a schematic diagram showing a fan-shaped detection box prediction model of a fan-shaped center network involved in the example of the present disclosure.

[0098] In some examples, such as Figure 8 As shown, after the input image 401 is input into the fan-shaped detection frame prediction model of the fan-shaped center network, the U-shaped feature pyramid network 701 performs a series of convolution processing on the input image 401 to extract convolution features. The object detection module 602 can identify the fan-shaped detection frame based on the convolution features. For the object detection module 602, please refer to the above Figure 6 The description is not repeated here.

[0099] It should be noted that the above examples and descriptions of the deep learning model (ie, the fan-shaped detection box prediction model) are not restrictive descriptions of the types of deep learning models. The deep learning model may also be of other types.

[0100] In some examples, the present disclosure also provides a method for training the object information determination model 1102 .

[0101] The training method (also referred to as training method) of the object information determination model 1102 is described in detail below by taking the above-mentioned fan-shaped residual network as an example and combining with the accompanying drawings.

[0102] Fig. 9 is a flow chart showing a training method involved in the examples of the present disclosure.

[0103] In some examples, such as Fig. 9As shown, the specific steps of the training method may include: manually identifying and marking medical objects in training samples (step S210); extracting object information of the training samples based on the marking results, and taking the object information of the training samples obtained after extraction as the gold standard result (step S220); inputting the training samples into the deep learning model to obtain the prediction results of the object information of the training samples (step S230); calculating the loss between the prediction results and the gold standard results (step S240); adjusting the network parameters of the deep learning model based on the loss and continuing to train the deep learning model until the loss is below a preset threshold or convergence is reached (step S250).

[0104] In the training method disclosed herein, the machine learning model is trained based on the results of manual recognition and labeling, which can make full use of the experience of manual recognition. At the same time, the use of detection frames to label medical objects can reduce the use of pixel-level annotation and recognition, further reducing labor costs.

[0105] In some examples, in step S210 , medical objects in the training samples may be manually identified and labeled.

[0106] In some examples, the subject of manual identification and labeling may include an interventionalist. In this case, the interventionalist has more relevant knowledge and experience, and thus can use the relevant knowledge and experience to achieve better training results.

[0107] In some examples, the manual labeling method may include arc mask labeling (described later). Fig. 10A is a schematic diagram of a training sample of vascular calcification lesions before labeling involved in the examples of the present disclosure. Fig. 10B is a schematic diagram of a training sample of labeled vascular calcification lesions involved in the examples of the present disclosure. Fig. 10C Schematic diagram of a training sample of a blood vessel branch before labeling involved in the example of the present disclosure. Fig. 10D Schematic diagram of a training sample of labeled blood vessel branches involved in the example of the present disclosure.

[0108] In some examples, such as Fig. 10A As shown, the training sample before labeling may have an arc mask A and a calcified plaque B.

[0109] In some examples, the arc mask A may be a circular dark line located at the periphery of the blood vessel cross-sectional image. Fig. 10B As shown, arc C can be marked on arc mask A based on the position, size and shape of calcified plaque B. In some examples, the brightness of the pixels on arc C can be greater than the brightness of the pixels on arc mask A (including arc mask A before and after marking). In some examples, arc C can have a starting position and an ending position.

[0110] In other examples, such as Fig. 10C As shown, the training sample before labeling may have an arc mask A and a blood vessel branch D. In some examples, such as Fig. 10D As shown, arc C can be marked on arc mask A based on the position, size and shape of blood vessel branch D.

[0111] In this case, the object information is obtained based on the arc mask marking method, which can obtain the object information more simply and quickly.

[0112] In some examples, in step S220, object information (ie, the fan-shaped detection box) of the training sample may be extracted based on the labeling result, and the object information of the training sample obtained after the extraction is used as the gold standard result.

[0113] In some examples, in step S220 , relevant information of the sector-shaped detection frame may be extracted based on the results of recognition and labeling in step S210 .

[0114] In some examples, the relevant information may include the angle θ between the arc length bisector of the sector detection frame and the 0 angle line (i.e., the horizontal line passing through the center of the circle corresponding to the arc), and the value range of the angle θ may be [0, 2π). In some examples, the relevant information may also include the half arc length L of the sector detection frame.

[0115] In some examples, the relevant information of the sector detection frame extracted above can be used as the gold standard result of the object information of the training sample. In this case, obtaining the gold standard result of the object information based on the angle and the half arc length can reduce the use of parameters, making the acquisition process simpler and faster.

[0116] In some examples, in step S230, the training sample can be input into the deep learning model to obtain the prediction result of the object information of the training sample. For the introduction of the deep learning model, please refer to the description of the fan-shaped detection box prediction model above, which will not be repeated here.

[0117] In some examples, in step S240, the loss between the predicted result and the gold standard result may be calculated. In some examples, when calculating the loss, the functions that may be used include the mean square error loss function (MeanSquaredErrorLoss), the mean absolute error loss function (MeanAbsoluteErrorLoss), and the like.

[0118] In step S250, the network parameters of the deep learning model can be adjusted based on the loss and the deep learning model can be continuously trained until the loss is below a preset threshold or convergence is reached.

[0119] In some examples, optimizers that can be used to adjust network parameters of deep learning models include stochastic gradient descent (SGD) optimizer and adaptive moment estimation (Adam) optimizer, among others.

[0120] In some examples, when the loss is below a preset threshold, the training of the deep learning model can be considered complete and can be put into use. In other examples, when the loss reaches convergence, the training of the deep learning model can be considered complete and can be put into use.

[0121] In some examples, training of a deep learning model can be considered complete when the loss reaches convergence.

[0122] Return to reference Figure 3 In step S300, the measurement parameter acquisition module 230 can acquire the measurement parameters of the medical object based on the object information.

[0123] In some examples, the types of measurement parameters included may be selected based on the acquisition method and usage requirements of the medical image (in this case, these selected measurement parameters may be referred to as measurement parameters of interest). For example, for a medical image containing vascular calcified plaques acquired by the intravascular ultrasound system 100, the aspects of interest to the interventional physician include measurement parameters of interest such as the location, angle, grade, length, and score of the vascular calcified plaques. The description of these parameters of interest is detailed above.

[0124] In some examples, the measurement parameters can be used for preoperative auxiliary diagnosis and evaluation of percutaneous coronary intervention (PCI), intraoperative treatment guidance, and postoperative evaluation of treatment effects.

[0125] In some examples, the measured parameters can provide a more effective reference for clinical research activities including predicting stent malexpansion based on the calcium score of the intravascular ultrasound system 100 .

[0126] In some examples, the measurement parameter acquisition module 230 may further include a measurement parameter calculation model 1104. In some examples, the measurement parameter calculation model 1104 converts the object information into the measurement parameter based on the conversion relationship. The conversion relationship may refer to a model used when the measurement parameter is obtained by conversion based on the object information. In this case, the measurement parameter is obtained using the measurement parameter calculation model 1104, and the measurement parameter can be automatically obtained.

[0127] In some examples, the measurement parameter calculation model 1104 may include a model established based on a formula and a model established based on machine learning.

[0128] Fig.11It is a schematic diagram showing the process of obtaining measurement parameters of a medical image 1101 containing calcified plaques involved in the example of the present disclosure. Fig.12 1 is a schematic diagram showing a process of acquiring measurement parameters of a medical image 1201 containing blood vessel branches according to an example of the present disclosure.

[0129] In some examples, reference Fig.11 Taking the cross-sectional image of a blood vessel acquired by the intravascular ultrasound system 100 as an example, the process of acquiring the measurement parameters of the medical image 1101 containing calcified lesions may be as follows: the medical image 1101 containing calcified lesions is input into the object information determination model 1102 to acquire the medical image 1103 containing calcified lesions with label information. The label information of the medical image 1103 containing calcified lesions with label information includes the catheter center (that is, the center of the catheter of the imaging device 110), the object sector detection frame, the 0-angle line, the arc length bisector, the semi-arc length L, and the angle θ between the arc length bisector and the 0-angle line.

[0130] In some examples, the measurement parameter calculation model 1104 may obtain the measurement parameters of interest of the calcified lesions based on the medical image 1103 containing the calcified lesions with label information.

[0131] In the above example, the results of the measurement parameters of interest include: the location of the calcified lesion is between 2 and 3 o'clock; the opening angle of the calcified lesion is 115°; the grade of the calcified lesion is grade II, etc.

[0132] In some examples, reference Fig.12 Taking the blood vessel cross-sectional image acquired by the intravascular ultrasound system 100 as an example, the process of acquiring the measurement parameters of the medical image 1201 containing the blood vessel branch may be: inputting the medical image 1201 containing the blood vessel branch into the object information determination model 1102 to acquire the medical image 1202 containing the blood vessel branch with label information. The label information of the medical image 1202 containing the blood vessel branch with label information includes the catheter center, the object sector detection frame, the 0-angle line, the arc length bisector, the half arc length L, and the angle θ between the arc length bisector and the 0-angle line.

[0133] In some examples, the measurement parameter calculation model 1104 may obtain the parameter of interest of the vascular branch based on the medical image 1202 containing the vascular branch and having label information.

[0134] In the above example, the results of the measurement parameters of interest include: the position of the blood vessel branch D is between 10 and 11 o'clock; the opening angle of the blood vessel branch is 85°, etc.

[0135] In some examples, the medical image 1101 including the calcified plaque and the medical image 1201 including the blood vessel branch may also be acquired by a vascular optical coherence tomography system.

[0136] In some examples, the apparatus 200 and acquisition method for determining intravascular image object information involved in the examples of the present disclosure may also be used to obtain measurement parameters of uneven rotational deformation artifacts in medical images acquired by the intravascular ultrasound system 100.

[0137] Fig.13 1 is a schematic diagram showing a process of obtaining measurement parameters of a medical image 1301 containing uneven rotation deformation artifacts according to an example of the present disclosure.

[0138] In some examples, reference Fig.13 Taking the blood vessel cross-sectional image acquired by the intravascular ultrasound system 100 as an example, the process of acquiring the measurement parameters of the medical image 1301 containing uneven rotation deformation artifacts may be as follows: the medical image 1301 containing uneven rotation deformation artifacts is input into the object information determination model 1102 to acquire the medical image 1302 containing uneven rotation deformation artifacts with label information. The label information of the medical image 1302 containing uneven rotation deformation artifacts with label information includes the catheter center, the object sector detection frame, the 0-angle line, the arc length bisector, the half arc length L, and the angle θ between the arc length bisector and the 0-angle line.

[0139] In some examples, the measurement parameter calculation model 1104 may obtain the non-uniform rotational deformation artifact parameter of interest based on the medical image 1302 containing non-uniform rotational deformation artifacts with label information.

[0140] In the above example, the parameters of interest that can be obtained include: the location of the uneven rotation deformation artifact is 5 to 8 o'clock; the opening angle of the uneven rotation deformation artifact is 95°, etc.

[0141] In summary, in the present disclosure, the measurement parameters of the medical object can be obtained based on the detection frame. Among them, the medical image acquisition module 210, the object information acquisition module 220 and the measurement parameter acquisition module 230 cooperate with each other to achieve the acquisition and analysis of medical images, which can not only reduce the influence of subjective factors, but also realize automation, and at the same time make the recognition and analysis process have higher speed and certainty.

[0142] In addition, since the rough outline of the medical object located in the blood vessel cavity has certain regularities, for example, the medical object may be distributed on the blood vessel wall. Therefore, using a detection frame with a preset shape can conform to the rough outline of the medical object, so that the shape parameters and position parameters of the detection frame can be used to characterize the characteristics of the medical object.

[0143] In this case, using the detection box as the object information of the medical object can reduce the use of pixel-level annotation and recognition in the early stage of training, or can eliminate the need for pixel-level annotation and recognition, thereby reducing labor costs and time costs.

[0144] In addition, the example of the present disclosure further provides a storage medium, which can store computer program instructions. When the processor runs the computer program instructions, it can perform at least one step of the above acquisition method. In this case, the present disclosure can be applied to a system that can run a computer program.

[0145] Although the present disclosure is specifically described above in conjunction with the accompanying drawings and examples, it is to be understood that the above description does not limit the present disclosure in any form. Those skilled in the art may modify and change the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes all fall within the scope of the present disclosure.

Claims

1. A device for determining information of an object in a blood vessel lumen, It is characterized in that The device includes: a medical image acquisition module, an object information acquisition module and a measurement parameter acquisition module; the medical image acquisition module is used to acquire medical images, and the medical images include intravascular images; the object information acquisition module is used to acquire object information of medical objects in the medical images, and the object information includes a detection frame with a preset shape for selecting the medical object, and shape parameters and position parameters of the detection frame; the measurement parameter acquisition module acquires measurement parameters of the medical object based on the object information, and the measurement parameters include the size and position of the medical object.

2. The device according to claim 1, It is characterized in that The intravascular image includes image data acquired by intravascular ultrasound and vascular optical coherence tomography and image data of the same blood vessel acquired by intravascular ultrasound and vascular optical coherence tomography, and the image data includes one frame of image data, multiple frames of image data and image pullback data.

3. The device according to claim 1, It is characterized in that The object information acquisition module includes an object information determination model, and the object information determination model includes a trained deep learning model, and the deep learning model is trained using training samples with the object information.

4. The device according to claim 3, It is characterized in that The deep learning model includes a sector fast regional convolutional network (Sector Faster R-CNN), a sector retina U-net (Sector Retina U-Net), a sector center network (Sector CenterNet) and a sector residual network (Sector ResNet).

5. The device according to claim 3, It is characterized in that The training method of the object information determination model includes: manually identifying and marking the medical objects in the training samples; extracting the object information of the training samples based on the marking results, and taking the object information of the training samples obtained after extraction as the gold standard result; inputting the training samples into the deep learning model to obtain the predicted results of the object information of the training samples; calculating the loss between the predicted results and the gold standard results; adjusting the network parameters of the deep learning model based on the loss and continuing to train the deep learning model until the loss is below a preset threshold or convergence is reached.

6. The device according to claim 1, It is characterized in that The detection frame is a fan-shaped detection frame.

7. The device according to claim 1, It is characterized in that The measured parameters also include the length, grade and score of the medical object.

8. The device according to claim 1, It is characterized in that The measurement parameter acquisition module includes a measurement parameter calculation model, and the measurement parameter calculation model converts the object information into the measurement parameters based on a conversion relationship.

9. A method for determining information of an intravascular image object, It is characterized in that The method includes: acquiring a medical image, wherein the medical image includes an intravascular image; acquiring object information of a medical object in the medical image, wherein the object information includes a detection frame having a preset shape for selecting the medical object, and shape parameters and position parameters of the detection frame; and acquiring measurement parameters of the medical object based on the object information, wherein the measurement parameters include the size and position of the medical object.

10. A storage medium, It is characterized in that The storage medium stores computer program instructions, and when the processor runs the computer program instructions, the method according to claim 9 is executed.

Images