Placenta accreta assessment system based on multimodal sign recognition and its construction method and construction device

Abstract

Disclosed is a placenta accreta assessment system based on multimodal sign recognition, belonging to the field of placenta accreta detection technology. The system includes a data collection module for obtaining detection text from multiple sets of placenta ultrasound images; a sign extraction module for respectively obtaining a first feature map and a second feature map using the placenta ultrasound images and the detection text; a cross-attention unit for fusing the multimodal features of the feature maps to obtain a third feature map; a feature aggregation unit for fusing the multi-stage feature maps to obtain a fourth feature map; a classification unit for outputting a segmentation mask for placenta accreta signs based on the fourth feature map; and an assessment module for determining the severity of the placenta accreta sign corresponding to the segmentation mask to obtain a placenta accreta assessment result for the subject. The system provides a placenta accreta sign segmentation result based on the subject's ultrasound imaging and text indicators, providing medical personnel with an effective placenta accreta assessment reference and avoiding diagnostic errors caused by subjective misjudgment by medical personnel in primary healthcare institutions.

Description

Technical Field

[0001] The present application belongs to the technical field of placenta accreta detection, and specifically relates to a placenta accreta assessment system based on multimodal sign recognition. Background Art

[0002] Placenta accreta refers to a condition in which placental villi invade beyond the upper third of the uterine myometrium, or even penetrate the myometrium or serosa. In severe cases, this condition may cause rectal or bladder invasion. The resulting placenta accreta disease is a major cause of serious adverse outcomes in women of childbearing age, including hysterectomy, postpartum hemorrhage, and death. This condition has been increasing with the rise in cesarean section rates in recent years. Currently, risk assessment of placenta accreta forms the basis for tiered management of placenta accreta disease and surgical decision-making, and ultrasound examination is the most economical and commonly used preoperative assessment method for placenta accreta.

[0003] Retrospective studies have established placenta accreta evaluation criteria based on clinical ultrasound images, such as the Placenta Accreta Type Ultrasound Scoring Scale proposed in patent CN106308851B. This provides quantitative scoring criteria for multiple ultrasound image indicators and demonstrates that this scoring scale can effectively assess the type of placenta accreta and accurately predict the risk of intraoperative bleeding and hysterectomy. However, the feasibility of this placenta accreta evaluation standard is limited. For primary care hospitals, existing evaluation standards are complex, and ultrasound examinations rely heavily on the experience and skills of ultrasound instrument operators. Consequently, scoring between primary care hospitals and general hospitals inevitably differs due to subjective personnel, leading to inconsistent evaluation results and misdiagnoses in referrals. Furthermore, the assessment of some ultrasound signs in existing ultrasound image evaluation systems remains highly subjective, making it difficult for different physicians to achieve stable and consistent assessments. This makes the effectiveness of placenta accreta risk assessment highly dependent on the physician's individual ultrasound sign segmentation skills.

[0004] The quality of placental ultrasound images is easily affected by various factors, such as intestinal gas and body shape. Ultrasound images are also prone to artifacts, resulting in relatively low resolution and signal-to-noise ratio. The use of ultrasound images to segment signs of placenta accreta is generally subject to inaccuracy due to individual differences. Multimodal data is complementary and can better reflect individual differences from multiple perspectives. For example, biochemical marker detection data can show the depth of placental invasion into the myometrium, ultrasound blood flow imaging can assess placental blood supply, and CT images can complement ultrasound images. Multimodal data can help neural networks capture features more comprehensively and improve the accuracy of localizing features of the placenta accreta region.

[0005] Therefore, an assessment system based on multimodal placental imaging for ultrasonic sign recognition and effective automatic segmentation of placenta accreta signs is needed to solve the problem of inaccurate existing placenta accreta risk assessment. Summary of the Invention

[0006] To address the problems existing in the prior art, this application aims to provide a placenta accreta assessment system based on multimodal sign recognition. This system uses a specific imaging method to acquire placental ultrasound images. A deep neural network, combined with biochemical markers of placenta accreta and medical records, is then used to mine pathological features from the subject's placenta ultrasound images. This system accurately and automatically segments, locates, and scores signs of placenta accreta, enabling a more accurate assessment of the severity of placenta accreta. By inputting the system's cross-sectional ultrasound images and biochemical test results, medical personnel can quickly obtain effective assessment information, including the location, extent, depth, and surrounding organ invasion of the placenta. This helps them quickly determine the extent of placenta accreta, estimate intraoperative blood loss, and determine surgical plans for placenta accreta, thereby preventing adverse maternal outcomes.

[0007] Specifically, this application involves the following aspects:

[0008] According to one aspect of the present application, a placenta implantation assessment system based on multimodal sign recognition is provided, comprising: a data collection module, which collects abdominal ultrasound scan data of a subject to obtain a plurality of placenta ultrasound images; collects biochemical test and medical record data of the subject to obtain a plurality of test texts; a sign extraction module, which inputs the plurality of placenta ultrasound images into a first feature extraction unit to obtain a first feature map, and inputs the plurality of test texts into a second feature extraction unit to obtain a second feature map; upsampling the first feature map and the second feature map multiple times, and after each upsampling, fusing the first feature map and the second feature map through a cross attention unit to obtain a third feature map after multiple upsampling; utilizing a feature aggregation unit to skip-connect and weightedly fuse the first feature map, the second feature map and the third feature map to obtain a fourth feature map; and inputting the fourth feature map into a classification unit. to obtain multiple segmentation masks of signs of placenta accreta; an evaluation module determines the severity of the signs of placenta accreta corresponding to the multiple segmentation masks, and obtains a placenta accreta evaluation result of the subject based on the severity; wherein, collecting the subject's abdominal ultrasound scanning data to obtain multiple placenta ultrasound images includes: using the left and right sides and / or the upper and lower edges of the subject's supine abdominal uterine contour as boundaries, starting from one side and / or one side, scanning the sagittal placenta once every first distance and / or second distance until the other side and / or the other side is scanned to obtain multiple placenta ultrasound images; the cross-attention unit includes a first cross-attention unit and a second cross-attention unit, which are used to calculate the similarity of the first feature map and the second feature map based on the first feature map and the second feature map, respectively, to generate a first weight matrix and a second weight matrix, respectively.

[0009] According to some embodiments of the present application, the value range of the first distance is 1 cm ~ 10 cm; the value range of the second distance is 1 cm ~ 8 cm.

[0010] According to some embodiments of the present application, at least one of the first feature extraction unit and / or the second feature extraction unit includes a shape-automatic adjustment convolution layer; the shape-automatic adjustment convolution layer adjusts the shape of one or more convolution kernels based on learning the offset of one or more convolution kernels at each sampling position.

[0011] According to some embodiments of the present application, the cross-attention unit is further used to: extract placenta implantation feature information of the first feature map and / or the second feature map based on the Top-k setting of the first weight matrix and the second weight matrix to obtain a third feature map.

[0012] According to some embodiments of the present application, the first cross-attention unit and / or the second cross-attention unit includes a cross-attention decoder.

[0013] According to some embodiments of the present application, the feature aggregation unit establishes a jump connection between the output of the first feature extraction unit, the output of the second feature extraction unit, and the output of the cross-attention unit after the last upsampling in multiple upsamplings, and fuses the first feature map, the second feature map, and the third feature map through gated weighted fusion to obtain a fourth feature map.

[0014] According to some embodiments of the present application, the feature aggregation unit includes a multi-scale feature aggregation block; the multi-scale feature aggregation block calculates the channel attention and spatial attention of the first feature map and the second feature map respectively before the jump connection between the first feature extraction unit and the second feature extraction unit to optimize the channel weight distribution and spatial weight distribution of the first feature map and the second feature map.

[0015] According to some embodiments of the present application, determining the severity of placenta accreta signs corresponding to multiple segmentation masks, and obtaining a placenta accreta assessment result for the subject based on the severity includes: providing, by a classification unit, one of three classification labels to the placenta accreta signs corresponding to each of the multiple segmentation masks to determine the severity of each placenta accreta sign; and taking the sum of the severities of all placenta accreta signs as the placenta accreta assessment result.

[0016] According to another aspect of the present application, a method for constructing a placenta accreta assessment system based on multimodal sign recognition is provided, including: obtaining training data, annotating multiple placental ultrasound images to obtain a first training sample set, and annotating multiple detection texts to obtain a second training sample set; obtaining segmentation results, processing the first training sample set using a first feature extraction unit to obtain a first feature map, and processing the second training sample set using a second feature extraction unit to obtain a second feature map; performing multiple upsampling on the first feature map and the second feature map, and fusing the first feature map and the second feature map through a cross-attention unit after each upsampling, to obtain a third feature map after multiple upsampling; using a feature aggregation unit to skip-connect and weightedly fuse the first feature map, the second feature map, and the third feature map to obtain a fourth feature map; inputting the fourth feature map into a classification unit to obtain multiple segmentation masks of placenta accreta signs; iterating the segmentation model, determining the loss values ​​of the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit according to the multiple segmentation masks, and iterating the weight parameters of the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit using gradient back propagation.

[0017] According to some embodiments of the present application, labeling multiple placental ultrasound images to obtain a first training sample set, and labeling multiple detection texts to obtain a second training sample set includes: determining a label for each placental ultrasound image in the multiple placental ultrasound images and a label for each detection text in the multiple detection texts based on evaluation parameters; wherein the evaluation parameters include: placental position, placental thickness, retroplacental hypoechoic band, bladder line, placental fossa, placental base blood flow, cervical sinusoids, cervical morphology and / or history of cesarean section.

[0018] According to some embodiments of the present application, the method for constructing a placenta accreta assessment system based on multimodal sign recognition further includes: determining whether the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit have converged after iteration; and stopping the iteration of the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit in response to the convergence of the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit after iteration.

[0019] According to some embodiments of the present application, the method for constructing a placenta accreta assessment system based on multimodal sign recognition further includes:

[0020] Determine the maximum number of iterations of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit; in response to the number of iterations of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit reaching the maximum number of iterations, stop the iterations of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit.

[0021] According to another aspect of the present application, a construction device for a placenta accreta assessment system based on multimodal sign recognition is also provided, including: a training data acquisition unit, which annotates multiple placental ultrasound images to obtain a first training sample set, and annotates multiple detection texts to obtain a second training sample set; a segmentation result acquisition unit, which uses a first feature extraction unit to process the first training sample set to obtain a first feature map, and uses a second feature extraction unit to process the second training sample set to obtain a second feature map; the first feature map and the second feature map are upsampled multiple times, and after each upsampling, the first feature map and the second feature map are fused through a cross-attention unit to obtain a third feature map after multiple upsampling; the first feature map, the second feature map and the third feature map are skipped and weightedly fused through a feature aggregation unit to obtain a fourth feature map; the fourth feature map is input into a classification unit to obtain multiple segmentation masks of placenta accreta signs; and a segmentation model iteration unit, which determines the loss values ​​of the first feature extraction unit, the second feature extraction unit, the cross-attention unit and the feature aggregation unit according to the multiple segmentation masks, so as to iterate the weight parameters of the first feature extraction unit, the second feature extraction unit, the cross-attention unit and the feature aggregation unit using gradient back propagation.

[0022] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the aforementioned method for constructing a placenta accreta assessment system based on multimodal sign recognition.

[0023] According to another aspect of the present application, a computer program product is provided, comprising computer program instructions. When the computer program instructions are executed by a processor, the processor executes the aforementioned method for constructing a placenta accreta assessment system based on multimodal sign recognition.

[0024] According to another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the aforementioned method for constructing a placenta accreta assessment system based on multimodal sign recognition.

[0025] Thus, the placenta accreta assessment system based on multimodal sign recognition provided by the present application, as an intelligent assessment tool, can conveniently predict the risk of placenta accreta for subjects, especially those in areas with scarce medical resources, thereby achieving the following goals: 1. Based on the assessment results, the primary hospital where the subject is located is guided to determine the risk level of placenta accreta and make timely referrals; 2. The superior hospital formulates reasonable preoperative preparations required for the risk level based on the assessment results, including blood source preparation, intraoperative monitoring equipment and drugs, etc., to avoid insufficient material preparation or excessive waste; 3. Based on the assessment results, a detailed surgical plan is formulated before surgery to minimize hysterectomy in women of childbearing age, while reducing bleeding and blood product transfusion, thereby improving the prognosis of the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The figure shows a block diagram of a placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application.

[0027] Figure 2 The figure shows a schematic structural diagram of a first feature extraction unit and a second feature extraction unit according to an embodiment of the present application.

[0028] Figure 3 The diagram shows a schematic diagram of the structure of the cross attention unit according to an embodiment of the present application.

[0029] Figure 4 The figure illustrates a flow chart of a method for constructing a placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application.

[0030] Figure 5 The figure shows a block diagram of a device for constructing a placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application.

[0031] Figure 6 The figure shows a block diagram of an electronic device according to an embodiment of the present application.

[0032] Figure 7 The figure shows a schematic diagram of an accuracy curve of a feature extraction module according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The present application is further described below with reference to examples. It should be understood that the examples are only used to further illustrate and explain the present application and are not intended to limit the present application.

[0034] Unless otherwise defined, technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art. Although methods and materials similar or identical to those described herein may be used in experiments or practical applications, the materials and methods are described herein below. In the event of a conflict, the present specification, including definitions, will prevail. In addition, the materials, methods, and examples are provided for illustrative purposes only and are not intended to be limiting. The present application is further described below with reference to specific examples, which are not intended to limit the scope of this application.

[0035] Application Overview

[0036] As mentioned above, existing ultrasound assessment systems typically rely on manual scoring of several items, which can easily lead to distortion and loss of assessment information. Ultrasound imaging assessment of placenta accreta should be based on comprehensive and consistent interpretation of the images collected from the subject, and artificial intelligence offers advantages in this regard. With the development of deep neural networks, computer vision technology has advanced rapidly, and the application of image semantic segmentation and object detection has become increasingly widespread, providing technical support for computer recognition of medical image information. Specifically, Sun Yat-sen University in Guangdong Province has developed a device that uses machine learning algorithms to automatically extract the fetal nuchal translucency. Currently, MRI / 3D imaging technology can also be used to image patients with placenta previa, and network analysis of MRI images can be used to determine the appropriate surgical incision for placenta accreta. Furthermore, multiple prenatal features, such as medical history and color Doppler ultrasound data, can be correlated to form a sequence of observed states and construct models, such as hidden Markov algorithms, to diagnose placenta accreta.

[0037] Despite progress in existing technologies, most still rely on manually acquired placental data to improve the model's sensitivity to texture features in ultrasound images. These methods fail to fully consider the global contextual characteristics of the lesion region within the image, limiting their ability to accurately locate the lesion. While single-modality AI models can effectively process ultrasound image data of the placenta, due to inconsistent image quality and the limitations of neural networks in identifying signs, there is still a lack of effective intelligent assessment tools for placenta accreta risk assessment.

[0038] The placenta accreta assessment system based on multimodal sign recognition provided by the present application can first perform an ultrasound scan on the abdomen of a subject in a specific manner to obtain a high-quality placenta ultrasound image, input the placenta ultrasound image into a first feature extraction unit to obtain a first feature map, and input the detection text into a second feature extraction unit to obtain a second feature map, thereby completing the preliminary extraction of information representing the placenta accreta sign in the image and text; the first feature map and the second feature map are upsampled multiple times to restore the resolution of the feature maps, and after each upsampling, the ultrasound modal information of the first feature map and the text modal information of the second feature map are fused through a cross-attention unit to extract more effective feature information with high correlation, thereby obtaining a third feature map; the feature aggregation unit is used to skip-connect and weightedly fuse the first, second and third feature maps to further strengthen the focus on key sign areas in the feature information, thereby obtaining a fourth feature map; the fourth feature map is input into a classification unit to obtain multiple segmentation masks of the placenta accreta sign, thereby using them to assess the specific condition and severity of the placenta accreta.

[0039] Specifically, this application introduces an attention mechanism into the convolutional layer of the neural network to capture more global contextual features of the placenta accreta lesion area in the feature map, redistributes the weights between the feature maps of the placenta ultrasound image extracted by the convolutional layer, focuses on key features, and reduces the representation of non-important features, thereby more accurately identifying the lesion area of ​​the placenta accreta, providing a solution for the segmentation of placenta accreta signs. In addition, a feature aggregation unit is introduced at the decoder end of the neural network to perform multi-scale feature fusion, fusing simple unimodal feature maps with complex multimodal feature maps and optimizing the feature map weight distribution to accurately locate key sign areas and avoid noise interference or overfitting caused by simply splicing multimodal features. In this way, the system obtains semantic segmentation results for multiple ultrasound signs, namely multiple lesion areas, replacing manual and automatic calibration of lesion locations, providing medical personnel with multiple key information for grading the severity of placenta accreta and predicting surgical risks.

[0040] The system described in this application can more effectively capture complex texture information in image data with limited data due to its high-quality acquisition of ultrasound image data, effective combination of neural networks and attention mechanisms, and use of multimodal placental data to optimize segmentation results. It can achieve in-depth extraction of signs of placental lesions, and can provide comprehensive and accurate automated support for corresponding diagnosis and medical decision-making, avoiding errors caused by subjective misjudgment by medical personnel in diagnostic situations, especially in primary medical institutions.

[0041] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0042] Exemplary Systems

[0043] Figure 1 The figure shows a placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application.

[0044] like Figure 1 As shown, the placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application includes the following modules.

[0045] The data collection module is used to collect the subject's abdominal ultrasound scan data to obtain multiple placental ultrasound images; collect the subject's biochemical test and medical record data to obtain multiple test texts. This module supports ultrasound equipment or endoscopic equipment that complies with the existing DICOM standard, such as existing ultrasound equipment that supports video acquisition functions, and can cooperate with medical personnel to collect static and / or dynamic ultrasound images of the subject's abdomen in real time. Data acquisition supports acquisition through acquisition cards, network ports or USB interfaces, etc. The module can also save data, for example, in a variety of video data formats, such as AVI, MP4 data, etc., to process the data through an acquisition card or any frame processing algorithm and obtain ultrasound images in DICOM format, and establish and store the subject's placental ultrasound image set.

[0046] Furthermore, the data collection module can be electrically connected to a computer storage device, which can be connected to the biochemical testing device and can preprocess and store the test results of the subject collected by the biochemical testing device. The data collection module described in this application can obtain the test results of the subject stored in the computer storage device. It is understood that preprocessing steps for the test results of the subject can also be performed directly on the data collection module, such as denoising, enhancement, normalization, word segmentation, etc.

[0047] The system described herein also includes a sign extraction module that inputs the multiple placental ultrasound images into a first feature extraction unit to obtain a first feature map, and inputs the multiple detected texts into a second feature extraction unit to obtain a second feature map; performs multiple upsampling on the first and second feature maps, and fuses the first and second feature maps via a cross-attention unit after each upsampling, to obtain a third feature map after the multiple upsampling; utilizes a feature aggregation unit to perform a weighted fusion of the first, second, and third feature maps via a skip connection to obtain a fourth feature map; and inputs the fourth feature map into a classification unit to obtain multiple segmentation masks for placenta accreta signs. The sign extraction module can be exemplified by a device, electronic device, or computer storage device with an input / output interface, such as an MCU, PLC, computer, or server. The module can execute functions including a first extraction unit for extracting image modality features, a second feature unit for extracting text modality features, and a cross-attention unit. The aforementioned multiple units are multi-parameter neural networks with complex structures, and the sign extraction module can execute these units. The structure and function of these units will be described in detail below.

[0048] The system described herein also includes an assessment module for determining the severity of the placenta accreta signs corresponding to the multiple segmentation masks and obtaining a placenta accreta assessment result for the subject based on the severity. Furthermore, the assessment module described herein can store the image segmentation masks output by the sign extraction module and can be electrically connected to a display device, suitable for visualizing these image segmentation results on the display device. This also facilitates medical personnel to manually assign a severity score to the corresponding placenta accreta lesion area based on the displayed results. The segmentation mask includes semantic recognition results of the subject's placental lesion signs, including multiple different mask values ​​for placental position, placental thickness, retroplacental hypoechoic zone, bladder line, placental fossa, placental base blood flow, cervical sinusoids, and the like.

[0049] In particular, abdominal ultrasound can clearly show the location, morphology, and relationship of the placenta with the myometrium. Therefore, an optimal ultrasound scanning scheme is needed to better capture characteristic medical signs such as abundant retroplacental blood flow, disappearance of the placental space, or abnormal expansion, and improve the quality of the placental ultrasound image. Therefore, according to one embodiment of the system described in the present application, the data collection module performs an ultrasound scan on the subject's abdomen to obtain multiple placental ultrasound images, which may include the following two methods: a first scanning method, with the left and right sides of the abdominal uterine contour of the subject in a supine position as the boundary, starting from one side, and scanning the sagittal placental image once every first distance using the ultrasound device or endoscope of the data collection module until the other side of the abdominal uterine contour is scanned and the placental image disappears from the device's field of view, thereby obtaining multiple placental ultrasound images; and a second scanning method in another embodiment, with the upper and lower edges of the abdominal uterine contour of the subject in a supine position as the boundary, starting from one side, and scanning the sagittal placental image once every second distance using the ultrasound device or endoscope of the data collection module until the other side of the abdominal uterine contour is scanned and the placental image disappears from the device's field of view, thereby obtaining multiple placental ultrasound images.

[0050] Thus, through the above-described embodiment, the placental ultrasound images scanned and stored by the data collection module can maximize information about the subject's placental integrity, particularly information about the areas where signs are located. Furthermore, the data collection module can preset overlapping areas between the two scanning methods and perform deduplication on the images scanned in these overlapping areas, thereby providing high-quality images to the sign extraction module.

[0051] Specifically, in the above-mentioned scanning method, the first distance can be set within a range of 1 to 10 cm. A smaller first distance increases the data dimension of the placental ultrasound image set obtained using this method. However, due to over-division of the image size, some placental accreta texture features may not be prominent. A larger first distance may result in loss of valid information in the image. Therefore, the first distance is preferably set within a range of 4 to 5 cm to ensure that the information provided by the image is consistent and not missed, while maintaining an appropriate data volume, thereby improving the training efficiency of each unit in the sign extraction module. Similarly, the second distance can also be set within a range of 1 to 8 cm; preferably, the second distance is set within a range of 4 to 5 cm. Since there is generally no significant difference in the extraction of placental position features between horizontal and vertical scanning, it is more preferable that the first and second distances remain the same to avoid asymmetric placental accreta feature information in the two directions, thereby preventing the generalization ability of the sign extraction module trained using placental ultrasound images.

[0052] Each of the multiple detection texts can be a spliced ​​text after preprocessing of the subject's multiple biochemical indicator detection data and medical history query data, that is, a structured text; it can also be the original text of the subject's multiple biochemical indicator detection data and medical history query data, that is, an unstructured text. Those skilled in the art will understand that the neural network can receive text data with a standard format after preprocessing (such as word embedding or other structured processing through Word2Vec), or it can directly process these data (such as selecting the second feature extraction unit as an existing natural language processing neural network). The use of detection text is because it can serve as a supplement to ultrasound images, reducing the problem of variable lesion morphology in ultrasound images due to individual differences in subjects, so that the entire sign extraction module can better adapt to changes in lesion morphology and suppress interference from low signal-to-noise ratio areas of ultrasound images.

[0053] Preferably, the system described in the present application selects serum human chorionic gonadotropin (hCG), more preferably β-hCG concentration, placental growth factor (PIGF) concentration, subject's cesarean section history, and subject's uterine fibroid removal history as detection text. It can be understood that after preprocessing, serializing, and vectorizing the above-mentioned structured / unstructured text, the convolutional layer can also be used to extract features therein. In the present application, the detection text is mainly used as a supplement to the ultrasound image, that is, to affect the feature selection of the feature map of the ultrasound image in the cross-attention unit, that is, the process of generating the third feature map.

[0054] refer to Figure 2 The shape-adjustable convolution layer can adjust the shape of one or more convolution kernels based on the offset of one or more convolution kernels at each sampling position in the learned convolution layer, that is, adjust the size of the sampled original ultrasound image and / or feature map. Specifically, for a two-dimensional convolution kernel of size K x K, an offset learning region of size 2 x K x K is set. The convolution kernel learns the offset in the x and y directions for each sampling point in the feature map through the offset learning region, so that each point on the feature map output by the two-dimensional convolution kernel corresponds to all points in the K x K sampling range of the input feature map for which the offset has been learned.

[0055] In this way, the convolution kernel used by the first feature extraction unit and / or the second feature extraction unit to extract sign features can adaptively adjust its size through previous offset learning, that is, adaptively adjust the receptive field during sampling, to automatically resolve the problem of irregular infiltration / penetration boundaries between the placenta and uterus in the ultrasound image. Compared to standard convolutional layers, the convolutional layers of the first feature extraction unit or the second feature extraction unit are more sensitive to small lesions in ultrasound images (such as areas of placental tissue adhesion) and can also better adapt to text instances with extreme aspect ratios (such as non-standard format data present in the input text data), thereby improving the quality of the first feature map and the second feature map.

[0056] That is, according to the embodiment of the present application, at least one of the first feature extraction unit and / or the second feature extraction unit included in the sign extraction module includes a convolution layer with automatic shape adjustment, and preferably both feature extraction units include a convolution layer with automatic shape adjustment.

[0057] refer to Figure 3 The cross attention unit is a bimodal cross attention decoder, whose input is the tokens mapped to the input dimension by the first feature map or the second feature map and the sequence of word tokens directly segmented by tools such as tokenizer, that is, and ;

[0058] In this way, the cross attention unit obtains the input features F from two modalities U and F T , namely the ultrasonic image features of the first feature map and the detected text features of the second feature map. U Based on this, calculate its cross-modal attention weight:

[0059]

[0060] (Formula 1)

[0061] in , In this way, the feature weight matrix of the first feature map is established. The text information can be fused into the ultrasound sign information through the attention weight, and the contribution of different attention heads can be adjusted to select and retain the high-contribution attention heads through the preset Top-k (i.e., the first k features with the highest attention weights) to further extract the most effective ultrasound sign features, thereby suppressing the interference of the mixed noise mode in the first feature map and improving the sign segmentation performance of the sign extraction module. Similarly, when the order of magnitude of the detected text is large, F T Based on F UFurther extract important features from the test text; and the features extracted from the two parts can also be weighted to retain more feature information. For the system described in this application, since four types of test text are generally selected and their confidence in clinical placenta accreta assessment is high, it is preferred to fuse the test text information with the ultrasound sign information to obtain a feature weight matrix for one of the first feature map and the second feature map corresponding to the ultrasound image, and then obtain a third feature map after fusing multimodal feature information based on this feature map.

[0062] Continue to refer Figure 3 The bimodal cross-attention decoding part can also calculate two attention weights in succession to output important features, including channel cross attention (CCA) and spatial cross attention (SCA). CCA is mainly used to calculate the global channel dependency between the features of the feature maps corresponding to different modal data, and can calculate the channel weight:

[0063]

[0064] (Formula 2)

[0065] Where σ is the activation function, W1 and W2 are the dimensionality reduction / recovery weight matrices of the fully connected layer of CCA, and GAP(F) is the global average pooling result for each channel of the tokens corresponding to the selected feature map. This generates a channel weight matrix as the feature weight matrix, which can suppress irrelevant features in some channels and further enhance the weights of important features. The generated features are then passed through SCA to calculate the spatial mask in the feature map:

[0066]

[0067] (Formula 3)

[0068] Among them, f 7×7 is a 7 x 7 convolution operation, F avg is the average pooling result of tokens corresponding to the selected feature map, F max The maximum pooling result of the tokens corresponding to the selected feature map. By calculating the weight of the spatial position corresponding to each element of the feature map, the bimodal cross-attention decoding can help focus on key areas in the ultrasound image, the target ultrasound feature, or the edge position of the target ultrasound feature, thereby increasing the effective feature information of the extracted third feature map.

[0069] In particular, the attention calculation part of CCA and SCA can also set residual connections so that the cross-attention unit can better fuse the shallow input and the deep semantic features obtained through the attention head, and align the features from the ultrasound image with the cross-modal language embedding from the detection text, thereby improving the data utilization efficiency of the cross-attention unit.

[0070] Then, according to the cross-attention unit described in this application, the feature map can be reconstructed by mapping the output features, such as the attention map, to the original dimensions of the input feature map, that is, outputting a fourth feature map. This is a technical solution that can be implemented by those skilled in the art. In this way, a fourth feature map containing deep multimodal placenta accreta sign features is obtained. This fourth feature map can be subsequently jump-connected with the aforementioned first feature map and / or second feature map to obtain a feature representation of the placenta accreta lesion area that highlights important signs and covers multimodal lesion information, which can be used for the most efficient and automatic segmentation of these lesion areas.

[0071] That is, according to an embodiment of the present application, the cross-attention unit includes a first cross-attention unit and a second cross-attention unit, which are used to calculate the similarity of the first feature map and the second feature map based on the first feature map and the second feature map, respectively, to generate a first weight matrix and a second weight matrix, respectively; and select information of the first feature map and / or the second feature map based on the Top-k setting of the first weight matrix and the second weight matrix to obtain a third feature map.

[0072] And, the first cross-attention unit and / or the second cross-attention unit includes a cross-attention decoder.

[0073] In order to better fuse the low-level features of each modality and the multimodal high-level features, a jump connection can be established between the fourth feature map and the first feature map and the second feature map. The jump connection can also introduce a multi-scale feature aggregation block as described above, which also includes a CCA calculation part and an SCA calculation part. The fused first feature map, the second feature map and / or the fourth feature map can be adjusted by adjusting the channel attention weight and the spatial attention weight to gate the weighted average of each feature map (such as using a pooling layer) to adjust the final fused feature, and then through bilinear interpolation, 1×1 convolution compression feature scale and other operations that can be understood by those skilled in the art, the fused feature is input into the classifier or generator (such as a fully connected layer). In this way, the feature features extracted by the entire feature extraction module focus on multimodal key information without missing the effective information of each modal input, so that accurate segmentation of placenta implantation signs can be provided.

[0074] That is, according to the embodiment of the present application, the feature aggregation unit establishes a jump connection between the output of the first feature extraction unit, the output of the second feature extraction unit and the output of the cross-attention unit after the last upsampling in the multiple upsamplings, and fuses the first feature map, the second feature map and the third feature map through gated weighted fusion to obtain the fourth feature map.

[0075] In addition, according to an embodiment of the present application, the feature aggregation unit includes a multi-scale feature aggregation block; before the multi-scale feature aggregation block jumps between the first feature extraction unit and the second feature extraction unit, it calculates the channel attention and spatial attention of the first feature map and the second feature map respectively to optimize the channel weight distribution and spatial weight distribution of the first feature map and the second feature map.

[0076] Finally, the fourth feature map containing the fused multimodal features is processed and input into a classification unit to obtain classification labels for the multiple ultrasound signs of placenta accreta learned by the sign extraction module with the aid of text. For example, the fourth feature map is processed through a convolutional layer with 1x1 convolution and a pooling layer to adjust the number of channels and size. It is then appropriately input into a classification unit to complete multi-label classification. The classification unit can be, for example, a combination of a fully connected layer and a softmax layer, or a sigmoid layer. In this way, a classification label is obtained for each ultrasound sign of placenta accreta. The evaluation module can obtain these classification labels and use the value of each classification label as the severity of the corresponding sign to reflect the pathological characteristics of placenta accreta represented by the sign. For example, the multiple labels can be 1, 2, 3, etc., and the value represented by the label is directly used as the severity of the sign. A larger value indicates a more severe placenta accreta pathology.

[0077] Table 1 Placenta accreta ultrasound sign assessment scale

[0078]

[0079] Table 1 shows the criteria for determining the placenta accreta assessment result of the corresponding placenta ultrasound image according to the severity of the placenta accreta ultrasound sign. In an exemplary embodiment, the sum of the label values ​​of all signs is used as a score to determine the assessment result:

[0080] 1. If the score is >0 and ≤5, the assessment result indicates that the patient corresponding to the placental ultrasound image and test text is a patient with mild placenta accreta, which can be a high-risk non-accreta / adhesion type;

[0081] 2. A score of >5 and <10 indicates that the patient is classified as having moderate placenta accreta, which may be a high-risk type.

[0082] 3. A score of ≥10 indicates a severe case of placenta accreta, which may be a high-risk percreta case.

[0083] 4. A score of 0 indicates no placenta accreta. In this way, the placenta accreta assessment results for the subjects from whom the placental ultrasound image set and the test text set were derived are automatically determined through the data collection module, the sign extraction module, and the assessment module. This can serve as an effective basis for assisting doctors in their diagnosis in clinical practice.

[0084] That is, determining the severity of the placenta accreta signs corresponding to each of the multiple segmentation masks, and obtaining a placenta accreta assessment result for the subject based on the severity includes: providing, by the classification unit, one of three classification labels to the placenta accreta signs corresponding to each of the multiple segmentation masks to determine the severity of each placenta accreta sign; and using the sum of the severities of all placenta accreta signs as the placenta accreta assessment result.

[0085] Exemplary Methods

[0086] Figure 4 The figure illustrates a method for constructing a placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application.

[0087] like Figure 4 As shown, the method for constructing a placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application includes the following steps.

[0088] Step S110, acquiring training data, annotating multiple placental ultrasound images in a placental ultrasound image set to obtain a first training sample set, and annotating multiple detection texts in a detection text set to obtain a second training sample set;

[0089] Step S120: Obtain a segmentation result, process the first training sample set using a first feature extraction unit to obtain a first feature map, and process the second training sample set using a second feature extraction unit to obtain a second feature map; perform multiple upsampling on the first feature map and the second feature map, and provide a cross attention unit to fuse the first feature map and the second feature map after each upsampling, to obtain a third feature map after the multiple upsampling; utilize a feature aggregation unit to skip-connect and weightedly fuse the first feature map, the second feature map, and the third feature map to obtain a fourth feature map; and input the fourth feature map into a classification unit to obtain multiple segmentation masks for signs of placenta accreta.

[0090] Step S130, iterate the segmentation model, determine the loss values ​​of the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit according to the multiple segmentation masks, and use gradient back propagation to iterate the weight parameters of the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit.

[0091] Among them, labeling multiple placental ultrasound images in the placental ultrasound image set to obtain a first training sample set, and labeling multiple detection texts in the detection text set to obtain a second training sample set includes: determining a label for each placental ultrasound image in the multiple placental ultrasound images and a label for each detection text in the multiple detection texts based on evaluation parameters; wherein, the evaluation parameters include: placental position, placental thickness, retroplacental hypoechoic band, bladder line, placental fossa, placental base blood flow, cervical sinusoids, cervical morphology and / or cesarean section history.

[0092] The specific method of iterative segmentation model is: determining whether the first feature extraction unit, the second feature extraction unit, the cross-attention unit and the feature aggregation unit converge after iteration; in response to the first feature extraction unit, the second feature extraction unit, the cross-attention unit and the feature aggregation unit converge after iteration, stopping the iteration of the first feature extraction unit, the second feature extraction unit, the cross-attention unit and the feature aggregation unit.

[0093] Alternatively, an alternative approach to the iterative segmentation model may also be: determining the maximum number of iterations of the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit; in response to the number of iterations of the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit reaching the maximum number of iterations, stopping the iterations of the first feature extraction unit, the second feature extraction unit, the cross-attention unit, and the feature aggregation unit.

[0094] It can be understood that the segmentation model is a complete module that includes the first feature extraction unit, the second feature extraction unit, the cross-attention unit, the feature aggregation unit and the classification unit, which has the function of receiving input data and outputting segmentation results. When using gradient back propagation to iterate the weight parameters of the first feature extraction unit, the second feature extraction unit, the cross-attention unit and the feature aggregation unit, the cross-entropy loss function can be selected to measure the loss value of each unit.

[0095] In particular, the first feature extraction unit, the second feature extraction unit, the cross-attention unit and / or the feature aggregation unit in the placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application can be the first feature extraction unit, the second feature extraction unit, the cross-attention unit and / or the feature aggregation unit obtained by training the aforementioned segmentation model according to the construction method of the placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application.

[0096] Example

[0097] This application provides general and / or specific descriptions of the materials and experimental methods used in the experiments. Reagents or instruments used without manufacturer indication are all commercially available conventional products.

[0098] Example 1: Training sample collection

[0099] Medical record data collected by Peking University Third Hospital were used: female patients diagnosed with placenta accreta between January 2021 and July 2024 were included, and those suffering from preeclampsia, HELLP syndrome, acute fatty liver of pregnancy, NYHA grade III or above heart disease, end-stage renal disease, and those who underwent uterine artery embolization within 3 months were excluded. A total of 334 patients with placenta accreta (including placenta percreta) and 189 patients without placenta accreta (excluding placenta accreta accreta) were enrolled; all patients signed an informed consent form and agreed to include ultrasound images in subsequent data calculations and modeling. Ultrasound images were acquired using the scanning method described herein, with the first and second distances serving as the interval benchmarks. A total of 3,156 placental ultrasound images were obtained, with each patient receiving between 7 and 12 ultrasound images. Professional physicians at Peking University Third Hospital manually segmented and annotated the ultrasound images of the placenta accreta, including areas of placental tissue invading the myometrium, areas of outward bulging or rupture of the serosal layer, abnormal blood flow pools within the lumen, serpentine vascular clusters, areas of bladder wall infiltration, and localized bulging of the uterine contour. These images corresponded to signs such as placental location, thickness, invagination, retroplacental hypoechoic zone, blood flow at the base of the placenta, bladder line, and cervical sinusoids. The ultrasound images served as the first training sample set, which was input into the first feature extraction unit.

[0100] Electronic medical records of enrolled patients were collected and the following information was extracted: serum human chorionic gonadotropin (β-hCG) concentration, placental growth factor (PIGF) concentration, history of cesarean section, and history of uterine fibroid removal. The above text data were preprocessed, including denoising, standardization, and BERT-WWM word segmentation, to obtain test texts. All test texts were annotated with the placenta accreta status of the patients from whom they came.

[0101] Example 2 Segmentation Model Training and Validation

[0102] The annotated detection text obtained in Example 1 is subjected to serialization operations, including text truncation, word encoding, and position encoding, and the sequence corresponding to each detection text is aligned; a word vector is constructed using Word2Vec to obtain a second training sample set for input into the second feature extraction unit.

[0103] The segmentation model consists of an encoder and a decoder. The encoder consists of a feature extraction unit (i.e., a first feature extraction unit and a second feature extraction unit, similarly hereinafter) and multiple downsampling steps, each followed by a feature extraction unit. The decoder consists of a cross-attention unit and multiple upsampling steps, each followed by a cross-attention unit. Skip connections are used between each layer of the encoder and decoder, with a feature aggregation unit incorporated at each skip connection. The resulting fourth feature map is fed into a spatial pyramid pooling layer (with a 1x1 pooling kernel), which then uses a 1x1 convolutional layer to reduce the number of channels before feeding into a fully connected layer to output the segmentation mask.

[0104] The first training sample set and the second training sample set are divided into training set and test set respectively in a ratio of 4:1. The loss function of the segmentation model is set to the cross entropy loss function; the initial learning rate is set to 0.01, the batch size is set to 1, the optimizer is selected as Adam, and the training round is set to 150 epochs; the segmentation model automatically stops iteration after reaching the maximum number of iterations. The accuracy curve of the model is shown in the figure below. Figure 7 As shown in the figure, after 10 training rounds, the accuracy has reached 0.95. In addition, the segmentation model has a high AUC for the ROC curve. This shows that the segmentation model can effectively distinguish samples with different annotated regions in the test set through semantic segmentation. Finally, the segmentation model has a high recall rate, which enables it to more effectively detect placenta accreta lesions and facilitates the screening of positive samples.

[0105] Exemplary devices

[0106] Figure 5 The figure shows a block diagram of a device for constructing a placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application.

[0107] like Figure 5 As shown, the construction device 200 of the placenta accreta assessment system based on multimodal sign recognition according to an embodiment of the present application includes:

[0108] A training data acquisition unit 210 annotates a plurality of placental ultrasound images to obtain a first training sample set, and annotates a plurality of detection texts to obtain a second training sample set;

[0109] The segmentation result acquisition unit 220 processes the first training sample set using the first feature extraction unit to obtain a first feature map, and processes the second training sample set using the second feature extraction unit to obtain a second feature map; performs multiple upsampling on the first feature map and the second feature map, and fuses the first feature map and the second feature map through a cross attention unit after each upsampling, to obtain a third feature map after the multiple upsampling; uses a feature aggregation unit to skip-connect and weightedly fuse the first feature map, the second feature map, and the third feature map to obtain a fourth feature map; and inputs the fourth feature map into a classification unit to obtain multiple segmentation masks for signs of placenta accreta.

[0110] The segmentation model iteration unit 230 determines the loss values ​​of the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit according to the multiple segmentation masks, so as to iterate the weight parameters of the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit using gradient back propagation.

[0111] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the construction device 200 of the placenta accreta assessment system based on multimodal sign recognition have been described above with reference to Figure 4 The method for constructing a placenta accreta assessment system based on multimodal sign recognition has been described in detail, and therefore, its repeated description will be omitted.

[0112] As described above, the apparatus 200 for constructing a placenta accreta assessment system based on multimodal sign recognition according to embodiments of the present application can be implemented in various terminal devices, such as a server for storing training sample sets and multiple segmentation masks. In some examples, the apparatus 200 for constructing a placenta accreta assessment system based on multimodal sign recognition according to embodiments of the present application can be integrated into a terminal device as a software module and / or a hardware module. For example, the apparatus 200 for constructing a placenta accreta assessment system based on multimodal sign recognition can be a software module within the terminal device's operating system, or an application developed specifically for the terminal device. Alternatively, the apparatus 200 for constructing a placenta accreta assessment system based on multimodal sign recognition can be one of the terminal device's hardware modules.

[0113] Alternatively, in other examples, the apparatus 200 for constructing a placenta accreta assessment system based on multimodal sign recognition and the terminal device may be separate devices, and the apparatus 200 for constructing a placenta accreta assessment system based on multimodal sign recognition may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0114] Exemplary electronic devices

[0115] Below, reference Figure 6 To describe the electronic device according to the embodiment of the present application.

[0116] Figure 6 The figure shows a block diagram of an electronic device according to an embodiment of the present application.

[0117] like Figure 6 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0118] The processor 13 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0119] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute these program instructions to implement the methods for constructing a placenta accreta assessment system based on multimodal sign recognition according to various embodiments of the present application described above and / or other desired functions. The computer-readable storage medium may also store various contents, such as placental ultrasound images, test text, and cross-attention units.

[0120] In one example, the electronic device 10 may further include an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0121] The input device 13 may include, for example, a keyboard, a mouse, and the like.

[0122] The output device 14 can output various information to the outside, including calibration coefficients obtained through linear regression, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0123] Of course, to simplify, Figure 6 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 10 may further include any other appropriate components according to specific application scenarios.

[0124] Exemplary computer program products and computer-readable storage media

[0125] In addition to the above-mentioned methods and apparatuses, embodiments of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor is caused to perform the steps of the method for constructing a placenta accreta assessment system based on multimodal sign recognition according to various embodiments of the present application as described in the "Exemplary Methods" section above.

[0126] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C, Python, or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0127] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is caused to execute the steps of the method for constructing a placenta accreta assessment system based on multimodal sign recognition according to various embodiments of the present application as described in the above “Exemplary Method” section of this specification.

[0128] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0129] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0130] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0131] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0132] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0133] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A placenta accreta assessment system based on multimodal sign recognition, characterized in that: include: a data collection module for collecting abdominal ultrasound scan data of the subject to obtain a plurality of placental ultrasound images; Collecting the subjects' biochemical test and medical record data to obtain multiple test texts; a feature extraction module, inputting the multiple placental ultrasound images into a first feature extraction unit to obtain a first feature map, and inputting the multiple detection texts into a second feature extraction unit to obtain a second feature map; performing multiple upsampling on the first feature map and the second feature map, fusing the first feature map and the second feature map through a cross attention unit after each upsampling, and obtaining a third feature map after the multiple upsampling; Using a feature aggregation unit to skip connection and weightedly fuse the first feature map, the second feature map, and the third feature map to obtain a fourth feature map; inputting the fourth feature map into a classification unit to obtain multiple segmentation masks of placenta accreta signs; an assessment module, configured to determine a severity of the placenta accreta signs corresponding to the plurality of segmentation masks, and obtain a placenta accreta assessment result of the subject based on the severity; The step of collecting the subject's abdominal ultrasound scan data to obtain multiple placental ultrasound images includes: Taking the left and right sides and / or the upper and lower edges of the uterine outline of the subject in the supine position as boundaries, starting from one side and / or one edge, scanning the placenta in the sagittal position once every first distance and / or second distance until the other side and / or the other edge is scanned to obtain the multiple placental ultrasound images; The cross attention unit includes a first cross attention unit and a second cross attention unit, configured to calculate similarities between the first feature map and the second feature map based on the first feature map and the second feature map, respectively, to generate a first weight matrix and a second weight matrix, respectively; The first distance has a value range of 4 cm to 5 cm, and the second distance has a value range of 4 cm to 5 cm; At least one of the first feature extraction unit and / or the second feature extraction unit includes a shape-automatic adjustment convolutional layer, wherein the shape-automatic adjustment convolutional layer adjusts the shape of one or more convolution kernels according to a learned offset of the one or more convolution kernels at each sampling position; The plurality of segmentation masks include: placental position, placental thickness, retroplacental hypoechoic zone, bladder line, placental fossa, placental base blood flow and / or cervical sinusoids.

2. The placenta accreta assessment system based on multimodal sign recognition according to claim 1, characterized in that: The cross attention unit is further used to extract placenta implantation feature information of the first feature map and / or the second feature map based on the Top-k setting of the first weight matrix and the second weight matrix to obtain a third feature map.

3. The placenta accreta assessment system based on multimodal sign recognition according to claim 1, characterized in that: The first cross-attention unit and / or the second cross-attention unit includes a cross-attention decoder.

4. The placenta accreta assessment system based on multimodal sign recognition according to claim 1, characterized in that: The feature aggregation unit establishes a jump connection between the output of the first feature extraction unit, the output of the second feature extraction unit and the output of the cross attention unit after the last upsampling in the multiple upsamplings, and fuses the first feature map, the second feature map and the third feature map through gated weighting to obtain the fourth feature map.

5. The placenta accreta assessment system based on multimodal sign recognition according to claim 4, characterized in that: The feature aggregation unit includes a multi-scale feature aggregation block; The multi-scale feature aggregation block calculates the channel attention and spatial attention of the first feature map and the second feature map respectively before the jump connection between the first feature extraction unit and the second feature extraction unit to optimize the channel weight distribution and spatial weight distribution of the first feature map and the second feature map.

6. The placenta accreta assessment system based on multimodal sign recognition according to claim 1, characterized in that: Determining the severity of the placenta accreta signs corresponding to the multiple segmentation masks, and obtaining a placenta accreta assessment result for the subject based on the severity includes: providing, by the classification unit, one of three classification labels to the placenta accreta signs corresponding to each of the plurality of segmentation masks, so as to determine the severity of each placenta accreta sign; The sum of the severity of all signs of placenta accreta was used as the placenta accreta assessment outcome.

7. A method for constructing a placenta accreta assessment system based on multimodal sign recognition, characterized in that: include: Acquire training data, annotate multiple placental ultrasound images to obtain a first training sample set, and annotate multiple detection texts to obtain a second training sample set; Obtaining a segmentation result, processing the first training sample set using a first feature extraction unit to obtain a first feature map, and processing the second training sample set using a second feature extraction unit to obtain a second feature map; performing multiple upsampling on the first feature map and the second feature map, fusing the first feature map and the second feature map through a cross attention unit after each upsampling, and obtaining a third feature map after the multiple upsampling; Using a feature aggregation unit to skip connection and weightedly fuse the first feature map, the second feature map, and the third feature map to obtain a fourth feature map; inputting the fourth feature map into a classification unit to obtain multiple segmentation masks of placenta accreta signs; Iterate the segmentation model, determine the loss values ​​of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit according to the multiple segmentation masks, and iterate the weight parameters of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit by using gradient back propagation; The first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit are the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit of the placenta accreta assessment system based on multimodal sign recognition according to any one of claims 1 to 6; The multiple placental ultrasound images include: a plurality of placental ultrasound images obtained by scanning the placenta in a sagittal position at intervals of a first distance and / or a second distance, with the left and right sides and / or upper and lower edges of the uterine outline of the subject in a supine position as boundaries, starting from one side and / or one edge, until the other side and / or one edge is scanned; The cross attention unit includes a first cross attention unit and a second cross attention unit, configured to calculate similarities between the first feature map and the second feature map based on the first feature map and the second feature map, respectively, to generate a first weight matrix and a second weight matrix, respectively; The first distance has a value range of 4 cm to 5 cm, and the second distance has a value range of 4 cm to 5 cm; At least one of the first feature extraction unit and / or the second feature extraction unit includes a shape-automatic adjustment convolutional layer, wherein the shape-automatic adjustment convolutional layer adjusts the shape of one or more convolution kernels according to a learned offset of the one or more convolution kernels at each sampling position; The plurality of segmentation masks include: placental position, placental thickness, retroplacental hypoechoic zone, bladder line, placental fossa, placental base blood flow and / or cervical sinusoids.

8. The method for constructing a placenta accreta assessment system based on multimodal sign recognition according to claim 7, characterized in that: Labeling a plurality of placental ultrasound images to obtain a first training sample set, and labeling a plurality of detection texts to obtain a second training sample set includes: The label of each placental ultrasound image in the multiple placental ultrasound images and the label of each detection text in the multiple detection texts are determined based on the evaluation parameters, wherein the evaluation parameters include: placental position, placental thickness, retroplacental hypoechoic band, bladder line, placental fossa, placental base blood flow, cervical sinusoids, cervical morphology and / or history of cesarean section.

9. The method for constructing a placenta accreta assessment system based on multimodal sign recognition according to claim 7, characterized in that: Also includes: Determining whether the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit converge after iteration; In response to the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit converging after iteration, the iteration of the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit is stopped.

10. The method for constructing a placenta accreta assessment system based on multimodal sign recognition according to claim 7, characterized in that: Also includes: Determining a maximum number of iterations of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit; In response to the number of iterations of the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit reaching the maximum number of iterations, the iterations of the first feature extraction unit, the second feature extraction unit, the cross attention unit and the feature aggregation unit are stopped.

11. A device for constructing a placenta accreta assessment system based on multimodal sign recognition, characterized in that: include: a training data acquisition unit, annotating a plurality of placental ultrasound images to obtain a first training sample set, and annotating a plurality of detection texts to obtain a second training sample set; a segmentation result acquisition unit, processing the first training sample set using the first feature extraction unit to obtain a first feature map, and processing the second training sample set using the second feature extraction unit to obtain a second feature map; performing multiple upsampling on the first feature map and the second feature map, fusing the first feature map and the second feature map through a cross attention unit after each upsampling, and obtaining a third feature map after the multiple upsampling; Using a feature aggregation unit to skip connection and weightedly fuse the first feature map, the second feature map, and the third feature map to obtain a fourth feature map; inputting the fourth feature map into a classification unit to obtain multiple segmentation masks of placenta accreta signs; a segmentation model iteration unit, which determines loss values ​​of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit according to the multiple segmentation masks, so as to iterate weight parameters of the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit by using gradient back propagation; The first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit are the first feature extraction unit, the second feature extraction unit, the cross attention unit, and the feature aggregation unit of the placenta accreta assessment system based on multimodal sign recognition according to any one of claims 1 to 6; The multiple placental ultrasound images include: a plurality of placental ultrasound images obtained by scanning the placenta in a sagittal position at intervals of a first distance and / or a second distance, with the left and right sides and / or upper and lower edges of the uterine outline of the subject in a supine position as boundaries, starting from one side and / or one edge, until the other side and / or one edge is scanned; The cross attention unit includes a first cross attention unit and a second cross attention unit, configured to calculate similarities between the first feature map and the second feature map based on the first feature map and the second feature map, respectively, to generate a first weight matrix and a second weight matrix, respectively; The first distance has a value range of 4 cm to 5 cm, and the second distance has a value range of 4 cm to 5 cm; At least one of the first feature extraction unit and / or the second feature extraction unit includes a shape-automatic adjustment convolutional layer, wherein the shape-automatic adjustment convolutional layer adjusts the shape of one or more convolution kernels according to a learned offset of the one or more convolution kernels at each sampling position; The plurality of segmentation masks include: placental position, placental thickness, retroplacental hypoechoic zone, bladder line, placental fossa, placental base blood flow and / or cervical sinusoids.

12. An electronic device, characterized in that: include: processor; as well as A memory having computer program instructions stored therein, wherein the computer program instructions, when executed by the processor, cause the processor to execute the method for constructing a placenta accreta assessment system based on multimodal sign recognition according to any one of claims 7 to 10.

13. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the method for constructing a placenta accreta assessment system based on multimodal sign recognition according to any one of claims 7 to 10.

14. A computer-readable storage medium, characterized in that Computer program instructions are stored thereon, and when the computer program instructions are executed by a processor, the processor is caused to execute the method for constructing a placenta accreta assessment system based on multimodal sign recognition according to any one of claims 7 to 10.

Images