Image segmentation methods, electronic devices and storage media

Abstract

This application discloses an image segmentation method, an electronic device, and a storage medium. The method includes: acquiring a first image and a second image of a target region of a biological object, wherein the target region includes a lesion area; the first image is obtained by scanning the target region using computed tomography (CT) technology, and the second image is obtained by scanning the target region using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the target region in the first image to obtain a cropped image; and performing image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain a target segmentation result. This application solves the technical problem of low accuracy in segmenting lesion regions in CT and MRI images in related technologies.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to an image segmentation method, an electronic device, and a storage medium. Background Technology

[0002] Currently, CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) images of biological subjects can be registered, and then staff can segment the lesion areas in these images based on the registration results. However, registering CT and MRI images is currently quite difficult, leading to significant errors in the registration results and consequently, low accuracy in segmenting the lesion areas based on the registration results.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an image segmentation method, electronic device, and storage medium to at least solve the technical problem of low accuracy in segmenting lesion regions in CT and MRI images in related technologies.

[0005] According to one aspect of the embodiments of this application, an image segmentation method is provided, comprising: acquiring a first image and a second image of a target part of a biological object, wherein the target part includes a lesion region, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the target part in the first image to obtain a cropped image; and performing image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result.

[0006] According to one aspect of the embodiments of this application, an image segmentation method is provided, comprising: responding to an input command applied to an operating interface, displaying a first image and a second image of a target part of a biological object on the operating interface, wherein the target part includes a lesion region, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology; and responding to an image segmentation command applied to the operating interface, displaying a target segmentation result on the operating interface, wherein the target segmentation result is obtained by image segmentation of the target part contained in the second image based on the first image and a cropped image, and the cropped image is obtained by cropping the second image based on the region of the target part in the first image.

[0007] According to one aspect of the embodiments of this application, an image segmentation method is provided, comprising: displaying a first image and a second image of a target part of a biological object on the display screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the target part includes a lesion region, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the target part in the first image to obtain a cropped image; performing image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result; and driving the VR device or AR device to render and display the target segmentation result.

[0008] According to one aspect of the embodiments of this application, an image segmentation method is provided, comprising: acquiring a first image and a second image of a target part of a biological object by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the first image and the second image, the target part including a lesion region, the first image being an image obtained by scanning the target part using computed tomography (CT) technology, and the second image being an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the target part in the first image to obtain a cropped image; performing image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result; and outputting the target segmentation result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the target segmentation result.

[0009] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the image segmentation method of any one of the above-mentioned methods when it runs.

[0010] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the image segmentation method of any of the above.

[0011] In this embodiment, a first image and a second image of the target part of a biological object are acquired; based on the region of the target part in the first image, the second image is cropped to obtain a cropped image; image segmentation is performed on the target part contained in the second image based on the first image and the cropped image to obtain the target segmentation result. By using the region of the target part in the first image to crop the second image, the cropped image of the target part in the second image is determined, thereby reducing the influence of other images in the second image other than the cropped image on the segmentation of the lesion region. Finally, the first image and the cropped image are used to segment the second image to obtain the target segmentation result, improving the clarity of local details in the lesion region, ensuring that the segmented target segmentation result is not distorted, thereby improving the segmentation accuracy of the lesion region, and thus solving the technical problem of low accuracy in segmenting lesion regions in CT and MRI images in related technologies.

[0012] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0014] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an image segmentation method of Embodiment 1 of this application;

[0015] Figure 2 This is a structural block diagram of the computational environment of an image segmentation method according to Embodiment 1 of this application;

[0016] Figure 3 This is a flowchart of an image segmentation method according to Embodiment 1 of this application;

[0017] Figure 4 This is a schematic diagram of a candidate segmentation result determination process according to Embodiment 1 of this application;

[0018] Figure 5 This is a schematic diagram of a target segmentation result determination process according to Embodiment 1 of this application;

[0019] Figure 6 This is a schematic diagram comparing segmentation results according to Embodiment 1 of this application;

[0020] Figure 7 This is a flowchart of an image segmentation method according to Embodiment 2 of this application;

[0021] Figure 8 This is a flowchart of an image segmentation method according to Embodiment 3 of this application;

[0022] Figure 9 This is a flowchart of an image segmentation method according to Embodiment 4 of this application;

[0023] Figure 10 This is a structural block diagram of an image segmentation apparatus according to Embodiment 5 of this application;

[0024] Figure 11 This is a structural block diagram of an image segmentation apparatus according to Embodiment 6 of this application;

[0025] Figure 12 This is a structural block diagram of an image segmentation apparatus according to Embodiment 7 of this application;

[0026] Figure 13 This is a structural block diagram of an image segmentation apparatus according to Embodiment 8 of this application;

[0027] Figure 14 This is a structural block diagram of an electronic device according to Embodiment 9 of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0031] The Probabilistic U-Net model is a model that processes images by progressively reducing and then progressively increasing the image resolution. Compared with the traditional U-Net model, this application adds a new encoder for modifying and intervening in the image. This encoder enables differences between the processed images, thereby making the segmentation of lesion regions from the images more accurate. The modification and intervention operations can include, but are not limited to, user data augmentation operations such as scaling, rotation, mirroring, and gamma transformation.

[0032] Magnetic Resonance Imaging (MRI) is a technique that effectively scans and images organs or tissues in biological subjects that have developed lesions at target locations, clearly showing the current state of the lesion area. The grayscale values ​​of MRI images represent the MRI signal intensity and reflect the relaxation time.

[0033] Computed Tomography (CT) is a technique that uses computed tomography (CT) scans to image diseased organs or tissues, clearly showing the current state of the lesion area. Unlike magnetic resonance imaging (MRI), the grayscale values ​​of CT images typically indicate the density of tissues, organs, and lesions.

[0034] Example 1

[0035] According to an embodiment of this application, an image segmentation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an image segmentation method of Embodiment 1 of this application. Figure 1 As shown, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 via a network. The virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc. The terminal 104 is not limited to PCs, mobile phones, tablets, etc. The server 102 can be a server corresponding to a media file operator. The network includes, but is not limited to: wide area network, metropolitan area network, or local area network.

[0037] Optionally, the virtual reality device 104 in this embodiment includes a memory, a processor, and a transmission device. The memory stores an application program that can be used to perform the following: acquiring a first image and a second image of a target area of ​​a biological object, wherein the target area includes a lesion region; the first image is an image obtained by scanning the target area using computed tomography (CT) technology, and the second image is an image obtained by scanning the target area using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the target area in the first image to obtain a cropped image; and performing image segmentation on the target area contained in the second image based on the first image and the cropped image to obtain a target segmentation result. This solves the technical problem of low accuracy in segmenting lesion regions in CT and MRI images in related technologies, achieving the goal of accurately identifying lesion regions in images.

[0038] The terminal in this embodiment can be used to display a first image and a second image of a target part of a biological object on the display screen of a virtual reality (VR) device or an augmented reality (AR) device. The target part includes a lesion area. The first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology. The terminal sends a cropped image to the virtual reality device 104 based on the area of ​​the target part in the first image, and performs image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result. After receiving the cropped image and the target segmentation result, the virtual reality device 104 displays the cropped image and the target segmentation result at the target projection position.

[0039] Optionally, the eye-tracking HMD (Head-Mounted Display) and eye-tracking module in the virtual reality device 104 of this embodiment function the same as in the embodiments described above. That is, the screen in the HMD is used to display real-time images, and the eye-tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The terminal in this embodiment acquires the user's position and movement information in real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in virtual three-dimensional space, as well as the user's field of vision orientation in virtual three-dimensional space.

[0040] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned AR / VR device (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1The AR / VR device (or mobile device) shown is an example of a computing node in computing environment 201. Figure 2 This is a structural block diagram of the computational environment for an image segmentation method according to Embodiment 1 of this application, as shown below. Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (represented as 210-1, 210-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 202 can remotely run applications or store data within computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0041] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0042] The services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. VM-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0043] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, such as Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers within a Pod handle requests related to one or more corresponding functions of the service. Proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with similar Pods.

[0044] During operation, executing a user request from end user 202 may require calling one or more services in computing environment 201, and executing one or more functions of one service may require calling one or more functions of another service. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0045] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0046] Under the aforementioned operating environment, this application provides the following: Figure 3 The image segmentation method shown is illustrated. It should be noted that the image segmentation method in this embodiment can be derived from... Figure 1 The mobile terminal in the illustrated embodiment is executed. Figure 3 This is a flowchart of an image segmentation method according to Embodiment 1 of this application. Figure 3 As shown, the method may include the following steps:

[0047] Step S302: Obtain a first image and a second image of the target part of the biological object.

[0048] The target area includes the lesion region. The first image is obtained by scanning the target area using computed tomography (CT) technology, and the second image is obtained by scanning the target area using magnetic resonance imaging (MRI) technology.

[0049] The aforementioned lesion area can refer to the lesion area caused by the pathological changes of the tissues and organs of a biological object, such as the lesion area of ​​appendicitis or nasopharyngeal carcinoma. The aforementioned target site can be the tissue or organ containing the lesion area. The aforementioned first image can refer to an image obtained by tomographic scanning of the lesion area, such as a CT image. The aforementioned second image can refer to an image obtained by magnetic resonance imaging of the lesion area, such as an MRI image. For ease of understanding, the first image will be referred to as a CT image and the second image as an MRI image in the following text.

[0050] Considering that some tissues and organs of biological subjects are difficult to segment using only CT or MRI images when diseased, for example, in the early stages of nasopharyngeal carcinoma, the lesion area is generally located in the superior or posterior wall. Simple tomographic imaging may not be able to directly segment the specific location of the lesion. Since tomography and MRI are common techniques in the medical field for determining lesion areas, to ensure the accuracy of the segmented lesion area, the image segmentation system can first acquire a first image and a second image containing the target area, i.e., the aforementioned CT and MRI images. Then, by combining the CT and MRI images, the lesion area is segmented to determine its specific location. The target area includes at least the lesion region where the disease has occurred. The first image can be obtained by scanning the target area using computed tomography (CT), and the second image can be obtained by scanning the target area using MRI.

[0051] Step S304: Based on the region of the target part in the first image, the second image is cropped to obtain a cropped image.

[0052] The target area in the first image can refer to the region where the diseased organ or tissue is located in the first image, i.e., the region where the target area is located, which also includes the lesion area. Considering that the lesion area is generally located in the center of the aforementioned region, and the edge portion can provide a basis for distinguishing between healthy and diseased tissue, the target area in the first image can be divided into at least two parts: the edge portion and the central portion of the target area. In addition, it can also be a portion within a preset distance from the boundary of the lesion area. The specific portion included in the target area in the first image can be set by the staff according to actual needs, and is not limited here. Among them, the probability of the lesion area being in the central portion is generally greater than that of the region in the edge portion. In order to avoid missed identification, the edge portion and the central portion of the target area can be considered simultaneously during the segmentation of the lesion area. The cropped image can refer to the image of the region corresponding to the target area in the second image. If the target area in the first image includes the edge portion and the central portion, then the corresponding cropped image also includes the edge portion and the central portion of the target area in the second image.

[0053] Currently, when segmenting lesion areas by combining CT and MRI images, the location of the lesion area is usually segmented by cross-modal registration of the CT and MRI images. The accuracy of cross-modal registration is one of the important factors affecting the accuracy of lesion area segmentation. However, since rigid or affine transformation is usually used to register CT and MRI images, even if rigid transformation is performed on the MRI image to improve the registration accuracy between MRI and CT images, the final registration result may still fail to display the important local details of the lesion area that can be effectively reflected in the CT or MRI images, resulting in serious distortion of the registered image.

[0054] Therefore, in one alternative embodiment, in order to improve the accuracy of the segmented lesion region, the second image, i.e. the MRI image, can be cropped to obtain the cropped region, which can then be used to segment the lesion region.

[0055] Specifically, considering that both CT and MRI images can reflect the diseased organs or tissues within a biological object, and thus segment the aforementioned lesion region, CT images are digitally reconstructed analog images with high density resolution. They can display the density of organs and lesions at different gray levels and can be easily post-processed by computers, such as performing two-dimensional and three-dimensional display analysis, thereby clearly showing the location of the lesion region in the image. MRI images have multiple imaging sequences, and can display multiple different images with high tissue resolution and multiple parameters for the same lesion region. Therefore, when determining the cropped image, image recognition can be performed on the first image, i.e., the CT image, to analyze and process the region of the target area in the first image, i.e., the region where the organ or tissue disease occurs in the first image. Then, this region is used to crop the second image, obtaining the image corresponding to the edge and center of the lesion region in the second image, i.e., the cropped image.

[0056] Step S306: Based on the first image and the cropped image, perform image segmentation on the target region contained in the second image to obtain the target segmentation result.

[0057] The above target segmentation result can refer to the image of the segmented lesion region.

[0058] In one optional embodiment, as described above, considering that the second image, i.e., the MRI image, has higher tissue resolution and no bone artifacts compared to the first image, i.e., the CT image, some lesion areas, such as the lesion area corresponding to the nasopharyngeal carcinoma mentioned above, will not be affected by the neck bones, after obtaining the cropped image, the first image and the cropped image can be used to perform image registration and image segmentation on the area containing the target part in the second image. That is, the first image and the cropped image can be registered, and then the area containing the target part in the second image can be segmented according to the registration result to obtain the lesion area, and thus obtain the target segmentation result mentioned above.

[0059] In one optional embodiment, since the errors generated during the registration of CT and MRI images are unavoidable, when performing image segmentation on the target area to determine the target segmentation result, multiple image recognition and image segmentation can be performed on the same second image to determine multiple first registration errors. Alternatively, multiple substitute images can be determined from other images in the imaging sequence where the second image is located. Then, image recognition and image segmentation can be performed on these multiple substitute images to determine multiple second registration errors. Finally, the segmentation result with the smallest registration error is selected as the target segmentation result, thereby improving the accuracy of the determined target segmentation result and solving the technical problem of low accuracy in segmenting lesion areas in related technologies.

[0060] In this embodiment, a first image and a second image of the target part of a biological object are acquired; based on the region of the target part in the first image, the second image is cropped to obtain a cropped image; image segmentation is performed on the target part contained in the second image based on the first image and the cropped image to obtain the target segmentation result. By using the region of the target part in the first image to crop the second image, the cropped image of the target part in the second image is determined, thereby reducing the influence of other images in the second image other than the cropped image on the segmentation of the lesion region. Finally, the first image and the cropped image are used to segment the second image to obtain the target segmentation result, improving the clarity of local details in the lesion region, ensuring that the segmented target segmentation result is not distorted, thereby improving the segmentation accuracy of the lesion region, and thus solving the technical problem of low accuracy in segmenting lesion regions in CT and MRI images in related technologies.

[0061] In this embodiment of the application, cropping a second image based on the region of the target part in a first image to obtain a cropped image includes: performing image segmentation on the target part contained in the first image to obtain an image segmentation result of the first image, wherein the image segmentation result is used to characterize the region of the target part in the first image; and cropping the second image based on the image segmentation result to obtain a cropped image.

[0062] The image segmentation result mentioned above can refer to the region of the target area in the first image, which may include the edge and center parts of the lesion region.

[0063] When cropping the second image using the region of the target part in the first image to obtain the cropped image described above, the first image can first be segmented to obtain the region that can represent the target part in the first image, i.e., the image segmentation result described above. Then, as mentioned above, this region can be used to crop the second image to obtain an image that can represent the region of the target part in the second image that corresponds to the region of the target part in the first image, i.e., the cropped image.

[0064] In one optional embodiment, a pre-trained first image segmentation model can be used to segment the first image to obtain the aforementioned image segmentation result, thereby improving the accuracy of the image segmentation result and thus improving the accuracy of image cropping. The image segmentation result can be obtained using the following formula:

[0065] .

[0066] in, This represents the image segmentation result. This represents the first image segmentation model. This represents the first image. This represents a mask value of 1 representing the true segmentation result in the second image. This represents the network parameters in the first segmentation model.

[0067] In this embodiment of the application, when the image segmentation result is used to characterize the boundary region of the target part in the first image, the second image is cropped based on the image segmentation result to obtain a cropped image. This includes: performing dilation processing on the image segmentation result based on a preset dilation parameter to obtain a dilation processing result; performing erosion processing on the image segmentation result based on a preset erosion parameter to obtain an erosion processing result; obtaining the difference between the dilation processing result and the erosion processing result to obtain the target boundary region; and cropping the second image based on the target boundary region to obtain a cropped image.

[0068] In one optional embodiment, as described above, the target region in the first image may include an edge portion and a central portion. In order to better segment the lesion region, corresponding cropped images can be generated for the edge portion and the central portion respectively, and then the cropped images can be used to determine the aforementioned target segmentation result.

[0069] Specifically, when the image segmentation result is the boundary region of the target area in the first image, considering that the first image, such as a CT image, has a high density resolution and can represent the target area and other areas in the first image in the form of 0 and 1, when cropping the second image based on the image segmentation result to obtain the cropped image, the image segmentation result can first be expanded based on a preset expansion parameter, such as expansion of 5mm, to obtain the above-mentioned expansion processing result. Then, the image segmentation result can be eroded based on a preset erosion parameter, such as erosion of 5mm, to obtain the above-mentioned erosion processing result. Finally, the difference between the above-mentioned expansion processing result and the erosion processing result can be obtained to obtain the boundary region of the target area in the first image, that is, the above-mentioned target boundary region. After obtaining the target boundary region, the second image can be cropped based on the target boundary region to obtain the image corresponding to the boundary region of the target area, that is, the edge part, in the second image, that is, the above-mentioned cropped image, thereby improving the accuracy of the cropped image. It should be noted that the specific image expansion and image erosion processes can be referred to in relevant literature, and will not be elaborated here.

[0070] In this embodiment of the application, when the image segmentation result is used to characterize the center of the target region in the central region of the first image, the second image is cropped based on the image segmentation result to obtain a cropped image, including: performing erosion processing on the image segmentation result based on preset erosion parameters to obtain the target center region; and cropping the second image based on the target center region to obtain the cropped image.

[0071] In one optional embodiment, corresponding to the foregoing, when the image segmentation result is the edge of the target region in the central region of the first image, the image segmentation result can be directly eroded according to preset erosion parameters to quickly obtain the central region of the target region in the first image, i.e., the aforementioned target central region. After obtaining the target central region, the second image can be cropped based on the target central region to obtain the image corresponding to the central region of the target region, i.e., the central part, in the second image, i.e., the aforementioned cropped image, thereby improving the accuracy of the cropped image.

[0072] In one optional embodiment, considering that the probability of a lesion being in the central region of the target site is greater than that in the peripheral region, the preset erosion parameter used to determine the central region can be different from the preset erosion parameter used to determine the peripheral region. The specific value of the preset erosion parameter can be determined according to the lesion type of the organ or tissue to be detected. For example, if the lesion type is one where the lesion area is easily segmented, such as appendicitis, then the corresponding preset erosion parameter used to determine the central region can be greater than the preset erosion parameter used to determine the peripheral region. If the lesion type is one where the lesion area is not easily segmented, such as nasopharyngeal carcinoma, then the corresponding preset erosion parameter used to determine the central region can be smaller than the preset erosion parameter used to determine the peripheral region.

[0073] In one alternative embodiment, to avoid missed identification, the images corresponding to the edge regions and the center regions are usually used as cropped images. Therefore, in order to improve the efficiency of determining the cropped image, the region corresponding to the eroded image segmentation result can be directly used as the center region during the process of determining the edge region.

[0074] In this embodiment of the application, image segmentation is performed on the target region contained in the second image based on the first image and the cropped image to obtain the target segmentation result. This includes: performing image segmentation on the first image, and performing image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain multiple candidate segmentation results; performing error prediction on the multiple candidate segmentation results based on the first image and the second image to obtain the segmentation error of the multiple candidate segmentation results, wherein different segmentation errors are used to characterize the errors between different candidate segmentation results and preset segmentation results; and selecting the target segmentation result from the multiple candidate segmentation results based on the segmentation errors of the multiple candidate segmentation results.

[0075] The aforementioned candidate segmentation results can include not only the segmentation results obtained by segmenting the second image using a pre-trained second image segmentation model, but also the image segmentation results obtained by segmenting the first image, i.e., the regions corresponding to the target parts segmented in the CT image. To ensure applicability during image segmentation, the second image segmentation model can be a probabilistic U-Net model. When segmenting the second image, the cropped images corresponding to the edge and center regions can be used as prior knowledge, thereby improving the accuracy of the candidate segmentation results.

[0076] As mentioned above, when determining the target segmentation result, since the error generated during the registration of CT and MRI images is unavoidable, in order to ensure the accuracy of the determined target segmentation result, multiple segmentation methods can be used, and the segmentation result with the smallest error can be taken as the target segmentation result. In addition to performing multiple segmentations on the second image as shown in step S306 above to determine the segmentation result with the smallest error, the target parts contained in the second image can also be segmented using the first image and the cropped image to obtain the above-mentioned multiple candidate segmentation results. Then, corresponding to the determination of the registration error, the first image and the second image can be used to predict the error of the above-mentioned multiple candidate segmentation results to determine the segmentation error corresponding to each of the different candidate segmentation results. The segmentation error can be determined by comparing the candidate segmentation result with the preset standard segmentation result, or it can be obtained by using a pre-trained error prediction model to predict the error of multiple candidate segmentation results based on the first image and the second image. The specific error prediction model will be introduced below. The standard segmentation error can be determined by the data generated when the staff manually segmented the image. Finally, after obtaining the segmentation error mentioned above, the target segmentation result can be determined from multiple candidate segmentation results according to preset filtering rules. The preset filtering rules can either mean selecting the candidate segmentation result with the smallest segmentation error as the target segmentation result, thereby improving the accuracy of the determined target segmentation result, or it can mean displaying candidate segmentation results with segmentation errors less than a preset threshold to the staff, who can then actively determine the target segmentation result from the filtered candidate results, thus facilitating the staff's examination of the specific lesions in organs or tissues.

[0077] In this embodiment of the application, image segmentation is performed on the first image, and image segmentation is performed on the target parts contained in the second image based on the first image and the cropped image to obtain multiple candidate segmentation results, including: image segmentation of the target parts contained in the first image to obtain a first candidate segmentation result; stitching the first image and the second image to obtain a stitched image; image segmentation is performed on the stitched image to obtain a second candidate segmentation result; image segmentation is performed on the stitched image based on the cropped image to obtain a third candidate segmentation result; and multiple candidate segmentation results are obtained based on the first candidate segmentation result, the second candidate segmentation result, and the third candidate segmentation result.

[0078] The aforementioned first candidate result may refer to the image segmentation result obtained by performing image segmentation on the first image.

[0079] In one optional embodiment, the edge regions of the first image and the center regions of the second image can be stitched together, or the center regions of the first image and the edge regions of the second image can be stitched together, to obtain multiple stitched images. Then, image segmentation is performed on the stitched images to obtain multiple candidate segmentation results, namely the second candidate segmentation result and the third candidate segmentation result. In one optional embodiment, to improve the efficiency of determining candidate segmentation results, the previously determined cropped image can be used instead of prior knowledge; instead, image segmentation processing can be directly performed on the stitched images to obtain the corresponding segmentation result, namely the second candidate segmentation result. To ensure the accuracy and rationality of the determined candidate segmentation results, the cropped image can be further used as prior knowledge to perform image segmentation on the stitched images to obtain the corresponding segmentation result, namely the third candidate segmentation result.

[0080] After obtaining the second and third candidate segmentation results, considering that the first candidate result obtained by segmenting the first image can also reflect the lesion area of ​​the target region, in order to improve the accuracy of the target segmentation result, the above-mentioned candidate segmentation result can be determined based on the first, second, and third candidate segmentation results. In an optional embodiment, the determined first, second, and third candidate segmentation results can be displayed to the user in a preset display interface, and the user can select a specific candidate segmentation result. Alternatively, based on the clarity of the obtained first, second, and third candidate segmentation results, the segmentation result with a clarity greater than a preset threshold can be used as the above-mentioned candidate segmentation result. Corresponding to the aforementioned segmentation formula, the candidate segmentation result can also be expressed in the form of a segmentation formula, with the candidate segmentation result as... For example, among which, This represents the first candidate segmentation result. This represents the second candidate segmentation result. This represents the third candidate segmentation result, and the specific calculation formula is as follows:

[0081] ,

[0082] ,

[0083] ,

[0084] .

[0085] in, and This represents the second image based on the boundary region. The image obtained after cropping. The second image is based on the central region. The image obtained by cropping.

[0086] In this embodiment of the application, image segmentation of the stitched image based on the cropped image to obtain a third candidate segmentation result includes: extracting features from the stitched image to obtain stitched image features; extracting features from the cropped image to obtain cropped image features; modifying the stitched image features based on the cropped image features to obtain modified image features; and decoding the modified image features to obtain the third candidate segmentation result.

[0087] In one optional embodiment, to avoid excessively high similarity among the determined candidate segmentation results, which could prevent the accuracy of the determined target segmentation result from improving, when segmenting the stitched image using a cropped image, the image features of the stitched image can be modified based on the image features of the cropped image. This results in multiple modified image features of the stitched image, ensuring that there are certain differences between the modified image features. Decoding these modified image features then yields the aforementioned third candidate segmentation result with lower similarity. .

[0088] Specifically, when segmenting a stitched image using a cropped image, image recognition and feature extraction can be performed on the stitched image first to determine its image features, i.e., the stitched image features mentioned above. Similarly, image recognition and feature extraction can be performed on the cropped image to determine its image features, i.e., the cropped image features mentioned above. These cropped image features can include edge region features of the edge regions and center region features of the center regions. Then, to ensure that the modified image features of the stitched image are different from the stitched image features, the stitched image features can be modified based on the cropped image features. For example, the color, shape, and texture features of the cropped image can be used to modify the color, shape, and texture features of the stitched image. During modification, the edge region features mentioned above can be used to modify the stitched image features, making the edge region features of the modified stitched image different from the edge region features in the original stitched image features; or the center region features mentioned above can be used to modify the stitched image features, making the center region features of the modified stitched image different from the center region features in the original stitched image features; or both edge region features and center region features, i.e., cropped image features, can be used simultaneously to modify the stitched image features, thereby changing both the edge region features and the center region features in the stitched image features, thus obtaining new image features for the stitched image. Finally, by decoding the modified image features, the aforementioned third candidate segmentation result can be obtained.

[0089] In this embodiment of the application, error prediction is performed on multiple candidate segmentation results based on a first image and a second image to obtain the segmentation error of multiple candidate segmentation results, including: superimposing multiple candidate segmentation results with the first image and the second image respectively to obtain multiple superimposed images; and using an error prediction model to perform error prediction on the multiple superimposed images respectively to obtain the segmentation error of multiple candidate segmentation results.

[0090] The aforementioned superimposed image can refer to a first superimposed image consisting of the candidate segmentation result superimposed on the first image, and a second superimposed image consisting of the candidate segmentation result superimposed on the second image.

[0091] In one optional embodiment, multiple segmentation errors corresponding to multiple candidate segmentation results can be determined by superimposing multiple candidate segmentation results with a first image and a second image, and using an error prediction model to predict the error of the superimposed image. Specifically, the first image and the second image can be superimposed according to a preset superimposition rule, such as the edge region of the target part in the first image and the edge region of the target part in the second image, to obtain multiple first superimposed images. At the same time, multiple candidate segmentation results and the second image can be superimposed according to a preset superimposition rule to obtain multiple second superimposed images.

[0092] After obtaining the aforementioned multiple overlay images, a pre-trained error prediction model can be used to predict the errors of each overlay image, thereby determining the segmentation errors corresponding to each of the multiple candidate segmentation results. To ensure the accuracy of the predicted segmentation errors, the differences between the candidate segmentation results and the actual data can be utilized, for example... The segmentation error corresponding to the candidate segmentation result is denoted by the following formula:

[0093] .

[0094] in, This represents the candidate segmentation results. The corresponding segmentation error. Corresponding to the aforementioned candidate segmentation results, the segmentation error may include: { , , , }

[0095] In this embodiment of the application, the target segmentation result is selected from the multiple candidate segmentation results based on the segmentation error of the multiple candidate segmentation results, including: determining the volume of the target part in the multiple candidate segmentation results; and selecting the target segmentation result from the multiple candidate segmentation results based on the segmentation error and the volume.

[0096] In one alternative embodiment, in addition to directly using the candidate segmentation result with the smallest segmentation error as the target segmentation result, in order to ensure the accuracy of the determined target segmentation result, the volume of the target part in the candidate segmentation results can also be combined to determine the target segmentation result from multiple candidate segmentation results.

[0097] Specifically, after determining the segmentation error corresponding to each of the multiple candidate segmentation results, the volume occupied by the target part in the multiple candidate segmentation results can be further determined. Then, based on the ratio of the segmentation error to the volume, the target segmentation result can be selected from the multiple candidate segmentation results. That is, the candidate segmentation result with the smallest ratio of segmentation error to volume can be used as the above-mentioned target segmentation result, thereby improving the rationality and accuracy of the determined target segmentation result.

[0098] In one optional embodiment, multiple first segmentation results can be selected from multiple candidate segmentation results based on the aforementioned segmentation error. For example, multiple segmentation results with segmentation errors less than a preset error threshold can be selected, or the first n segmentation results after sorting the segmentation errors in ascending order can be used as the multiple first segmentation results. At the same time, multiple second segmentation results can be selected from multiple candidate segmentation results based on the volume occupied by the target part. For example, the first m segmentation results after sorting the volume in descending order can be used as the second segmentation results. Then, the segmentation results that are included in both the first and second segmentation results can be used as the target segmentation result.

[0099] In one optional embodiment, after determining multiple segmentation errors, the segmentation errors can be adjusted according to the volume of the target part corresponding to each segmentation error in the candidate segmentation result. For example, a standard volume can be determined according to the aforementioned standard segmentation result, and then the segmentation error can be adjusted according to the ratio of the target part in the candidate segmentation result to the standard volume. Finally, the segmentation result corresponding to the smallest adjusted segmentation error is taken as the aforementioned target segmentation result.

[0100] To facilitate understanding of the image segmentation process described above Figure 4 This is a schematic diagram of a candidate segmentation result determination process according to Embodiment 1 of this application. Figure 5 This is a schematic diagram of a target segmentation result determination process according to Embodiment 1 of this application.

[0101] in, Figure 4 In this context, CT represents a CT image, i.e., the first image; T1 and T2 represent MRI images, i.e., the second image; Candidate 1 refers to the lesion region obtained by image segmentation of the first image, i.e., the aforementioned first candidate segmentation result; Candidate 2 refers to the lesion region obtained by image segmentation of the stitched image obtained by stitching the first and second images together, i.e., the aforementioned second candidate segmentation result; Candidates 3 and 4 refer to the lesion regions obtained by modifying and decoding the image features of the stitched image based on the image features of the cropped image, i.e., the aforementioned third candidate segmentation result; Model 1 refers to the aforementioned first image segmentation model; Model 2 refers to the aforementioned second image segmentation model, i.e., the probabilistic U-NET model; and Model 3 refers to the feature modification model used to modify the image features of the stitched image. To improve segmentation efficiency, Model 3 can be inserted as a modification module into Model 2. Figure 5Model 4 in the text refers to the aforementioned error prediction model. Errors 1, 2, 3, and 4 correspond to the segmentation errors of candidates 1, 2, 3, and 4 when superimposed with different CT or MRI images.

[0102] Taking the segmentation of the primary nasopharyngeal carcinoma lesion as an example, when segmenting the nasopharyngeal carcinoma image to obtain the location of the primary lesion, the following steps can be performed first: Figure 4 The process shown yields multiple candidate segmentation results. Specifically, Figure 4 The process of obtaining multiple candidate segmentation results can be divided into five steps, specifically including: the image segmentation system can first acquire CT and MRI images of biological objects, such as the nasal cavity and neck of a person, i.e. Figure 4 The image segmentation system can segment the CT image (CT, T1, and T2) using a first image segmentation model (Model 1) to obtain a first candidate segmentation result (Candidate 1). Based on the location of the nasopharynx in the CT image, it can then segment the MRI image using a second image segmentation model (Model 2) to obtain a cropped image. The system can also stitch the CT and MRI images together (either CT and T1, CT and T2, or simultaneously) to obtain a stitched image. This stitched image is then further segmented using the second image segmentation model (Model 2). The image segmentation system can extract features from the cropped image, namely, the edge region features and the center region features of the cropped image, as well as the stitched image features. The image segmentation system can modify the features of the stitched image based on the features of the cropped image, and then decode the modified stitched image features to obtain the third candidate segmentation results, namely, candidate 3 and candidate 4. For example, the edge region features can be used to modify the stitched image features, and after decoding, candidate 3 can be obtained; the center region features can be used to modify the stitched image features, and after decoding, candidate 4 can be obtained.

[0103] After obtaining candidate 1, candidate 2, candidate 3, and candidate 4, the image segmentation system can then perform... Figure 5 The process shown involves selecting the target segmentation result from multiple candidate segmentation results to determine the location of the primary nasopharyngeal carcinoma lesion. Specifically, Figure 5The process of selecting the target segmentation result can be divided into three steps, specifically: First, the image segmentation system can perform image recognition on multiple candidate segmentation results to determine the target region in multiple volumes of the multiple candidate segmentation results. Simultaneously, the multiple candidate segmentation results are overlaid with CT and MRI images respectively to obtain multiple overlaid images. Next, the image segmentation system can use an error prediction model, i.e., Model 4, to predict errors in the multiple overlaid images, obtaining multiple segmentation errors corresponding to each of the multiple candidate segmentation results. The system then determines the smallest error among these errors as the target error corresponding to each of the multiple candidate segmentation results. Finally, based on the target error and volume of each candidate segmentation result, the image segmentation system selects the target segmentation result from the multiple candidate segmentation results according to a preset selection method. Figure 5 The target result.

[0104] In this embodiment, a method for segmenting the primary nasopharyngeal carcinoma lesion is also provided. The method includes: acquiring a first image and a second image of the patient's head and neck, wherein the first image is an image obtained by scanning the head and neck using computed tomography (CT) technology, and the second image is an image obtained by scanning the head and neck using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the primary nasopharyngeal carcinoma lesion in the first image to obtain a cropped image; and performing image segmentation on the primary nasopharyngeal carcinoma lesion contained in the second image based on the first image and the cropped image to obtain a target segmentation result.

[0105] Nasopharyngeal carcinoma is a common cancer. The first image mentioned above can refer to an image obtained by scanning the target area using computed tomography (CT) technology, such as a CT image. The second image can refer to an image obtained by scanning the target area using magnetic resonance imaging (MRI) technology, such as an MRI image. It should be noted that the above method for segmenting the primary lesion of nasopharyngeal carcinoma is not only applicable to nasopharyngeal carcinoma but can also be applied to other cancers where the location of the primary lesion can be determined by scanning, such as lung cancer. The corresponding scanning area for the first and second images can be the patient's thoracic cavity. Nasopharyngeal carcinoma is chosen as the cancer to be detected to illustrate the segmentation method for the primary lesion for ease of understanding, not because this method can only segment the primary lesion of nasopharyngeal carcinoma.

[0106] The above-mentioned target segmentation results can refer to the image of the primary nasopharyngeal carcinoma lesion, which can be used to determine the specific location of the primary nasopharyngeal carcinoma lesion.

[0107] In one optional embodiment, in order to accurately segment the primary nasopharyngeal carcinoma lesion and improve the accuracy of nasopharyngeal carcinoma diagnosis, the patient's head and neck can first be scanned using computed tomography (CT) and magnetic resonance imaging (MRI) techniques to obtain the first and second images, namely, CT and MRI images of the patient's head and neck. Then, to further improve the accuracy of identifying the primary nasopharyngeal carcinoma lesion using the first and second images, image recognition can be performed on the first image, i.e., the CT image, to analyze and process the area of ​​the head and neck in the first image. Then, the second image is cropped using this area to obtain the images corresponding to the edge and center parts of the patient's head and neck in the second image, i.e., the cropped image. Finally, the first image and the cropped image are used to segment the area of ​​the head and neck contained in the second image to obtain the target segmentation result. The specific cropping and segmentation process can be as described above and will not be repeated here.

[0108] To facilitate understanding of the segmentation process for nasopharyngeal carcinoma primary lesions, the segmentation process can be divided into at least three parts. The first part is the dataset collection and annotation, where the collected dataset can be used to train multiple models, such as the first image segmentation model, the second image segmentation model, and the error prediction model. The second part is the detailed implementation, which involves using multiple models to process the first and second images to determine the target segmented image, i.e., the nasopharyngeal carcinoma primary lesion. The third part is the comparison and evaluation, which is the process of evaluating the above-mentioned nasopharyngeal carcinoma primary lesion segmentation methods.

[0109] Specifically, in the first part, CT and MRI images generated during the diagnosis and treatment of nasopharyngeal carcinoma patients can be collected to construct the aforementioned dataset. The primary nasopharyngeal carcinoma lesions in these images are then labeled to perform data annotation. For example, when constructing the dataset, multiple CT and MRI images generated during radiotherapy or chemoradiotherapy for 124 nasopharyngeal carcinoma patients can be obtained and used to construct the dataset. The MRI images must include at least T1-weighted and T2-weighted images. Then, when labeling the primary nasopharyngeal carcinoma lesions in the CT and MRI images, several experienced radiation oncology experts can manually label the primary nasopharyngeal carcinoma lesions in the CT and MRI images, referring to the clinical information and relevant diagnostic documents of the aforementioned 124 nasopharyngeal carcinoma patients. This reduces annotation errors caused by labeling differences and ensures the accuracy of the annotation results. In one optional embodiment, to improve the convenience of manual annotation and subsequent use of the annotated data, the collected CT and MRI images can be standardized. For example, the scan size of the CT images can be set to 513×513×145 voxels and the average resolution to 0.98×0.98×2.89 mm, and the scan size of the MRI images can be set to 448×448×19 voxels and the average resolution to 0.57×0.57×6.00 mm. Furthermore, to improve the accuracy of the trained models when using the dataset to train the aforementioned multiple models, the data in the dataset can be randomly grouped. For example, 60% of the data can be randomly grouped into one group, 30% into another, and 10% into yet another. Then, the grouped data can be used for model training, validation, and testing.

[0110] In the second part, a probabilistic U-Net model can be used to segment nasopharyngeal carcinoma lesions in CT and MRI images. As mentioned earlier, to ensure the accuracy of the segmented nasopharyngeal carcinoma lesions, an encoder can be added to the probabilistic U-Net model, namely the aforementioned adjustment and modification model. This encoder modifies the stitched image features using the cropped image features to obtain modified image features. Then, the decoder in the probabilistic U-Net model is used to decode the modified image features to obtain multiple candidate segmentation results. Finally, the segmentation error of the multiple candidate segmentation results is used to determine the target segmentation result. In an optional scheme of this embodiment, to improve the applicability of the modified probabilistic U-Net model, the dimension of the latent space of the modified probabilistic U-Net model can be set to 6 dimensions, the input size of the training input patch can be set to (64, 192, 192), and the data augmentation of the encoder can be performed. That is, the parameter transformation methods used when modifying the image features can be set to at least include: rotation, scaling, mirroring, and gamma transformation, and the batch size can be set to 2. When training the probabilistic U-Net model, the initial learning rate can be set to 0.01, and a polygon decay scheme with an exponent of 0.9 can be used to train the probabilistic U-Net model for 1000 training cycles, thereby improving the accuracy of nasopharyngeal carcinoma primary lesions segmented by the probabilistic U-Net model.

[0111] In the third part, the primary lesions manually marked by the aforementioned radiation oncology experts can be used to compare the above-mentioned nasopharyngeal carcinoma segmentation method with the traditional primary lesion segmentation method using four quantitative indicators. The above-mentioned nasopharyngeal carcinoma segmentation method can be evaluated, analyzed, and improved. The four quantitative indicators can be: Dice score (a medical indicator, abbreviated as DSC%), 95% Hausdorff distance (HD95), average surface distance (ASD), and median surface distance (MedianSD).

[0112] Table 1

[0113]

[0114] Table 1 above is an evaluation table of a nasopharyngeal carcinoma segmentation method according to an embodiment of this application. In the table, EF represents early fusion, TB represents tumor boundary (the edge region of the head and neck), and TC represents tumor center (the central region of the head and neck). Table 1 shows the quantitative evaluation results of different segmentation methods. The data in rows 2-5 of Table 1 are the evaluation results of different quantitative indicators under single-modal CT, multimodal CT & MRI, context-aware multimodal CT & MRI TB, and MRI TC states, respectively. As shown in Table 1, the single-modal segmentation model achieved a DSC of 76.6%. With the early fusion of CT and MRI, the multimodal segmentation model slightly improved the DSC to 77.5%, and the corresponding distance metrics (i.e., HD95, ASD, and MedianSD) all decreased slightly, indicating enhanced segmentation performance. The segmentation model with context-aware input (CT using MRI at the tumor boundary / center) was generally comparable to the early fusion model, with DSCs of 77.3% and 77.8%, respectively, and no significant change in segmentation performance. This means that when potential registration errors exist, the improvement that the multimodal segmentation model can achieve compared to the single-modal segmentation model is limited. The sixth row of Table 1 shows the quantitative evaluation results of the segmentation of the primary nasopharyngeal carcinoma lesion after the dual-stream fusion of the single-modal and multi-modal methods described above. The data in rows 7 and 8 of Table 1 are the quantitative evaluation results of the target segmentation result after constructing multiple candidate segmentation results as described above in this application. Among them, the seventh row shows the evaluation results when there are only two candidate segmentation results, and the eighth row shows the evaluation results when there are four candidate segmentation results. When there are only two candidate segmentation results, the DSC of the segmentation model, namely the aforementioned probabilistic U-Net model, can be improved to 78.5%, which is 1.9% higher than that of the single-modal segmentation model. When there are four candidate segmentation results, the DSC of the segmentation model is further improved to 79.6%, which is a significant improvement compared to the DSC under any modality. At the same time, compared with the single-modal segmentation model, the HD95 is reduced by 1.08 mm and the ASD is reduced by 0.42 mm. Compared with the multi-modal segmentation model, the HD95 is reduced by 0.92 mm and the ASD is reduced by 0.23 mm, which are also significant reductions. The last row in Table 1 represents the quantitative evaluation results of the nasopharyngeal carcinoma segmentation method proposed in this application. As can be seen from Table 1, the nasopharyngeal carcinoma primary lesions segmented by the lesion segmentation method proposed in this application have better results.

[0115] For ease of understanding, Figure 6 This is a schematic diagram comparing segmentation results according to Embodiment 1 of this application, as shown below. Figure 6As shown, the first column represents the segmentation results of the single-modal segmentation model (CT), the second column represents the segmentation results of the multimodal segmentation model (CT & MRI), the third column represents the segmentation results of the multimodal segmentation model (CT & MRI TB) with early fusion of edge regions, and the fourth column represents the segmentation results of the multimodal segmentation model (CT & MRI TC) with early fusion of edge regions. The first row represents the segmentation results predicted using CT images, the second row represents the segmentation results predicted using CT and MRI images, and the third row represents the final target segmentation result shown on the CT image. The image includes the extent of the nasopharyngeal carcinoma primary lesion manually annotated by radiation oncology experts, as well as the extent of the nasopharyngeal carcinoma primary lesion predicted by the above different segmentation models. The range corresponding to the black irregular box in the middle of each image is the predicted extent of the nasopharyngeal carcinoma primary lesion, and the range corresponding to the gray irregular box next to the black irregular box is the manually annotated extent of the nasopharyngeal carcinoma primary lesion. The DSC scores of different images can be seen in the upper right corner of the image.

[0116] As can be seen from the above, the method for segmenting primary nasopharyngeal carcinoma lesions proposed in this application, by constructing a context-aware multimodal selection framework using CT and MRI images, achieves higher segmentation accuracy and better segmentation results compared to traditional primary lesion segmentation methods. Compared to traditional single-modal segmentation models and multimodal segmentation models, it improves the DSC score by 3.0% and 2.1%, respectively. Furthermore, it exhibits extremely high similarity to primary nasopharyngeal carcinoma lesions manually annotated by radiation oncology experts, better aligning with the annotation capabilities of radiation oncology experts, and demonstrating excellent segmentation results for primary nasopharyngeal carcinoma lesions.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0118] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0120] Example 2

[0121] According to an embodiment of this application, an image segmentation method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0122] Figure 7 This is a flowchart of an image segmentation method according to Embodiment 2 of this application, as follows: Figure 7 As shown, the method may include the following steps:

[0123] In step S702, in response to an input command applied to the operation interface, a first image and a second image of the target part of the biological object are displayed on the operation interface.

[0124] The target area includes the lesion region. The first image is obtained by scanning the target area using computed tomography (CT) technology, and the second image is obtained by scanning the target area using magnetic resonance imaging (MRI) technology.

[0125] The aforementioned input command may refer to a command used to display a first image and a second image in a preset operation interface. The first image may refer to an image obtained by tomographic scanning of the lesion area, such as a CT image, and the second image may refer to an image obtained by magnetic resonance imaging of the lesion area, such as an MRI image.

[0126] In one optional embodiment, after receiving the input command, the image segmentation system can first acquire a first image and a second image containing the target area, namely the CT image and MRI image mentioned above, and display the first image and the second image in a preset operation interface for easy viewing by staff.

[0127] Step S704: In response to the image segmentation command applied to the operation interface, the target segmentation result is displayed on the operation interface. The target segmentation result is obtained by segmenting the target part contained in the second image based on the first image and the cropped image. The cropped image is obtained by cropping the second image based on the region of the target part in the first image.

[0128] The aforementioned image segmentation instruction can refer to an instruction to segment the target segmentation result from the second image, that is, the image of the region where the lesion area is located. The region of the target location in the first image can refer to the region where the diseased organ or tissue is located in the first image, that is, the region where the aforementioned target location is located, which also includes the lesion area.

[0129] In one optional embodiment, after receiving the image segmentation command, the image segmentation system can first perform image recognition on the first image, i.e., the CT image, to analyze and process the region of the target site in the first image, i.e., the region in the first image where an organ or tissue lesion occurs. Then, it uses this region to crop the second image, obtaining the images corresponding to the edge and center portions of the lesion region in the second image, i.e., the cropped image. Then, it uses the first image and the cropped image to perform image registration and image segmentation on the region of the target site in the second image. That is, it can register the first image and the cropped image, and finally, it performs image segmentation on the region of the target site in the second image based on the registration result, thereby determining the location of the primary lesion in the lesion region, and thus determining the target segmentation result. After determining the target segmentation result, it can also output the target segmentation result in a preset operation interface for easy viewing by staff.

[0130] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0131] Example 3

[0132] According to the embodiments of this application, an image segmentation method applicable to virtual reality scenarios such as virtual reality (VR) devices and augmented reality (AR) devices is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0133] Figure 8 This is a flowchart of an image segmentation method according to Embodiment 3 of this application. Figure 8 As shown, the method may include the following steps:

[0134] Step S802: Display a first image and a second image of the target part of the biological object on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device.

[0135] The target area includes the lesion region. The first image is an image obtained by scanning the target area using computed tomography (CT) technology, and the second image is an image obtained by scanning the target area using magnetic resonance imaging (MRI) technology.

[0136] In one optional embodiment, to make the determined target segmentation result more accurate, a first image and a second image containing the target area of ​​the biological object can be acquired using a virtual reality (VR) device or an augmented reality (AR) device, and the first and second images can be displayed in the VR or AR device to facilitate observation of the lesion area in the image by staff. The first image can be an image obtained by tomographic scanning of the lesion area, such as a CT image, and the second image can be an image obtained by magnetic resonance imaging of the lesion area, such as an MRI image.

[0137] Step S804: Based on the region of the target part in the first image, the second image is cropped to obtain a cropped image.

[0138] In one optional embodiment, in order to improve the accuracy of the location of the primary lesion in the segmented lesion region, the first image and the second image, namely the CT image and the MRI image, can be cropped to obtain the cropped region mentioned above, and the location of the primary lesion in the lesion region can be determined by using the cropped region.

[0139] Specifically, considering that both CT and MRI images can reflect the diseased organs or tissues within a biological object, thereby identifying the aforementioned lesion area, and that CT images are digitally reconstructed analog images with high density resolution, capable of displaying the density of organs and lesions at different gray levels, and easily undergoing computer post-processing, such as two-dimensional or three-dimensional display analysis, to clearly show the location of the lesion area in the image, and that MRI images have multiple imaging sequences, which can display multiple different images with high tissue resolution and multiple parameters for the same lesion area, when determining the cropped image, image recognition can be performed on the first image, i.e., the CT image, to analyze and process the area of ​​the target location in the first image, i.e., the area where the organ or tissue disease occurs in the first image, and then this area can be used to crop the second image to obtain the image corresponding to the edge and center of the lesion area in the second image, i.e., the aforementioned cropped image.

[0140] Step S806: Based on the first image and the cropped image, perform image segmentation on the target region contained in the second image to obtain the target segmentation result;

[0141] In one optional embodiment, since the errors generated during the registration of CT and MRI images are unavoidable, when performing image segmentation on the target area to determine the target segmentation result, multiple image recognition and image segmentation can be performed on the same second image to determine multiple first registration errors. Alternatively, multiple substitute images can be determined from other images in the imaging sequence where the second image is located. Then, image recognition and image segmentation can be performed on these multiple substitute images to determine multiple second registration errors. Finally, the segmentation result with the smallest registration error is selected as the target segmentation result, thereby improving the accuracy of the determined target segmentation result and solving the technical problem of low accuracy in segmenting the location of the primary lesion in the lesion area in related technologies.

[0142] Step S808: Drive the VR or AR device to render and display the target segmentation results.

[0143] In one optional embodiment, a VR or AR device can be driven to render the target segmentation results, enabling staff to more intuitively observe relevant information about the primary lesion in the target area.

[0144] Optionally, in this embodiment, the image segmentation method described above can be applied to a hardware environment consisting of a server and a virtual reality device. The first image, second image, cropped image, and target segmentation result of the target part of the biological object are displayed on the presentation screen of the virtual reality (VR) device or augmented reality (AR) device. The server can be a server corresponding to a media file operator. The aforementioned network includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The aforementioned virtual reality device is not limited to, for example, a virtual reality headset, virtual reality glasses, or a standalone virtual reality device.

[0145] In this embodiment, the virtual reality device includes a memory, a processor, and a transmission device. The memory stores an application program that can be used to perform the following: cropping a second image based on a region of a target part in a first image to obtain a cropped image, including: performing image segmentation on the target part contained in the first image to obtain an image segmentation result of the first image, wherein the image segmentation result is used to characterize the region of the target part in the first image; and cropping the second image based on the image segmentation result to obtain the cropped image.

[0146] In this embodiment of the application, the application is further configured to perform: cropping the second image based on the image segmentation result to obtain a cropped image, where the image segmentation result is used to characterize the edge of the target region in the first image as a boundary region. This includes: performing dilation processing on the image segmentation result based on a preset dilation parameter to obtain a dilation processing result; performing erosion processing on the image segmentation result based on a preset erosion parameter to obtain an erosion processing result; obtaining the difference between the dilation processing result and the erosion processing result to obtain the target boundary region; and cropping the second image based on the target boundary region to obtain a cropped image.

[0147] In this embodiment of the application, the application is further configured to perform: cropping the second image based on the image segmentation result to obtain a cropped image, where the image segmentation result is used to characterize the center of the target region in the central region of the first image, including: performing erosion processing on the image segmentation result based on preset erosion parameters to obtain the target central region; and cropping the second image based on the target central region to obtain the cropped image.

[0148] In this embodiment of the application, the application is further configured to perform: image segmentation of the target region contained in the second image based on the first image and the cropped image to obtain a target segmentation result, including: performing image segmentation on the first image, and performing image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain multiple candidate segmentation results; performing error prediction on the multiple candidate segmentation results based on the first image and the second image to obtain the segmentation error of the multiple candidate segmentation results, wherein different segmentation errors are used to characterize the errors between different candidate segmentation results and preset segmentation results; and selecting the target segmentation result from the multiple candidate segmentation results based on the segmentation error of the multiple candidate segmentation results.

[0149] In this embodiment of the application, the application is further configured to perform: image segmentation on the first image, and image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain multiple candidate segmentation results, including: image segmentation on the target region contained in the first image to obtain a first candidate segmentation result; stitching the first image and the second image to obtain a stitched image; image segmentation on the stitched image to obtain a second candidate segmentation result; image segmentation on the stitched image based on the cropped image to obtain a third candidate segmentation result; and obtaining multiple candidate segmentation results based on the first candidate segmentation result, the second candidate segmentation result, and the third candidate segmentation result.

[0150] In this embodiment of the application, the application is further configured to perform: image segmentation of the stitched image based on the cropped image to obtain a third candidate segmentation result, including: extracting features from the stitched image to obtain stitched image features; extracting features from the cropped image to obtain cropped image features; modifying the stitched image features based on the cropped image features to obtain modified image features; and decoding the modified image features to obtain the third candidate segmentation result.

[0151] In this embodiment of the application, the application is further configured to perform: error prediction on multiple candidate segmentation results based on the first image and the second image to obtain the segmentation error of the multiple candidate segmentation results, including: superimposing the multiple candidate segmentation results with the first image and the second image respectively to obtain multiple superimposed images; and using an error prediction model to perform error prediction on the multiple superimposed images respectively to obtain the segmentation error of the multiple candidate segmentation results.

[0152] In this embodiment of the application, the application is also used to perform: selecting a target segmentation result from multiple candidate segmentation results based on the segmentation error of multiple candidate segmentation results, including: determining the volume of the target part in multiple candidate segmentation results; and selecting the target segmentation result from multiple candidate segmentation results based on the segmentation error and the volume.

[0153] It should be noted that the image segmentation method described above for use in VR or AR devices in this embodiment may include... Figure 8 The method of the illustrated embodiment is intended to drive a VR device or AR device to display a first image, a second image, a cropped image, and a target segmentation result of a target part of a biological object.

[0154] Optionally, the processor in this embodiment can invoke the application stored in the memory via the transmission device to perform the above steps. The transmission device can receive media files sent by the server via a network, and can also be used for data transmission between the processor and the memory.

[0155] Optionally, in a virtual reality device, there is a head-mounted display with eye tracking. The screen in the HMD is used to display the video footage. The eye tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The tracking system is used to track the user's position and movement information in real three-dimensional space. The computing and processing unit is used to acquire the user's real-time position and movement information from the tracking system and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of vision orientation in the virtual three-dimensional space.

[0156] In this embodiment, the virtual reality device can be connected to a terminal, and the terminal and the server are connected via a network. The virtual reality device is not limited to virtual reality headsets, virtual reality glasses, virtual reality all-in-one machines, etc., and the terminal is not limited to PCs, mobile phones, tablets, etc. The server can be a server corresponding to a media file operator, and the network includes, but is not limited to, wide area networks, metropolitan area networks, or local area networks.

[0157] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0158] Example 4

[0159] According to an embodiment of this application, an image segmentation method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0160] Figure 9 This is a flowchart of an image segmentation method according to Embodiment 4 of this application, as follows: Figure 9 As shown, the method may include the following steps:

[0161] Step S902: Obtain the first image and the second image of the target part of the biological object by calling the first interface.

[0162] The first interface includes a first parameter, the parameter value of which is a first image and a second image. The target area includes a lesion region. The first image is an image obtained by scanning the target area using computed tomography (CT) technology, and the second image is an image obtained by scanning the target area using magnetic resonance imaging (MRI) technology.

[0163] The aforementioned lesion area can refer to the lesion area caused by the disease of the tissues and organs of a biological object, such as the lesion area of ​​appendicitis, the lesion area of ​​nasopharyngeal carcinoma, etc. The aforementioned first image can refer to the image obtained by tomographic scanning of the lesion area, such as a CT image. The aforementioned second image can refer to the image obtained by magnetic resonance imaging of the lesion area, such as an MRI image. For ease of understanding, the following text will refer to the first image as a CT image and the second image as an MRI image.

[0164] In one optional embodiment, to ensure the accuracy of determining the location of the primary lesion within the lesion area, the image segmentation system can first acquire a first image and a second image containing the target area, namely the aforementioned CT image and MRI image. Then, by combining the CT and MRI images, the lesion area is analyzed to determine the specific location of the primary lesion. The target area includes at least the lesion region where the disease has occurred. The first image can be obtained by scanning the target area using computed tomography (CT) technology, and the second image can be obtained by scanning the target area using magnetic resonance imaging (MRI).

[0165] Step S904: Based on the region of the target part in the first image, the second image is cropped to obtain a cropped image.

[0166] Considering that both CT and MRI images can reflect the diseased organs or tissues within a biological object, thereby identifying the aforementioned lesion area, and that CT images are digitally reconstructed analog images with high density resolution, capable of displaying the density of organs and lesions at different gray levels, and easily undergoing computer post-processing, such as two-dimensional or three-dimensional display analysis, to clearly show the location of the lesion area in the image, and that MRI images have multiple imaging sequences, which can display multiple different images with high tissue resolution and multiple parameters for the same lesion area, when determining the cropped image, image recognition can be performed on the first image, i.e., the CT image, to analyze and process the area of ​​the target site in the first image, i.e., the area where the organ or tissue disease occurs in the first image, and then this area can be used to crop the second image to obtain the image corresponding to the edge and center of the lesion area in the second image, i.e., the cropped image.

[0167] Step S906: Based on the first image and the cropped image, perform image segmentation on the target region contained in the second image to obtain the target segmentation result.

[0168] The above-mentioned target segmentation result can refer to the segmented image that can represent the information of the primary lesion in the lesion area.

[0169] In one optional embodiment, as described above, considering that the second image, i.e., the MRI image, has higher tissue resolution and no bone artifacts compared to the first image, i.e., the CT image, some lesion areas, such as the lesion area corresponding to the nasopharyngeal carcinoma mentioned above, will not be affected by the bony structure of the neck, after obtaining the cropped image, the first image and the cropped image can be used to perform image registration and image segmentation on the area containing the target part in the second image. That is, the first image and the cropped image can be registered, and then the area containing the target part in the second image can be segmented according to the registration result, thereby determining the location of the primary lesion in the lesion area, and thus determining the target segmentation result.

[0170] Step S908: Output the target segmentation result by calling the second interface.

[0171] The second interface includes a second parameter, the value of which is the target segmentation result.

[0172] After obtaining the target segmentation result, the image segmentation system can output the target segmentation to a preset operation interface through a preset second interface, so that staff can easily view the target segmentation result.

[0173] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0174] Example 5

[0175] According to an embodiment of this application, an apparatus for implementing the above-described image segmentation method is also provided, which can be deployed in a target client. Figure 10 This is a structural block diagram of an image segmentation apparatus according to Embodiment 5 of this application, as follows: Figure 10 As shown, the device 1000 includes: an acquisition module 1002, a cutting module 1004, and a segmentation module 1006.

[0176] The acquisition module 1002 is used to acquire a first image and a second image of the target part of the biological object. The target part includes a lesion area. The first image is obtained by scanning the target part using computed tomography (CT) technology, and the second image is obtained by scanning the target part using magnetic resonance imaging (MRI) technology. The cropping module 1004 is used to crop the second image based on the area of ​​the target part in the first image to obtain a cropped image. The segmentation module 1006 is used to perform image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result.

[0177] In the above embodiments of this application, the cropping module 1004 includes: a first segmentation unit, used to perform image segmentation on the target part contained in the first image to obtain an image segmentation result of the first image, wherein the image segmentation result is used to characterize the region of the target part in the first image; and an image cropping unit, used to crop the second image based on the image segmentation result to obtain a cropped image.

[0178] In the above embodiments of this application, when the image segmentation result is used to characterize the boundary region of the target part in the first image, the image cropping unit is further configured to: perform expansion processing on the image segmentation result based on a preset expansion parameter to obtain an expansion processing result; perform erosion processing on the image segmentation result based on a preset erosion parameter to obtain an erosion processing result; obtain the difference between the expansion processing result and the erosion processing result to obtain the target boundary region; and crop the second image based on the target boundary region to obtain a cropped image.

[0179] In the above embodiments of this application, when the image segmentation result is used to characterize the center of the target part in the central region of the first image, the image cropping unit is further configured to: perform erosion processing on the image segmentation result based on preset erosion parameters to obtain the target central region; and crop the second image based on the target central region to obtain a cropped image.

[0180] In the above embodiments of this application, the segmentation module 1006 includes: a second segmentation unit, used to perform image segmentation on the first image and to perform image segmentation on the target part contained in the second image based on the first image and the cropped image, to obtain multiple candidate segmentation results; an error prediction unit, used to perform error prediction on the multiple candidate segmentation results based on the first image and the second image, to obtain the segmentation error of the multiple candidate segmentation results, wherein different segmentation errors are used to characterize the error between different candidate segmentation results and preset segmentation results; and a result filtering unit, used to filter out the target segmentation result from the multiple candidate segmentation results based on the segmentation error of the multiple candidate segmentation results.

[0181] In the above embodiments of this application, the second segmentation unit is further configured to: perform image segmentation on the target part contained in the first image to obtain a first candidate segmentation result; stitch the first image and the second image together to obtain a stitched image; perform image segmentation on the stitched image to obtain a second candidate segmentation result; perform image segmentation on the stitched image based on the cropped image to obtain a third candidate segmentation result; and obtain multiple candidate segmentation results based on the first candidate segmentation result, the second candidate segmentation result, and the third candidate segmentation result.

[0182] In the above embodiments of this application, the second segmentation unit is further configured to: extract features from the stitched image to obtain stitched image features; extract features from the cropped image to obtain cropped image features; modify the stitched image features based on the cropped image features to obtain modified image features; and decode the modified image features to obtain a second candidate segmentation result.

[0183] In the above embodiments of this application, the error prediction unit is further configured to: superimpose multiple candidate segmentation results with the first image and the second image respectively to obtain multiple superimposed images; and use the error prediction model to perform error prediction on the multiple superimposed images respectively to obtain the segmentation error of the multiple candidate segmentation results.

[0184] In the above embodiments of this application, the result filtering unit is further configured to: determine the volume of the target part in multiple candidate segmentation results; and filter out the target segmentation result from multiple candidate segmentation results based on the segmentation error and the volume.

[0185] It should be noted that the acquisition module 1002, cropping module 1004, and segmentation module 1006 mentioned above correspond to steps S302 to S306 in Embodiment 1. The instances and application scenarios implemented by these three modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that the aforementioned modules or units can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of a device in the AR / VR device provided in Embodiment 1.

[0186] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0187] Example 6

[0188] According to an embodiment of this application, an apparatus for implementing the above-described image segmentation method is also provided, which can be deployed in a target client. Figure 11 This is a structural block diagram of an image segmentation apparatus according to Embodiment 7 of this application, as follows: Figure 11 As shown, the device 1100 includes: a first display module 1102 and a second display module 1104.

[0189] The first display module 1102 is used to respond to input commands applied to the operation interface and display a first image and a second image of the target part of the biological object on the operation interface. The target part includes a lesion area. The first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology. The second display module 1104 is used to respond to image segmentation commands applied to the operation interface and display the target segmentation result on the operation interface. The target segmentation result is obtained by segmenting the target part contained in the second image based on the first image and the cropped image. The cropped image is obtained by cropping the second image based on the area of ​​the target part in the first image.

[0190] It should be noted that the first display module 1102 and the second display module 1104 mentioned above correspond to steps S702 to S704 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 2. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the AR / VR device provided in Embodiment 1.

[0191] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0192] Example 7

[0193] According to an embodiment of this application, an apparatus for implementing the above-described image segmentation method is also provided, which can be deployed in a target client. Figure 12 This is a structural block diagram of an image segmentation apparatus according to Embodiment 8 of this application, as follows: Figure 12 As shown, the device 1200 includes: a first display module 1202, a cutting module 1204, a segmentation module 1206, and a second display module 1208.

[0194] The first display module 1202 is used to display a first image and a second image of a target part of a biological object on the display screen of a virtual reality (VR) device or an augmented reality (AR) device. The target part includes a lesion area. The first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology. The cropping module 1204 is used to crop the second image based on the area of ​​the target part in the first image to obtain a cropped image. The segmentation module 1206 is used to perform image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result. The second display module 1208 is used to drive the VR device or AR device to render and display the target segmentation result.

[0195] It should be noted that the first display module 1202, the cropping module 1204, the segmentation module 1206, and the second display module 1208 mentioned above correspond to steps S802 to S808 in Embodiment 3. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 3. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the AR / VR device provided in Embodiment 1.

[0196] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0197] Example 8

[0198] According to an embodiment of this application, an apparatus for implementing the above-described image segmentation method is also provided, which can be deployed in a target client. Figure 13 This is a structural block diagram of an image segmentation apparatus according to Embodiment 9 of this application, as follows: Figure 13 As shown, the device 1300 includes: an acquisition module 1302, a cutting module 1304, a segmentation module 1306, and an output module 1308.

[0199] The acquisition module 1302 is used to acquire a first image and a second image of the target part of the biological object by calling a first interface. The first interface includes a first parameter, the value of which is the first image and the second image. The target part includes a lesion area. The first image is obtained by scanning the target part using computed tomography (CT) technology, and the second image is obtained by scanning the target part using magnetic resonance imaging (MRI). The cropping module 1304 is used to crop the second image based on the area of ​​the target part in the first image to obtain a cropped image. The segmentation module 1306 is used to perform image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result. The output module 1308 is used to output the target segmentation result by calling a second interface. The second interface includes a second parameter, the value of which is the target segmentation result.

[0200] It should be noted that the acquisition module 1302, cropping module 1304, segmentation module 1306, and output module 1308 mentioned above correspond to steps S902 to S908 in Embodiment 4. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 4. It should also be noted that the modules or units mentioned above can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of a device in the AR / VR device provided in Embodiment 1.

[0201] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0202] Example 9

[0203] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0204] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of grid devices in a computer grid.

[0205] In this embodiment, the aforementioned electronic device can execute the program code for the following steps in the image segmentation method: acquiring a first image and a second image of the target part of a biological object, wherein the target part includes a lesion region, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the target part in the first image to obtain a cropped image; and performing image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result.

[0206] Optionally, Figure 14 This is a structural block diagram of an electronic device according to Embodiment 10 of this application. As shown in the figure, the electronic device A may include: a processor 1402 and a memory 1404, wherein an executable program is stored; the processor is used to run the program, wherein the program executes the virtual object generation method shown in Embodiment 1 when it runs.

[0207] Optionally, such as Figure 14 As shown, electronic device A may also include a storage controller and a peripheral interface, wherein the peripheral interface is connected to the radio frequency module, the audio module and the display.

[0208] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image segmentation method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image segmentation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a grid. Examples of such grids include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0209] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: cropping the second image based on the region of the target part in the first image to obtain a cropped image, including: performing image segmentation on the target part contained in the first image to obtain an image segmentation result of the first image, wherein the image segmentation result is used to characterize the region of the target part in the first image; and cropping the second image based on the image segmentation result to obtain a cropped image.

[0210] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: when the image segmentation result is used to characterize the boundary region of the target part in the first image, cropping the second image based on the image segmentation result to obtain a cropped image, including: performing dilation processing on the image segmentation result based on preset dilation parameters to obtain a dilation processing result; performing erosion processing on the image segmentation result based on preset erosion parameters to obtain an erosion processing result; obtaining the difference between the dilation processing result and the erosion processing result to obtain the target boundary region; and cropping the second image based on the target boundary region to obtain a cropped image.

[0211] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: when the image segmentation result is used to characterize the center of the target part in the central region of the first image, the second image is cropped based on the image segmentation result to obtain a cropped image, including: performing erosion processing on the image segmentation result based on preset erosion parameters to obtain the target central region; and cropping the second image based on the target central region to obtain a cropped image.

[0212] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: performing image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain a target segmentation result, including: performing image segmentation on the first image, and performing image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain multiple candidate segmentation results; performing error prediction on the multiple candidate segmentation results based on the first image and the second image to obtain the segmentation error of the multiple candidate segmentation results, wherein different segmentation errors are used to characterize the errors between different candidate segmentation results and preset segmentation results; and selecting the target segmentation result from the multiple candidate segmentation results based on the segmentation errors of the multiple candidate segmentation results.

[0213] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: performing image segmentation on a first image, and performing image segmentation on the target region contained in a second image based on the first image and the cropped image to obtain multiple candidate segmentation results, including: performing image segmentation on the target region contained in the first image to obtain a first candidate segmentation result; stitching the first image and the second image to obtain a stitched image; performing image segmentation on the stitched image to obtain a second candidate segmentation result; performing image segmentation on the stitched image based on the cropped image to obtain a third candidate segmentation result; and obtaining multiple candidate segmentation results based on the first candidate segmentation result, the second candidate segmentation result, and the third candidate segmentation result.

[0214] The processor can access the information and application program stored in the memory via the transmission device to perform the following steps: performing image segmentation on the stitched image based on the cropped image to obtain a third candidate segmentation result, including: extracting features from the stitched image to obtain stitched image features; extracting features from the cropped image to obtain cropped image features; modifying the stitched image features based on the cropped image features to obtain modified image features; and decoding the modified image features to obtain the third candidate segmentation result.

[0215] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: perform error prediction on multiple candidate segmentation results based on the first image and the second image to obtain the segmentation error of multiple candidate segmentation results, including: superimposing multiple candidate segmentation results with the first image and the second image respectively to obtain multiple superimposed images; and using the error prediction model to perform error prediction on the multiple superimposed images respectively to obtain the segmentation error of multiple candidate segmentation results.

[0216] The processor can access information and application programs stored in memory via a transmission device to perform the following steps: selecting a target segmentation result from multiple candidate segmentation results based on the segmentation error of multiple candidate segmentation results, including: determining the volume of the target region in multiple candidate segmentation results; and selecting the target segmentation result from multiple candidate segmentation results based on the segmentation error and the volume.

[0217] In this embodiment, a first image and a second image of the target part of a biological object are acquired; based on the region of the target part in the first image, the second image is cropped to obtain a cropped image; image segmentation is performed on the target part contained in the second image based on the first image and the cropped image to obtain the target segmentation result. By using the region of the target part in the first image to crop the second image, the cropped image of the target part in the second image is determined, thereby reducing the influence of other images in the second image other than the cropped image on the segmentation of the lesion region. Finally, the first image and the cropped image are used to segment the second image to obtain the target segmentation result, improving the clarity of local details in the lesion region, ensuring that the segmented target segmentation result is not distorted, thereby improving the segmentation accuracy of the lesion region, and thus solving the technical problem of low accuracy in segmenting lesion regions in CT and MRI images in related technologies.

[0218] Those skilled in the art will understand that Figure 14 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 14This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more... Figure 14 The more or fewer components shown (such as mesh interfaces, display devices, etc.), or having the same Figure 14 The different configurations shown.

[0219] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0220] Example 10

[0221] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image segmentation method provided in Embodiment 1.

[0222] In this embodiment, the storage medium can be located in any computer terminal in the group of computer terminals in the computer grid, or in any mobile terminal in the group of mobile terminals.

[0223] In this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a first image and a second image of a target part of a biological object, wherein the target part includes a lesion area, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology; cropping the second image based on the region of the target part in the first image to obtain a cropped image; and performing image segmentation on the target part contained in the second image based on the first image and the cropped image to obtain a target segmentation result.

[0224] In this embodiment, the storage medium is configured to store program code for performing the following steps: cropping a second image based on the region of the target part in the first image to obtain a cropped image, including: performing image segmentation on the target part contained in the first image to obtain an image segmentation result of the first image, wherein the image segmentation result is used to characterize the region of the target part in the first image; and cropping the second image based on the image segmentation result to obtain a cropped image.

[0225] In this embodiment, the storage medium is configured to store program code for performing the following steps: when the image segmentation result is used to characterize the boundary region of the target part in the first image, cropping the second image based on the image segmentation result to obtain a cropped image, including: performing dilation processing on the image segmentation result based on a preset dilation parameter to obtain a dilation processing result; performing erosion processing on the image segmentation result based on a preset erosion parameter to obtain an erosion processing result; obtaining the difference between the dilation processing result and the erosion processing result to obtain the target boundary region; and cropping the second image based on the target boundary region to obtain a cropped image.

[0226] In this embodiment, the storage medium is configured to store program code for performing the following steps: when the image segmentation result is used to characterize the center of the target region in the central region of the first image, the second image is cropped based on the image segmentation result to obtain a cropped image, including: performing erosion processing on the image segmentation result based on preset erosion parameters to obtain the target central region; and cropping the second image based on the target central region to obtain the cropped image.

[0227] In this embodiment, the storage medium is configured to store program code for performing the following steps: performing image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain a target segmentation result, including: performing image segmentation on the first image, and performing image segmentation on the target region contained in the second image based on the first image and the cropped image to obtain multiple candidate segmentation results; performing error prediction on the multiple candidate segmentation results based on the first image and the second image to obtain the segmentation error of the multiple candidate segmentation results, wherein different segmentation errors are used to characterize the errors between different candidate segmentation results and preset segmentation results; and selecting the target segmentation result from the multiple candidate segmentation results based on the segmentation errors of the multiple candidate segmentation results.

[0228] In this embodiment, the storage medium is configured to store program code for performing the following steps: performing image segmentation on a first image, and performing image segmentation on a target region contained in a second image based on the first image and a cropped image to obtain multiple candidate segmentation results, including: performing image segmentation on the target region contained in the first image to obtain a first candidate segmentation result; stitching the first image and the second image to obtain a stitched image; performing image segmentation on the stitched image to obtain a second candidate segmentation result; performing image segmentation on the stitched image based on the cropped image to obtain a third candidate segmentation result; and obtaining multiple candidate segmentation results based on the first candidate segmentation result, the second candidate segmentation result, and the third candidate segmentation result.

[0229] In this embodiment, the storage medium is configured to store program code for performing the following steps: performing image segmentation on the stitched image based on the cropped image to obtain a third candidate segmentation result, including: extracting features from the stitched image to obtain stitched image features; extracting features from the cropped image to obtain cropped image features; modifying the stitched image features based on the cropped image features to obtain modified image features; and decoding the modified image features to obtain the third candidate segmentation result.

[0230] In this embodiment, the storage medium is configured to store program code for performing the following steps: performing error prediction on multiple candidate segmentation results based on a first image and a second image to obtain the segmentation error of multiple candidate segmentation results, including: superimposing multiple candidate segmentation results with the first image and the second image respectively to obtain multiple superimposed images; and using an error prediction model to perform error prediction on the multiple superimposed images respectively to obtain the segmentation error of multiple candidate segmentation results.

[0231] In this embodiment, the storage medium is configured to store program code for performing the following steps: selecting a target segmentation result from multiple candidate segmentation results based on the segmentation error of multiple candidate segmentation results, including: determining the volume of the target part in multiple candidate segmentation results; and selecting the target segmentation result from multiple candidate segmentation results based on the segmentation error and the volume.

[0232] In this embodiment, a first image and a second image of the target part of a biological object are acquired; based on the region of the target part in the first image, the second image is cropped to obtain a cropped image; image segmentation is performed on the target part contained in the second image based on the first image and the cropped image to obtain the target segmentation result. By using the region of the target part in the first image to crop the second image, the cropped image of the target part in the second image is determined, thereby reducing the influence of other images in the second image other than the cropped image on the segmentation of the lesion region. Finally, the first image and the cropped image are used to segment the second image to obtain the target segmentation result, improving the clarity of local details in the lesion region, ensuring that the segmented target segmentation result is not distorted, thereby improving the segmentation accuracy of the lesion region, and thus solving the technical problem of low accuracy in segmenting lesion regions in CT and MRI images in related technologies.

[0233] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0234] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0236] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0237] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0238] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0239] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image segmentation method, characterized in that, include: Acquire a first image and a second image of a target part of a biological object, wherein the target part includes a lesion area, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology; Based on the region of the target part in the first image, the second image is cropped to obtain a cropped image, wherein the density resolution of the first image is greater than the density resolution of the second image; Based on the registration results of the first image and the cropped image, image segmentation is performed on the target region contained in the second image to obtain the target segmentation result.

2. The method according to claim 1, characterized in that, Based on the region of the target part in the first image, the second image is cropped to obtain a cropped image, including: The target region contained in the first image is segmented to obtain an image segmentation result of the first image, wherein the image segmentation result is used to characterize the region of the target region in the first image; The second image is cropped based on the image segmentation result to obtain the cropped image.

3. The method according to claim 2, characterized in that, When the image segmentation result is used to characterize the boundary region of the target region in the first image, the second image is cropped based on the image segmentation result to obtain the cropped image, including: The image segmentation result is expanded based on preset expansion parameters to obtain the expanded processing result; The image segmentation result is subjected to erosion processing based on preset erosion parameters to obtain the erosion processing result; The difference between the expansion processing result and the erosion processing result is obtained to determine the target boundary region; The second image is cropped based on the target boundary region to obtain the cropped image.

4. The method according to claim 2, characterized in that, When the image segmentation result is used to characterize the center of the target region as being in the central region of the first image, the second image is cropped based on the image segmentation result to obtain the cropped image, including: The image segmentation result is subjected to erosion processing based on preset erosion parameters to obtain the target center region; The second image is cropped based on the target center region to obtain the cropped image.

5. The method according to claim 1, characterized in that, Based on the first image and the cropped image, image segmentation is performed on the target region contained in the second image to obtain the target segmentation result, including: The first image is segmented, and the target region contained in the second image is segmented based on the first image and the cropped image to obtain multiple candidate segmentation results; Based on the first image and the second image, error prediction is performed on the plurality of candidate segmentation results to obtain the segmentation error of the plurality of candidate segmentation results, wherein different segmentation errors are used to characterize the error between different candidate segmentation results and preset segmentation results; Based on the segmentation error of the multiple candidate segmentation results, the target segmentation result is selected from the multiple candidate segmentation results.

6. The method according to claim 5, characterized in that, The first image is segmented, and the target region contained in the second image is segmented based on the first image and the cropped image to obtain multiple candidate segmentation results, including: The target region contained in the first image is segmented to obtain a first candidate segmentation result; The first image and the second image are stitched together to obtain a stitched image; The stitched image is segmented to obtain a second candidate segmentation result; Based on the cropped image, the stitched image is segmented to obtain a third candidate segmentation result; Based on the first candidate segmentation result, the second candidate segmentation result, and the third candidate segmentation result, the plurality of candidate segmentation results are obtained.

7. The method according to claim 6, characterized in that, Based on the cropped image, the stitched image is segmented to obtain a third candidate segmentation result, including: Feature extraction is performed on the stitched image to obtain the stitched image features; Feature extraction is performed on the cropped image to obtain the cropped image features; Based on the cropped image features, the stitched image features are modified to obtain the modified image features; The modified image features are decoded to obtain the third candidate segmentation result.

8. The method according to claim 5, characterized in that, Based on the first image and the second image, error prediction is performed on the plurality of candidate segmentation results to obtain the segmentation error of the plurality of candidate segmentation results, including: The multiple candidate segmentation results are superimposed on the first image and the second image respectively to obtain multiple superimposed images; Error prediction models are used to predict the errors of the multiple superimposed images to obtain the segmentation errors of the multiple candidate segmentation results.

9. The method according to claim 5, characterized in that, Based on the segmentation error of the multiple candidate segmentation results, the target segmentation result is selected from the multiple candidate segmentation results, including: Determine the volume of the target region in the multiple candidate segmentation results; Based on the segmentation error and the volume, the target segmentation result is selected from the plurality of candidate segmentation results.

10. An image segmentation method, characterized in that, include: In response to input commands applied to the operating interface, a first image and a second image of the target part of the biological object are displayed on the operating interface, wherein the target part includes a lesion area, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology. In response to an image segmentation command applied to the operation interface, a target segmentation result is displayed on the operation interface. The target segmentation result is obtained by segmenting the target region contained in the second image based on the registration result of the first image and the cropped image. The cropped image is obtained by cropping the second image based on the region of the target region in the first image. The density resolution of the first image is greater than that of the second image.

11. An image segmentation method, characterized in that, include: A first image and a second image of a target part of a biological object are displayed on the display screen of a virtual reality (VR) device or an augmented reality (AR) device. The target part includes a lesion area. The first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology. Based on the region of the target part in the first image, the second image is cropped to obtain a cropped image, wherein the density resolution of the first image is greater than the density resolution of the second image; Based on the registration results of the first image and the cropped image, image segmentation is performed on the target region contained in the second image to obtain the target segmentation result; The VR device or AR device is driven to render and display the target segmentation result.

12. An image segmentation method, characterized in that, include: A first image and a second image of a target part of a biological object are obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the first image and the second image, the target part includes a lesion area, the first image is an image obtained by scanning the target part using computed tomography (CT) technology, and the second image is an image obtained by scanning the target part using magnetic resonance imaging (MRI) technology. Based on the region of the target part in the first image, the second image is cropped to obtain a cropped image, wherein the density resolution of the first image is greater than the density resolution of the second image; Based on the registration results of the first image and the cropped image, image segmentation is performed on the target region contained in the second image to obtain the target segmentation result; The target segmentation result is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target segmentation result.

13. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 12.

Images