Data processing method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a data processing method, device and equipment and medium, which can be applied to various fields such as medical navigation and automatic driving. The method comprises: obtaining an object scan image containing a reference marker, obtaining a mask image corresponding to the reference marker from the object scan image, the mask image being used to represent the shape of the reference marker in the object scan image; obtaining a simulation marker model corresponding to the reference marker, determining the body surface center point of the reference marker in the object scan image according to the simulation marker model and the mask image; obtaining a spatial marker position corresponding to the reference marker through a visual navigation tool, and performing a navigation operation on a spatial object indicated by the object scan image according to the spatial marker position and the body surface center point. By using the embodiments of the present application, the positioning accuracy of the marker can be improved, and the navigation effect can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] A surgical navigation system is a system that organically combines modern imaging technology, stereotactic technology, computer technology, and artificial intelligence technology with surgeons to make full use of information to enable patients to receive safe and precise surgical treatment.

[0003] Current surgical navigation systems involve manually selecting key points from preoperative medical scans and then matching these manually selected points with actual intraoperative key points for navigation. However, manually selecting key points from medical scans is difficult, often resulting in inaccurate placement. Once the key points are initially determined, adjustments are not easily made, creating a heavy reliance on the operator and ultimately affecting navigation effectiveness. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and medium that can improve the accuracy of marker positioning and thus improve navigation performance.

[0005] One embodiment of this application provides a data processing method, including:

[0006] Obtain an object scan image containing reference markers, and extract the mask image corresponding to the reference markers from the object scan image; the mask image is used to characterize the shape of the reference markers in the object scan image;

[0007] Obtain the simulation marker model corresponding to the reference marker, and determine the center point of the reference marker on the object surface in the scanned image based on the simulation marker model and the mask image;

[0008] The spatial marker position corresponding to the reference marker is obtained through a visual navigation tool. Based on the spatial marker position and the center point of the body surface, navigation operations are performed on the spatial object indicated by the object scan image.

[0009] One embodiment of this application provides a data processing apparatus, including:

[0010] The acquisition module is used to acquire an object scan image containing reference marks, and to acquire the mask image corresponding to the reference marks from the object scan image; the mask image is used to characterize the shape of the reference marks in the object scan image;

[0011] The determination module is used to obtain the simulation mark model corresponding to the reference mark, and determine the center point of the reference mark on the object surface in the object scan image based on the simulation mark model and the mask image;

[0012] The navigation module is used to obtain the spatial marker position corresponding to the reference marker through a visual navigation tool, and to perform navigation operations on the spatial object indicated by the object scan image based on the spatial marker position and the center point of the body surface.

[0013] The acquisition module includes:

[0014] The image segmentation unit is used to segment the scanned image of the object using a segmentation threshold to obtain a region image containing the reference marker;

[0015] The mask image acquisition unit is used to acquire M connected regions contained in the region image, and based on the volume of each of the M connected regions, acquire the mask image corresponding to the reference marker in the M connected regions; M is a positive integer.

[0016] The mask image acquisition unit includes:

[0017] The volume acquisition subunit is used to acquire the volume of each of the M connected components, and to mark each of the M connected components based on the volume of the connected components to obtain the identification information corresponding to each of the M connected components.

[0018] The connected component filtering subunit is used to identify, based on the identification information, connected components whose volume falls within the volume threshold range as reference markers contained in the object scan image.

[0019] The binary image acquisition subunit is used to acquire the binary image of the reference marker and determine the binary image as the mask image corresponding to the reference marker.

[0020] The acquisition module includes:

[0021] The probability map output unit is used to input the object scan image into the semantic segmentation model and output the labeled segmentation probability map corresponding to the object scan image through the semantic segmentation model;

[0022] The label separation unit is used to separate the connected components in the label segmentation probability map to obtain the mask image corresponding to the reference label.

[0023] The probability graph output unit includes:

[0024] The downsampling subunit is used to downsample the object scan image through the first network layer in the semantic segmentation model to obtain the abstract semantic feature map corresponding to the object scan image;

[0025] The upsampling subunit is used to upsample the abstract semantic feature map through the second network layer in the semantic segmentation model to obtain the labeled feature map corresponding to the object scan image;

[0026] The activation subunit is used to perform feature selection on the labeled feature map through the activation network layer in the semantic segmentation model to obtain the labeled segmentation probability map corresponding to the object scan image.

[0027] The upsampling subunit is used for:

[0028] By upsampling the semantic feature map through the second network layer in the semantic segmentation model, the labeled localization feature map corresponding to the object scan image is obtained.

[0029] The marker localization feature map and the object scan image are fused to obtain the marker feature map.

[0030] The device also includes:

[0031] The sample acquisition module is used to acquire sample scan images of sample markers; the sample scan images include label information corresponding to the sample markers;

[0032] The sample probability map output module is used to input the sample scan image into the initial segmentation model and output the sample segmentation probability map corresponding to the sample scan image through the initial segmentation model;

[0033] The metric function determination module is used to obtain the similarity between the sample segmentation probability map and the label information, and determine the metric function corresponding to the initial segmentation model based on the similarity.

[0034] The parameter correction module is used to correct the network parameters of the initial segmentation model based on the metric function, and to determine the initial segmentation model containing the corrected network parameters as the semantic segmentation model.

[0035] The sample probability map output module includes:

[0036] The first output unit is used to input the sample scan image into the initial segmentation model and output the first segmentation probability map corresponding to the sample scan image through the initial segmentation model.

[0037] The second output unit is used to perform data transformation on the sample scan image to obtain a deformed scan image, input the deformed scan image into the initial segmentation model, and output the second segmentation probability map corresponding to the deformed scan image through the initial segmentation model.

[0038] The probability map fusion unit is used to obtain the sample segmentation probability map corresponding to the sample scan image based on the first segmentation probability map and the second segmentation probability map.

[0039] The determining module includes:

[0040] The filtering operation unit is used to obtain the spatial dimension information corresponding to the reference mark, perform filtering operation on the spatial dimension information, and obtain the filtered data corresponding to the reference mark.

[0041] The rendering unit is used to encapsulate the filtered data into a set of key-value pairs, and to render the set of key-value pairs based on the rendering window to obtain the simulation label model corresponding to the baseline label.

[0042] The determining module includes:

[0043] The transformation matrix construction unit is used to convert the mask image into a mesh image, and constructs the transformation matrix based on the corresponding points between the simulation marker model and the mesh image.

[0044] The body surface center determination unit is used to obtain the body surface center point of the reference marker in the object scan image based on the transformation matrix and the center point of the simulation marker model.

[0045] Among them, the center points of the simulation marker model include center point P1 and center point P2;

[0046] The body surface center determination unit includes:

[0047] The operation subunit is used to perform a product operation on the transformation matrix and the array composed of center point P1 and center point P2 to obtain the center point F1 and center point F2 of the reference mark;

[0048] The centroid determination subunit is used to obtain the centroid of the object contained in the object scan image, obtain the first distance between the center point F1 and the centroid, and the second distance between the center point F2 and the centroid;

[0049] The distance comparison sub-unit is used to determine the center point F1 as the reference mark on the body surface center point in the object scan image if the first distance is less than the second distance.

[0050] The navigation module includes:

[0051] The marker registration unit is used to register the spatial marker position and the center point of the body surface to obtain the marker registration result between the spatial object indicated by the object scan image and the object scan image.

[0052] The navigation operation unit is used to perform navigation operations on spatial objects in the image navigation scene based on the marker registration results.

[0053] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method provided in one aspect of this application.

[0054] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in one aspect of this application.

[0055] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the above aspect.

[0056] In this embodiment, an object scan image containing reference markers can be acquired. A mask image corresponding to the reference markers is then obtained from the object scan image. This mask image is used to characterize the morphology of the reference markers in the object scan image. Furthermore, a simulation marker model corresponding to the reference markers can be acquired. Based on the simulation marker model and the mask image, the surface center point of the reference markers in the object scan image is determined. The spatial marker position corresponding to the reference markers is obtained using a visual navigation tool. Based on the spatial marker position and the surface center point, navigation operations are performed on the spatial object indicated by the object scan image. It can be seen that by segmenting the reference markers in the object scan image, a mask image corresponding to each reference marker can be obtained. Based on this mask image, the surface center point of the reference markers can be further fitted. By registering the surface center point with the spatial marker position determined by the visual navigation tool, i.e., by automatically locating and detecting the reference markers in the object scan image, the positioning accuracy of the surface center point can be improved. Therefore, navigation operations can be performed on the spatial object indicated by the object scan image in the image scanning scene, improving the navigation effect. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0059] Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0060] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0061] Figure 4 This is a schematic diagram of a reference mark detection process provided in an embodiment of this application;

[0062] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0063] Figure 6 This is a schematic diagram of segmentation processing of an object scan image provided in an embodiment of this application;

[0064] Figure 7 This is a schematic diagram of another object scanning image segmentation process provided in an embodiment of this application;

[0065] Figure 8 This is a schematic diagram of determining the center point of a reference marker on the body surface, provided in an embodiment of this application.

[0066] Figure 9 This is a schematic diagram of the automatic detection result of a reference mark provided in an embodiment of this application;

[0067] Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0068] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0070] This application relates to Computer Vision (CV). Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data.

[0071] This application relates to object localization within computer vision, a crucial task in the field. By segmenting objects (e.g., reference markers) in an image (e.g., a scanned image of an object), a mask image corresponding to the object is obtained. Then, based on the mask image, the center point of the object in the image (e.g., the surface center point of the reference marker) can be fitted, thus determining the object's position in the image. The scanned image of the object may include, but is not limited to, medical scan images of different human body parts, road scan images, etc.; the reference marker can refer to labeled key points in the image.

[0072] This application also involves the following concepts:

[0073] Computed Tomography (CT) is a medical imaging modality that uses an X-ray beam to scan a slice of human body with a certain thickness, ranging from 0.5 to 1 cm. The thinner the slice, the higher the accuracy. The resulting CT image can be one type of scanned image of the object involved in this application.

[0074] Magnetic Resource Imaging (MRI): a medical imaging modality that uses magnetic resonance to obtain electromagnetic signals from the human body and reconstruct human information; MRI does not produce radiation and can obtain tomographic images in any direction, and the MRI images obtained can be one type of scanned image of the object involved in this application.

[0075] Fiducial Marker: Also known as a finite marker, it is a marker used for surgical navigation. It is affixed to key points on the body surface and can be clearly imaged in CT or MRI images. It is used to mark key points in the preoperative planning system.

[0076] Preoperative planning system: Prescribes a surgical plan before surgery. This plan may include locating key points and segmenting key structures from medical scan images (object scan images), specifying the surgical approach, surgical incision, and other steps.

[0077] Iterative Closest Point (ICP) algorithm: A classic data registration algorithm used to obtain the transformation relationship between point clouds or point sets in two input 3D spaces.

[0078] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. Figure 1As shown, this network architecture may include a server 10d and a user terminal cluster. The user terminal cluster may include one or more user terminals; the number of user terminals is not limited here. Figure 1 As shown, the user terminal cluster may specifically include user terminal 10a, user terminal 10b, and user terminal 10c, etc.

[0079] Among them, server 10d can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0080] User terminals 10a, 10b, and 10c can all include: smartphones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches and smart bracelets), smart voice interaction devices, smart home appliances (such as smart TVs), in-vehicle devices, aircraft, and other electronic devices with object positioning capabilities. Figure 1 As shown, user terminals 10a, 10b, and 10c can each connect to server 10d via a network, so that each user terminal can interact with server 10d through the network connection.

[0081] Please see Figure 2 , Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. This embodiment uses a medical surgical navigation scenario as an example to describe the data processing process; for example... Figure 2As shown, assuming the target user needs to undergo brain surgery, before the surgery, the target user can first undergo a computed tomography (CT) scan of the head using medical equipment to obtain a head CT image; or, the target user can undergo magnetic resonance imaging (MRI) of the head using medical equipment to obtain a head MRI image; or, the target user can undergo ultrasound imaging of the head using medical equipment to obtain a head ultrasound image. The head CT image, head MRI image, or head ultrasound image can then be used as the target user's object scan image 20a. The object scan image 20a may include reference markers for annotating key points. These reference markers can be affixed to the head surface in the object scan image 20a and can be clearly imaged in the CT or MRI images. The object scan image 20a may contain one or more reference markers, and the number of reference markers can be determined based on actual needs; this application does not limit this.

[0082] Furthermore, after acquiring the object scan image 20a, reference marker segmentation processing can be performed on the object scan image 20a to obtain a mask image 20b. The mask image 20b can be considered a template for an image filter, used to extract the region of interest (e.g., the region where the reference marker is located) in the object scan image 20a; that is, the mask image 20a can be used to characterize the shape of the reference marker in the object scan image 20a. When the mask image 20b includes multiple reference markers, the mask corresponding to each individual reference marker can be extracted using the mask image 20b; when the mask image 20b includes only one reference marker, the mask image 20b is the mask for that single reference marker. It should be noted that the above reference marker segmentation processing can be considered as the process of extracting reference markers from the object scan image 20a. For example, threshold segmentation can be used to extract the reference markers from the object scan image 20a, or a semantic segmentation model can be used to extract the reference markers from the object scan image 20a. This application embodiment does not limit the method of reference marker segmentation processing.

[0083] The reference marker can be a real object, such as a cylinder with different inner and outer diameters. Based on the actual size information of the reference marker, a simulated reference marker model, i.e., a simulated marker model, can be generated. Simultaneously, the mask of the reference marker (e.g., mask image 20b) can be converted into a mesh image. By registering the simulated marker model onto the mesh image, the surface center point of each reference marker in the object scan image 20a can be determined. This surface center point is used to characterize the positional information of the reference marker in the object scan image 20a.

[0084] During brain surgery on a target user, a camera device 20c (e.g., a camera) can be used to acquire real-time images of the target user's brain 20d, and the positional information of the actual reference markers can be obtained through the tip of a visual navigation tool (e.g., a visual navigation stick), i.e., the spatial positional information of the reference markers during brain surgery. By registering the center point of the body surface with the spatial positional information, surgical navigation operations can be performed in the medical surgical navigation scenario. That is, the reference markers in the preoperative scan image 20a are registered with the actual reference markers during surgery, and subsequent surgical navigation operations are performed based on the registration result, which can improve the surgical navigation effect.

[0085] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application. It can be understood that the data processing method can be executed by a computer device, which can be a server (e.g., Figure 1 The corresponding embodiment is a server 10d), or a user terminal (e.g., Figure 1 The user terminal in the user terminal cluster shown can be any user terminal, or a computer program (including program code); such as Figure 3 As shown, the data processing method may include the following steps S101-S103:

[0086] Step S101: Obtain an object scan image containing reference marks, and obtain a mask image corresponding to the reference marks from the object scan image. The mask image is used to characterize the shape of the reference marks in the object scan image.

[0087] Specifically, in object navigation scenarios, a specific device can be used to acquire a scanned image of the target object, which is associated with that target object (e.g., Figure 2 The corresponding embodiment shows the object scan image 20a. The specific device mentioned above may include camera equipment, medical scanning equipment (e.g., CT equipment, MRI equipment, ultrasound equipment, etc.), etc. Different specific devices can be used for different object navigation scenarios, and this application does not limit this. The target object can refer to the object being scanned, which may include human body parts, road vehicles, objects, etc., and this application does not limit this. It should be noted that the surface of the target object in the object scan image has reference marks. These reference marks can be used to mark the key points of the target object, and the reference marks can be clearly imaged in the object scan image. For example, the reference marks can be bone nails, or hollow cylinders with different inner and outer diameters, etc.

[0088] In a planning system targeting a specific object, an automatic reference marker detection algorithm can be introduced to automatically calculate the position of the reference markers from the object scan image. For example, the object scan image can be input into the planning system, where the system can segment the target object and reference markers within the scan image to obtain mask images corresponding to the reference markers. These mask images can characterize the shape of the reference markers in the object scan image; for instance, the region to which the reference marker belongs can be extracted from the mask image. When multiple reference markers are detected in the object scan image, the reference markers can be segmented, and the masks corresponding to each individual reference marker can be extracted.

[0089] It is understood that computer devices can select the segmentation processing method for an object scan image based on its imaging modality; that is, object scan images with different imaging modalities can be processed using different segmentation methods. These segmentation methods can include threshold segmentation, semantic segmentation models, etc. The semantic segmentation model can include, but is not limited to: FCN (Fully Convolutional Networks, a fully convolutional network without fully connected layers), UNet (using a U-shaped structure that first downsamples and then upsamples), SegNet (using an encoder-decoder structure), and RefineNet (using a residual network for downsampling and then chained residual pooling in the RefineNet module for upsampling). This application does not limit the specific implementation of these methods.

[0090] Step S102: Obtain the simulation marker model corresponding to the reference marker, and determine the center point of the reference marker on the object surface in the object scan image based on the simulation marker model and the mask image.

[0091] Specifically, the computer equipment can acquire the actual spatial dimensions of the reference markers. Based on this spatial dimensions, a simulated reference marker model, or simulated marker model, can be generated. Further, a mesh generation algorithm can be used to convert the mask image into a mesh image, and a registration algorithm can be used to register the simulated marker model onto the mesh model to determine the center point of the reference marker on the object's surface in the scanned image. This center point can be used to characterize the position of the reference marker in the scanned image.

[0092] Mesh generation can be considered as dividing a specific region of interest (e.g., the region to which the reference marker belongs) into a series of small sub-regions (elements) to meet certain requirements; the shape and distribution of each element in the mesh can be determined by an automatic mesh generation algorithm. This mesh generation algorithm may include, but is not limited to: TSDF (truncated signed distance function, a common method for calculating hidden potential surfaces in 3D reconstruction), Wavelet (wavelet transform), Poisson (Poisson process), FSSR (super-resolution network), multi-level flowing cytometry surface reconstruction, and marching cubes (also known as isosurface extraction algorithms), and this application does not limit the specific algorithms used.

[0093] The above-mentioned registration algorithm can be called a point cloud registration algorithm. This registration algorithm may include: Iterative Closest Point (ICP) algorithm, Kernel Correlation (KC) algorithm, Robust Point Matching (RPM) algorithm, and point cloud registration algorithm using U-particle Filter (UPF). This application does not limit the specific algorithm to this one.

[0094] For example, when the reference marker is a hollow cylinder with different inner and outer diameters, the spatial dimension information includes the cylinder's inner diameter, outer diameter, and height. The simulation marker model generated based on this spatial dimension information can include the center point of the plane containing the inner diameter and the center point of the plane containing the outer diameter; that is, the simulation marker model can include two center points. Furthermore, the transformation matrix from the simulation marker model to the mesh model can be obtained. Using the transformation matrix and the two center points of the simulation marker model, the two center points of the reference marker in the mesh image can be determined, and one of these center points can be selected as the surface center point, such as selecting the center point closer to the centroid of the target object.

[0095] Step S103: Obtain the spatial marker position corresponding to the reference marker through the visual navigation tool, and perform navigation operation on the spatial object indicated by the object scan image based on the spatial marker position and the center point of the body surface.

[0096] Specifically, in object navigation scenarios, visual navigation tools can be used to obtain the true location of reference markers, such as spatial marker locations. Then, based on the spatial marker locations and the center of the body surface, the target object in the scanned image can be registered with the target object in real space (which can be simply referred to as the spatial object), allowing subsequent navigation operations to be performed. For example, spatial marker locations and the center point of the body surface can be registered to obtain the registration result between the spatial object indicated by the scanned image and the scanned image; based on the registration result, navigation operations can be performed on the spatial object in the image navigation scenario.

[0097] The object scan image may include one or more reference markers. Each reference marker may correspond to a center point on the object surface. The center points on the object surface of all reference markers in the object scan image may form a point set S1. The spatial position information of each reference marker in the object navigation scene may form a point set S2. Rigid body registration is performed using a registration algorithm to obtain the transformation relationship T between the spatial objects in the object scan image and the object navigation image. Based on this transformation relationship T, subsequent navigation operations can be performed, such as S2 = T * S1.

[0098] Please see Figure 4 , Figure 4 This is a schematic diagram of a reference marker detection process provided in an embodiment of this application. For example... Figure 4 As shown, taking head surgery navigation in a medical surgical navigation scenario as an example, a medical scanning device can be used to obtain a head scan image 30a (object scan image) of the target user, and the head scan image 30a is transmitted into the preoperative planning system. An automatic benchmark detection algorithm is introduced into the preoperative planning system. Through this automatic benchmark detection algorithm, the position of the benchmark can be automatically determined from the head scan image 30a.

[0099] In the preoperative planning system, the head and reference markers in the head scan image 30a can be segmented (i.e., reference marker instance segmentation) to obtain a mask image 30b. This mask image 30b can include all reference markers contained in the head scan image 30a, and a mask for a single reference marker can be extracted based on this mask image 30b. Furthermore, using the mask of a single reference marker and a simulated marker model of that single reference marker, the body surface center point of the reference marker in the head scan image 30a can be fitted, i.e., the positional information of the reference marker in the head scan image 30a.

[0100] After obtaining the center point of the reference marker on the body surface before surgery, the actual spatial position (i.e., spatial position information) of the reference marker during the head surgery of the target user can be obtained. Based on the center point on the body surface and the actual spatial position, the head scan image 30a and the head of the target user during surgery are registered to perform subsequent navigation operations, such as performing surgical operations on the head of the target user based on the marker registration results.

[0101] In this embodiment, by segmenting the reference marks in the object scanning image, a mask image corresponding to each reference mark can be obtained. Based on the mask image, the body surface center point of the reference mark can be further fitted. By registering the body surface center point with the spatial mark position determined by the visual navigation tool, that is, by automatically locating and detecting the reference marks in the object scanning image, the positioning accuracy of the body surface center point can be improved. Furthermore, in the image scanning scene, navigation operations can be performed on the spatial object indicated by the object scanning image, thereby improving the navigation effect.

[0102] Please see Figure 5 , Figure 5 This is a schematic flowchart illustrating another data processing method provided in an embodiment of this application. It can be understood that the data processing method can be executed by a computer device, which can be a server (e.g., Figure 1 The corresponding embodiment is a server 10d), or a user terminal (e.g., Figure 1 The user terminal in the user terminal cluster shown can be any user terminal, or a computer program (including program code); such as Figure 5 As shown, the data processing method may include the following steps S201-S210:

[0103] Step S201: Obtain the sample scan image containing sample labels, input the sample scan image into the initial segmentation model, and output the sample segmentation probability map corresponding to the sample scan image through the initial segmentation model.

[0104] For ease of understanding, this application uses a medical surgical navigation scenario as an example to describe the data processing process in detail. During data processing, a semantic segmentation model can be used to segment the acquired object scan image to obtain a mask image corresponding to the reference marker in the object scan image. Before use, this semantic segmentation model needs to be trained; only a trained semantic segmentation model can be introduced into the preoperative planning system.

[0105] For ease of understanding, taking the Unet model as an example, the training process of the semantic segmentation model is described through steps S201 to S202.

[0106] Computer equipment can acquire sample scan images containing sample markers. These sample scan images can be medical images obtained using medical equipment, such as CT images or MRI images. The sample scan images can include one or more reference markers, which can be referred to as sample markers. The computer equipment can annotate the sample markers in the sample scan images with label information, which can be used to characterize the positional information of the sample markers in the sample scan images.

[0107] It is understood that computer devices can acquire a large number of sample scan images and use all the acquired sample scan images as a sample dataset for training a semantic segmentation model. Each sample scan image in this dataset can carry a label information. During training, the number of sample scan images input for each training iteration can be one or multiple (by batch processing the sample training set), and this application does not limit this.

[0108] The computer device can acquire an initial segmentation model and input the sample scan image into it. The initial segmentation model can output a sample segmentation probability map corresponding to the sample scan image. This initial segmentation model can refer to the model of the aforementioned semantic segmentation model during the training phase. The initial segmentation model may include an encoding component, a decoding component, and activation function layers (e.g., sigmoid function layers). The encoding component may include one or more convolutional layers for downsampling the sample scan image. The decoding component may include one or more deconvolutional layers for upsampling the features output by the encoding component. The number of convolutional layers in the encoding component can be the same as the number of deconvolutional layers in the decoding component. The activation function layers can be used to perform nonlinear transformations on the features output by the decoding component and to perform feature selection on the features output by the decoding component.

[0109] For example, the encoding component in the initial segmentation model may include four convolutional layers. The output features of the previous convolutional layer can be used as the input features of the next convolutional layer. Different convolutional layers can output different numbers of convolutional feature maps. The decoding component in the initial segmentation model may include four deconvolutional layers. The output features of the previous deconvolutional layer can also be used as input to the next deconvolutional layer. The four convolutional layers in the above encoding component can be denoted as convolutional layer c1, convolutional layer c2, convolutional layer c3, and convolutional layer c4. Convolutional layer c1 can extract 16 feature maps, that is, convolutional layer c1 can output 16 feature maps. Convolutional layer c2 can extract 32 feature maps. Convolutional layer c3 can extract 64 feature maps. Convolutional layer c4 can extract 128 feature maps.

[0110] The four deconvolutional layers of the aforementioned decoding component can be designated as deconvolutional layer v1, v2, v3, and v4. The feature map output by deconvolutional layer v1 has the same resolution (size information) as the feature map output by convolutional layer c4; the feature map output by deconvolutional layer v2 has the same resolution as the feature map output by convolutional layer c3; the feature map output by deconvolutional layer v3 has the same resolution as the feature map output by convolutional layer c2; and the feature map output by deconvolutional layer v4 has the same resolution as the feature map output by convolutional layer c1. In other words, the resolution of the feature map output by deconvolutional layer v4 is the same as the resolution of the sample scan image (e.g., 176*176*176). The feature map output by deconvolutional layer v4 is then input into an activation function layer, through which the sample segmentation probability map corresponding to the sample scan image can be obtained.

[0111] Optionally, in addition to the four convolutional layers mentioned above, each convolutional layer in the encoding component may be followed by a pooling layer. This pooling layer can reduce the number of parameters in the encoding component. It should be noted that the foregoing description is only an example of one network structure for the initial segmentation model. Other network results are also possible, such as an encoding component including eight convolutional layers and a decoder including eight deconvolutional layers. This application does not limit this to any particular type.

[0112] In one or more embodiments, the output of the initial segmentation model on the sample scan image can be determined as a first segmentation probability map. Then, data transformation can be performed on the sample scan image to obtain a deformed scan image. This data transformation may include, but is not limited to, random flipping, transposition, rotation, etc. The deformed scan image is input into the initial segmentation model, which outputs a second segmentation probability map corresponding to the deformed scan image. Based on the first and second segmentation probability maps, a sample segmentation probability map corresponding to the sample scan image can be obtained. For example, information fusion can be performed on the first and second segmentation probability maps to obtain the sample segmentation probability map corresponding to the sample scan image. The processing procedure for the deformed scan image using the initial segmentation model is the same as the aforementioned processing procedure for the sample scan image using the initial segmentation model, and will not be repeated here.

[0113] Optionally, the feature output results corresponding to the sample scan image can be obtained through the encoding and decoding components in the initial segmentation model. After performing data transformation on the sample scan image to obtain a deformed scan image, the feature output results corresponding to the deformed scan image can be obtained through the encoding and decoding components in the initial segmentation model. Further, feature fusion can be performed on the feature output results of the sample scan image and the deformed scan image to obtain a fused output result. Through the activation function layer in the initial segmentation model, a nonlinear transformation can be applied to this fused output result to obtain a sample segmentation probability map. This embodiment of the application enhances the initial segmentation model online through data transformation, which can improve the quality of the sample segmentation probability map and thus improve the generalization performance of the initial segmentation model.

[0114] Step S202: Obtain the similarity between the sample segmentation probability map and the label information corresponding to the sample scan image, and determine the metric function corresponding to the initial segmentation model based on the similarity; modify the network parameters of the initial segmentation model based on the metric function, and determine the initial segmentation model containing the modified network parameters as the semantic segmentation model.

[0115] Specifically, the computer device can calculate the similarity between the sample segmentation probability map output by the initial segmentation model and the label information corresponding to the sample scan image. The methods for calculating the similarity here may include, but are not limited to: Euclidean distance, Manhattan distance, Chebyshev distance, and Pearson correlation coefficient.

[0116] Based on the similarity calculated above, the metric function corresponding to the initial segmentation model (e.g., DiceLoss, also known as the set similarity metric function) can be determined. This metric function can be called the objective function (or loss function) corresponding to the initial segmentation model. The metric function can be represented by the following formula (1):

[0117]

[0118] Where X represents the sample segmentation probability map, Y represents the label information corresponding to the sample markers in the sample scan image; |X∩Y| represents the sample segmentation probability. Figure X The intersection with the label information Y can be approximated as the sample segmentation probability. Figure X The dot product between the product and the label information Y is calculated, and the elements of the dot product are summed; |X| and |Y| represent the sample segmentation probabilities, respectively. Figure X The quantization calculation of the tag information Y can be obtained by directly adding the elements, or by summing the squares of the elements; L dice L represents the metric function.dice The larger the value, the higher the probability of sample segmentation. Figure X The more similar L is to the label information Y, dice The smaller the value, the higher the probability of sample splitting. Figure X The greater the difference between the label information Y and the label information Y.

[0119] Backpropagation is performed using this metric function to adjust the network parameters of the initial segmentation model. This means minimizing and optimizing the metric function to continuously correct the network parameters of the initial segmentation model, thus training the initial segmentation model. When the initial segmentation model reaches the training convergence condition or the preset maximum number of iterations, it can be determined that the initial segmentation model has been trained successfully, and the network parameters at this point are saved. The initial segmentation model containing these network parameters is then identified as the semantic segmentation model.

[0120] The metric function represented by the above formula (1) is only one of the objective functions of the initial segmentation model. Other objective functions, such as the cross-entropy loss function, can also be used. This application does not limit this.

[0121] Step S203: Obtain the object scan image containing the reference marker, input the object scan image into the semantic segmentation model, and output the marker segmentation probability map corresponding to the object scan image through the semantic segmentation model.

[0122] Specifically, a computer device can use a medical device to acquire a scanned image of an object containing reference markers. Using the semantic segmentation model trained as described above, the reference markers in the scanned image are segmented to obtain a marker segmentation probability map corresponding to the scanned image. This semantic segmentation model can include a trained encoding component, a decoding component, and an activation function layer. The convolutional layer in the encoding component can be called the first network layer, and the deconvolutional layer in the decoding component can be called the second network layer.

[0123] After the object scan image is input into the semantic segmentation model, it is first fed into the encoding component of the semantic segmentation model. The first network layer of the semantic segmentation model downsamples the object scan image to obtain an abstract semantic feature map corresponding to the object scan image. The second network layer of the semantic segmentation model upsamples the abstract semantic feature map to obtain a labeled feature map corresponding to the object scan image. Finally, the activation network layer of the semantic segmentation model performs feature selection on the labeled feature map to obtain a labeled segmentation probability map corresponding to the object scan image. Here, the abstract semantic feature map can refer to the feature map finally output by the encoding component, which can refer to the feature extraction part of the semantic segmentation model, used to extract contextual information from the object scan image; the labeled feature map can refer to the feature map finally output by the decoding component, which can refer to the image reconstruction part of the semantic segmentation model, used to extract the positional information of the reference markers in the object scan image.

[0124] In one or more embodiments, the semantic segmentation model may further introduce a skip connection between the encoding and decoding components to combine the feature map output from the decoding component with the feature map output from the encoding component, thereby obtaining a labeled feature map corresponding to the object scan image. For example, the semantic feature map can be upsampled through a second network layer in the semantic segmentation model to obtain a labeled localization feature map, which can be used to determine the position of the reference marker in the object scan image; the labeled feature map is obtained by feature fusion of the labeled localization feature map and the object scan image. For example, suppose the encoding component in a semantic segmentation model includes convolutional layers c1 and c2, and the decoding component includes deconvolutional layers v1 and v2. The feature map output by convolutional layer c2 can be called the abstract semantic feature. The abstract semantic feature is upsampled by the deconvolutional layer v1 in the decoding component to obtain the feature map output by the deconvolutional layer v1. Then, the feature map output by the deconvolutional layer v1 can be concatenated with the feature map output by the convolutional layer c2, that is, stacked in the channel dimension of the feature maps to obtain the first concatenated feature. Next, the first concatenated feature can be upsampled by the deconvolutional layer v2 to obtain the feature map output by the deconvolutional layer v2 (labeled localization feature map). Then, the feature map output by the deconvolutional layer v2 can be concatenated with the feature map output by the convolutional layer c1 to obtain the second concatenated feature, which can be called the labeled feature map.

[0125] Step S204: Separate the connected components in the marker segmentation probability map to obtain the mask image corresponding to the reference marker.

[0126] Specifically, after obtaining the label segmentation probability map, a mask image of a single reference label can be extracted based on connected components (closed regions). A reference label is a connected component, and each connected component in the label segmentation probability map can be considered as a reference label. By separating the connected components in the label segmentation probability map, a mask image of a single reference label can be obtained.

[0127] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating the segmentation processing of an object scan image provided in an embodiment of this application. For example... Figure 6 As shown, after obtaining the object scan image 40a (which can be the user's head scan image), a pre-trained semantic segmentation model 40b can be obtained. The semantic segmentation model 40b can include an encoding component and a decoding component. The encoding component can be composed of multiple convolutional layers, or multiple convolutional layers and pooling layers can be alternately composed. For example, the encoding component can include convolutional layer c1, convolutional layer c2, convolutional layer c3 and convolutional layer c4. The decoding component can be composed of multiple deconvolutional layers. For example, the decoding component can include deconvolutional layer v1, deconvolutional layer v2, deconvolutional layer v3 and deconvolutional layer v4.

[0128] The object scan image 40b is input into the semantic segmentation model 40b. First, it is input into the convolutional layer c1 of the encoding component of the semantic segmentation model 40b. The object scan image 40b is convolved by this convolutional layer c1 to obtain feature map D1. Then, feature map D2 is convolved by convolutional layer c2 to obtain feature map D2. Similarly, feature map D3 is output through convolutional layer c3, and feature map D4 is output through convolutional layer c4. Feature maps D1, D2, D3, and D4 have different numbers of channels.

[0129] Furthermore, feature map D4 can be input to deconvolution layer v1 in the decoding component. Deconvolution layer v1 upsamples feature map D4 to obtain feature map D5. Feature maps D4 and D5 have different resolutions, such as having the same height and width. Further, feature maps D4 and D5 can be concatenated to obtain a fused feature R1, where the number of channels in fused feature R1 is the sum of the number of channels in feature map D4 and feature map D5. This fused feature R1 can be used as input to deconvolution layer v2. Based on this deconvolution layer v2, fused feature R1 is upsampled to obtain feature map D6. Feature map D6 and feature map D3 are then concatenated to form fused feature R2. Similarly, deconvolution layer v3 can output feature map D7, which is concatenated to form fused feature R3. Deconvolution layer v4 can output feature map D8, which is concatenated to form fused feature R4. By performing a nonlinear mapping on the fused feature R4 through the activation function layer in the semantic segmentation model 40b, a labeled segmentation probability map 40c can be obtained. Then, connected components in the labeled segmentation probability map 40c can be separated to obtain the mask image 40d corresponding to the reference marker in the object scan image 40a. Segmenting the object scan image 40a using the semantic segmentation model can improve image segmentation accuracy.

[0130] It should be noted that after acquiring the object scan image, the computer device can select the corresponding segmentation processing method based on the imaging modality of the object scan image. This imaging modality can be determined by the type of device that generated the object scan image. Generally speaking, when the object scan image is an MRI image generated using an MRI device, the segmentation processing method can be determined to be a semantic segmentation model, that is, using this semantic segmentation model to segment the reference markers in the object scan image. When the object scan image is a CT image generated using a CT device, the segmentation processing method can be determined to be a threshold segmentation method, that is, using threshold segmentation to segment the reference markers in the object scan image.

[0131] In one or more embodiments, assuming the object scan image is a CT image, a segmentation threshold can be used to segment the object scan image to obtain a region image containing the reference marker. This segmentation threshold can be customized according to actual needs; for example, it can be set to 300. Here, the segmentation threshold can be a CT value (a unit of measurement for the density of a local tissue or organ in the human body). Based on this segmentation threshold, the reference marker and the target object in the object scan image can be initially segmented to obtain a region image containing the reference marker. When the target object is a human body part, the reference marker and the target object can be separated by human skin.

[0132] Furthermore, connected component analysis can be performed on the region image. For example, M connected components can be obtained from the region image. Based on the volume of each of the M connected components, a mask image corresponding to the reference marker can be extracted from the M connected components. M can be a positive integer, such as 1, 2, ... The reference marker and the target object in the object scan image belong to different connected components. For example, the LabelGeometryImageFilter (a component for finding the centroid and bounding box of each region in an object scan image) of ITK (The Insight Segmentation and Registration Toolkit, a medical image processing software package) can be used to obtain the M connected components.

[0133] The process of extracting the mask image corresponding to the reference marker from the M connected components may include: obtaining the connected component volume corresponding to each of the M connected components; marking the M connected components based on the connected component volume to obtain the identification information corresponding to each of the M connected components; determining the connected components whose volume belongs to the volume threshold range as the reference marker contained in the object scan image based on the identification information; obtaining the binary image of the reference marker and determining the binary image as the mask image corresponding to the reference marker.

[0134] The labels (label IDs) of the M connected components obtained based on LabelGeometryImageFilter may be discontinuous or even out of order. Therefore, ITK's RelabelComponentImageFilter (a component for relabeling connected components) can be used to relabel the M connected components to use continuous label information. For example, the label information can be represented in the form of numbers, with no gaps between the numbers labeled for the M connected components. By default, the relabeled label information can be sorted according to the volume of the connected components. Furthermore, a reference label can be selected from the M connected components. For example, the reference label has a fixed shape and its volume is usually around 550. Therefore, the volume threshold range for selecting the reference label can be set to 450 to 650, that is, connected components with a volume between 450 and 650 are selected as reference labels, and the binary image corresponding to each reference label is extracted and used as the mask image of the reference label.

[0135] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating another object scanning image segmentation process provided in an embodiment of this application. For example... Figure 7As shown, after acquiring the object scan image 50a (which can be a user's head scan image), the object scan image 50a can be segmented using a segmentation threshold (for example, the segmentation threshold can be set to 300) to separate the head (target object) and the reference marker in the object scan image 50a. Furthermore, connected component analysis can be performed on the segmentation results of the object scan image 50a to obtain M connected components from the object scan image 50a, and the volume of each connected component (i.e., the connected component volume) can be calculated. Then, the identification information can be relabeled for the M connected components, such as... Figure 7 Image 50b is shown.

[0136] Connected components with volumes between 450 and 650 are selected from the M connected components and used as reference markers in the object scan image 50a to obtain image 50c. Then, based on image 50c, the binary images corresponding to each reference marker can be extracted as mask images. Segmenting the object scan image 50a using a segmentation threshold can improve image segmentation efficiency.

[0137] Step S205: Obtain the spatial dimension information corresponding to the reference mark, perform a filtering operation on the spatial dimension information to obtain the filtered data corresponding to the reference mark; encapsulate the filtered data into a key-value pair set, and perform rendering processing on the key-value pair set based on the rendering window to obtain the simulation mark model corresponding to the reference mark.

[0138] Specifically, after obtaining the mask image corresponding to the reference marker, the spatial dimension information of the reference marker can be acquired. This spatial dimension information refers to the size of the reference marker in the objective world. A simulated reference marker model (which can be simply referred to as the simulated marker model) can be generated using this spatial dimension information. For example, when the reference marker is a hollow cylinder with an outer diameter and an inner diameter, the spatial dimension information can include the inner diameter, outer diameter, and height. The simulated marker model can be generated using VTK (Visualization Toolkit). For instance, the spatial dimension information can be used as the input data (Source) of VTK. By filtering the spatial dimension information, filtered data can be obtained. Then, the filtered data can be mapped (vtkMapper) and encapsulated into an object structure (vtkActor), which can be represented as a set of key-value pairs. This object structure is added to the rendering scene (vtkRenderer), and the object structure is rendered and displayed using the rendering window (vtkRenderWindow), thus obtaining the simulated marker model corresponding to the reference marker.

[0139] It should be noted that the aforementioned VTK is only one example of generating a simulation marker model as described in the embodiments of this application. Other visualization tools, such as PyBullet, Simulink, etc., can also be used, and this application does not limit them.

[0140] Step S206: Convert the mask image into a mesh image and construct a transformation matrix based on the corresponding points between the simulation marker model and the mesh image.

[0141] Specifically, computer devices can use mesh generation algorithms to convert mask images into mesh images. Taking the Marching Cube (MC) algorithm as an example, the MC algorithm first divides the space into numerous hexahedral meshes, which can be considered as dividing the space into many small blocks. Then, it finds the hexahedral mesh through which the isosurface passes, extracting one hexahedron as the current hexahedron and acquiring all its information, such as the values ​​and coordinates of its eight vertices. The function values ​​of the eight vertices of the current hexahedron are compared with the given isosurface value C to obtain the state table of the hexahedron. Based on the state table index of the current hexahedron, the edges of the hexahedrons intersecting the isosurface are found, and the coordinates of each intersection point are calculated using linear interpolation. Using the central difference method, the normal vectors of the eight vertices of the current hexahedron are calculated, and then linear interpolation is used to obtain the normal vectors of each vertex of the triangular facets. Based on the coordinates of the vertices and the vertex normal vectors of each triangular facet, the triangular faces are connected to obtain the mesh image.

[0142] Furthermore, a data registration algorithm can be used to register the simulated marker model onto the mesh image. For example, this data registration algorithm can be the ICP algorithm, which can find corresponding point pairs between the simulated marker model and the mesh image, and construct a transformation matrix between the simulated marker model and the mesh image based on these corresponding point pairs. The ICP algorithm can use the Iterative Closest Point Transform in VTK to obtain the transformation matrix. The process of obtaining this transformation matrix can include: setting the number of iterations (which can be set according to actual needs, such as 100 iterations), and using rigid body registration to obtain the transformation matrix A∈R. 3×4 That is, the size of the transformation matrix is ​​3×4.

[0143] Step S207: Perform a product operation on the transformation matrix and the array consisting of the center points P1 and P2 of the simulation marker model to obtain the center points F1 and F2 of the reference marker.

[0144] Specifically, the computer device can obtain the simulation center point of the simulation marker model. Based on the transformation matrix and the simulation center point, the center point of the reference marker in the object scan image can be obtained. For example, the center point of the reference marker can be shown in the following formula (2):

[0145] F = AP (2)

[0146] Wherein, P can represent the center point of the simulation marker model, and F can represent the center point of the reference marker in the object scan image. Assuming that the reference marker is a vertically symmetrical object, the simulation marker point can include center point P1 and center point P2. Then, the center point of the reference marker in the object scan image obtained by the above formula (2) also includes two center points (for example, center point F1 and center point F2).

[0147] Step S208: Obtain the centroid of the object contained in the object scan image, obtain the first distance between the center point F1 and the centroid, and the second distance between the center point F2 and the centroid.

[0148] Specifically, the computer device can obtain the centroid of the object (target object) contained in the object scan image, which may be located within the target object; and then obtain the first distance between the centroid and the center point F1, and the second distance between the centroid and the center point F2.

[0149] Step S209: If the first distance is less than the second distance, then the center point F1 is determined as the center point of the reference mark on the body surface in the object scan image.

[0150] Specifically, when the first distance is less than the second distance, the center point F1 can be determined as the center point of the reference marker on the object's surface in the scanned image; when the first distance is greater than the second distance, the center point F2 can be determined as the center point of the reference marker on the object's surface in the scanned image. In other words, the center point closest to the centroid can be used as the center point of the reference marker on the object's surface in the scanned image.

[0151] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating how to determine the center point of a reference marker on the body surface, as provided in an embodiment of this application. Figure 8 As shown, after extracting the mask image of a single reference marker, a mesh generation algorithm (e.g., the MC algorithm) can be used to convert the mask image into a mesh image 50b. Simultaneously, a visualization tool (e.g., VTK) can be used to generate a simulation marker model 50a from the spatial dimension information of the reference marker in the real world.

[0152] A data registration algorithm (e.g., ICP algorithm) is used to obtain the transformation matrix 50c between the simulated marker model 50a and the grid image 50b. Based on the transformation matrix 50c and the simulated center point of the simulated marker model 50a, the center points F1 and F2 of the reference marker in the object scan image can be obtained. The computer device can also obtain the centroid of the target object in the object scan image and calculate the distances between the centroid and the center points F1 and F2, respectively. The center point closest to the centroid (e.g., center point F1) is then determined as the body surface center point (also called the body surface point) of the reference marker. At this time, center point F2 can be called the external center point (also called the external point) of the reference marker. The external points, body surface points, and centroid in the object scan image are as follows: Figure 8 The image shown is 50f.

[0153] In this embodiment of the application, by introducing an automatic detection method for reference markers in an object navigation scenario (e.g., a medical surgical navigation scenario), the position of each reference marker in the object scan image (e.g., the center point of the reference marker on the body surface) can be automatically detected.

[0154] Please see Figure 9 , Figure 9 This is a schematic diagram illustrating the automatic detection result of a reference marker provided in an embodiment of this application. For example... Figure 9 As shown, when the scanned image of an object includes four reference markers (e.g., reference marker A, reference marker B, reference marker C, and reference marker D), the automatic detection process of the reference markers described above can determine the center points of the three reference markers on the body surface in the scanned image. For example, the center point of reference marker A on the body surface in the scanned image is recorded as F-7, the center point of reference marker B on the body surface in the scanned image is recorded as F-9, the center point of reference marker C on the body surface in the scanned image is recorded as F-11, and the center point of reference marker D on the body surface in the scanned image is recorded as F-13. Figure 9 Each row shown can be represented as a visualization of the body surface center point of the same reference mark in different cross-sections, and each column can be represented as a visualization of the body surface center of different reference marks in the same cross-section.

[0155] Step S210: Obtain the spatial marker position corresponding to the reference marker through a visual navigation tool, and perform navigation operation on the spatial object indicated by the object scan image based on the spatial marker position and the center point of the body surface.

[0156] For details on the implementation of step S210, please refer to [link / reference]. Figure 3 Step S103 in the corresponding embodiment will not be described again here.

[0157] In this embodiment, by segmenting the reference marks in the object scan image, a mask image corresponding to each reference mark can be obtained. Based on the mask image, the body surface center point of the reference mark can be further fitted. By registering the body surface center point with the spatial mark position determined by the visual navigation tool, that is, by automatically locating and detecting the reference marks in the object scan image, the positioning and detection accuracy of the body surface center point can be improved. Consequently, in the image scanning scene, navigation operations can be performed on the spatial object indicated by the object scan image, thereby improving the navigation effect.

[0158] It is understood that the specific implementation of this application may involve scanned images of user body parts (e.g., head, kidneys, stomach, etc.). When the above embodiments of this application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0159] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 10 As shown, the data processing device 1 includes: an acquisition module 11, a determination module 12, and a navigation module 13;

[0160] The acquisition module 11 is used to acquire an object scan image containing reference marks, and to acquire a mask image corresponding to the reference marks from the object scan image; the mask image is used to characterize the shape of the reference marks in the object scan image;

[0161] The determination module 12 is used to obtain the simulation mark model corresponding to the reference mark, and determine the center point of the reference mark on the object surface in the object scan image based on the simulation mark model and the mask image.

[0162] Navigation module 13 is used to obtain the spatial marker position corresponding to the reference marker through a visual navigation tool, and to perform navigation operations on the spatial object indicated by the object scan image based on the spatial marker position and the center point of the body surface.

[0163] The specific functional implementation methods of the acquisition module 11, determination module 12, and navigation module 13 can be found in [reference needed]. Figure 3 Steps S101-S103 in the corresponding embodiments will not be described again here.

[0164] In one or more embodiments, the acquisition module 11 includes: an image segmentation unit 111, a mask image acquisition unit 112, a probability map output unit 113, and a marker separation unit 114.

[0165] Image segmentation unit 111 is used to segment the object scan image using a segmentation threshold to obtain a region image containing a reference marker.

[0166] The mask image acquisition unit 112 is used to acquire M connected regions contained in the region image, and based on the volume of each of the M connected regions, acquire the mask image corresponding to the reference marker in the M connected regions; M is a positive integer.

[0167] The probability map output unit 113 is used to input the object scan image into the semantic segmentation model and output the labeled segmentation probability map corresponding to the object scan image through the semantic segmentation model.

[0168] The label separation unit 114 is used to separate the connected components in the label segmentation probability map to obtain the mask image corresponding to the reference label.

[0169] The specific functional implementation of the image segmentation unit 111, the mask image acquisition unit 112, the probability map output unit 113, and the marker separation unit 114 can be found in [reference needed]. Figure 5 The relevant descriptions in step S204 of the corresponding embodiment will not be repeated here. Specifically, when the image segmentation unit 111 and the mask image acquisition unit 112 are performing their respective operations, the probability map output unit 113 and the marker separation unit 114 both pause their operations; when the probability map output unit 113 and the marker separation unit 114 are performing their respective operations, the image segmentation unit 111 and the mask image acquisition unit 112 both pause their operations.

[0170] In one or more embodiments, the mask image acquisition unit 112 includes: a volume acquisition subunit 1121, a connected component filtering subunit 1122, and a binary image acquisition subunit 1123.

[0171] The volume acquisition subunit 1121 is used to acquire the volume of each of the M connected components, and to mark the M connected components based on their volumes to obtain the identification information corresponding to each of the M connected components.

[0172] The connected component filtering subunit 1122 is used to identify connected components whose volume belongs to the volume threshold range as reference markers contained in the object scan image based on the identification information.

[0173] Binary image acquisition subunit 1123 is used to acquire the binary image of the reference mark and determine the binary image as the mask image corresponding to the reference mark.

[0174] In one or more embodiments, the probability graph output unit 113 includes: a downsampling subunit 1131, an upsampling subunit 1132, and an activation subunit 1133;

[0175] The downsampling subunit 1131 is used to downsample the object scan image through the first network layer in the semantic segmentation model to obtain the abstract semantic feature map corresponding to the object scan image.

[0176] The upsampling subunit 1132 is used to upsample the abstract semantic feature map through the second network layer in the semantic segmentation model to obtain the labeled feature map corresponding to the object scan image;

[0177] The activation subunit 1133 is used to perform feature selection on the labeled feature map through the activation network layer in the semantic segmentation model to obtain the labeled segmentation probability map corresponding to the object scan image.

[0178] Optionally, the upsampling subunit 1132 is specifically used for:

[0179] By upsampling the semantic feature map through the second network layer in the semantic segmentation model, the labeled localization feature map corresponding to the object scan image is obtained.

[0180] The marker localization feature map and the object scan image are fused to obtain the marker feature map.

[0181] The specific implementation methods of the volume acquisition subunit 1121, connected component filtering subunit 1122, binary image acquisition subunit 1123, downsampling subunit 1131, upsampling subunit 1132, and activation subunit 1133 can be found in [link to relevant documentation]. Figure 5 Steps S203-S204 in the corresponding embodiments will not be described again here.

[0182] In one or more embodiments, the data processing device 1 further includes: a sample acquisition module 14, a sample probability map output module 15, a metric function determination module 16, and a parameter correction module 17;

[0183] Sample acquisition module 14 is used to acquire sample scan images of sample markers; the sample scan images include label information corresponding to the sample markers;

[0184] The sample probability map output module 15 is used to input the sample scan image into the initial segmentation model and output the sample segmentation probability map corresponding to the sample scan image through the initial segmentation model.

[0185] The metric function determination module 16 is used to obtain the similarity between the sample segmentation probability map and the label information, and to determine the metric function corresponding to the initial segmentation model based on the similarity.

[0186] The parameter correction module 17 is used to correct the network parameters of the initial segmentation model based on the metric function, and to determine the initial segmentation model containing the corrected network parameters as the semantic segmentation model.

[0187] The specific implementation methods of the sample acquisition module 14, sample probability map output module 15, metric function determination module 16, and parameter correction module 17 can be found in [reference needed]. Figure 5 Steps S201-S202 in the corresponding embodiments will not be described again here.

[0188] In one or more embodiments, the sample probability map output module 15 includes: a first output unit 151, a second output unit 152, and a probability map fusion unit 153;

[0189] The first output unit 151 is used to input the sample scan image into the initial segmentation model and output the first segmentation probability map corresponding to the sample scan image through the initial segmentation model.

[0190] The second output unit 152 is used to perform data transformation on the sample scan image to obtain a deformed scan image, input the deformed scan image into the initial segmentation model, and output the second segmentation probability map corresponding to the deformed scan image through the initial segmentation model.

[0191] The probability map fusion unit 153 is used to obtain the sample segmentation probability map corresponding to the sample scan image based on the first segmentation probability map and the second segmentation probability map.

[0192] The specific functional implementations of the first output unit 151, the second output unit 152, and the probability graph fusion unit 153 can be found in [reference needed]. Figure 5 Step S201 in the corresponding embodiment will not be described again here.

[0193] In one or more embodiments, the determining module 12 includes: a filtering operation unit 121, a rendering unit 122, a transformation matrix construction unit 123, and a body surface center determining unit 124.

[0194] The filtering operation unit 121 is used to obtain the spatial dimension information corresponding to the reference mark, perform a filtering operation on the spatial dimension information, and obtain the filtered data corresponding to the reference mark.

[0195] The rendering unit 122 is used to encapsulate the filtered data into a set of key-value pairs, and to perform rendering processing on the set of key-value pairs based on the rendering window to obtain the simulation mark model corresponding to the baseline mark.

[0196] Transformation matrix construction unit 123 is used to convert the mask image into a mesh image and construct the transformation matrix based on the corresponding points between the simulation marker model and the mesh image.

[0197] The body surface center determination unit 124 is used to obtain the body surface center point of the reference marker in the object scanning image based on the transformation matrix and the center point of the simulation marker model.

[0198] Optionally, the center points of the simulation marker model include center point P1 and center point P2;

[0199] The body surface center determination unit 124 includes: a calculation subunit 1241, a centroid determination subunit 1242, and a distance comparison subunit 1243;

[0200] The operation subunit 1241 is used to perform a product operation on the transformation matrix and the array composed of center point P1 and center point P2 to obtain the center point F1 and center point F2 of the reference mark;

[0201] The centroid determination subunit 1242 is used to obtain the centroid of the object contained in the object scan image, obtain the first distance between the center point F1 and the centroid, and the second distance between the center point F2 and the centroid.

[0202] The distance comparison subunit 1243 is used to determine the center point F1 as the reference mark on the body surface center point in the object scan image if the first distance is less than the second distance.

[0203] The specific functional implementation of the filtering operation unit 121, rendering unit 122, transformation matrix construction unit 123, body surface center determination unit 124, and the operation subunits 1241, centroid determination subunit 1242, and distance comparison subunit 1243 contained in the body surface center determination unit 124 can be found in [reference needed]. Figure 5 Steps S205-S209 in the corresponding embodiments will not be described again here.

[0204] In one or more embodiments, the navigation module 13 includes: a marker registration unit 131 and a navigation operation unit 132;

[0205] The marker registration unit 131 is used to perform marker registration on the spatial marker position and the center point of the body surface to obtain the marker registration result between the spatial object indicated by the object scan image and the object scan image.

[0206] The navigation operation unit 132 is used to perform navigation operations on spatial objects in the image navigation scene based on the marker registration results.

[0207] The specific functional implementation methods of the marker registration unit 131 and the navigation operation unit 132 can be found in [reference needed]. Figure 3 Step S103 in the corresponding embodiment will not be described again here.

[0208] In this embodiment, by segmenting the reference marks in the object scan image, a mask image corresponding to each reference mark can be obtained. Based on the mask image, the body surface center point of the reference mark can be further fitted. By registering the body surface center point with the spatial mark position determined by the visual navigation tool, that is, by automatically locating and detecting the reference marks in the object scan image, the positioning accuracy of the body surface center point can be improved. Consequently, in the image scanning scene, navigation operations can be performed on the spatial object indicated by the object scan image, thereby improving the navigation effect.

[0209] Further, please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 11 As shown, the computer device 1000 can be a user terminal, for example, the one described above. Figure 1 The user terminal 10a in the corresponding embodiment can also be a server, for example, as described above. Figure 1 The server 10d in the corresponding embodiment will not be limited here. For ease of understanding, this application takes a computer device as a user terminal as an example. The computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may also optionally be at least one storage device located away from the aforementioned processor 1001. Figure 11 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0210] The network interface 1004 in the computer device 1000 can also provide network communication functions, and the optional user interface 1003 can also include a display screen and a keyboard. Figure 11 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0211] Obtain an object scan image containing reference markers, and extract the mask image corresponding to the reference markers from the object scan image; the mask image is used to characterize the shape of the reference markers in the object scan image;

[0212] Obtain the simulation marker model corresponding to the reference marker, and determine the center point of the reference marker on the object surface in the scanned image based on the simulation marker model and the mask image;

[0213] The spatial marker position corresponding to the reference marker is obtained through a visual navigation tool. Based on the spatial marker position and the center point of the body surface, navigation operations are performed on the spatial object indicated by the object scan image.

[0214] It should be understood that the computer device 1000 described in the embodiments of this application can perform the foregoing... Figure 3 , Figure 5 as well as Figure 8 The description of the data processing method in any of the embodiments can also be performed as described above. Figure 10 The description of the data processing device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0215] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned data processing device 1. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figure 3 , Figure 5 as well as Figure 8 The description of the data processing method in any of the embodiments is redundant and will not be repeated here. Similarly, the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions may be deployed and executed on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network. These multiple computing devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0216] Furthermore, it should be noted that this application also provides a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, causing the computer device to perform the aforementioned actions. Figure 3 , Figure 5 as well as Figure 8 The description of the data processing method in any of the embodiments is redundant and will not be repeated here. Similarly, the beneficial effects of using the same method will not be repeated here either. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0217] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0218] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0219] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0220] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0221] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: The system acquires object scan images from the planning system and selects a segmentation processing method for the object scan images based on their imaging modality. The object scan images contain reference markers, and different segmentation processing methods are used for object scan images with different imaging modalities. The reference marker in the object scan image is segmented using the segmentation processing method of the imaging modality to obtain the mask image corresponding to the reference marker; Obtain the simulation marker model of the reference marker, convert the mask image into a mesh image, use the iterative nearest point transformation algorithm to determine the corresponding point pairs between the simulation marker model and the mesh image, and construct the transformation matrix between the simulation marker model and the mesh image based on the corresponding point pairs; Based on the transformation matrix and the two center points of the simulation marker model, the two center points of the reference marker are determined. Among the two center points of the reference marker, the center point closest to the centroid is determined as the body surface center point of the reference marker in the object scan image. The body surface center point is used to characterize the position of the reference marker in the object scan image. The spatial marker position corresponding to the reference marker is obtained through a visual navigation tool. Based on the spatial marker position and the center point of the body surface, a navigation operation is performed on the spatial object indicated by the object scan image.

2. The method according to claim 1, characterized in that, The segmentation process includes a segmentation threshold; The step of segmenting the reference marker in the scanned image of the object using the aforementioned segmentation processing method to obtain the mask image corresponding to the reference marker includes: The object scan image is segmented using the segmentation threshold to obtain a region image containing the reference marker; Obtain M connected components contained in the region image, and based on the volume of each of the M connected components, obtain the mask image corresponding to the reference marker in the M connected components; M is a positive integer.

3. The method according to claim 2, characterized in that, The step of obtaining the mask image corresponding to the reference marker in the M connected components based on the connected component volumes of the M connected components includes: Obtain the connected component volume corresponding to each of the M connected components, and mark each of the M connected components based on the connected component volume to obtain the identification information corresponding to each of the M connected components; Based on the identification information, connected components whose volumes fall within the volume threshold range are identified as reference markers contained in the object scan image. Obtain the binary image of the reference mark, and determine the binary image as the mask image corresponding to the reference mark.

4. The method according to claim 1, characterized in that, The segmentation processing method includes a semantic segmentation model; The step of segmenting the reference marker in the scanned image of the object to obtain the mask image corresponding to the reference marker includes: The object scan image is input into the semantic segmentation model, and the semantic segmentation model outputs a labeled segmentation probability map corresponding to the object scan image. The connected components in the marked segmentation probability graph are separated to obtain the mask image corresponding to the reference mark.

5. The method according to claim 4, characterized in that, The step of outputting the labeled segmentation probability map corresponding to the object scan image through the semantic segmentation model includes: The object scan image is downsampled through the first network layer in the semantic segmentation model to obtain the abstract semantic feature map corresponding to the object scan image; The abstract semantic feature map is upsampled through the second network layer in the semantic segmentation model to obtain the labeled feature map corresponding to the object scan image; By using the activation network layer in the semantic segmentation model, feature selection is performed on the labeled feature map to obtain the labeled segmentation probability map corresponding to the object scan image.

6. The method according to claim 5, characterized in that, The step of upsampling the semantic feature map through the second network layer of the semantic segmentation model to obtain the labeled feature map corresponding to the object scan image includes: The semantic feature map is upsampled through the second network layer in the semantic segmentation model to obtain the marker localization feature map corresponding to the object scan image; The marker localization feature map and the object scan image are fused to obtain the marker feature map.

7. The method according to claim 4, characterized in that, Also includes: Acquire a sample scan image containing sample markers; the sample scan image includes label information corresponding to the sample markers; The sample scan image is input into the initial segmentation model, and the initial segmentation model outputs the sample segmentation probability map corresponding to the sample scan image. Obtain the similarity between the sample segmentation probability map and the label information, and determine the metric function corresponding to the initial segmentation model based on the similarity; The network parameters of the initial segmentation model are corrected based on the metric function, and the initial segmentation model containing the corrected network parameters is determined as the semantic segmentation model.

8. The method according to claim 7, characterized in that, The step of inputting the sample scan image into the initial segmentation model and outputting the sample segmentation probability map corresponding to the sample scan image through the initial segmentation model includes: The sample scan image is input into the initial segmentation model, and the initial segmentation model outputs the first segmentation probability map corresponding to the sample scan image. The sample scan image is transformed to obtain a deformed scan image. The deformed scan image is input into the initial segmentation model. The initial segmentation model outputs a second segmentation probability map corresponding to the deformed scan image. Based on the first segmentation probability map and the second segmentation probability map, the sample segmentation probability map corresponding to the sample scan image is obtained.

9. The method according to claim 1, characterized in that, The step of obtaining the simulation marker model corresponding to the benchmark marker includes: Obtain the spatial dimension information corresponding to the reference mark, perform a filtering operation on the spatial dimension information, and obtain the filtered data corresponding to the reference mark; The filtered data is encapsulated into a set of key-value pairs, and the set of key-value pairs is rendered based on a rendering window to obtain the simulation tag model corresponding to the benchmark tag.

10. The method according to claim 1, characterized in that, The two center points of the simulation marker model include center point P1 and center point P2, and the two center points of the reference marker include center point F1 and center point F2. The step of determining the two center points of the reference marker based on the transformation matrix and the two center points of the simulation marker model, and determining the center point closest to the centroid of the reference marker as the body surface center point of the reference marker in the object scan image, includes: The transformation matrix and the array consisting of the center point P1 and the center point P2 are multiplied to obtain the center point F1 and the center point F2 of the reference mark. Obtain the centroid of the object contained in the object scan image, obtain the first distance between the center point F1 and the centroid, and the second distance between the center point F2 and the centroid; If the first distance is less than the second distance, then the center point F1 is determined as the center point of the reference mark on the body surface in the object scan image.

11. The method according to claim 1, characterized in that, The step of performing navigation operations on the spatial object indicated by the object scan image based on the spatial marker position and the body surface center point includes: The spatial marker position and the body surface center point are marked and registered to obtain the marking and registration result between the spatial object indicated by the object scan image and the object scan image; Based on the marker registration results, navigation operations are performed on the spatial objects in the image navigation scene.

12. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire object scan images input from the planning system, and select the segmentation processing method of the object scan images based on the imaging modality of the object scan images; the object scan images contain reference markers, and different segmentation processing methods are used for object scan images with different imaging modalities; The acquisition module is further configured to use the segmentation processing method of the imaging modality to segment the reference mark in the object scan image to obtain the mask image corresponding to the reference mark; The determination module is used to obtain the simulation mark model of the reference mark, convert the mask image into a mesh image, use the iterative nearest point transformation algorithm to determine the corresponding point pairs between the simulation mark model and the mesh image, and construct the transformation matrix between the simulation mark model and the mesh image based on the corresponding point pairs; The determining module is further configured to determine two center points of the reference mark based on the transformation matrix and two center points of the simulation mark model, and to determine the center point closest to the centroid of the reference mark as the body surface center point of the reference mark in the object scan image; the body surface center point is used to characterize the position of the reference mark in the object scan image; The navigation module is used to obtain the spatial marker position corresponding to the reference marker through a visual navigation tool, and to perform navigation operations on the spatial object indicated by the object scan image based on the spatial marker position and the body surface center point.

13. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method according to any one of claims 1-11.

Images