Image segmentation method and device, equipment and medium

Abstract

The invention relates to an image segmentation method and device, equipment and a medium. Comprising the following steps: acquiring an image position code of an original image, a prompt position code of user trigger information on the original image and a description text of the original image; processing the image position code, the prompt position code and the description text by using a pre-acquired encoder, and acquiring an image code feature corresponding to the image position code, a prompt code feature corresponding to the prompt position code and a text code feature corresponding to the description text; performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature; and processing the fused coding feature by using a pre-acquired decoder to generate a target segmentation result of the original image. Therefore, image segmentation is carried out by using the multi-modal information of the image, the accuracy of image segmentation is improved, the accuracy of a planning scheme is further improved, and the auditing cost of the planning scheme is reduced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image segmentation, and in particular to an image segmentation method, device, equipment and medium. Background Art

[0002] In the field of medical robot control, the robot usually needs to segment the collected images and then develop a planning plan. Then, the planning plan is reviewed manually to ensure the accuracy of the planning plan and the safety of operations based on the planning plan.

[0003] At present, robots usually segment images only based on the image features of the collected images. This segmentation method has low segmentation accuracy, resulting in inaccurate planning schemes, which in turn require manual correction of planning schemes or re-labeling of planning schemes. Therefore, it is necessary to provide a new image segmentation method to improve segmentation accuracy, thereby improving the accuracy of planning schemes and reducing the review cost of planning schemes. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides an image segmentation method, device, equipment and medium.

[0005] In a first aspect, the present disclosure provides an image segmentation method, comprising:

[0006] Obtaining an image position code of the original image, a prompt position code of user-triggered information on the original image, and a description text of the original image;

[0007] Using a pre-acquired encoder, the image position code, the prompt position code and the description text are processed to obtain image coding features corresponding to the image position code, prompt coding features corresponding to the prompt position code and text coding features corresponding to the description text;

[0008] Performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature;

[0009] The fused coding features are processed using a pre-acquired decoder to generate a target segmentation result of the original image.

[0010] In a second aspect, the present disclosure provides an image segmentation device, comprising:

[0011] An information acquisition module, used to acquire an image position code of an original image, a prompt position code of user-triggered information on the original image, and a description text of the original image;

[0012] An encoding module, used to process the image position code, the prompt position code and the description text using a pre-acquired encoder, to obtain image encoding features corresponding to the image position code, prompt encoding features corresponding to the prompt position code and text encoding features corresponding to the description text;

[0013] A feature fusion module, used for performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature;

[0014] The decoding module is used to process the fused coding features using a pre-acquired decoder to generate a target segmentation result of the original image.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When one or more programs are executed by one or more processors, the one or more processors implement the method provided in the first aspect.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0020] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0021] An image segmentation method, device, equipment and medium of the disclosed embodiment include: obtaining the image position code of the original image, the prompt position code of the user trigger information on the original image and the description text of the original image; using the pre-acquired encoder, processing the image position code, the prompt position code and the description text, obtaining the image coding features corresponding to the image position code, the prompt coding features corresponding to the prompt position code and the text coding features corresponding to the description text; performing feature fusion processing on the image coding features, the prompt coding features and the text coding features to generate fused coding features; using the pre-acquired decoder, processing the fused coding features to generate the target segmentation result of the original image. Thus, firstly, the multimodal information composed of the original image, the annotation information of the original image and the description information of the original image is encoded, and then the encoding results are fused and decoded to achieve image segmentation. Therefore, the multimodal information of the image is used for image segmentation, which improves the accuracy of image segmentation, thereby improving the accuracy of the planning scheme and reducing the review cost of the planning scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 A flowchart of an image segmentation method provided by an embodiment of the present disclosure;

[0025] Figure 2 A schematic diagram of the structure of a feature fuser provided in an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of the structure of a decoder provided in an embodiment of the present disclosure;

[0027] Figure 4 A logical schematic diagram of an image segmentation method provided by an embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram of the structure of an image segmentation device provided in an embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0032] In order to improve the accuracy of image segmentation, the following Figures 1 to 4 The image segmentation method provided in the embodiment of the present disclosure is described. In the embodiment of the present disclosure, the image segmentation method can be executed by an electronic device. The electronic device can be a control device of a medical robot.

[0033] Figure 1A flow chart of an image segmentation method provided by an embodiment of the present disclosure is shown.

[0034] like Figure 1 As shown, the image segmentation method may include the following steps.

[0035] S110, obtaining an image position code of the original image, a prompt position code of user trigger information on the original image, and a description text of the original image.

[0036] In this embodiment, the electronic device first obtains the image to be segmented as the original image, and then performs image block embedding processing on the original image to divide the original image into multiple image blocks. Next, the electronic device performs position encoding on all the image blocks of the original image respectively to obtain the position encoding of all the image blocks of the original image.

[0037] In order to improve the segmentation accuracy of the original image, multimodal information of the original image can also be obtained. The multimodal information includes user trigger information on the original image determined based on the trigger operation of the user on the interactive interface, and a description text of the original image. Further, the electronic device performs position encoding on the user trigger information to obtain a prompt position encoding of the user trigger information.

[0038] The image position encoding refers to the specific position corresponding to each pixel in the original image, which is used to enhance the image segmentation algorithm's ability to understand the pixel position in the original image. Optionally, the original image includes but is not limited to a tooth image, a bone image, a lung image, and the like.

[0039] The user trigger information refers to the content marked by the user on the original image. Optionally, the user trigger information includes one or both of key points and detection frames. Correspondingly, the prompt position encoding refers to the specific position corresponding to the pixel contained in the user trigger information, which is used to enhance the image segmentation algorithm's ability to understand the pixel position in the user trigger information.

[0040] The description text refers to the language description information of the original image, which may be a description of the image features of the original image, so as to facilitate the image segmentation algorithm to understand the original image based on the description text.

[0041] S120, using the pre-acquired encoder, process the image position code, the prompt position code and the description text to obtain image coding features corresponding to the image position code, prompt coding features corresponding to the prompt position code and text coding features corresponding to the description text.

[0042] In this embodiment, the electronic device calls a pre-acquired encoder, processes the image position encoding based on the original image encoding network in the encoder to obtain image encoding features, processes the prompt position encoding based on the prompt encoding network in the encoder to obtain prompt encoding features, and processes the description text based on the text encoding network in the encoder to obtain text encoding features.

[0043] Optionally, the original image encoding network in the encoder includes but is not limited to a Masked Autoencoder (MAE).

[0044] Among them, the prompt coding network in the encoder is used to accurately segment the prompt position coding of the user trigger information. Specifically, the electronic device can first obtain the classification vector of the key points contained in the user trigger information and the classification vector of the detection frame, select two key points (such as the upper left corner point and the lower right corner point) from the detection frame, and then use the prompt coding network to encode the prompt position of the key points contained in the user trigger information and the prompt position coding of the two key points contained in the detection frame, use the same encoding method for encoding, and then distinguish the prompt coding features corresponding to the prompt position coding of the key points and the prompt coding features corresponding to the prompt position coding of the detection frame based on the classification vector.

[0045] The text encoding network in the encoder is used to extract features from the description text to guide the accurate segmentation of the original image. Optionally, the text encoding network includes but is not limited to a Chinese language model (such as Taiyi-CLIP-Roberta-large-326M-Chinese).

[0046] In this way, the encoder can assist the encoder in improving its ability to understand the original image based on the multimodal information of the original image, which is beneficial to improving the segmentation accuracy of the original image.

[0047] S130, performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature.

[0048] In this embodiment, in order to achieve accurate image segmentation, the electronic device can also call a specific feature fusion algorithm to perform feature fusion processing on multimodal features including image coding features, prompt coding features and text coding features, so that the multimodal features can be efficiently fused to obtain fused coding features.

[0049] The fused coding feature refers to a feature vector that integrates the image features of the original image, the features of the user-triggered information, and the features describing the text representation, that is, the fused coding feature is a feature vector that integrates multimodal features.

[0050] S140: Using the pre-acquired decoder, the fused coding features are processed to generate a target segmentation result of the original image.

[0051] In this embodiment, the electronic device calls a pre-acquired decoder to decode the fused coding features to segment the target object from the original image as a target segmentation result.

[0052] The target segmentation result refers to the target object that is focused on in the original image. For example, if the original image is a tooth image, the target segmentation result is the root canal of the tooth image. For another example, if the original image is a lung image, the target segmentation result is the cancer cells in the lung image.

[0053] An image segmentation method of an embodiment of the present disclosure includes: obtaining an image position code of an original image, a prompt position code of user trigger information on the original image, and a description text of the original image; using a pre-acquired encoder, processing the image position code, the prompt position code, and the description text to obtain image coding features corresponding to the image position code, prompt coding features corresponding to the prompt position code, and text coding features corresponding to the description text; performing feature fusion processing on the image coding features, the prompt coding features, and the text coding features to generate fused coding features; using a pre-acquired decoder, processing the fused coding features to generate a target segmentation result of the original image. Thus, firstly, the multimodal information composed of the original image, the annotation information of the original image, and the description information of the original image is encoded, and then the encoding results are fused and decoded to achieve image segmentation. Therefore, the multimodal information of the image is used for image segmentation, which improves the accuracy of image segmentation, thereby improving the accuracy of the planning scheme and reducing the review cost of the planning scheme.

[0054] In order to expand the form of multimodal information input to the encoder, the method further includes:

[0055] Obtaining an initial segmentation result corresponding to the original image, wherein the initial segmentation result is obtained by processing the original image through a preset segmentation algorithm;

[0056] Using the encoder obtained in advance, the image position code, the initial segmentation result, the prompt position code, and the description text are processed to obtain image coding features corresponding to the image position code, segmentation coding features corresponding to the initial segmentation result, prompt coding features corresponding to the prompt position code, and text coding features corresponding to the description text;

[0057] Accordingly, the specific implementation method of S130 includes:

[0058] Add the image coding feature and the segmentation coding feature to obtain the image addition feature;

[0059] The image addition features, the prompt coding features and the text coding features are fused to generate fused coding features.

[0060] Specifically, the electronic device obtains an initial segmentation result obtained by segmenting the original image based on a preset segmentation algorithm, and then calls a pre-acquired encoder, and processes the initial segmentation result based on the segmentation image coding network in the encoder to obtain segmentation coding features. At the same time, based on the original image coding network in the encoder, the image position coding is processed to obtain image coding features, and based on the prompt coding network in the encoder, the prompt position coding is processed to obtain prompt coding features, and based on the text coding network in the encoder, the description text is processed to obtain text coding features, thereby obtaining richer coding features.

[0061] Furthermore, since both the image coding features and the segmentation coding features can represent the image features of the original image, the electronic device adds the image coding features and the segmentation coding features of the same pixel to obtain the image addition features. Then, the electronic device calls a specific feature fusion algorithm to perform feature fusion processing on the multimodal features including the image addition features, the prompt coding features and the text coding features, so that the multimodal features with more categories can be efficiently fused to obtain the fused coding features.

[0062] The preset segmentation algorithm is used to segment the original image based on image features (such as pixel values, texture values) of the original image. Optionally, the preset segmentation algorithm includes but is not limited to a threshold-based segmentation algorithm, an edge-based segmentation algorithm, a region-based segmentation algorithm, and an energy-based segmentation algorithm.

[0063] In this way, since more forms of multimodal features are obtained, it is beneficial to improve the segmentation accuracy of the original image based on more forms of multimodal features.

[0064] In another embodiment of the present disclosure, S110, S130 and S140 are explained in detail respectively.

[0065] In some embodiments of the present disclosure, the specific method of "obtaining the image position code of the original image and the prompt position code of the user trigger information on the original image" in S110 includes but is not limited to the following method: based on the coordinate data of the first pixel point in the original image, the dimension of the coding vector corresponding to the first pixel point, and the dimension index in the coding vector corresponding to the first pixel point, position encoding the original image to obtain the image position code; based on the coordinate data of the second pixel point in the user trigger information, the dimension of the coding vector corresponding to the second pixel point, and the dimension index in the coding vector corresponding to the second pixel point, position encoding the user trigger information to obtain the prompt position code.

[0066] Specifically, the electronic device takes each pixel in the original image as the first pixel, takes the number of channels as the dimension of the coding vector corresponding to the first pixel, and adopts a sinusoidal position coding method to position-code the original image based on the coordinate data of the first pixel, the dimension of the coding vector corresponding to the first pixel, and the dimension index in the coding vector corresponding to the first pixel, to obtain an image position code; at the same time, the electronic device takes the pixel contained in the user trigger information as the second pixel, takes the number of channels as the position of the coding vector corresponding to the second pixel, and adopts a sinusoidal position coding method to position-code the user trigger information based on the coordinate data of the second pixel, the dimension of the coding vector corresponding to the second pixel, and the dimension index in the coding vector corresponding to the second pixel, to obtain a prompt position code.

[0067] Optionally, the sinusoidal position encoding method can be implemented in one of the following ways:

[0068] PE(x,y,z,2i)=sin(x / 10000^(6i / D))

[0069] PE(x,y,z,2i+1)=cos(x / 10000^(6i / D))

[0070] PE(x,y,z,2j+D / 3)=sin(y / 10000^(6j / D))

[0071] PE(x,y,z,2j+1+D / 3)=cos(y / 10000^(6j / D))

[0072] PE(x,y,z,2k+2D / 3)=sin(z / 10000^(6k / D))

[0073] PE(x,y,z,2k+1+2D / 3)=cos(z / 10000^(6k / D))

[0074] Among them, x, y, z are the coordinate data of the first pixel point or the second pixel point, D is the number of channels, that is, the dimension of the encoding vector corresponding to the first pixel point or the second pixel point, 2i, 2i+1, 2j+D / 3, 2j+1+D / 3, 2k+2D / 3, 2k+1+2D / 3 are the dimension indexes in the encoding vector corresponding to the first pixel point or the second pixel point.

[0075] Therefore, the sinusoidal position coding method is adopted to respectively encode the image position coding of the original image and the user trigger information on the original image, so that the encoder can enhance the understanding ability of the pixel position in the original image based on the obtained image position coding and prompt position coding, which is further beneficial to improve the image segmentation accuracy.

[0076] In some embodiments of the present disclosure, specific implementation methods of S130 include but are not limited to the following methods:

[0077] S1301, stacking the prompt coding feature and the text coding feature to obtain the prompt coding feature;

[0078] S1302: Using a pre-acquired feature fuser, perform feature fusion processing on the prompt coding feature and the image coding feature to generate a fused coding feature.

[0079] Among them, the prompt coding features include key point coding features and detection box coding features; accordingly, the specific implementation method of S1301 includes but is not limited to the following method: stacking the key point coding features and the detection box coding features along a first stacking direction to obtain a first stacking feature; stacking the first stacking feature and the text coding feature along a second stacking direction to obtain a second stacking feature, and using the second stacking feature as the prompt coding feature.

[0080] The first stacking direction refers to the dimension (eg, token dimension) of the classification vector corresponding to the prompt coding feature and the image coding feature. The second stacking dimension refers to the dimension (eg, token dimension) of the classification vector corresponding to the first stacking feature and the text coding feature.

[0081] Among them, the specific implementation method of S1302 includes but is not limited to the following methods: based on the self-attention network in the feature fuser, the prompt coding feature is processed to obtain the self-attention feature; based on the prompt-to-image attention network in the feature fuser, the self-attention feature and the image coding feature are processed to obtain the first cross-attention feature; based on the first fully connected network in the feature fuser, the first cross-attention feature is processed to obtain the target prompt feature; based on the image-to-prompt attention network in the feature fuser, the target prompt feature and the image coding feature are processed to obtain the second cross-attention feature, and the second cross-self-attention feature is used as the target image feature; wherein, the target prompt feature and the target image feature constitute the fused coding feature.

[0082] For easier understanding, see Figure 2The structural diagram of the feature fusion device shown in FIG. Specifically, first, the electronic device inputs the prompt coding feature into the self-attention network for self-attention processing to obtain the self-attention feature, then inputs the self-attention feature and the image coding feature into the prompt-to-image attention network for prompt-to-image cross-attention processing to obtain the first cross-attention feature, then inputs the first cross-attention feature into the first fully connected network to obtain an updated prompt feature as the target prompt feature, at the same time, inputs the updated prompt feature and the image coding feature into the image-to-prompt attention network for image-to-prompt cross-attention processing to obtain an updated image feature as the target image feature, and the target prompt feature and the target image feature constitute a fused coding feature.

[0083] In this way, the feature fuser uses a cross-attention mechanism to process the prompt encoding features and image encoding features, which not only retains the detailed information of the original image itself, but also fully considers the user-triggered information including key points and detection boxes, which is conducive to improving the image segmentation accuracy.

[0084] In some embodiments of the present disclosure, the specific implementation method of S140 includes but is not limited to the following methods: based on the upsampling network in the decoder, the target image feature contained in the fused coding feature is processed to obtain the upsampled image feature; based on the prompt-to-image attention network in the decoder, the target prompt feature contained in the fused coding feature is processed to obtain the second cross-attention feature; based on the second fully connected network in the decoder, the second cross-attention feature is processed to obtain the confidence of the target prompt feature; based on the third fully connected network in the decoder, the second cross-attention feature is processed to obtain the fully connected feature; dot multiplication is performed on the upsampled image feature and the fully connected feature to obtain a candidate segmentation result of the original image; if the confidence of the target prompt feature is greater than a preset confidence threshold, the candidate segmentation result is used as the target segmentation result of the original image.

[0085] For easier understanding, see Figure 3 The schematic diagram of the decoder structure is shown. Specifically, first, the electronic device inputs the updated image features (i.e., the target image features) into the upsampling network for upsampling processing to obtain the upsampled image features; at the same time, the electronic device inputs the updated prompt features (i.e., the target prompt features) into the prompt image attention network for cross-attention processing to obtain the second cross-attention features, then, the second cross-attention features are input into the second fully connected network and the third fully connected network for processing to obtain the confidence and fully connected features of the target prompt features, then, the upsampled image features and the fully connected features are multiplied to obtain the candidate segmentation results of the original image, and finally, it is determined whether the confidence of the target prompt features is greater than the preset confidence threshold, if so, the candidate segmentation results are used as the target segmentation results of the original image.

[0086] The confidence of the target hint feature may specifically be an intersection-over-union ratio, which is used to evaluate whether the candidate segmentation result is reliable.

[0087] The second fully connected network and the third fully connected network are fully connected networks with different parameters and have different functions.

[0088] In this way, the decoder uses a cross-attention mechanism to process the target prompt features, which not only retains the detailed information of the original image itself, but also fully considers the user trigger information including key points and detection frames, and combines the upsampled image features corresponding to the target image features to perform image segmentation, which is beneficial to improve the image segmentation accuracy.

[0089] In yet another embodiment of the present disclosure, the overall logic of the image segmentation method is explained in detail.

[0090] Figure 4 A logical schematic diagram of an image segmentation method provided by an embodiment of the present disclosure is shown.

[0091] S410, obtaining an image position code of an original image, an initial segmentation result corresponding to the original image, a prompt position code of user trigger information on the original image, and a description text of the original image.

[0092] Optionally, the user trigger information includes key points and detection boxes.

[0093] S420. Using the pre-acquired encoder, the image position code, the initial segmentation result, the prompt position code, and the description text are processed to obtain image coding features corresponding to the image position code, segmentation coding features corresponding to the initial segmentation result, prompt coding features corresponding to the prompt position code, and text coding features corresponding to the description text.

[0094] Specifically, the electronic device calls a pre-acquired encoder, processes the image position encoding based on the original image encoding network in the encoder to obtain image encoding features, and at the same time, processes the initial segmentation result based on the segmentation image encoding network in the encoder to obtain segmentation encoding features. At the same time, based on the prompt encoding network in the encoder, processes the prompt position encoding to obtain prompt encoding features, and at the same time, based on the text encoding network in the encoder, processes the description text to obtain text encoding features.

[0095] Optionally, the hint position encoding includes key point encoding features and detection box encoding features.

[0096] S430, adding the image coding feature and the segmentation coding feature to obtain an image addition feature.

[0097] Specifically, the electronic device adds the image coding feature and the segmentation coding feature of the same pixel to obtain an image addition feature.

[0098] S440: Stack the key point coding features and the detection box coding features along a first stacking direction to obtain a first stacking feature.

[0099] In this way, by stacking the key point encoding features and the detection box encoding features, the key point information and the detection box information are fused into the first stacking feature.

[0100] S450: stacking the first stacking feature and the text coding feature along a second stacking direction to obtain a second stacking feature, and using the second stacking feature as a prompt coding feature.

[0101] In this way, by stacking the first stacking feature and the text encoding feature, the key point information, the detection box information and the text information are integrated into the second stacking feature as a prompt encoding feature.

[0102] S460: Perform feature fusion processing on the image addition feature and the prompt coding feature to generate a fused coding feature.

[0103] In this embodiment, the specific implementation method of S460 includes but is not limited to the following method: using a pre-acquired feature fuser to perform feature fusion processing on the prompt coding feature and the image coding feature to generate a fused coding feature.

[0104] Specifically, the electronic device processes the prompt coding feature based on the self-attention network in the feature fuser to obtain the self-attention feature; processes the self-attention feature and the image coding feature based on the prompt-to-image attention network in the feature fuser to obtain the first cross-attention feature; processes the first cross-attention feature based on the first fully connected network in the feature fuser to obtain the target prompt feature; processes the target prompt feature and the image coding feature based on the image-to-prompt attention network in the feature fuser to obtain an updated image feature, and uses the updated image feature as the target image feature; wherein the target prompt feature and the target image feature constitute the fused coding feature.

[0105] S470: Using the pre-acquired decoder, process the fused coding features to generate a target segmentation result of the original image.

[0106] Among them, the specific implementation method of S470 includes but is not limited to the following methods: based on the upsampling network in the decoder, the target image feature contained in the fused coding feature is processed to obtain the upsampled image feature; based on the prompt-to-image attention network in the decoder, the target prompt feature contained in the fused coding feature is processed to obtain the second cross-attention feature; based on the second fully connected network in the decoder, the second cross-attention feature is processed to obtain the confidence of the target prompt feature; based on the third fully connected network in the decoder, the second cross-attention feature is processed to obtain the fully connected feature; dot multiplication is performed on the upsampled image feature and the fully connected feature to obtain the candidate segmentation result of the original image; if the confidence of the target prompt feature is greater than the preset confidence threshold, the candidate segmentation result is used as the target segmentation result of the original image.

[0107] In some embodiments, when performing model training on the encoder and the decoder, the electronic device obtains pre-prepared training data, the training data including a reference image and annotation information of the segmented area in the reference image. For example, a pre-prepared oral and maxillofacial cone beam computed tomography (oral CBCT) image and corresponding segmentation annotations of each tooth and each root canal are obtained. Specifically, first, the electronic device randomly selects a segmentation object in the reference image, such as the root canal of tooth No. 11 and tooth No. 23, etc. Then, the user randomly selects a point from the segmentation object of the reference image as a key point, and randomly draws a detection box from the segmentation object of the reference image, thereby obtaining user trigger information. At the same time, the reference image is described using text to obtain a description text of the reference image, such as the description text includes the root canal in tooth No. 11 and tooth No. 23, etc. Then, the electronic device performs position encoding on a portion of the image blocks of the reference image and the user trigger information of the reference image, determines the image position encoding of the portion of the image blocks of the reference image and the prompt position encoding of the user trigger information on the reference image, then, the electronic device inputs the image position encoding of the portion of the image blocks of the reference image, the prompt position encoding of the user trigger information on the reference image, and the description text of the reference image into the encoder, fuses the image features output by the encoder, and inputs them into the decoder together with the remaining image blocks in the reference image that have not been position encoded, to obtain a predicted segmentation result, and then iteratively trains the model based on the predicted segmentation result and the annotation information, and calculates the loss function during the training process, which includes two parts, including diceloss and focal loss. loss, and the intersection-over-intersection ratio uses the mean square error. The two loss functions are added 1:1. Finally, the parameters of the encoder and decoder are optimized based on the loss function to obtain the trained encoder and decoder.

[0108] Optionally, the loss function includes but is not limited to a mean square error loss function (MSE), and may also be other loss functions.

[0109] The present disclosure also provides an image segmentation device for implementing the above-mentioned image segmentation method, wherein the device is configured in a control device of a robot. Figure 5 In the embodiment of the present disclosure, the image segmentation device may be an electronic device, wherein the electronic device may be a control device of a medical robot.

[0110] Figure 5 A schematic structural diagram of an image segmentation device provided in an embodiment of the present disclosure is shown.

[0111] like Figure 5 As shown, the image segmentation device 500 may include:

[0112] An information acquisition module 510 is used to acquire an image position code of an original image, a prompt position code of user-triggered information on the original image, and a description text of the original image;

[0113] The encoding module 520 is used to process the image position code, the prompt position code and the description text by using the pre-acquired encoder to obtain image coding features corresponding to the image position code, prompt coding features corresponding to the prompt position code and text coding features corresponding to the description text;

[0114] A feature fusion module 530 is used to perform feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature;

[0115] The decoding module 540 is used to process the fused coding features using a pre-acquired decoder to generate a target segmentation result of the original image.

[0116] An image segmentation device according to an embodiment of the present disclosure includes: obtaining an image position code of an original image, a prompt position code of user trigger information on the original image, and a description text of the original image; using a pre-acquired encoder, processing the image position code, the prompt position code, and the description text to obtain image coding features corresponding to the image position code, prompt coding features corresponding to the prompt position code, and text coding features corresponding to the description text; performing feature fusion processing on the image coding features, the prompt coding features, and the text coding features to generate fused coding features; using a pre-acquired decoder, processing the fused coding features to generate a target segmentation result of the original image. Thus, firstly, the multimodal information composed of the original image, the annotation information of the original image, and the description information of the original image is encoded, and then the encoding results are fused and decoded to achieve image segmentation. Therefore, the multimodal information of the image is used for image segmentation, which improves the accuracy of image segmentation, thereby improving the accuracy of the planning scheme and reducing the review cost of the planning scheme.

[0117] In some embodiments of the present disclosure, the information acquisition module 510 includes:

[0118] A first position encoding unit, configured to perform position encoding on the original image based on coordinate data of a first pixel point in the original image, a dimension of a coding vector corresponding to the first pixel point, and a dimension index in the coding vector corresponding to the first pixel point, so as to obtain the image position encoding;

[0119] The second position encoding unit is used to perform position encoding on the user trigger information based on the coordinate data of the second pixel point in the user trigger information, the dimension of the encoding vector corresponding to the second pixel point, and the dimension index in the encoding vector corresponding to the second pixel point to obtain the prompt position code.

[0120] In some embodiments of the present disclosure, the feature fusion module 530 includes:

[0121] A stacking processing unit, used for stacking the prompt coding feature and the text coding feature to obtain the prompt coding feature;

[0122] The feature fusion unit is used to use a pre-acquired feature fuser to perform feature fusion processing on the prompt coding feature and the image coding feature to generate the fused coding feature.

[0123] In some embodiments of the present disclosure, the hint coding features include key point coding features and detection box coding features;

[0124] Accordingly, the stacking processing unit is specifically used for:

[0125] Along a first stacking direction, stacking the key point coding feature and the detection frame coding feature to obtain a first stacking feature;

[0126] The first stacking feature and the text coding feature are stacked along a second stacking direction to obtain a second stacking feature, and the second stacking feature is used as the prompt coding feature.

[0127] In some embodiments of the present disclosure, the feature fusion unit is specifically used to:

[0128] Based on the self-attention network in the feature fuser, the prompt encoding feature is processed to obtain a self-attention feature;

[0129] Processing the self-attention feature and the image encoding feature based on a prompt-to-image attention network in the feature fuser to obtain a first cross-attention feature;

[0130] Based on the first fully connected network in the feature fuser, the first cross-attention feature is processed to obtain a target prompt feature;

[0131] Based on the image-to-cue attention network in the feature fuser, the target cue feature and the image encoding feature are processed to obtain an updated image feature, and the updated image feature is used as the target image feature;

[0132] The target prompt feature and the target image feature constitute the fused coding feature.

[0133] In some embodiments of the present disclosure, the decoding module 540 includes:

[0134] An upsampling unit, configured to process the target image features contained in the fused coding features based on the upsampling network in the decoder to obtain upsampled image features;

[0135] a cross attention processing unit, configured to process the target cue feature included in the fused coding feature based on the cue-to-image attention network in the decoder to obtain a second cross attention feature;

[0136] a first processing unit, configured to process the second cross-attention feature based on a second fully connected network in the decoder to obtain a confidence level of the target prompt feature;

[0137] A second processing unit, configured to process the second cross-attention feature based on a third fully connected network in the decoder to obtain a fully connected feature;

[0138] A dot multiplication unit, used to perform dot multiplication on the upsampled image feature and the fully connected feature to obtain a candidate segmentation result of the original image;

[0139] The segmentation result determination unit is used to use the candidate segmentation result as the target segmentation result of the original image if the confidence of the target prompt feature is greater than a preset confidence threshold.

[0140] In some embodiments of the present disclosure, the device further includes:

[0141] An initial segmentation result acquisition module is used to acquire an initial segmentation result corresponding to the original image, wherein the initial segmentation result is obtained by processing the original image with a preset segmentation algorithm;

[0142] A feature encoding module, used to process the image position code, the initial segmentation result, the prompt position code, and the description text using the pre-acquired encoder to obtain image coding features corresponding to the image position code, segmentation coding features corresponding to the initial segmentation result, prompt coding features corresponding to the prompt position code, and text coding features corresponding to the description text;

[0143] Accordingly, the feature fusion module 530 includes:

[0144] An adding unit, used for adding the image coding feature and the segmentation coding feature to obtain an image addition feature;

[0145] The feature fusion unit is used to perform feature fusion processing on the image addition feature, the prompt coding feature and the text coding feature to generate the fused coding feature.

[0146] It should be noted that Figure 5 The image segmentation apparatus 500 shown can perform Figures 1 to 4 The various steps in the method embodiment shown in the figure are implemented Figures 1 to 4 The various processes and effects in the method embodiment shown are not described in detail here.

[0147] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0148] like Figure 6 As shown, the electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0149] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0150] The memory 602 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 602 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 602 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 602 is a non-volatile solid-state memory. In a particular embodiment, the memory 602 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0151] The processor 601 reads and executes the computer program instructions stored in the memory 602 to perform the steps of the image segmentation method provided in the embodiment of the present disclosure.

[0152] In one example, the electronic device may further include a transceiver 603 and a bus 604. Figure 6 As shown, the processor 601, the memory 602 and the transceiver 603 are connected via a bus 604 and communicate with each other.

[0153] The bus 604 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 604 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0154] The following is an embodiment of a computer-readable storage medium provided in an embodiment of the present disclosure. The computer-readable storage medium and the image segmentation method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the computer-readable storage medium, reference can be made to the embodiment of the above-mentioned image segmentation method.

[0155] This embodiment provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to perform an image segmentation method, which is applied to an electronic device corresponding to a robot arm base. The method includes:

[0156] Obtaining an image position code of the original image, a prompt position code of user-triggered information on the original image, and a description text of the original image;

[0157] Using a pre-acquired encoder, the image position code, the prompt position code and the description text are processed to obtain image coding features corresponding to the image position code, prompt coding features corresponding to the prompt position code and text coding features corresponding to the description text;

[0158] Performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature;

[0159] The fused coding features are processed using a pre-acquired decoder to generate a target segmentation result of the original image.

[0160] Of course, the storage medium containing computer executable instructions provided by the embodiment of the present disclosure, whose computer executable instructions are not limited to the above method operations, can also execute related operations in the image segmentation method provided by any embodiment of the present disclosure.

[0161] Through the above description of the implementation methods, technicians in the relevant field can clearly understand that the present disclosure can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions to enable a computer cloud platform (which can be a personal computer, server, or network cloud platform, etc.) to execute the image segmentation method provided by each embodiment of the present disclosure.

[0162] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. An image segmentation method, characterized in that: include: Obtaining an image position code of the original image, a prompt position code of user-triggered information on the original image, and a description text of the original image; Using a pre-acquired encoder, the image position code, the prompt position code and the description text are processed to obtain image coding features corresponding to the image position code, prompt coding features corresponding to the prompt position code and text coding features corresponding to the description text; Performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature; The fused coding features are processed using a pre-acquired decoder to generate a target segmentation result of the original image.

2. The method according to claim 1, characterized in that The step of obtaining the image position code of the original image and the prompt position code of the user trigger information on the original image includes: Based on the coordinate data of a first pixel point in the original image, the dimension of the coding vector corresponding to the first pixel point, and the dimension index in the coding vector corresponding to the first pixel point, position encoding the original image is performed to obtain the image position encoding; Based on the coordinate data of the second pixel point in the user trigger information, the dimension of the coding vector corresponding to the second pixel point, and the dimension index in the coding vector corresponding to the second pixel point, the user trigger information is position encoded to obtain the prompt position code.

3. The method according to claim 1, characterized in that The step of performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature includes: The prompt coding feature and the text coding feature are stacked to obtain a prompt coding feature; The prompt coding feature and the image coding feature are subjected to feature fusion processing by using a pre-acquired feature fuser to generate the fused coding feature.

4. The method according to claim 3, characterized in that The prompt coding features include key point coding features and detection frame coding features; Correspondingly, the prompt coding feature and the text coding feature are stacked to obtain the prompt coding feature, including: Along a first stacking direction, stacking the key point coding feature and the detection frame coding feature to obtain a first stacking feature; The first stacking feature and the text coding feature are stacked along a second stacking direction to obtain a second stacking feature, and the second stacking feature is used as the prompt coding feature.

5. The method according to claim 3, characterized in that: The using of the pre-acquired feature fuser to perform feature fusion processing on the prompt coding feature and the image coding feature to generate the fused coding feature includes: Based on the self-attention network in the feature fuser, the prompt encoding feature is processed to obtain a self-attention feature; Processing the self-attention feature and the image encoding feature based on a prompt-to-image attention network in the feature fuser to obtain a first cross-attention feature; Based on the first fully connected network in the feature fuser, the first cross-attention feature is processed to obtain a target prompt feature; Based on the image-to-cue attention network in the feature fuser, the target cue feature and the image encoding feature are processed to obtain an updated image feature, and the updated image feature is used as the target image feature; The target prompt feature and the target image feature constitute the fused coding feature.

6. The method according to claim 1, characterized in that The using of the pre-acquired decoder to process the fused coding features to generate a target segmentation result of the original image includes: Based on the upsampling network in the decoder, the target image features included in the fused coding features are processed to obtain upsampling image features; Based on the cue-to-image attention network in the decoder, the target cue feature included in the fused encoding feature is processed to obtain a second cross-attention feature; Based on a second fully connected network in the decoder, processing the second cross-attention feature to obtain the confidence of the target prompt feature; Based on a third fully connected network in the decoder, processing the second cross attention feature to obtain a fully connected feature; Performing a dot product on the upsampled image feature and the fully connected feature to obtain a candidate segmentation result of the original image; If the confidence of the target prompt feature is greater than a preset confidence threshold, the candidate segmentation result is used as the target segmentation result of the original image.

7. The method according to claim 1, characterized in that Also includes: Obtaining an initial segmentation result corresponding to the original image, wherein the initial segmentation result is obtained by processing the original image with a preset segmentation algorithm; Using the pre-acquired encoder, the image position code, the initial segmentation result, the prompt position code, and the description text are processed to obtain image coding features corresponding to the image position code, segmentation coding features corresponding to the initial segmentation result, prompt coding features corresponding to the prompt position code, and text coding features corresponding to the description text; Accordingly, the performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature includes: Adding the image coding feature and the segmentation coding feature to obtain an image addition feature; The image addition feature, the prompt coding feature and the text coding feature are subjected to feature fusion processing to generate the fused coding feature.

8. An image segmentation device, characterized in that: include: An information acquisition module, used to acquire an image position code of an original image, a prompt position code of user-triggered information on the original image, and a description text of the original image; An encoding module, used to process the image position code, the prompt position code and the description text using a pre-acquired encoder, to obtain image encoding features corresponding to the image position code, prompt encoding features corresponding to the prompt position code and text encoding features corresponding to the description text; A feature fusion module, used for performing feature fusion processing on the image coding feature, the prompt coding feature and the text coding feature to generate a fused coding feature; The decoding module is used to process the fused coding features using a pre-acquired decoder to generate a target segmentation result of the original image.

9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method according to any one of claims 1 to 7.

Images